How to Implement a Technology Supported Free-Selection Peer Review Protocol

Design Implications from Two Studies on Computer Network Education

Pantelis M. Papadopoulos Thomas D. Lagkas Stavros N. Demetriadis

Informatics Department ICTE Department Informatics Department
Aristotle University of Thessaloniki University of Western University Aristotle University of Thessaloniki
Thessaloniki, Greece Kozani, Greece Thessaloniki, Greece
pmpapad@csd.auth.gr tlagkas@uowm.gr sdemetri@csd.auth.gr

Abstract—This paper presents design guidelines for
implementing a free-selection peer review protocol. “Free-
selection” (FS) refers to the ability of students freely access all
available peer work and choose which of them to read and
review. A series of two studies on the free-selection protocol
has provided evidence on the efficiency of the method. In the
First study, the FS protocol was compared against the widely
used assigned-pair one, where students work in instructor-
defined dyads. In the Second study, further issues of the FS
approach were evaluated, with our attention focused on
students who, due to the freedom element of the protocol, do
not receive reviews. Both studies paint a very promising
picture of free-selection. However, several issues were also
raised on how to effectively apply such a protocol. As the use of
technology is necessary in the FS approach, we provide in this
paper the design implications derived from the two studies
regarding, (a) peer work availability, (b) non-reviewed works,
(c) student population, (d) peer work visual representation, (e)
peer work length, (f) presentation order, (g) double-blinded
approach, (h) peer reviews availability, and (i) students’
approaches in selecting peer work for reading and reviewing.
CSCL, peer review, free-selection, assigned pair, technology-
enhanced learning, computer network education
I. INTRODUCTION
Peer review is an instructional method aiming to help
students elaborate on domain-specific knowledge, while
simultaneously developing methodological review skills.
Typically, in a peer review cycle, an author drafts a piece of
work which is then evaluated by a peer. The evaluation or
critique is carried out anonymously on the basis of explicitly
defined criteria and is subsequently returned to the author.
The author is free to review his or her final draft based on the
given critique. Yet, when practicing peer review in the
classroom the instructor has a number of alternative design
selections to choose from.
The literature abounds with relevant studies indicating
that the method is popular among educators inspired mainly
by the constructivist and socio-constructivist paradigms for
learning [e.g., 1, 2, 3] who want to challenge their students to
think critically, synthesize information, and communicate
science in nontechnical language. The method has been used
extensively in various fields [e.g., 4, 5, 6, 7, 8, 9] and in the
CS discipline [e.g., 10, 11, 12], with promising results
reported.
Researchers stress the fact that peer review offers to
students the chance of developing a range of skills important
in the development of language and writing ability, such as
meaningful interaction with peers, new perspectives on the
writing process, and a greater exposure to ideas [13, 14].
Based on that, we focused our attention on how to enhance
the peer review method by increasing even further students’
exposure on new perspectives and ideas.
As a reference point we used the widely used “assigned-
pair” (AP) protocol. While there are studies that assign
multiple reviewers to a single peer [15, 16], in the context of
our study we use the term “assigned-pair” to refer to a more
common research design where students in a author-reviewer
dyad are assigned exclusively to each other. Students can
play both roles and review each other’s work. The overhead
for the instructor is well contained and the activity is
straightforward for the students. This design, of course, is not
without drawbacks. One review for a single peer might not
be enough for the author to get valuable comments and
suggestions for improvement. Additionally, the benefit for
the reviewer may be limited, since she gets just one more
point of view (the author’s). Finally, the method requires a
level of stability in the dyads formation throughout the
activity, while a bad pairing may have negative results.
Our focus was to (a) enhance the learning benefits of
peer review for the students, without increasing the
minimum amount of work that they had to do, and (b) keep
instructor’s overhead low. Towards this direction, we
decided to investigate the efficiency of a “free-selection”
(FS) protocol, where there are no dyads and students are free
to browse all peer work and select what to review.
We tested the effectiveness of the FS protocol in a series
of two studies conducted in the same context. Results were
very promising suggesting that indeed such a protocol can
enhance the learning outcome. However, the implementation
of the FS protocol requires the use of technology and is more
complex from its AP counterpart. Through the two studies,
we were able to identify factors that can affect the impact of
the FS approach.
In the following, we present (a) information on the two
studies on the FS protocol, (b) useful design implications for
implementing the protocol, and (c) conclusions on the
approach.

This paper is available by IEEE Xplore at:

http://dx.doi.org/10.1109/ICALT.2012.12
 II. FIRST STUDY
The First study [17] compared the effectiveness of the
Free-Selection protocol against the typical Assigned-Pair
one, where students work in instructor-defined dyads
(playing both the roles of author-reviewer).
The study employed 37 sophomore students majoring in
Informatics and Telecommunication Engineering in a 5-year
study program. We randomly assigned students into two
study conditions: Free-Selection (FS) (n=17), and Assigned-
Pair (AP) (n=20). Pre-test results analysis showed that the
two groups were comparable in terms of prior domain
knowledge (p>0.05).
The activity lasted two weeks. During the first week, all
students studied individually online in a learning
environment that we developed. Students had to study cases
presenting past experienced in the domain of Network
Planning and Design (NP&D) and provide answers to three
realistic scenarios about various companies in need of a new
computer network. The first week was identical for the two
groups. During the second week, students in the AP group
worked in same-gender dyads that we randomly formatted.
Students in a dyad had to blindly review each other answers
and provide comments and suggestions for improvement.
Based on these suggestions, students could revise their initial
answer and complete their study in the learning environment.
On the other hand, students in the FS group were able to see
the answers of all the other students and freely select which
answer to review (they had to review at least one answer per
scenario). Because of the free format in selecting what to
review, a small number of answers did not receive a review
and we assigned these answers for review directly to selected
students.
Post-test result analysis on domain knowledge
acquisition showed that students in the FS group performed
significantly better than those in the AP group in acquisition
of domain knowledge (p<0.05). Also, FS students became
significantly better reviewers, by resembling the scoring
patterns of expert raters employed in the study.
The First study gave us clear indications of the learning
benefits of the Free-Selection approach in peer review
settings. However, it also highlighted a number of research
questions issues that needed further analysis. The most
important of them was the impact of the FS approach to
students that do not receive reviews for the answers they
submit.
III. SECOND STUDY
Following the findings of the First study, the Second
study [18] analyzed the effectiveness of the FS protocol on
students that do not receive review due to the free policy on
choosing answers to review.
The Second study was conducted a year after the First
one, on the exact same context, meaning the students’ year of
study, knowledge domain, and material. The study employed
38 sophomore students majoring in Informatics and
Telecommunication Engineering. In order to compare the
impact of not receiving a review, we needed similar
populations of students in the two conditions (review/no
review). To do so, we randomly selected 15 students. These
students participated as anybody else in the activity, but their
answers did not appear to others and thus they did not
received reviews. This means that all 38 students submitted
reviews, but the answers of only 23 students were available
for review. We informed students beforehand of this random
selection, although we did not divulge the identities of the
selected ones. Five students whose answers appeared to
others, but did not receive reviews to all their answers were
also added to the initial 15. The students that did not receive
reviews had to conduct a self-review process, following
instructions on analyzing their answers. Eventually, 20
students worked in the Self-Review (SR) group, and 18
students in the Peer-Review (PR) group. Pre-test results
analysis showed that the two groups were comparable in
terms of prior domain knowledge (p>0.05).
The activity followed the same pattern as before. During
the first week, students had to study individually in the
environment and provide answers to the three scenarios on
the NP&D domain. During the second week, all students
followed the Free-Selection review protocol, freely
reviewing at least one answer per scenario. At the end,
students in the PR group revised their initial answers based
on the comments they received, while students in the SR
group based their revisions only on what they read in others’
answers and in the self-review form. Post-test result analysis
on domain knowledge acquisition showed that the two
groups were comparable (p>0.05).
IV. DESIGN IMPLICATIONS
Based on the findings of these two studies, we propose a
set of design implications for the designers and instructors
who wish to implement the free-selection approach.
A. Make Students’ Work Freely Available to Support
Multiple Viewpoints
The main idea behind the Free-Selection review protocol
is that students benefit more when they are allowed to read
as many of their peers’ submissions as they want. By reading
multiple peer submissions on the same subject, students get
to know others’ opinions and perspectives. That way, they
become aware of alternative approaches and argumentation.
Ideally, a student in a free-selection setting should be able to
recognize solid arguments, distinct between high and low
quality works, and understand were the high quality works
converge. The degree that this process is actually happening
depends on the ability of each student to analyze and
compare various viewpoints.
Peer review protocols that strictly define and narrow
down the amount of peer work that a student has access to
diminish the impact multiple perspectives can have on
learning. In addition, in these protocols, access to peer work
also means an obligation to produce a review. Thus, the
obvious argument in favor of these protocols is that they are
easy to implement. Students have a contained amount of
work to perform, while the overhead for the instructor is
easily controlled. On the other hand, the free-selection
approach requires the use of technology in managing and
controlling the activity. However, this technology is readily
available. Although we developed our own technology-
enhanced learning environment for the research needs of the
two studies, an interested researcher could apply the method
using technological tools such as forums, and LMSs.
The important thing to note about the Free-Selection
protocol is that by granting access to all submissions, we do
not oppose additional workload to the students. We just give
them the option to get better informed about what others did,
before conducting a review. In the First study, the students
took advantage of this and read on average 8 (out of the 16
available) answers before selecting which ones to review.
They reviewed on average 2 answers in each scenario. This
was their choice and as they said in the interviews, the fact
that they were free to choose the number of answers to read
and review (they had to submit at least on review) made
them develop a positive attitude towards the activity. On the
other hand, there were also students that chose to apply a
minimum effort strategy and read and review only one
answer in each scenario. These students essentially shifted
from the FS setting to the AP one. In other words, even in
the extreme cases were students opt to read and review only
one peer work, the effectiveness of the FS method would be
at the same level as the typical AP approach.
B. Apply Self-Review Instead of an At-Least-One-Review-
Per-Submission Strategy
The most important critique against the Free-Selection
protocol is that the method is based on the randomness of the
students’ selection process, thus allowing the existence of
non-reviewed submissions. Applying a review protocol that
does not guarantee at least one review per submission may
seem counter-intuitive. On the contrary, though, we maintain
that the effectiveness of the FS approach relies more on
conducting the review process itself and not on receiving
review comments.
A student is benefited by a review process in three ways:
(a) by reading peer work, the student becomes aware of other
points of view, (b) the process of analyzing and critiquing
peer work engages student deeper with the learning material,
since this is the background on which review comments and
suggestions should be rooted, and (c) the student receives
review comments and suggestion to improve the initial
answer. Items (a) and (b) can be significantly enhanced in a
free-selection setting. As we already mentioned, in FS,
students have much more access to others’ views that in any
other review protocol. The results from the two studies
showed that students submit more reviews when in FS (First:
M=1.9; Second: M=1.7, per scenario for FS, M=1.0 for AP
in First). This means that student engagement is also
increased. Regarding item (c), there is indeed an
unpredictable number of students that may not receive
comments. This, of course, happens because the rest of the
students receive multiple reviews (First: max=3; Second:
max=6, in a single scenario). Thus, for some students, even
item (c) is enhanced in FS.
The question is whether the lack of review comments has
a significant effect on students. This was exactly the focus of
the Second study. We addressed the lack of review
comments with a self-review form where the students had to
analyze their initial answers and compare them to what they
have read in other answers. Results analysis showed that
students that did not receive reviews achieved the same level
of performance as the students that did. This is a clear
indication that the lack of reviews can easily be handled in
the FS protocol.
A designer could apply various strategies to ensure that
each submission will receive a review in FS (e.g., by
applying a first-come-first-served method on the availability
of the submissions). However, this affects the degree of
freedom the students have in selecting submissions to read
and review. Based on the findings of the Second study, we
believe that such strategies are not necessary.
C. Apply to Medium Size Groups
The size of the group in which the FS protocol is
implemented can affect the learning outcome. Although the
goal is to present multiple peer works to the students, we
need to provide students with an amount of data that they can
handle. In the First study, there were 17 students in the FS
group, meaning that each student had access to 16 answers in
each scenario. Usage data showed that students read on
average 8 answers per scenario. The higher the number of
available answers, the higher the probability that students
will select randomly which ones to read. That is why in the
Second study, we decided to show a random subset of 9
answers to each student. Each answer appeared the same
number of times and had equally chances of receiving review
comments. An interested designer, should either divide large
groups of students into smaller ones and implement the FS
independently in each one, or include all students in one
group and randomly provide access to a number of answers
that would not overwhelm the students.
D. Provide an Effective View of All the Submissions
Students working in a Free-Selection setting have to
make two choices. First, they have to select which answers to
read, and second, which of these answers to review. The
explanation behind the second choice is probably clearer for
us. The student has a picture of what others said and, based
on a personal criterion, makes a selection. As we will later
explain, this criterion (or set of criteria) can be different for
each student, but eventually explains student’s behavior in
the activity. What is not that clear is how the student selects
which of the available answers to read. This selection is very
important, because it defines student’s expose to others’
opinions and automatically excludes from reviewing the rest
of the answers.
We propose two design guidelines to assist students in
this selection. First, the designer must provide a visual
representation of the set of available answers in a way that it
will be easy for the students to get an idea of the available
material. For this, we chose to present the peers’ answers in a
grid, rather than in list, to avoid scrolling and make it easier
for the students to print this grid. We believe that a list gives
a sense of order, sequence, or importance. While using a
search engine or reading the news online, the student, as any
other user, is used (or even encouraged) to pay more
attention to the top of the list. The use of a grid was our
 Figure 1. Answer grid. According to this figure, the student has read answers 1, 2, 3, 5, and 9 (marked by an eye
icon), and has reviewed answers 1 and 3 (marked by a bullet icon). The complete grid had 16 answers for the Free-
Selection group.
approach to circumvent this bias. Other representations may
be used, having in mind to present all submissions as equally
important. The students had to click on an answer in the grid
to view the whole answer text.
The second design guideline is to give students the
chance to get a rough idea of what each answers says,
without reading the whole submission. In our case, the
knowledge domain was Network Planning and Design, and
students included characteristics of the network designs they
proposed even in the first sentences of their answers. Thus,
including only the first 3-4 lines of each answer in the grid
boxes was usually enough to give students a glimpse of what
their peers were suggesting (Fig. 1).
In both our studies, students’ answers were usually a
little less that a whole page. In settings were students
produce lengthier deliverables, additional information should
be given to the students. For example, in case the students
have to submit papers expanding in several pages, abstracts
should be required so that they can be presented in the grid.
The goal in FS is to present shortly the main points of each
answer without having the students reading the whole text.
That way, they will be able to take an informed decision on
which answers to read and which to skip.
E. Preferred for short deliverables
It was very important that the students’ answers were
relative short, rarely filling a whole page. This was
commented by the students in the interviews, saying that it
was easy to read a lot of answers because of their length.
One should expect that as the answers become longer, the
possibility of students reading more of them gets lower. This,
in turn, diminishes one of the main purposes of the FS
approach, which is to make the students learn more by
analyzing different opinions.
F. Random Appearance of Submissions
One guideline that is connected to the previous section is
to randomly present the answer into the answer grid to
enhance the spread of reviews, even if the students decide to
focus only on answers appearing in the first places. As a
good measure we applied a simple algorithm to randomly
present the answers. We used students’ username in the
system as the algorithm seed so that each student would view
a different answer grid that would remain the same each time
the student logins in the system.
Usage analysis showed that both the readings and the
reviews were widely spread throughout the grid and this was
one of the reasons we had so few non-reviewed answers in
the First study.
G. Double-Blinded Review to Move Focus on Submission
Content
We applied a double-blinded review process in both our
studies. Our goal was to analyze students’ choice in selecting
which answers to read and review when they had to base
selection solely on the answer text. This allowed us to avoid
any biases. In the First study, we asked students whether they
wanted to know the identity of the authors they reviewed and
of their reviewers. Students answered using a 5-step Likert
scale (1=No, 5=Yes). Students in the Assigned-Pair group
were divided regarding the authors’ identities (M=2.50), but
most of them wanted to know who commented on their
answers (M=3.43). On the contrary, students in the FS group
did not care about others’ identity (M=1.36, M=1.55,
respectively). Thus, one could choose to apply a non-blinded
approach. We believe, however, that students’ behavior
could change in case peers’ identities are freely available.
H. Control Access of Submitted Reviews
In our studies, we chose to present the review comments
only to the respective authors, thus controlling students’
interaction. In that way we were able to better control the
variables under investigation in each study. However, one
could opt to make the review comments public, thus
allowing deeper interaction between the students. This could
affect the impact of the FS approach in several ways.
First, students are better informed of each other’s
perspectives, since each review represents in a sense the
opinion of the reviewer on the subject. This can be beneficial
for the students.
On the other hand, available reviews may affect future
reviewers’ judgment. The first review may act as the
dominant one, while it is not likely that students would
choose to submit conflicting reviews. It is more probable that
students will submit converging reviews, and in that sense
reproduce same comments and suggestions.
Finally, having the reviews publicly available diminishes
the chance of non-reviewed answers. It will be clear which
answers have already been reviewed and which ones are left
non-reviewed. Thus, students will be able to provide one
review to each answer.
I. Factors that Affect Students’ Selection on Selecting a
Submission for Review
Base on usage data and interview statements in the two
studies, there are two main criteria students apply in
selecting which answers to review. Half of the students try to
avoid conflict and select answers that, according to them, are
of high quality. This way the students are able to provide
complementing reviews saying only nice things for their
fellow students. On the contrary, another half of the students
selects answers that, according to them, present obvious need
for improvement. This way it is easier for the students to
write more meaningful reviews. Of course, students’
opinions about high and low quality answers are not always
in line with the instructor’s judgment. However, these two
ways of thinking are, most of the times, behind students’
behavior. Other factors that, according to students, affect the
selection process are the length of an answer (medium size
answers are usually preferred), and how clear an answer is
(students in general refrain from reviewing answers that they
do not understand).
V. CONCLUSIONS
Results on the Free-Selection approach have been
encouraging so far, and as the availability of the necessary
technological tools makes the implementation easy, we
expect similar approach to gain researchers’ attention. In the
two studies, we tested a generic FS format. However, as we
presented here, there is a number of factors that can be
modified and affect the learning outcome. The 9 implications
presented in the previous sections can provide a useful
picture to the instructor who wishes to implement similar
peer review methods.
REFERENCES
[1] Topping, K. (1998). Peer assessment between students in colleges and
universities. Review of Educational Research, 68, 249-276.
[2] Falchikov, N. (2001). Learning together: Peer tutoring in higher
education. London: Routledge Falmer.
[3] Liu, C. C., & Tsai, C. M. (2005). Peer assessment through web-based
knowledge acquisition: tools to support conceptual awareness.
Innovations in Education and Teaching International, 42, 43-59.
[4] Falchikov, N., & Goldfinch, J. (2000). Student Peer Assessment in
Higher Education: A Meta-Analysis Comparing Peer and Teacher
Marks. Review of Educational Research, 70(3), 287-322.
[5] Dossin, M. M. (2003). Among Friends: Effective Peer Critiquing. The
Clearing House, 76 (4), 206-207.
[6] Anewalt, K. (2005). Using peer review as a vehicle for
communication skill development and active learning. Journal of
Computing in Small Colleges, 21(2), 148-155.
[7] Turner, S., Pérez-Quiñones, M.A., Edwards, S., & Chase, J (2010).
Peer Review in CS2: Conceptual Learning. Proceedings of
SIGCSE’10, March 10–13, 2010, Milwaukee, Wisconsin, USA.
[8] Liou, H. C., & Peng, Z. Y. (2009). Training effects on computer-
mediated peer review. System, 37, 514–525
[9] Goldin, I. M., & Ashley, K. D., (2011). Peering Inside Peer Review
with Bayesian Models. In G. Biswas et al. (Eds.): AIED 2011, LNAI
6738, pp. 90–97. Springer-Verlag: Berlin.
[10] Crespo, R. M., Pardo, A., & Kloos, C. D. (2004). An Adaptive
Strategy for Peer Review. Paper presented at ASEE/IEEE Frontiers in
Education Conference. Savannah, GA.
[11] Ziu, E.Z.F., Lin, S., Chiu, C.H., & Yuan, S.M. (2001). Web-based
peer review: the learner as both adapter and reviewer. IEEE
Transactions on Education, 44, 246-251.
[12] Demetriadis, S., Egerter, T., Hanisch, F., & Fischer, F. (2011). Peer
review-based scripted collaboration to support domain-specific and
domain-general knowledge acquisition in computer science.
Computer Science Education, 21(1), 29-56.
[13] Hansen, J., & Liu, J. (2005). Guiding principles for effective peer
response. ELT Journal, 59, 31-38.
[14] Lundstrom, K., & Baker, W. (2009). To give is better than to receive:
The benefits of peer review to the reviewer’s own writing. Journal of
Second Language Writing, 18, 30-43.
[15] Tseng, S., & Tsai, C-C. (2007). On-line peer assessment and the role
of peer feedback: A study of high school computer course. Computer
& Education, 49, 1161–1174.
[16] Tsai, C.-C., & Liang, J.-C. (2009). The development of science
activities via on-line peer assessment: The role of scientific
epistemological views. Instructional Science, 37, 293-310.
[17] XXXXXX (2012). Computers & Education.
[18] XXXXXX (in preparation).