2014 IEEE 14th International Conference on Advanced Learning Technologies
Usage Data and Group Rankings In Peer Review Settings
A Case Study on Students’ Behavior and Performance
Pantelis M. Papadopoulos
Aarhus University
Aarhus, Denmark
pmpapad@tdm.au.dk
Abstract—This paper focuses on the effect that usage data and
group ranking information may have when are presented to
students in a peer review setting. The study analyzes the
performance and attitudes of 56 sophomore students enrolled
in a Network Planning and Design course. The students,
grouped randomly in three different conditions, followed the
prescriptions of a free-selection peer review protocol that
guided students in double-blind review process, allowing them
to select on their own peer work for review. Students in the
first condition acted as the control group without any
information on their usage data or rankings. Students in the
second condition had access to usage data information
mirroring their activity in the study, while students in the third
condition received additional information on their rankings
inside their group. Result analysis suggests that, while there is
not difference in domain knowledge acquisition, engagement
was higher in the third group, with students spending more
time in the activity and expressing a more positive attitude
towards the study.
Keywords-peer review, free-selection, Computer Science
Education, engagement and motivation
I. INTRODUCTION
Peer review is a widely used instructional method that
aims to assist students in acquiring domain knowledge, while
also developing domain-independent skills in reviewing.
Typically, a peer review process includes four phases: (a)
production of initial student work, (b) assigning authors’
work to reviewers, (c) providing feedback back to authors in
their work, and (d) revising initial work according to review
comments and suggestions. McConnell [1] argues that peer
reviewing offers to students the opportunity for a
constructive and collaborative learning experience, by
engaging them in an active learning exercise. Peer review is
primarily expected to support higher-level learning skills
such as synthesis, analysis, and evaluation [2] as the students
have the opportunity to analyze and evaluate peer work.
Scardamalia and Bereiter [3] have provided evidence that
higher cognitive processes of learning are stimulated and
guided by the peer review procedure, by implementing the
method into school classes. The method has been used
extensively in various fields (e.g., [4][5]), including
Computer Science (e.g., [6][7], the primary domain of our
work.
978-1-4799-4038-7/14 $31.00 © 2014 IEEE
DOI 10.1109/ICALT.2014.92
This paper is available by IEEE Xplore at:
http://dx.doi.org/10.1109/ICALT.2014.92
Thomas D. Lagkas
The University of Sheffield
International Faculty, CITY College, Greece
T.Lagkas@sheffield.ac.uk
Our research scope is to enhance the learning experience
for students in peer review settings, focusing mainly on the
assigning phase, allowing students to freely select the peer
work they want to analyze and review. We call this Free-
Selection review protocol and it has been in the center of
attention in a series of studies. Results showed that students
that acted both as authors and reviewers in a free-selection
review setting outperformed students that assumed the same
roles in teacher-defined pairs [8]. In addition, result analysis
of a recent study on the way peer comments and self-
reflection affect students revealed that intrinsic feedback
may have a stronger effect than extrinsic feedback
(reviewers’ comments) [9]. We refer to intrinsic feedback as
the deeper understanding constructed by a reviewer on the
quality of her work, through critical reflection on peers’
contribution.
In this paper, we focus on ways to enhance students’
engagement in peer review, by providing usage and ranking
information. Usage information refers to the effort put by the
students in the activity, as evident by the number of logins in
the learning environment used, the number of peer
submissions read, and the number of reviews submitted.
Ranking information provides additional value to these data,
by also informing students on their relative position in the
group for each of the usage metrics. The study examined
whether this additional information would trigger additional
motives to the students, or whether ranking information
could have a detrimental effect on students’ behavior.
II. METHOD
A. Participants
The study employed 56 volunteering sophomore students
(32m, 24f) majoring in Informatics and Telecommunications
Engineering in a 5-year study program. We awarded a bonus
grade for the lab section of the course to each student that
successfully completed all the phases of the activity. We
randomly assigned student population into three study
conditions: Control: 18 students (11m, 7f); Usage: 20
students (11m, 9f); Ranking: 18 students (10m, 8f). All
students were enrolled in the “Network Planning and
Design” (NP&D) course, teaching students how to analyze
clients’ needs, identify system specifications, and design an
appropriate computer network.
297
B. Material
1) The learning environment: We designed and
developed the learning environment we used in this study to
be able to taylor more ealisy the study conditions to our
research needs. The learning material was presented as
realistic open-ended scenarios (acting as the problems
students had to solve) accompanied by a list of supporting
material presented as past cases (acting as the resources to
solve the problems). The scenarios referred to various
installations of computer network systems in
new/restructured facilities, while the supporting material
referred to similar past projects highlighting important
domain factors, such as the cost of the project, efficiency
requirements, expansion requirements, and the traffic type
and end-users’ profile. Students had to take into account the
specific conditions and the context presented and propose
their own computer networks, as answers to 3 scenarios.
2) Pre-test and post-test: The pre-test instrument was
comprised by 6 open-ended questions (e.g., “How can the
security requirements of a network affect its architecture?”)
measuring students’ prior knowledge of the domain.
Similarly, the post-test instrument was comprised by 3
open-ended questions (e.g., “Which network characteristics
are affected by the end-users’ profile?”) used to record the
level of domain knolwdge students acquired in the study.
3) Attitude questionnaire: An online questionnaire was
used after the study to record students’ opinions and
attitudes towards different aspects of the activity.
C. Procedure
The whole activity lasted 2 weeks, starting with the Pre-
test phase and the written test in the classroom. During the
Study phase that started right after and lasted 1 week,
students logged in the environment (from wherever and
whenever they wanted) and worked on the 3 available
scenarios proposing their own computer networks.
Next, the students entered the Review phase that lasted 4
days. During this time, students gained access to all the peer
work in their groups and they had to review, in a double-
blind process, at least 1 peer submission per scenario.
Students were free to select the submissions they wanted.
Along with the comments, the reviewer had to also suggest a
grade according to the following scale: (1: Rejected/Wrong
answer; 2: Major revisions needed; 3: Minor revisions
needed; 4: Acceptable answer; 5: Very good answer).
During the Review phase, each student in the Usage
group was able to see (a) the number of logins, (b) the
number of answers read, and (c) the number of answers
reviewed. These 3 metrics did not provide any additional
information to the students, since a student could also find
out the respective values by taking notes of her activity. On
the contrary, a student in the Ranking group was able to see,
in addition to the previous information, her rankings and the
average values in her group. This was something that the
students could not have measured themselves. Chromatic
code was used to denote the quartile according to the ranking
(1st: green; 2nd: yellow; 3rd: orange; 4th: red).
In the Revise phase that followed and lasted 3 days, the
students were able to revise their own answers, according to
298
the comments received from their peers. In case a student did
not receive peer comments for her answers, the system was
asking her to perform self-review before revising the initial
answer. The self-review form was asking students to analyze
their own answers having in mind also the networks other
students proposed. During the Revise phase, one more metric
was available to Usage and Ranking groups, the average
grade received from peers, along with the number of
received reviews. After the Review and Revise phases, the
students took a written post-test in class, followed by the
attitudes questionnaire that concluded the activity.
III. RESULTS
Students’ answers in the written tests (pre- and post-test)
and the learning environment were independently assessed
by two raters that followed predefined grading instructions.
Table I presents students’ scores in pre- and post-test and
in the learning environment (initial/revised answers).
Students scored very low in the pre-test instrument,
providing evidence that were, indeed, novices in the domain.
One-way analysis of variance results showed that the 3
groups were comparable both in pre- and post-test, and in the
initial and revised answers (p>.05)
TABLE I. STUDENTS’ PERFORMANCE
Control Usage Ranking
(0-10) M SD n M SD n M SD n
Pre-test 2.11 (0.98) 18 1.98 (1.01) 20 2.02 (1.09) 18
Post-test 8.17 (1.01) 18 8.10 (0.96) 20 8.11 (0.93) 18
(1-5)
Initial 2.94 (0.58) 18 3.12 (0.75) 20 2.80 (0.77) 18
Revised 3.44 (0.67) 18 3.64 (0.58) 20 3.49 (0.77) 18
The lack of significant difference for the initial answers
was expected, since study conditions were the same up to
that point for everyone. However, the lack of statistical
difference in the scores of the revised answers suggests that
the instructional intervention applied did not have an effect
on students’ domain knowledge acquisition. Despite the
latter, paired-samples t-test results showed that scores in the
revised answers increased significantly (p<.05) for all the
groups. In addition, no statistically significant difference
between the groups was found on the usage metrics
performance (logins, reads, reviews, and grades).
Table II presents students’ responses in the most
important items of the attitudes questionnaire.
When it comes to hours spent in the learning
environment, we asked the students to keep track of time
throughout the activity and report this time in the
questionnaire. According to students’ responses, the 3 groups
were comparable (p>.05) on the time spent during the first
week (Q4). On the contrary, pairwise analysis showed that
students in the Ranking group spent significantly more time
(p<.05) in the learning environment during the second week
(Q5). The big drop between the first and the second week
was expected. During the first week, students have to read a
lot of material and answer the scenario questions. On the
contrary, during the second week, students’ workload is
 significantly lower, since peers’ answers are shorter, while
revising a scenario answer also takes less time than writing
an initial one.
TABLE II. QUESTIONNAIRE RESPONSES
Control Usage Ranking
(n=18) (n=20) (n=18)
M (SD) M (SD) M (SD)
Q1. Would you like to know the identity of your peers that
reviewed your answers? (1: No; 5:Yes)
1.70 (0.80) 1.95 (1.16) 1.84 (1.30)
Q2. Would you characterize the usage information presented
during the second week as useful or useless? (1: Useless; 5: Useful)
n.a. 4.33 (1.06) 4.26 (0.81)
Q3. Would you characterize the ranking information presented
during the second week as useful or useless? (1: Useless; 5: Useful)
n.a. n.a. 3.53 (1.07)
Q4. How many hours did you spend approximately in the activity
during the first week (Study phase)? (in hours)
10.08 (3.09) 10.12 (3.80) 9.59 (3.94)
Q5. How many hours did you spend approximately in the activity
during the second week (Review and Revision phases)? (in hours)
2.48 (1.40) 2.33 (1.01) 3.31 (1.13)*
*. p<.05
IV. DISCUSSION
The usage metrics we presented covered several aspects
of the activity. By presenting them to the students we are
also making a statement on what is important in a setting like
this. Result analysis suggests that presenting the usage data
alone may create a positive attitude, but this is not adequate
to affect students’ behavior and performance in the activity.
Students of both related groups appreciated having an
organized view of the metrics of their activity, but this did
not result in higher levels of knowledge acquisition, either in
the learning environment or the post-test.
Usage data was something that students could estimate
themselves. For example, a student could count the total
number of reviews she submitted, by going though each
scenario. As such, providing just a value without a reference
point was “nice” but rather unimportant. On the contrary,
students’ behavior was affected when this reference point
was presented. Students in the Ranking group were able to
compare themselves with the group average. This provided
the motive to some of the students in the group to actively try
improving their positions. We argue that this was the main
reason behind the statistical difference found in the time
spent during the second week. Nevertheless, this attitude
towards the ranking information was not universal in the
group, as many students chose to ignore it. We had to
perform a case-by-case analysis to get a better picture of how
and whether rankings affected students. This deeper analysis
revealed that there was a wide range of different approaches
followed in the Ranking group.
299
View publication stats
As we mentioned, there were students that reacted to low
ranking and tried to better their positions by reading and
reviewing more peer answers. On the other extreme, there
were students that performed the bare minimum required by
the activity. Rankings for those students remained low
throughout the activity. Between those two attitudes there
were students that tried only partially (just in the beginning,
or in a specific metric) to improve their rankings. For
example, students stated that they usually got alarmed by a
low ranking in the Grade metric that appeared in the
Revision phase. This was the only metric that students could
not change, as it was depicting peers’ opinions on the
student’s answers in the previous Review phase. However, if
the Grade ranking was high, little attention was given to the
rest of the metrics. This may also be problematic. Students
clearly stated that knowing that their Grade metric was high
in ranking was reassuring. However, high in rankings may
not always mean a correct answer. For example, since the
score of the Revised answers was significantly higher, a
student that decided not to revise her answers, based solely
on the ranking her Initial answer received, may be misled.
In conclusion, providing students with usage and ranking
information is one more type of feedback. Showing this
information could provide an additional motive for deeper
engagement to some students. However, students appropriate
this information according to their intrinsic motives. Highly
engaged students may identify the areas they need to
improve, while less engaged students would either ignore the
additional data, or, worse, find an excuse to disengage even
further.
REFERENCES
[1] McConnell, J. (2001). Active and cooperative learning. Analysis of
Algorithms: An Active Learning Approach. Jones & Bartlett Pub
[2] Anderson, L. W., & Krathwohl, D. R. (Eds.) (2001). A Taxonomy for
Learning, Teaching, and Assessing: A Revision of Bloom's
Taxonomy of Educational Objectives. NY: Longman
[3] Scardamalia, M., & Bereiter, C. (1994). Computer support for
knowledge-building communities. The Journal of the Learning
Sciences, 3(3), 265-283.
[4] Dossin, M. M. (2003). Among Friends: Effective Peer Critiquing. The
Clearing House, 76 (4), 206-207.
[5] Goldin, I. M., & Ashley, K. D., (2011). Peering Inside Peer-Review
with Bayesian Models. In G. Biswas et al. (Eds.): AIED 2011, LNAI
6738, pp. 90–97. Springer-Verlag: Berlin.
[6] Turner, S., Pérez-Quiñones, M.A., Edwards, S., & Chase, J (2010).
Peer-Review in CS2: Conceptual Learning. Proceedings of
SIGCSE’10, March 10–13, 2010, Milwaukee, Wisconsin, USA
[7] Liou, H. C., & Peng, Z. Y. (2009). Training effects on computer-
mediated peer review. System, 37, 514–525
[8] Papadopoulos, P. M., Lagkas, T. D., & Demetriadis, S. N., (2012).
How to Improve the Peer Review Method: Free-Selection vs
Assigned-Pair Protocol Evaluated in a Computer Networking Course.
Computers & Education, 59, 182 – 195. Elsevier.
[9] Papadopoulos, P. M., Lagkas, T. D., & Demetriadis, S. N., The
Impact of Instrinsic and Extrinsic Feedback in Free-Selection Peer
Review Settings (submitted).