Original article

doi: 10.1111/jcal.12073

bs_bs_banner

A delineation of the cognitive processes

manifested in a social annotation environment
S.C. Li, J.W.C. Pow & W.C. Cheung
Department of Education Studies, Faculty of Social Science, Hong Kong Baptist University, Kowloon Tong, Kowloon, Hong Kong

Abstract This study aims to examine how students’ learning trajectories progress in an online social
annotation environment, and how their cognitive processes and levels of interaction correlate
with their learning outcomes. Three different types of activities (cognitive, metacognitive and
social) were identified in the online environment. The time series of different levels of
cognitive and metacognitive activities correlate with one another, indicating that a kind of
symbiotic or ecological relationship exists among the cognitive and metacognitive processes.
The multiple regression results show that low-cognitive level activities mediate the effect of
high-cognitive level activities on student’s performance in the group inquiry learning project.
The results of multilevel analysis indicate that the average number of highlighted texts posted
per person in a group accounted for over 50% of the variance of the group inquiry learning
project scores. Further, the results also indicate that the level of collaboration within a group
explains over 70% of the variance of the high-level cognitive and metacognitive activities
across different groups. In sum, group collaboration is found to be conducive to fostering
high-level cognitive and metacognitive activities.

Keywords collaborative learning, Diigo, inquiry learning, knowledge building, social annotation, social
bookmarking.

Web technology is drastically changing the ways

Introduction
The advancement in ubiquitous computing has created
huge challenges as well as opportunities to education.
It is believed that the affordance and pervasiveness of
technology can transform traditional classroom learn-
ing to more ubiquitous learning, enabling learners to
acquire right digital content at the right time and at the
right place (Chen & Li, 2010). Comas-Quinn,
Mardomingo, and Valentine (2009); Tan, Lin, Chu, and
Liu (2012); and Traxler (2007) argue that ubiquitous
Accepted: 27 April 2014
Correspondence: Sandy C. Li, Department of Education Studies,
Faculty of Social Science, Hong Kong Baptist University, Kowloon
Tong, Kowloon, Hong Kong. Email: sandyli@hkbu.edu.hk
people acquire information and construct knowledge. It
provides a platform that seamlessly connects private
and public learning spaces across formal and informal
contexts (Chan, Roschelle, Hsi, Kinshuk, & Sharples,
2006; Looi et al., 2010; Sharples, Taylor, & Vavoula,
2007). As such, there has been a proliferation of studies
on the use of Web 2.0 tools to facilitate collaborative
learning across contexts, ranging from English vocabu-
lary learning (Chen & Li, 2010) to scientific inquiry
learning (Liu, Peng, Wu, & Lin, 2009a; Liu, Tan, &
Chu, 2009b). Recently, the use of online social anno-
tation to facilitate collaborative learning has received
considerable attention (Huang, Huang, & Hsieh, 2008;
Huang, Jeng, & Huang, 2009; Lu & Deng, 2012,
2013). However, most of the earlier studies focus more

© 2014 John Wiley & Sons Ltd Journal of Computer Assisted Learning (2015), 31, 1–13 1
2 S.C. Li et al.
on the architecture of the system (Chao, Chen, &
Chang, 2010; Huang et al., 2008; Nokelainen,
Miettinen, Kurhila, Floréen, & Tirri, 2005; Wolfe,
2008), with only a few address the issues associated
with student learning performance (Lin & Lai, 2013;
Lu & Deng, 2013; Yang, Yu, & Sun, 2013). Little is
known about the interplay of cognitive, metacognitive
as well as other non-cognitive processes manifested in
the systems. In our present study, we aim to examine
how students’ learning trajectories progress in Diigo,
an online social bookmarking and annotation environ-
ment (Nelson, Christopher, & Mims, 2009; Twu,
2009), and how their cognitive processes and level of
interaction correlate with their learning outcomes. Spe-
cifically, the research questions for this study are three-
fold: (1) how do students’ cognitive and metacognitive
processes change over time? (2) how do students’ cog-
nitive and metacognitive processes manifested in the
environment correlate with students’ learning out-
comes? and (3) how does the level of collaboration
within a group affect students’ learning outcomes?
Through addressing these three questions, we could
deepen our understanding of how the use of an online
social annotation environment can facilitate collabora-
tive inquiry learning.
Web 2.0 for learning
In the first Web 2.0 Conference held in 2004, Tim
O’Reily (2005) outlined some preliminary principles to
help distinguish the first generation from the second
generation of World Wide Web. As one of the most
eye-catching principles among others, ‘Harnessing
Collective Intelligence’ has become a unique feature of
Web 2.0. With this ‘collective intelligence’ principle
(The New Media Consortium and the EDUCAUSE
Learning Initiative, 2008), Web 2.0 has been developed
in a way to provide more and more collaboration
opportunities in many areas of human development on
an unprecedented scale. This is made possible by the
user-friendly technologies to ‘publish’ people’s knowl-
edge, ideas, opinions and comments. As in education,
we witness the power of Web 2.0’s collective intelli-
gence with a growing number of opportunities for stu-
dents around the world to learn together through a
variety of social media and online collaboration plat-
forms. Tu, Blocher, and Roberts (2008) and Collis and
Moonen (2008) also suggest that Web 2.0 technologies,
such as social bookmarking and social annotation,
provide the affordance for promoting high-level cogni-
tive activities. The World Wide Web has become not
just our largest information repository but an indispen-
sable platform for exchanging information and
working collaboratively, making it the key information
source for inquiry and exploratory learning.
Social annotation
There have been findings showing that annotation can
improve students’ performance in reading and writing
(Yang et al., 2013). Marshall (1997) indicates that
annotation provides a myriad of useful functions in the
process of reading. They include procedural signals,
place markings and aids to memory, in situ problem
working, records of interpretive activities, traces of
readers’ attention and incidental reflections. The use of
annotation can promote deep learning by helping stu-
dents encode, decode and transform useful input infor-
mation (Nokelainen et al., 2005). However, Chao et al.
(2010) point out that traditional paper-based annotation
may pose physical and cognitive barriers to learners
when they want to organize or integrate information
across different printed media. In parallel with the
advancement of Web technology, a lot of work has been
done on the development of online annotation systems
(Chao et al., 2010; Glahn, Specht, & Koper, 2009;
Hoff, Wehling, & Rothkugel, 2009; Hsi, 2007; Huang
et al., 2008; Wolfe, 2008). Online annotation systems
can be categorized into two basic classes: the
discussion-oriented system where users can browse
and annotate threaded discussion messages on the
margins; and the document-oriented system where
users can browse and annotate Web-based documents
and media (Nokelainen et al., 2005). Most online anno-
tation systems provide a central repository where users
can store, retrieve and collate the annotations they
made in and across different documents. Some of the
Web-based annotation systems also provide social net-
working and threaded discussion facilities that enable
users to share and co-construct annotations with their
collaborators (Lu & Deng, 2013; Nelson et al., 2009;
Twu, 2009; Yang et al., 2013). There exists a body of
research studies on online annotation, but most of the
studies centre on the system design issues (Chao et al.,
2010; Huang et al., 2008; Nokelainen et al., 2005;
Wolfe, 2008), with a few looking into students’
© 2014 John Wiley & Sons Ltd
Social annotation and learning
performance and behaviours in using collaborative
annotations to support their learning (Lin & Lai, 2013;
Lu & Deng, 2013; Yang et al., 2013). Most of the
findings were derived from student perception surveys
or quasi-experiments that compared students’ sum-
mative assessment scores between the experimental
and control groups. Little is known about the cognitive,
metacognitive and other non-cognitive processes of
learning and how these processes are related to the
activities students are engaged in collaborative annota-
tion. A more detailed delineation of the students’ learn-
ing outcomes as well as the interplay of these processes
would deepen our understanding of the affordance of
collaborative annotation.
Inquiry-based learning
Inquiry-based learning can find its origin in construc-
tivist learning theories, such as Dewey (1938), Freire
(1984) and Vygotsky (1978). In his work on Science
education, Schwab (1962) construed inquiry-based
learning as ‘inquiry into inquiry’. Inquiry-based learn-
ing falls under the realm of inductive approaches to
learning which involves a process of exploration
through which learners ask questions, formulate and
rigorously test their hypotheses, and make conjectures
and refutations (Chang & Wang, 2009). Under these
premises, learners construct meanings and advance
their understanding through problem solving and
knowledge synthesis. Wilhelm (2007) identifies a set of
key elements for a successful inquiry, which include
(1) defining authentic and ill-structured problem(s); (2)
working in groups to enable students to develop multi-
ple perspectives; (3) students’ ownership over the pro-
blems; and (4) setting milestones. Jonassen (1997)
defines well-structured problems as restricted problems
with convergent solutions, and ill-structured problems
as those that possess multiple solutions and approaches
to solutions. Ill-structured problems usually contain
ambiguity about which principles, rules or theories are
essential for the solution. Thus, well-crafted inquiry-
based learning is instrumental in fostering students’
high-level cognitive and metacognitive abilities.
Cognitive, metacognitive and social processes
Meaningful inquiry learning requires less structured
problems through which learners are expected to
© 2014 John Wiley & Sons Ltd
3
orchestrate a multitude of cognitive, metacognitive and
social processes such as employing appropriate strat-
egies and modifying one’s behaviours to address the
changing demands of the tasks (Hammouri, 2003;
Montague, 1992). While there is an increasing reper-
toire of knowledge about the intricate nature of these
processes associated with learning (Billing, 2007;
Greene & Azevedo, 2010; Tsai, 2009; Wolfe & Flewitt,
2010; Yang, 2012), little is known about the relative
contributions of cognitive, metacognitive and social
processes to learning outcomes when students are
engaged in online collaborative inquiry activities. Pre-
vious studies indicate that proper use of metacognitive
strategies, to a large extent, enhances the likelihood
of success in problem solving (Hammouri, 2003;
Hartman & Everson, 1996). The term ‘metacognition’
first appears in the early work of Flavell (1976) who
defines it as the self-consciousness of one’s knowledge,
and self-monitoring and self-regulation of one’s own
cognitive processes and outcomes or anything related
to them. Biggs (1985) refers to meta-learners as those
who are aware of their learning goals, task demands
and personal cognitive resources, and able to deploy
appropriate strategies to accomplish the given tasks.
While cognitive strategies are related to integrating
new knowledge with one’s prior or existing knowledge,
metacognitive strategies involve planning and monitor-
ing reflective and regulatory activities that take place
before, during and after any cognitive activities (Li,
Kong, Lee, & Henri, 2006; Yang, 2012). In the context
of reading, Yang (2012) expounds that cognitive pro-
cesses refer to the actions readers take while interacting
directly with the text, whereas metacognitive processes
are deliberate, planned strategies by which readers
regulate and remediate their reading. In his study on
how cognitive and metacognitive processes affect
undergraduate students’ mathematical problem
solving, Hammouri (2003) modified Montague’s
(1992) coding scheme and defined cognitive processes
as (1) read to comprehend the problem situation; (2)
paraphrase the problem in one’s own words; (3) visu-
alize the problem with mental images; (4) make
hypotheses; (5) make conjectures and predictions; and
(6) evaluate the validity of solutions. To understand
students’ online interaction, Song and McNary (2011)
encode different levels of cognitive processes with the
keywords such as agree/disagree, inform, elaborate,
argue, explain, summarize and infer. Apart from
4 S.C. Li et al.
cognitive and metacognitive processes, the literature on
online collaborative learning indicates that successful
online problem solving is also linked to a wide range of
non-cognitive factors such as mutual trust, accountabil-
ity and social support (Hung, Lee, & Lim, 2012; Soller,
2001; Yang, 2012). To address the social dimension
of online learning, Song and McNary (2011) use ‘moti-
vate’, ‘appreciate’, ‘encourage’ and ‘reinforce’ to
encode behaviours that acknowledge peer actions
within a group. Hammouri (2003) asserts that the three
types of processes interlink with one another and that
metacognitive processes provide the impetus forces
which influence both cognitive and social behaviours.
Interestingly, it was found in some studies that
metacognitive processes showed no ‘direct’ effect on
student learning outcomes, yet they produced signifi-
cant ‘indirect’ effects on learning by means of cog-
nitive processes (Purpura, 1999; Yang, 2012). This sug-
gests that further work to unfold the interaction among
these processes is deemed necessary.
Methods
Context and participants
The present study scrutinizes how online social anno-
tation facilitates group-based inquiry learning. The
context of the study was a university-level course on
technology in education delivered with the support of
Moodle, an open-source course management system
adopted by the university concerned. This is one of the
core foundation courses of a teacher education pro-
gramme with an aim to develop in students the essen-
tial theories and pedagogical skills for integrating
technology in teaching and learning. The course cur-
riculum consists of an assessment component which
required students to engage themselves in a collabora-
tive inquiry-based learning project. The objective of
this group project is to enable students to develop first-
hand experience in using technology to support col-
laborative learning, and to help them identify and
reflect on the key issues arising from the process of
learning. Two classes on this foundation course were
offered respectively in the first and second semesters
during the academic year 2012–2013, with each class
comprising 24 students. Thus, a convenient sample of a
total of 48 students (14 male and 34 female) was
involved in this study. Each class was divided into
five groups, with each group comprising four to six
members.
Pedagogical design
To enhance students’ ownership over the inquiry, each
group was required to research into a chosen social
issue. To facilitate meaningful learning, feedback from
the teacher was provided to each group to ensure that
their chosen issue was authentic, complex and ill struc-
tured. The chosen topics range from social and eco-
nomic issues to environmental issues, for instance: (1)
Does globalization bring about prosperity or poverty?
(2) Should the right to public knowledge override the
right to privacy or vice versa? (3) Is the currently pro-
posed moral and national education an angel or a devil
to the Hong Kong society? (4) Should urban develop-
ment override environmental conservation or vice
versa? Each group was required to co-author a report
on the chosen issue to explicate the perspectives of
different stakeholders as well as the collective views of
the group with arguments and supporting evidence. To
facilitate collaborative inquiry, each group was
required to use Diigo to share their bookmarks, high-
light and tag relevant texts, make annotations with
sticky notes, and co-construct argumentation wherever
appropriate. Diigo provides students with a toolbar to
plug into their Web browsers across different platforms
such as desktop computers, tablets and smartphones.
Students can bookmark Web pages, highlight texts or
images, and make annotation with sticky notes while
surfing the Web with their mobile devices. As shown in
Figure 1, each sticky note is associated with a discus-
sion forum where they can build on the artefacts
created by their collaborators. For better organization,
retrieval and sorting of information, the system allows
students to assign a set of tags or keywords to each
bookmark. Students can also subscribe to Diigo’s Rich
Site Summary service to keep themselves informed of
other group members’ postings on the system.
The entire project was divided into four different
stages: Stage 1 – initialization, Stage 2 – exploration,
Stage 3 – consolidation, Stage 4 – celebration. At Stage
1, the teacher helped students identify a social issue of
interest, define each member’s role, set goals and mile-
stones for their learning, familiarize themselves with
the social annotation system, and set guidelines for
group collaboration. At Stage 2, each member was
© 2014 John Wiley & Sons Ltd
Social annotation and learning 5

Figure 1 An Example of an Annotated Website with Highlight Texts and Sticky Notes

expected to contribute at least a minimum number of
relevant websites with annotations, and to respond to or
cross-annotate at least one member’s postings each
week. At Stage 3, students were required to collate the
information they solicited as well as the annotations
and the content of their discussions. Based on the dis-
tilled information, they were expected to construct their
own mind maps and develop their arguments. At the
final stage, each group was invited to present their
findings and collective views and arguments over the
issue with supporting facts and evidence.
against the coding scheme given in Table 1. The unit of
analysis was a student’s posting on Diigo, which could
be a bookmark, a highlighted text or a sticky note.
To evaluate the quality of each posting, we adopted
Kim and Lee’s (2012) approach by attaching a three-
level Quality Score (QS) labelled zero, one and two
according to its levels of relevance and accuracy. The
posting was then further categorized and assigned
a Weighted Score according to its cognitive or
metacognitive levels given in Table 1. The data were
Table 1. Coding Scheme for Analysing Different Levels of
Activities Students were Engaged in Diigo

Data and coding Weighted

Dimension Label Code Score
Students’ bookmarks, highlighted texts, sticky notes,
discussion postings and their project scores were the Cognitive1 c01 Agree 1
data sources for the study. As suggested by Hammouri c02 Inform 1
c03 Elaborate 1
(2003), Purpura (1999) and Yang (2012), cognitive and c04 Classify 1
metacognitive processes have respectively direct and c05 Illustrate 1
indirect effects on student learning outcomes. To Cognitive2 c06 Question 3
c07 Criticize 3
unfold the interaction among these processes, we
c08 Summarize 5
modified Song and McNary’s (2011) coding scheme to c09 Synthesize 5
incorporate a metacognitive dimension into the frame- c10 Evaluate 5
work as shown in Table 1. As proposed by Li et al. Metacognitive m01 Reflect 7
m02 Manage 7
(2006), the metacognitive dimension encompasses m03 Plan 7
reflection and self-regulation, planning, and manage- Social s01 Appreciate –
ment of students’ own learning. All the highlighted s02 Request –
s03 Encourage –
texts, sticky notes and discussion postings were coded

© 2014 John Wiley & Sons Ltd

6 S.C. Li et al.

input into a spreadsheet manually and coded by one
researcher and one research assistant with an inter-rater
reliability of above 0.7. The Aggregate Score of each
posted item was defined as the product of its QS and
Weighted Score.
To delineate the progression of students’ cognitive
processes, we computed three scores (Cognitive1
Score, Cognitive2 Score, Metacognitive Score) to char-
acterize different levels and types of activities identi-
fied in their inquiry learning. For each student, we then
computed a Cognitive1 Score by adding the Aggregate
Scores of all the items posted by the student under the
Cognitive1 dimension. Likewise, Cognitive2 Score and
Metacognitive Score were computed in the same
fashion to reflect the levels of students’ engagement in
the collaborative inquiry learning.
Results
Students’ activity profiles in Diigo environment
The numbers of bookmarks, highlighted texts, sticky
notes and discussion threads posted by each group
were depicted in Table 2. The average number of
posted items per person in each group (M) ranged from
7.8 to 72, and the standard deviation (sd) varied from
5.50 to 46.1. The average number of posted items, M,
can be used as a measure to gauge the intensity of each
group’s activities in Diigo. Based on the M values
given in Table 2, Group F and Group G were respec-
tively the most active and inert groups. To better illus-
trate the variation in members’ contribution in each
group, we defined the normalized standard deviation,
sd’, as the sd divided by the total number of posted
items. sd’ thus represents the sd of the fraction of
postings contributed by members within a group. In
this case, a smaller sd’ value indicates that the number
of posted items is more evenly distributed among
members of a group. As given in Table 2, Group A had
the largest sd’ value, signifying that the posted items
were contributed mostly by a few group members. To
gauge the level of collaboration within a group, we
computed a new quantity, Collab, by dividing M by
sd’. A high score of Collab implies that the group
activities in Diigo are intensive (with a large M value),
and/or each group member has a fair contribution to the
activities (a low sd’ value). As such, Group A and
Group F exhibited a high level of collaboration as com-
pared to the other groups (see Table 2). ANOVA was
also conducted on the total number of posted items
against gender, and no statistically significant differ-
ence was found between the male and female groups.
To delineate the levels of activities students were
engaged in, all posted items including bookmarks,
highlighted texts and sticky notes were coded with
respect to the scheme given in Table 1. The cognitive
dimension is divided into ‘Cognitive1’ referring to
Level-1 cognitive processes such as ‘agree’, ‘disagree’,
‘inform’, ‘classify’ and ‘elaborate’, and ‘Cognitive2’
referring to processes of a higher cognitive level such
as ‘question’, ‘criticize’, ‘summarize’, ‘synthesize’
and ‘evaluate’. The processes associated with the
metacognitive dimension were categorized into three
groups, namely ‘reflect’, ‘manage’ and ‘plan’, while
the social processes were coded with the keywords:
‘appreciate’, ‘request’ and ‘encourage’. With the
scheme given in Table 1, most of the text-highlighting
activities were coded under ‘Cognitive1’. A sample of
sticky notes posted by students is given in Table 3. The

Table 2. The Mean and Standard Deviation of Each Group’s Postings in Diigo

Highlighted Sticky Discussion Collab

Group Bookmark text note thread Total M SD SD’ (M/SD’)

A 41 91 96 0 228 45.6 46.1 0.202 225.4

B 9 13 20 1 43 8.6 4.31 0.102 84.12
C 28 41 76 1 146 29.2 17.1 0.117 248.7
D 20 65 52 0 137 34.3 10.5 0.077 446.8
E 31 64 65 0 160 32.0 13.8 0.087 368.5
F 29 100 159 0 288 72.0 24.9 0.087 829.6
G 11 10 10 0 31 7.8 5.50 0.179 43.25
H 16 140 36 0 192 38.4 38.7 0.202 190.2
I 24 104 27 0 155 31.0 14.5 0.094 329.2
J 10 66 33 0 109 18.2 8.43 0.078 233.7

SD’ = normalized standard deviation.

© 2014 John Wiley & Sons Ltd

Social annotation and learning 7

Table 3. A Sample of Sticky Notes Posted by Students

Dimension Code Sticky note

Cognitive1 Agree Agreed! The radioactive waste disposal is a big problem for our next generation!
Disagree I don’t agree with Kelvin! Wiki only provides information, all you need is critical
thinking to identify the information is fake or not.
Inform For more information about nuclear disaster, please refer to the website’s pdf file.
Elaborate Wow! I see, nuclear power is not clean and 100% environmental, the waste cannot be
destroyed and, it may leak out and cause troubles.
Classify . . . these refer to problems arising from GM food . . . .
Illustrate WHO claimed that ‘GM foods currently available on the international market have
passed risk assessments and are not likely to present risks to human health.’
Cognitive2 Question Can students develop their critical thinking if the national education curriculum is
implemented? It really depends on the teaching materials and teachers’ standpoints.
Criticize The views presented in this website cannot represent the opinions of the majority of the
residents living at Tai Po. We even don’t know the background of this society and its
composition. The author doesn’t indicate the sample size for their opinion poll.
Summarize To sum up, the development of north east areas in Hong Kong would probably bring
lots of benefits according to writer’s opinions: (1) increase the residential uses to
satisfy the increasing demand of population; (2) promote the development of
high-tech industry . . . and (3) promote the long-term economic development . . . .
Synthesize According to the statistics provided, we can see that the development of north east area
of Hong Kong can largely increase the housing supply, both in public and private
housing, which can greatly reduce the existing pressure on accommodation.
Evaluate Excellent article from the Conservation Group of Northeast New Territories. It provided
their detailed reasons to oppose the development plan. More importantly, suggestions
were given for further improvement of the plan. Please read it all of you!
Metacognitive Reflect It’s a kind of an irony that it is more likely to let student become less capable to
distinguish right from wrong as some wrong information or information that had not
been proven right is included in the curriculum.
Manage We need to have a good division of labour. Apart from working on your assigned area,
please also forward the information to us whenever you come across useful websites.
Plan The words no. should be about 2000 words, so I think the background of the issue
should be 400 words (250 words include Hong Kong context), . . . I will first make a
draft about the background of the issue, than you all make some comment or edit,
ok?
Social Appreciate Thanks to John. It is the official document that I am looking for! It is of great help to
me . . . . . . .
Request Sorry that I cannot open the link of the website! Can you check this link for me?
Encourage It is great! . . . . The pictures are well presented and I do like that! Well done!

figures given in Table 4 indicate that most of the activ-
ities in Diigo were of ‘Cognitive1’ type. As the level
of cognitive activities escalated, the number of posted
items decreased. The numbers of posted items relating
to metacognitive and social activities were small as
compared to other types of activities. As 5 out of the 10
groups showed little evidence of social activities, our
discussion will mainly focus on activities associated
with Cognitive1, Cognitive2 and Metacognitive
dimensions.
Figure 2 shows the variation of different types of
activities over time. It was evident that the number of
Cognitive1, Cognitive2 and Metacognitive activities
recorded during a day had a strong association
with each other. The more Cognitive1 activities stu-
dents were engaged in, the more Cognitive2 and
Metacognitive activities were recorded in Diigo. As
mentioned above, Cognitive1 activities comprised
mainly of text-highlighting activities. Thus, the
results depicted in Figure 2 seem to indicate that text-
highlighting activities help provide a context
for students to develop high-level cognitive and
metacognitive activities. To further elaborate the rela-
tionship among the three types of activities over time,
we used IBM SPSS 21 package to examine the cross-
correlation of the three time series, Cognitive1 Activ-
ities (T_C1), Cognitive2 Activities (T_C2) and
Metacognitive Activities (T_MC), across different

© 2014 John Wiley & Sons Ltd

8 S.C. Li et al.

Table 4. Numbers of Cognitive1,

Dimension Cognitive2, Metacognitive and Social
Activities Identified in Each Group
Group Cognitive1 Cognitive2 Metacognitive Social Total

A 172 28 5 23 228
B 27 9 1 6 43
C 102 33 11 0 146
D 108 25 3 1 137
E 115 33 1 11 160
F 161 102 11 14 288
G 24 7 0 0 31
H 165 26 1 0 192
I 143 8 4 0 155
J 86 22 1 0 109

groups. The results presented in Table 5 indicate that

the three types of activities interweaved with one
another: T_C1 correlated strongly with T_C2 and
T_MC across different groups, except for Group G, as
no metacognitive activity was identified in the group.
In other words, the more Cognitive1 activities a group
was engaged in during a day, the more Cognitive2 and
Metacognitive activities were promoted.
Multiple regression
To quantify and differentiate different levels of activ-
ities, we computed for each student three scores, Cog-
nitive1 Score (C1), Cognitive2 Score (C2) and
Metacognitive Score (MC), by multiplying each posted
item’s cognitive level score with its QS. The Total Score
(TS) was calculated by adding Cognitive1, Cognitive2

Figure 2 Variation of the Numbers of Cognitive1, Cognitive2 and Metacognitive Activities Over Time

© 2014 John Wiley & Sons Ltd

Social annotation and learning 9

Table 5. Cross-Correlation of the Time Series for Cognitive1, Cognitive2 and Metacognitive Activities Across Different Groups

Group

A B C D E F G H I J

r1 .816** 1.0** .70** .97** .75** .88** .66* .93** .93** .99**
r2 .300** 1.0** .36** .78** .51** .61** – .27* .54** .95**

r1 = cross-correlation coefficients between T_C1 and T_C2. r2 = cross-correlation coefficients between T_C1 and T_MC.
*p < 0.05; **p < 0.01.

and Metacognitive Score together. The results given in
Table 6 indicate that the numbers of bookmarks (BM),
highlighted texts (HL) and sticky notes (SN) posted by
each student strongly correlated with each other. Book-
marks, highlighted texts and sticky notes also have
strong correlation with TS, Cognitive1, Cognitive2 and
Metacognitive Score. In addition, highlighted texts,
sticky notes, Cognitive1, Cognitive2 and TS correlated
strongly with Project Score (PS), while the correlation
between bookmarks and PS, and between Metaco-
gnitive Score and PS, was not statistically significant.
To further examine the interplay among students’
different levels of cognitive activities and their project
scores, multiple regression analysis was conducted
with IBM SPSS 21 package. Four regression models
were depicted in Table 7. The R2 values given in Model
1, Model 2 and Model 3 indicate that TS, Weighted
Cognitive1 Score and Cognitive2 Score are strongly
associated with PS, explaining respectively 14%, 15%
and 8% of the variance in PS. In Model 4, Cognitive1
and Cognitive2 were put together as the two independ-
ent variables for the analysis. The results indicate that
the regression coefficient for Cognitive2 is statistically
insignificant. After controlling for the effect of Cogni-
tive1 on PS, the % variance of PS explained by Cog-
nitive2 drops from 8% to 3.7% (16.8%–15.1%),
indicating that Cognitive1 mediates the association
between Cognitive2 and PS.

Table 6. Correlation Among Different Types of Activities Students Engaged in and Their Project Scores

Correlation coefficient

BM HL SN TS C1 C2 MC PS

Bookmark (BM) 1 .456** .620** .603** .728** .351* .508** .254

Highlight (HL) .456** 1 .462** .597** .898** .307* .305* .348*
Sticky note (SN) .620** .462** 1 .934** .701** .843** .594** .316*
Total Score (TS) .603** .597** .934** 1 .748** .898** .654** .370**
Cognitive1 Score (C1) .728** .898** .701** .748** 1 .424** .486** .389**
Cognitive2 Score (C2) .351* .307* .843** .898** .424** 1 .415** .283*
Metacognitive Score (MC) .508** .305* .594** .654** .486** .415** 1 .196
Project Score (PS) .254 .348* .316* .370** .389** .283* .196 1

*p < 0.05 level (two tailed); **p < 0.01 level (two tailed).

Table 7. Multiple Regression Models with

Project Score as the Dependent Variable Regression (Project Score as dependent variable)

Model Independent variable β t R2 F

1 Total Score (TS) .370* 2.70** .137** 7.30**

2 Cognitive1 Score (C1) .389** 2.86** .151** 8.20**
3 Cognitive2 Score (C2) .283* 2.00* .080* 4.01*
4 Cognitive1 Score (C1) .328* 2.19* .168* 4.55*
Cognitive2 Score (C2) .144 0.96

*p < 0.05; **p < 0.01.

© 2014 John Wiley & Sons Ltd

10 S.C. Li et al.

Table 8. Two-Level Models for Examining

Variance component the Group Effects on Project Score,
Model Description γ01 of u0j P-value Cognitive2 Score and Metacognitive Score

A0 (PS)ij = β0j + rij – 7.62 <0.001

β0j = γ00 + u0j
A1 (PS)ij = β0j + rij 0.05 3.75 0.003
β0j = γ00 + γ01 (G_HL)j + u0j
B0 (WC2)ij = β0j + rij – 4160 0.020
β0j = γ00 + u0j
B1 (WC2)ij = β0j + rij 0.26 972 0.002
β0j = γ00 + γ01 (Collab)j + u0j
C0 (WMC)ij = β0j + rij – 95.2 0.019
β0j = γ00 + u0j
C1 (WMC)ij = β0j + rij 0.04 28.0 <0.001
β0j = γ00 + γ01(Collab)j + u0j

results were obtained, and the % variance of (PS)ij

Multilevel analysis
Three sets of two-level models (A0, A1, B0, B1, C0, C1)
were constructed to examine the group effect of the
Cognitive1 Score, Cognitive2 Score, Metacognitive
Score and PS using HLM 7 (Scientific Software Inter-
national) package. As shown in Table 8, Model A0,
Model B0 and Model C0 are the fully unconditional
models in which PS, Cognitive2 Score and
Metacognitive Score are the respective outcome vari-
ables, and no Level-1 and Level-2 predictors are
chosen. In these fully unconditional models, the vari-
ance of the outcome variable in each model is divided
into two components: the within-group variance (rij)
and the between-group variance (u0j). Models A1, B1
and C1 are models with no Level-1 predictors but a
single Level-2 predictor. The Level-2 predictor is used
to examine how much the unconditional between-
group variance can be explained by the model. In
Model A1, the total number of highlighted texts posted
by each group (G_HL)j is chosen as the Level-2 pre-
dictor. The % difference between the unconditional
variance in Project Score, (PS)ij, over groups and vari-
ance in (PS)ij over groups after each group’s total
number of highlighted texts posted has been taken into
account is equal to 50.7% ([7.62 – 3.75]/7.62*100%).
Thus, the average number of highlighted texts posted
per person in a group, (G_HL)j, has a high predictive
power over the variation in group-project performance
among the groups. Similar models were constructed
using the total number of bookmarks and the total
number of sticky notes posted by each group as the
Level-2 predictors, but no statistically significant
explained by the predictors is negligible. In Model B1,
each group’s level of collaboration score, (Collab)j, is
chosen as the Level-2 predictor. The % difference
between the unconditional variance in Cognitive2
Score, (C2)ij, over groups and variance in (C2)ij over
groups after each group’s (Collab)j has been taken into
account is equal to 76.6% ([4160 – 972]/4160*100%).
Likewise, each group’s level of collaboration score,
(Collab)j, is also chosen as the Level-2 predictor for
Model C1. The % difference between the unconditional
variance in Metacognitive Score, (MC)ij, over groups
and variance in (MC)ij over groups after each group’s
(Collab)j has been taken into account is equal to 70.6%
([95.2 – 28]/95.2*100%). Thus, the level of collabora-
tion of each group accounted for the variation in
each group’s high-level cognitive and metacognitive
activities. A similar two-level model was constructed
to examine the group-level effect on Cognitive1 Score.
However, no statistically significant result was derived.
Discussion and conclusion
The results presented above deepen our understanding
of how the use of online social annotation facilitates
collaborative inquiry learning. Three different types of
activities, namely cognitive, metacognitive and social,
were identified in the online environment. The time
series of the Cognitive1 activities correlates strongly
with those of the Cognitive2 and Metacognitive activ-
ities, indicating that a kind of symbiotic or ecological
relationship exists among the three types of activities.
The more Cognitive1 activities students were engaged

© 2014 John Wiley & Sons Ltd

Social annotation and learning
in during a day, the more Cognitive2 and metacognitive
activities were fostered in a group. The numbers of the
three types of activities formed a pyramid similar to
the population pyramid found in an ecosystem, with the
number of Cognitive1 activities at the bottom and the
number of metacognitive activities at the top. The cor-
relation analysis indicates that the Cognitive1 activ-
ities, such as highlighting text, correlate with higher
level of activities and also student’s project score. That
is, the more highlighted texts a student posted, the more
Cognitive2 and Metacognitive activities he/she
engaged in, and the higher project score he/she
attained. The multiple regression analysis also shows
that Cognitive1 activities mediate the effect of Cogni-
tive2 activities on student’s performance in the group
project. The effect of Metacognitive activities was
found to be statistically not significant. The results
echo the findings derived from the studies conducted
by Hammouri (2003), Purpura (1999) and Yang
(2012), that metacognitive processes have only indirect
effect on student learning outcomes. It could be attrib-
uted to the fact that the aggregated number of
metacognitive activities was relatively small as com-
pared to the number of the cognitive activities, so it was
not sufficient to account for the variation in students’
project scores. To look further into the effect of
metacognitive activities on students’ learning out-
comes, a longitudinal study with a larger sample size
and a longer project span is required.
The effect of text-highlighting activities on student
learning was further elaborated with the results of the
multilevel analysis. It was shown that the average
number of highlighted texts posted per person in a group
accounted for most of the variation of project scores
across the groups, indicating that text-highlighting
helped enhance group inquiry and student learning out-
comes. Our interpretation is that the highlight function
provided by the online annotation environment helps
students to have a more in-depth comprehension of
information and the texts highlighted provide a rich
context for students to engage themselves in further
group deliberations and high levels of cognitive and
metacognitive activities. Our findings contrast with the
results of some studies on traditional discussion boards
(Nisbet, 2004; Song & McNary, 2011) that students’
postings are of little value and did not contribute to
learning. The results of the multilevel analysis further
signify that the level of collaboration within a group
© 2014 John Wiley & Sons Ltd
11
accounts for most of the variation of Cognitive2 Score
and Metacognitive Score across the groups, suggesting
that active participation and a distributive share of con-
tribution among group members help promote high
levels of cognitive and metacognitive activities. This
finding echoes with the notion of social constructivism
that culture and the social context is of critical impor-
tance for cognitive development. It is evident that the
online social annotation environment provides the
affordance for promoting high-level cognitive and
metacognitive activities. In particular, the text-
highlighting facility not only levers learners’ compre-
hension ability but also provides the necessary anchors
and scaffolds for learners to engage in meaning nego-
tiations and knowledge building.
While discussing the affordance of the social anno-
tation system, we should not play down the role of
pedagogical and social factors in engendering mean-
ingful inquiry learning experiences. Obviously, tech-
nology per se may not suffice to boost learning
motivation nor foster high-level cognitive and
metacognitive processes. The pedagogical factors such
as the authenticity and the ill-structured nature of prob-
lems, students’ ownership over the problems, and teac-
her’s facilitation remain crucial for designing inquiry
learning and engaging learners.
Limitations and suggestions
The purpose of this research is not to make any gener-
alized claims on the use of online social annotation
environment to support learning. Thus, the results of
our study should be interpreted as illustrative rather
than definitive. Although the use of online social anno-
tation environment seems to have a positive effect on
student learning outcomes, yet our knowledge of what
factors are conducive to fostering collaboration and
different levels of cognitive and metacognitive activ-
ities is still limited. Furthermore, a more detailed
analysis of the time series by shortening the time inter-
val for event recording from 1 day to an hour or even a
minute will deepen our understanding of the interplay
of different levels of cognitive and metacognitive activ-
ities. Last but not least, the pedagogical factors such as
teacher’s facilitation and the pedagogical design
warrant our attention. To shed light on the above issues,
a longitudinal study with in-depth interviews for both
teacher and students is deemed necessary.
12
References
Biggs, J. B. (1985). The role of metalearning in study pro-
cesses. The British Journal of Educational Psychology, 55,
185–212.
Billing, D. (2007). Teaching for transfer of core/key skills in
higher education: Cognitive skills. Higher Education, 53,
483–516.
Chan, T.-W., Roschelle, J., Hsi, S., Kinshuk, K., & Sharples,
M. (2006). One-to-one technology enhanced learning: An
opportunity for global research collaboration. Research
and Practice of Technology Enhanced Learning, 1,
3–29.
Chang, C.-Y., & Wang, H.-C. (2009). Issues of inquiry learn-
ing in digital learning environments. British Journal of
Educational Technology, 40, 169–173.
Chao, P.-Y., Chen, G.-D., & Chang, C.-W. (2010). Develop-
ing a cross-media system to facilitate question-driven
digital annotations on paper textbooks. Educational Tech-
nology & Society, 13, 38–49.
Chen, C.-M., & Li, Y.-L. (2010). Personalised context-aware
ubiquitous learning system for supporting effective
English vocabulary learning. Interactive Learning Envi-
ronments, 18, 341–364.
Collis, B., & Moonen, J. (2008). Web 2.0 tools and processes
in higher education: Quality perspectives. Educational
Media International, 45, 93–106.
Comas-Quinn, A., Mardomingo, R., & Valentine, C. (2009).
Mobile blogs in language learning: Making the most of
informal and situated learning opportunities. ReCALL, 21,
96–112.
Dewey, J. (1938). Logic: the theory of inquiry. New York,
NY: Rinehart and Winston.
Flavell, J. H. (1976). Metacognitive aspects of problem
solving. In L. B. Resnick (Ed.), The nature of intelligence
(pp. 231–235). Hillsdale, NJ: Erlbaum.
Freire, P. (1984). Pedagogy of the oppressed. New York, NY:
Continuum Publishing Company.
Glahn, C., Specht, M., & Koper, R. (2009). Visualisation
of interaction footprints for engagement in online
communities. Educational Technology & Society, 12,
44–57.
Greene, J. A., & Azevedo, R. (2010). The measurement of
learners’ self-regulated cognitive and metacognitive pro-
cesses while using computer-based learning environments.
Educational Psychologist, 45, 203–209.
Hammouri, H. A. M. (2003). An investigation of undergradu-
ates’ transformational problem solving strategies:
Cognitive/metacognitive processes as predictors of
holistic/analytic strategies. Assessment & Evaluation in
Higher Education, 28, 571–586.
S.C. Li et al.
Hartman, H., & Everson, H. (1996). Self-concept and
metacognition in ethnic minorities. Urban Education, 31,
222–237.
Hoff, C., Wehling, U., & Rothkugel, S. (2009). From paper-
and-pen annotations to artefact-based mobile learning.
Journal of Computer Assisted Learning, 25, 219–237.
Hsi, S. (2007). Conceptualizing learning from the everyday
activities of digital kids. International Journal of Science
Education, 29, 1509–1529.
Huang, Y.-M., Huang, T.-C., & Hsieh, M.-Y. (2008). Using
annotation services in a ubiquitous Jigsaw cooperative
learning environment. Educational Technology & Society,
11, 3–15.
Huang, Y.-M., Jeng, Y.-L., & Huang, T.-C. (2009). An edu-
cational mobile blogging system for supporting collabora-
tive learning. Journal of Educational Technology &
Society, 12, 163–175.
Hung, D., Lee, S.-S., & Lim, K. Y. T. (2012). Authenticity in
learning for the twenty-first century: Bridging the formal
and the informal. Educational Technology Research and
Development, 60, 1071–1091.
Jonassen, D. H. (1997). Instructional design models for
well-structured and ill-structured problem-solving learn-
ing outcomes. Educational Technology: Research and
Development, 45, 65–95.
Kim, M., & Lee, E. (2012). A multidimensional analysis tool
for visualizing online interactions. Educational Technol-
ogy & Society, 15, 89–102.
Li, S. C., Kong, S. C., Lee, F. L., & Henri, J. (2006). Capacity
building for lifelong learning: A study of practitioners’
perceptions on information literacy. Informatics in Educa-
tion, 5, 231–244.
Lin, J.-W., & Lai, Y.-C. (2013). Harnessing collaborative
annotations on online formative assessments. Educational
Technology & Society, 16, 263–274.
Liu, T.-C., Peng, H., Wu, W.-H., & Lin, M.-S. (2009a). The
effects of mobile natural-science learning based on the 5E
learning cycle: A case study. Journal of Educational Tech-
nology & Society, 12, 344–358.
Liu, T.-Y., Tan, T.-H., & Chu, Y.-L. (2009b). Outdoor natural
science learning with an RFID-supported immersive
ubiquitous learning environment. Journal of Educational
Technology & Society, 12, 161–175.
Looi, C.-K., Seow, P., Zhang, B., So, H.-J., Chen, W., &
Wong, L.-H. (2010). Leveraging mobile technology for
sustainable seamless learning: A research agenda. British
Journal of Educational Technology, 41, 154–169.
Lu, J., & Deng, L. (2012). Reading actively online: An
exploratory investigation of online annotation tools for
inquiry learning. Canadian Journal of Learning and Tech-
nology, 38, 1–16.
© 2014 John Wiley & Sons Ltd
Social annotation and learning
Lu, J., & Deng, L. (2013). Examining students’ use of online
annotation tools in support of argumentative reading.
Australasian Journal of Educational Technology, 29, 161–
171.
Marshall, C. C. (1997) Annotation: From paper books to the
digital library. Retrieved from http://www.csdl.tamu.edu/
∼marshall/dl97.pdf (last accessed 26 February 2014).
Montague, M. (1992). The effect of cognitive and
metacognitive strategy instruction on the mathematical
problem solving of middle school students with learning
disabilities. Journal of Learning Disabilities, 25, 230–
248.
Nelson, J., Christopher, A., & Mims, C. (2009). TPACK and
Web 2.0: Transformation of Teaching and Learning.
TechTrends: Linking Research & Practice to Improve
Learning, 53, 80–85.
Nisbet, D. (2004). Measuring the quantity and quality of
online discussion group interaction. Journal of eLiteracy,
1, 122–139.
Nokelainen, P., Miettinen, M., Kurhila, J., Floréen, P., &
Tirri, H. (2005). A shared document-based annotation tool
to support learner-centred collaborative learning. British
Journal of Educational Technology, 36, 757–770.
O’Reily, T. (2005) What is Web 2.0? O’Reilly. Retrieved
from http://oreilly.com/web2/archive/what-is-web-20.html
(last accessed 4 September).
Purpura, J. E. (1999). Learner strategy use and performance
on language tests: A structural equation modeling
approach. Cambridge: University of Cambridge Local
Examinations Syndicate and Cambridge University Press.
Schwab, J. (1962). The teaching of science as inquiry. In
J. Schwab & P. Braandwein (Eds.), The teaching of science
(pp. 3–103). Cambridge, MA: Harvard University Press.
Sharples, M., Taylor, J., & Vavoula, G. (2007). A theory of
learning for the mobile age. In R. Andrews & C.
Haythornthwaite (Eds.), Sage handbook of Elearning
research (pp. 221–247). London: Sage Publications.
Soller, A. L. (2001). Supporting social interaction in an intel-
ligent collaborative learning system. International Journal
of Artificial Intelligence in Education, 12, 40–62.
Song, L., & McNary, S. W. (2011). Understanding students’
online interaction: Analysis of discussion board postings.
Journal of Interactive Online Learning, 10, 1–14.
© 2014 John Wiley & Sons Ltd
13
Tan, T.-H., Lin, M.-S., Chu, Y.-L., & Liu, T.-Y. (2012). Edu-
cational affordances of a ubiquitous learning environment
in a natural science course. Educational Technology &
Society, 15, 206–219.
The New Media Consortium and the EDUCAUSE Learning
Initiative. (2008) The Horizon Report 2008. 2008 ed. CA.
Traxler, J. (2007). Defining, discussing and evaluating
mobile learning: The moving finger writes and having writ.
International Review of Research in Open and Distance
Learning, 8, 1–12.
Tsai, M.-J. (2009). The model of strategic e-Learning:
Understanding and evaluating student e-Learning from
metacognitive perspectives. Educational Technology &
Society, 12, 34–48.
Tu, C.-H., Blocher, M., & Roberts, G. (2008). Constructs for
Web 2.0 learning environments: A theatrical metaphor.
Educational Media International, 45, 253–269.
Twu, H.-L. (2009). Effective wiki strategies to support high-
context culture learners. TechTrends: Linking Research &
Practice to Improve Learning, 53, 16–22.
Vygotsky, L. S. (1978). Mind in society: The development of
higher psychological processes. Cambridge, MA: Harvard
University Press.
Wilhelm, J. D. (2007). Engaging readers & writers with
inquiry (theory and practice). New York, NY: Scholastic
Inc.
Wolfe, J. (2008). Annotations and the collaborative digital
library: Effects of an aligned annotation interface on
student argumentation and reading strategies. Interna-
tional Journal of Computer-Supported Collaborative
Learning, 3, 141–164.
Wolfe, S., & Flewitt, R. (2010). New technologies, new
multimodal literacy practices and young children’s
metacognitive development. Cambridge Journal of Educa-
tion, 40, 387–399.
Yang, K.-L. (2012). Structures of cognitive and
metacognitive reading strategy use for reading comprehen-
sion of geometry proof. Educational Studies in Mathemat-
ics, 80, 307–326.
Yang, X., Yu, S., & Sun, Z. (2013). The effect of collabora-
tive annotation on Chinese reading level in primary
schools in China. British Journal of Educational Technol-
ogy, 44, 95–111.