You are on page 1of 11

Using Logical Sensor Network to Support Context-Aware Learning Assessment

Gislaine C.M. Rosales, Delano Beder, Joice L. Otsuka, Regina B. Araujo Department of Computer Science University of So Carlos So Carlos, SP Brazil
{gislaine, delano, regina}@dc.ufscar.br, joice@ufscar.br

Abstract. Early detection of learning issues can contribute to mitigate problems such as poor school performance, low participation of students and high dropout rates. Most of the existing solutions to learning assessment are based on a post evaluation of data and does not support real-time interpretation and feedback. This paper presents a context-aware learning assessment system that can provide a more precise monitoring of the students learning process. The novelty of our system lies on the use of physical and logical sensors for the accurate tracking of events that can influence the teaching and learning process. The events captured by sensors are modeled as contexts in ontologies. A simple and usable interface is offered to students, tutors, teachers and course coordinators.

1. Introduction
The continuous emergence of new information and communication technologies, from interactive, collaborative, context-aware and immersive multimedia and 3D simulations, to multi-modal interfaces, augmented reality and broadband wireless networks, to mention a few, makes it possible for teachers and students, to envisage new and powerful forms of creation, distribution and consumption of content for distance education courses. Despite these exciting technologies, making courses content more accessible to students is often achieved by replicating information in different formats and to different virtual spaces what generates a mass of content, version conflict and maintenance difficulty. This way of organizing theoretical contents and exercises, which makes up a course, makes it more difficult for both students and teachers to track academic progress and organize activities in an orderly manner. Moreover, it is often observed in distance education production environments the difficulty in the management of information for teachers, course coordinators, instructional designers and other managers of the process, mainly due to asynchrony of study intervals. It is well known the benefits of indirect mediation in space and time provided by distance education, where students can organize their own activities according to their convenience, allowing some students to attend few disciplines in a condensed manner, while others can attend various parallel disciplines. However, the monitoring and the assessment of all students of a discipline/course become laborious and complex. Also, with existing tools for assessing learning process, it is a complex task to identify ongoing learning difficulties of each student. As a consequence, poor school performance, low students participation and high dropout rates are expected.

This paper introduces a powerful system for the monitoring and assessing of the teaching and learning process in distance education courses. The novelty of our system lies on the use of physical and logical sensors (the latter as a software abstraction) for the accurate tracking of events that can influence the teaching and learning process. Our approach considers not only the logical environment the user is in but also his/her surrounding physical environment. The main characteristics of our system include: accurate monitoring of localized events; real-time and posterior feedback to LMSs users on the learning process; more secure online evaluations through accurate evaluation and monitoring of both physical locations where online evaluations will take place (e.g., computer laboratory) and logical environments (LMS and other computer applications running in parallel); LMS platform independence, among others. The main goals of our system are: to help teachers, tutors and course coordinators to better understand students learning difficulties and lack of motivation so that early actions can be taken; to provide a better understand of users behavior patterns and their different learning styles; to help students to better organize themselves along the course and improve performance. Similar goals have been pursued by researchers in the area, such as Chen et al. (2007a); Chen et al. (2007b); Chen and Chen (2009); Dringus and Ellis (2005); Enright et al. (2008); Huang et al. (2009); Otsuka (2007); Zaane and Luo (2001). However, existing solutions focus typically on storage and posterior data analysis, providing little or no support for real-time feedback. They also lack support for the monitoring of special types of actions, such as users interaction intention.

This paper is organized as follows: Section 2 presents the related works. Section 3 introduces our system and the use of contexts and ontologies for the learning process monitoring. Section 4 presents a brief description of our systems architecture. Three case studies to be used to validate the proposed system are described in Section 5, followed by Conclusions and References.

2. Related Works
There are several interesting works that propose solutions for the monitoring and assessment of learning in LMSs. Dringus and Ellis (2005) proposed a data mining system that identifies common participation indicators in order to judge the entries in the forum tool and check adjustment of the teaching and learning process during its development. The performed analysis is based on the data mes-sages exchanged by students during the discussion, which are stored in the LMS database. Zaane and Luo (2001), presents a framework based on data mining techniques to analyze web server access logs. By tracking the students actions, logs are obtained and behavior patterns are identified; they are essential because assist in the learning assessment process. The system provides a configured interface through which the user can set restrictions for the collection, the steps of discovery and analysis of patterns of behavior, applying filters according to their research interest. Enright et al. (2008) proposes an intelligent tutoring system, applying data mining techniques that provides the teacher the possibility of detecting possible patterns of repeated actions by students that may indicate problems with the adopted teaching strategies; it also enables the teacher to make changes in the classroom and suggestions for new content. A model designed to provide formative assessment support in LMSs, based in a multi-agent architecture and the use of ontologies, is presented by Otsuka et

al. (2007). Its architecture is primarily based on two types of agents: (i) activity monitoring agents, which collect data on student participation in each activity, according to specified criteria, such as regular access and postings in forums, and the number of posts; (ii) analyzer agents, which are responsible for analyzing the participation indicators collected from a particular activity (such as Discussion Forums, Chat and Portfolio), on a knowledge base. Those agents may indicate patterns of behavior which might infer in the occurrence of problems, alerts and quality in holdings. Chen et al. (2007a) presents a framework that provides a communication interface based on intelligent agents: interaction, instruction, information, assessment, and log agents. The model is based on data analysis from databases and web logs that contain information about the interactions of participants in the LMS. It also presents the users performance in the activities, their efforts and progresses. The system sends messages automatically to teachers when a student performance is low, which is a situation that requires more attention and monitoring. Chen et al. (2007b) combines techniques of data mining to identify factors of learning; they also use fuzzy analysis to identify the students' performance during the assessment process in portfolios. The system presented by Huang et al. (2009) supports formative assessment and self-evaluation in mobile learning environments. Based on the answers given by students, the system performs adaptive assessment, which is based on successes and mistakes, and also the technological capacity of users' devices to refine the assessment by presenting issues of levels more or less complex. The system uses access to the databases that store information about students interactions in order to define the issues to be included in the ratings. In a similar proposal to the previous one, Chen and Chen (2009) presents a tool for formative assessment techniques using mobile data mining, and rules of inference based on fuzzy logic, which enables the teacher to monitor the students' learning. Both teacher and students can access the environment using a mobile device. The system is based on agents, such as an agent of formative assessment and feedback to staff who work in the students responses analysis to define evaluation rules that are used to infer the level of individual learning in each student. It can be observed that the presented solutions focus on storage and subsequent analysis of data that were stored in databases or obtained from system logs generated by users or by applications. However, the proposed tools do not support the monitoring of special types of actions performed in the environment, nor the contexts that may be important in the process of students assessment. Examples of such actions include: movements performed with the mouse, information entered and then deleted in a form and so on. Examples of context include: different times the student performs his/her activities, actions he/she is performing in parallel, local where activities are performed, network connection quality, server access latency, among others. In our system, this information is collected, interpreted and delivered, via a web-based interface, to a teacher or tutor to help him/her, to better know each students educational profile. This can help not only to pinpoint problems during the educational process, but also to create the basis for personalized education. Moreover, it is also expected that early detection of education problems can reduce high dropout rates in distance learning courses, low teaching quality, and low interaction among students, teachers and tutors.

Next section introduces the logical sensor network abstraction and the support of contexts and ontologies concepts in the modeling of our system.

3. Accurate Monitoring Using Physical and Logical Sensors


Our system makes use of the concept of sensors and sensor networks to track specific and localized events. In our system, a sensor can be either physical or logical. A physical sensor is a device that can be deployed in a physical environment to capture information of interest (e.g., temperature, humidity, movement, presence of a person in an environment). A set of physical sensors that are deployed on an area of interest (e.g., a building, a room, a soya plantation) and can collaborate among themselves and forward information to a recipient node (sink) is defined as a sensor network. A sensor network can be deployed, for instance, in a room that will be used for online examinations to evaluate and monitor the room for abnormal activities and actions (e.g., from personal identification to detection of cell phone activity). The information (events) collected by a sensor network can be combined with information collected from digital environments to provide a more accurate and powerful view of what is occurring. For the detection of events in a digital environment, the concepts of logical sensor and logical sensor network are used. A logical sensor is a software component that can capture specific and relevant information about actions and intentions of a user in a digital environment. A logical sensor network is defined in our work as a set of logical sensors that can collaborate and forward information to a recipient node. A logical sensor network can be deployed, for instance, on both: an LMS and a remote users device for localized and more refined detection of users and LMS applications events, as shown in figure 1.

Figure 1. Physical and Logical Sensors Network 3.1. Use of Physical and Logical Sensors Examples of events that can be collected by the sensors of our system include: users navigation path along the course resources and activities; the order an information was

typed in a form (e.g., an evaluative questionnaire); how long the user took to fill in a text box; how many times a checkbox field was checked and unchecked; number of mouse clicks in a forum; content types visited, and visiting frequency; time an activity was submitted; discipline status; what students are doing and in which virtual environments (disciplines) they are browsing; what actions/activities are being performed at the same time (listening to music, for example); statistical reports (activities submitted, activities that are late, hits, posts, etc), access to a chat (including hours of access, time remained, posted messages, etc.), sending messages to a forum (considering the subject, message body, time of delivery, number of characters of the message, time needed to type the message), among other events required for the desired monitoring of the application. Examples of events generated by users and/or their applications outside the virtual environment and collected by our system include: access to the scroll bar while reading a text or participate in a chat, movements of the mouse over the list of participants in a forum or other activity; if the user has oscillated by clicking on the submit button while participating in an questionnaire activity or another activity that involves web form, users parallel activities while studying (e.g., listening to music, watching a video, browsing on other sites); among others. We identify user events that can be collected and interpreted during the study activities, which can help managers, such as pedagogical coordinators and teachers, to understand the teachinglearning process. We can also identify the users preferences and provide support for future implementations that allow the selection of content based on the learning style of each user. A simple example of how physical and logical sensors can be used together to provide a better view of the actions performed in an exam room is the deployment of cell phone activity sensor (physical sensor) and the deployment of content types visited and activities accessed by the student (both logical sensors) on the LMS being used by the students at exam time. The co-relation and interpretation of the collected data is made using context and ontologies concepts. Next sub-sections introduce these concepts. 3.2. Context Aware Learning Assessment Users information, actions and intentions are captured by sensors and mapped into contexts. According to Dey (1999), context is defined as any information that can be used to characterize the situation of an entity. An entity is the focus of interest of an application. In our system an entity can be an LMS, a user (student, tutor, teacher, pedagogical), activities (task), and disciplines, among others. A system is context aware if it uses context to provide relevant information and/or services to users, Dey et al. (2001). Our system is context-aware because it uses information generated by LMSs users, or its applications, from their access devices, which are collected by logical sensors and interpreted to infer relevant information. Our learning monitoring system provides an interaction interface, added to the virtual environment, as illustrated in Figure 4, which provides additional information relevant to the user. Abowd and Mynatt (2000) define some categories for context-aware applications, called 5Ws, which are: who, where, when, what and why. We use this abstraction of the 5 Ws to classify the learning contexts of our system and support the process of software development. The who, where, when and what dimensions are mapped as classes of an ontology and can be combined in order to determine the reason a situation occurs. The use of ontologies allows us to explicitly define the concepts in the context of learning and its restrictions.

3.3. Use of Ontologies for Context Modeling Our ontology represents concepts defined and proposed from an extensive research on various models of distance education used in universities and colleges worldwide. Terms such as coordinator, teacher, tutor, student, course, discipline, resource and activity were established from both: the distance education models studied and concepts often used in traditional LMS such as Moodle (Modular Object Oriented Dynamic Learning Environment) and Sakai. The who dimension indicates the LMS user on which the monitoring system will collect information. The profiles mapped to the who category follow a hierarchy of monitoring, - a user with "course coordinator" profile has the highest priority and can carry out the monitoring of her/his own events as well as events from other users at lower levels. Students occupy the lowest level in the hierarchy of tracking options; they can only monitor events related to their own activities and courses. The restrictions established by the hierarchy of LMS users are determined in the properties of context ontology. Where indicates the virtual space(s) in which a user is while navigating on the LMS (e.g., a course, a discipline, a study material or an activity). Our monitoring system has a logical sensor that detects where the user is to determine what types of information can be presented to him/her. For instance, if the system identifies that the user is browsing for an activity, information about that activity will be presented according to predetermined settings for the user profile. If the user is in a discipline, another information set will be presented, and so on. The when dimension indicates the relationship of time associated to events that are collected by the logical sensors. From this information, relevant information can be extracted, such as time spent on the development of activities a users routine can be created from her/his activities history. In addition, the when dimension is useful to assist in determining preferences and possible user learning difficulties, which will consider the temporal information combined with time that remained connected and with the number of accesses made to materials, activities, and course. What is mapped as a set of information concerning people, their activities and devices used to interact with the LMS. These can be monitored by our system and have a set of rules applied to them in order to infer what is happening. This information set that make up the what dimension is mapped to a class of Learning ontology, as shown in Figure 2. Some examples of subclasses used are: Access, AcademicPerformance, ConnectedTime, FeedbackTime, Post and Avoidance. The why dimension helps to determine more complex situations, such as the reason of a particular situation or an action performed by the user. We can say that the dimension why is the outcome of the inferences made from the application of a set of rules on the information collected by the sensors to provide possible causes and consequences for academic standing and the users behavior in the virtual environment. Examples include: possible causes for a poor school performance and reasons for absence in synchronous activities. The modeling of contexts using ontologies based on the taxonomy of the 5W's allows us to represent knowledge about the teaching-learning process in a satisfactory manner to provide a more refined monitoring than provided by traditional LMSs. More complex situations can be inferred when more contexts are considered, such as social, professional, personal and family users contexts. These new contexts can also be mapped on ontologies to enhance the results of the system inferences, but due to space restrictions, will not be discussed in this paper. Next section presents a brief description of the proposed system architecture.

Figure 2. Partial view of the Learning Ontology

4. System Architecture
Our system architecture follows a hybrid model, as shown in Figure 3. It is hierarchical, but the communication between some layers, such as Applications, Middleware and Logical Sensors, follows a publish/subscribe model, instead of traditional client/server model. On the other hand, communication between the layers LMS and Logical Sensors follows the traditional client/server model. The proposed architecture aims to facilitate the communication between the applications and middleware services. One of the main services offered by the middleware is the Context Interpretation Service, composed of an Inference System and Queries Ontologies. This service is responsible for providing high-level information about what is perceived by the sensors in the monitoring process of a user's actions on his/her device and LMS. In order to increase the level of abstraction of information to facilitate understanding of the teaching-learning process, the service uses a knowledge base composed of ontologies and a set of rules applied to the data collected. The system architecture has the following main components: LMS: In the first (bottom) layer of the architecture is the LMS, which represents the platform for storage and delivery (in the web) of digital study content. LMS and User Device can be monitored by logical and physical sensors to capture events that can be of interest for the learning process monitoring. Sensor Network: The second layer consists of logical (and physical, when required by the application) sensors that monitor users' actions. The logical and physical sensor network can be deployed on the users device and on the LMS. Middleware: A middleware is systems software that resides between the applications and the underlying operating systems and networks, and provides reusable services that can be composed, configured, and deployed to create networked applications rapidly and robustly, Blair et al. (2004). Our middleware has the following services: Learning Object Service, Context Interpretation Service, Data Mining Service, Logic Sensors Configuration Service, Subscribe Management Service and Content Adaptation Service.

Application: the top layer of the architecture consists of applications that make use of our learning monitoring system.

The middleware interacts with the sensor network to get the events from the learning environment. It then uses its interpretation service to make inferences about the entity being monitored.

Figure 3. System Architecture showing the Context Interpretation Service

The publish/subscribe mechanism is used for the communication among applications, modules, and services of the middleware. Subscribers are entities interested in receiving events generated by publishers. Each new generated event is published to all interested subscribers. A subscription performed by an application to a sensor or multiple sensors happens every time a learning action is configured in the LMS (a forum, a chat, a questionnaire, among other tools). Upon subscription, the events generated by the sensors that monitor a specific action will be retrieved and sent to the middleware. Three case studies are being developed to validate the system, which are described in the following section.

5. Case Studies
Three case studies are being developed to validate the system: 1) Monitoring performance and study organization (student view); 2) Monitoring of online examinations (teacher view); and 3) Monitoring of students actions and progress (tutor view) and are described below. 5.1. Monitoring Performance and Study Organization This case study seeks to identify the main processes and event flows that can better support students in their own performance status and work organization, such as:

activities status (general and individual: late, in progress, finalized); errors versus success in activities (general and individual); suggestions for study agenda organization considering deadlines and exams proximity, etc. After performing the login in the LMS, our system identifies the user and their access permissions. At this moment, other logical sensors are activated, according to the predetermined profile and settings for the user, who will receive information of interest showed as she/he navigates the virtual environment. The information presented is specific to the virtual space he/she is, i.e., it refers to a course or a discipline or a particular activity that is being browsed at the time. This case study is being implemented. 5.2. Monitoring of Online Examinations This case study involves the monitoring of online examinations whose aim is to meet security requirements for exams application. This study has great relevance, especially for situations in which online exams are not applied due to the lack of secure mechanisms (vulnerability present on the platforms; difficulty to manage aspects of reliability and legitimacy of the process). In this context, our system can support the monitoring of online exams carried out remotely. For this, a logical set of sensors is placed in the user's device. This device can be either a computer of the institution, located in a lab where the teacher has access only remotely, or a student's own equipment. The logical sensors can monitor events that can jeopardize the security of the process, such as attempts to access applications or unauthorized files, access to unauthorized sites, access to different users accounts from the same device during the test, among others. Sensors can also be deployed on the LMS to monitor students actions, such as the use of communication tools not allowed for this exam, access to LMS`s unauthorized resources, among others. Physical sensors that can detect cell phone activity, film the room, even process face recognition can be used in combination with the above mentioned logical sensors. This work is being developed with the support of the network technical staff, teachers and students (face to face and at distance/virtually) of our Institution. 5.3. Monitoring Students Actions and Progress This case study considers the monitoring of students and tutors actions and progress from the teachers point of view. The teacher can monitor students' progress in activities and disciplines by means of a friendlier interface that presents information interpreted, inferred from the data collected by logical sensors. This interface provides better understanding of the students actions and progress in the LMS for early problems identification. Figure 4 shows an interface prototype being developed in our laboratory. The system adds a toolbar to the LMS, which is displayed at the bottom of the page. The toolbar shows information obtained from logical sensors, configured for the monitoring of forum activity, identified with the title AS6-1 Doubts Forum. In this example, a user with the teacher profile accesses the forum activity by following the LMS standard navigation and view complementary information. This information was added by our monitoring system and includes: the amount of posts in this forum; the number of participants with interactions in this activity; a chart with the amount of tutors and students interactions. It also shows the number of participants missing the discussion. For a more refined query, the teacher can click on the toolbar, included by the

monitoring system, and send parameters that meet her/his interest, such as: participation bellow expected average of an individual or general participations, feedback time, etc. This case study is being implemented the inference rules are being defined to be integrated with the other layers of our system architecture.

Figure 4. A Embedded Interface to the LMS

6. Conclusions and Future Works


This paper presented a learning assessment system that can provide accurate information on users of a Learning Management System. In this system, sensors can be deployed on digital and physical environments that surround users to capture events of interest for the accurate monitoring of different aspects and views of the teaching and learning process in distance education courses. These events are mapped as contexts in ontologies that can facilitate the interpretation of complex queries to the system. The main contribution of this work is to make use of a powerful and novel abstraction (logical sensor networks) that can mainly help tutors, teachers and course coordinators to take early actions against the first signs of students learning difficulties. The system can also assist students in their own performance monitoring and studies organization. This system is being integrated to the LMS used in a public Institution, which offers distance education courses for more than three thousand students. A middleware is being implemented that performs complex interpretation service, with different inference techniques, from simple logic to fuzzy logic, integrated to ontologies. As future work the system should answer higher level queries of the following type: Why do most of the students missed question 4 in the anatomy exam? Why was students performance in Calculus course so low? What are the possible reasons that led to a high quantity of grades below average in 3 of a total of 6 subjects in the first half of the 2010 academic year in the Engineering course? To obtain this level of induction, we intend to

map out new context situations in the existing ontologies and to use new rules to context interpretation. Also, as further work, the system will deliver personalized content to students according to their learning characteristics, which are identified by the system.

References
Abowd G. and Mynatt, D, 2000. Charting past, present, and future research in ubiquitous computing. In ACM Transactions Computing Human Interact (ACM Press), Vol. 7, pp. 29-58. Blair, G. et a. Middleware technologies for future communication networks. IEEE Network, v. 18, n.1, p. 4 - 5, 2004 Chen, H.-M. et al, 2007a. E-Homebook System: A web-based interactive education interface., Vol. 49, pp.160-17. Chen, C.-M. et al, 2007b.Mining Key Formative Assessment Rules based on Learner Profiles for Web-based Learning Systems. Proceedings of the 7th IEEE International Conference on Advanced Learning Technologies . Niigata, pp. 584-588. Chen, C.-M. and Chen, M. C., 2009. Mobile formative assessment tool based on data mining techniques for supporting web-based learning. Vol. 52, pp.256-273. Dey, A.K., 2001. Understanding and Using Context. In Personal Ubiquitous Computing, Vol. 5, N. 1, pp. 47. Dey, A.K. and Abowd, G.D, 1999. Towards a better understanding of context and context-awareness. GVU Technical Report GITGVU- 99-22, College of Computing, Georgia Institute of Technology. Dey, A. et al, 2001. A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. In Human-Computer Interaction Journal, Vol.16, N. 2, pp.97-166. Dringus, L. P. and Ellis, T., 2005. Using data mining as a strategy for assessing asynchronous discussion forums. In Computer & Education, Vol. 45, pp. 141-160. Enright, J. et al, 2008. Intelligent Functional Dependency Tutoring Tool. 16th International Conference on Computers in Education (ICCE 2008). Taipei, Taiwan, pp. 171-172. Huang, Y. M. et al, 2009. An adaptive testing system for supporting versatile educational assessment. In Computer & Education, Vol. 52, pp.53-67. Otsuka, J. L. et al, 2007. A Multi-Agent Formative Assessment Support Model for Learning Management Systems. Proceedings of the 7th IEEE International Conference on Advanced Learning Technologies. Niigata. Zaane, O. and Luo, J., 2001. Towards Evaluating Learners Behaviour in a Web-Based Distance Learning Environment. International Conference on Advanced Learning Technologies (ICALT 2001. Madison, USA, pp. 6-8.

You might also like