New tools for assessment in distance education

Liane Tarouco

Graduate Program Informatics on Education

University Federal of Santa Catarina

Brazil

Liane@penta.ufrgs.br

Luciano Hack

Graduate Program on Comuter Science

University Federal of Rio Grande do Sul

Brazil

 

 

Abstract: This paper describes a strategy for improving the quality of distance education assessment provinding a set of complementary tools to help on formative evaluation. Regular forms to assess learning are analyzed and Kirkpatrick's model for evaluation is described as well as a set of tools to provide capability to asses according this model. The set of complementary tools includes consensus, tracking, voting and self-evaluation.

 

 

1.Introduction

Learning assessment became a challenge to those working with distance education. In its definition of learning assessment, Bloom, Hastings and Madaus (Mendes 1998), proposed that the evaluation process should includes a great variety of evidences that go beyond the traditional final examination pencil and paper based. In a face-to-face context, professors use more than just formal mechanisms to evaluate students. Body language, participation, quality of questions proposed by students are good indicators of learning, In distance education what normally occurs it is the use of formal mechanisms only. It occurs not only because of the lack of accessible mechanisms that can assist instructors on this task, but also, because many professors prefer to work with more traditional methods of evaluation.

However, thanks to evolution of networks and computers and especially the Internet, mechanisms, that can fill this gap started to show up. Tools became available, to keep track of students activities and interactions with learning environment, with colleagues and with the professor. These kind of information present new opportunities for monitoring the way students learn and for learning assessment.

The use of the Internet for distance education assessment offers many advantages:

A research conducted by Dirks [Dirks] with instructors of distance learning MBA programs presented some questions like: Considering all the time you spend on this course (100%), how much of that time do you think you spend on student assessment (writing tests, grading, giving feedback, and reporting scores)? Answers indicated that the newer teachers, 2-9 years teaching, spent over half their time on assessment (56%), those with 10-19 years experience spent less time on assessment (46%), and the career instructors, 20+ years teaching, spent the least amount of time doing assessment (35%). The primary reasons given for having assessments include providing feedback, giving grades, and motivation. The four types of assessment most commonly mentioned are: case studies, exams, papers, and projects. All of them requires extensive time to grade and are very difficult to get automated help to do the evaluation.

Other researches point also that what is often used for assessment in distance education are the following:

Learning theories state that group learning has significant relevance and must be supported also in distance education. Participation in group activities, cooperation and collaboration must be supported and graded. But there are a lack of good tools to help evaluate the participation of the distance education student in group activities.

Slowly, thanks to evolution of networks and computers and especially the Internet, mechanisms, that can fill this gap started to show up. Tools became available, to keep track of students activities and interactions with learning environment, with colleagues and with the professor as indicated in figure 1.

Figure 1: Internet as mediating tool

These kind of information present new opportunities for monitoring the way students learn and for learning assessment.

As stated by Tucker (Tucker 1998) the CMDL (Computer Mediated Distance Learning) classroom is routinely a far more scrutable educational environment than the physical classroom. The virtual classroom's electronic data storage, retrieval and exchange system (i.e., the text of student and faculty transactions, communication logs, file structures and information presentation algorithms that exist on the file server's hard disk drives) represent concentrated, structured and highly accessible artifacts of the learning transactions. These artifacts can be retrieved, analyzed and reported via highly economical, automatic processes operating in the background of the communications system.

This work presents a proposed solution for assessment on distance education mediated by Internet using innovative approaches based on Kirkpatrick model for evaluate training problems and will describe how they where implemented.

 

2. Assessment models and tools

There are today some systems, available in the market, that provide detailed information on the evolution of the student working at distance mediated by Internet. Examples of this kind of system include, CyberQ, WebCT and AulaNet. Statistics of usage and regular approaches for testing like multiple choice quiz and term papers are usually possible in these kind of systems.

Donald Kirkpatrick developed a model of evaluation, used in training programs. It recommends a division of the evaluation process in four levels (Kirkpatrick 1998):

Mendoza is other researcher that that it is also relevant to investigate student context to find out which aspects of course contents he/she is in the know (Mendoza 1999). So an aspect to be also considered is the lack of suitable tools to asses their current level of knowledge and skills. Mendoza recommends utilization of techniques, other that conventional ones, to collect data and suggest formative evaluation as well as summative evaluation. To achieve formative evaluation is important to make easy for the students not only inform their opinions and feelings about how course is going on but see this contributions being handled and considered.

To help in all these tasks and also support evaluation as proposed in Kirkpatrick model one should have a tool that would help organize polling and receive data in a suitable form to allow counting and summarizing results. To do this, it was designed a system aimed to support flexible and dynamically built of enquete for polling. The system has a WWW interface that allows designing of different kinds of multiple choice questions. Basic data informed are question and alternatives. Depending on type of desired measure the system ask for:

  1. Feedback for each alternative
  2. Indication of correct answer
  3. Nothing more, just counting incidence of selections on each alternative

Option a is used for questions inserted in the learning path to ask for active response. Option b is used for testing and give grades. Option c is used for polling purposed. This system was designed to complement regular tools used to assess distance education students. The following mechanisms have been selected to assist in the evaluation process in this framework:

It is important to stand out that this proposed work is not an alternative approach to assessment rather it is a complementary one and is supposed that may be used together. Each on of this tools are supposed to be used to help evaluate at more than one level in Kirkpatrick model

To implement the tracking tool, a generic interface was designed to intermediate all the interaction of the student with the server. Interactions are processed in real time (for authentication purposes) and periodically in batch to generate records in a student data-base.

To implement the voting, consensus and self-evaluating tools a set of CGI programs where developed to allow dynamically create pages with questions and propositions (HTML forms) that are presented to students and collect their answers (Tarouco 1997).

 

Conclusions

It is our believe that formative evaluation plays an important role to get distance students motivated because they feel a sense of not being lost in space. Just the opposite, they feel that being closely monitored and having their opinions being asked often is more like having a personal tutor that pay attention on way they do and what they say. This kind of attention provided by the system helps to minimize solitude that sometimes is referred as main desisting factor from students.

 

References