PREST Module Program

PREST
Practitioner Research and Evaluation Skills Training in Open and Distance Learning a
HANDBOOK
Programme evaluation and its role in quality assurance
B1
The PREST training resources aim to help open and distance learning practitioners develop and extend their research and evaluation skills. They can e used on a sel!"study asis or y training providers. The resources consist o! t#o sets o! materials$ a six"module !oundation course in research and evaluation skills and six hand ooks in speci!ic research areas o! ODL. There is an accompanying user guide. % !ull list appears on the ack cover. The print" ased materials are !reely do#nloada le !rom the &ommon#ealth o! Learning '&OL( #e site '###.col.org)prest(. Providers #ishing to print and ind copies can apply !or camera" ready copy #hich includes colour covers 'in!o*col.org(. They #ere developed y the +nternational Research ,oundation !or Open Learning '###.ir!ol.ac.uk( on ehal! o! &OL.
The PREST core team

&harlotte &reed 'Programme coordinator( Richard ,reeman '+nstructional designer- editor and author( Pro!essor .ernadette Ro inson '%cademic editor and author( %lan /oodley '%cademic editor and author(
%dditional mem ers

Terry %llsop '&ritical revie#er( %licia ,entiman '.asic education adviser( 0raham 1iles 'Page layout( 1elen Lentell '&ommon#ealth o! Learning Training Programme 2anager( Santosh Panda 'External academic editor( Reehana Ra3a '1igher education adviser(
Steering group
The PREST programme has een guided y a distinguished international steering group including$ Peter &ookson- Ra4 Dhanara4an- Tony Dodds- Terry Evans- Olug emiro 5egede- David 2urphyEvie 6onyongo- Santosh Panda and 1ilary Perraton.
%ckno#ledgements
/e are particularly grate!ul to 1ilary Perraton and Ra4 Dhanara4an #ho originally conceived o! the PREST programme and have supported the pro4ect throughout. %mong those to #hom #e are inde ted !or support- in!ormation and ideas are 1onor &arter- 7ate &ro!ts- 5ohn Daniel- 6ick 0ao5enny 0lennie- 7eith 1arry- &olin Latchem- Lydia 2eister- Roger 2ills- San4aya 2ishra- Ros 2orpeth- Rod Tyrer- Paul /est and Dave /ilson. +n developing the materials- #e have dra#n inspiration !rom the lead provided y Roger 2itton in his hand ook- 2itton- R. 89:; Practical research in distance education- &am ridge$ +nternational Extension &ollege.
1and ook .<$ Programme evaluation and its role in quality assurance
%uthors$ 2erridy /ilson"Strydom. &ritical revie#ers$ Terry %llsop- Richard ,reeman and .ernadette Ro inson. &opyright$ &reative&ommons- #ith %ttri ution- Share"%like- =.> 'http$))creativecommons.org)licenses) y"sa)=.>)( +S.6 8":9<9?@">;";
Permissions
See the last page o! the module.
&ontents
Programme evaluation and its role in quality assurance.......................................1 1and ook overvie# .................................................................................................. 8 Learning outcomes ................................................................................................... 8 1and ook organisation ............................................................................................ ; 1o# to use the materials .......................................................................................... = Evaluation examples ................................................................................................ A Resources ................................................................................................................ : Status o! the readings .............................................................................................. : Unit 1: What is programme evaluation and why is it important?.........................11 Bnit overvie# .......................................................................................................... 88 Bnit outcomes ........................................................................................................ 8; /hat is programme evaluationC ............................................................................ 8; Evaluation as a !orm o! research ........................................................................... 8@ The relationship et#een programme monitoring and evaluation ......................... 89 Programme monitoring in ODL ...............................................................................89 /hy should distance education practitioners conduct evaluationsC ..................... ;> Bnit summary ......................................................................................................... ;8 Re!erences ............................................................................................................. ;; ,eed ack to selected activities .............................................................................. ;; Unit 2: Evaluation conceptuali ation: de!ining the purpose o! an evaluation....2" Bnit overvie# .......................................................................................................... ;@ Bnit outcomes ........................................................................................................ ;@ Purposes and types o! evaluation .......................................................................... ;@ +denti!ying an evaluationDs target audience ............................................................ => ,ormulating key questions .................................................................................... =8 &onsultation E a key to success ............................................................................ =@ Bnit summary ......................................................................................................... =? Re!erences ............................................................................................................. =: ,eed ack to selected activities .............................................................................. =9 Unit #: $esigning and implementing an e!!ective evaluation...............................%# Bnit overvie# ...........................................................................................................<= Bnit outcomes ........................................................................................................ <= Evaluation planning- !orming a conceptual !rame#ork .......................................... << De!ining the evaluation delivera les and activities .................................................<A &hoosing evaluation research methods ................................................................. <: .udgeting !or evaluations$ keeping expectations realistic ..................................... @= Evaluation logistics ................................................................................................. @< ,rom data gathering to analysis and reporting ...................................................... @@ Bnit summary ......................................................................................................... @A Re!erences ............................................................................................................. @? ,eed ack to selected activities .............................................................................. @? Unit %: &uality assurance' quality control and other views..................................(# Bnit overvie# .......................................................................................................... A= Bnit outcomes .........................................................................................................A= /hat is quality assuranceC .................................................................................... A= Di!!erentiating et#een evaluation and quality assurance ..................................... AA
&ommon#ealth o! Learning
iii
&ontents
&onnecting evaluation to programme planning and quality assurance processes Bnit summary ......................................................................................................... &onclusion ............................................................................................................. Re!erences ............................................................................................................. ,eed ack to selected activities ..............................................................................
A: ?= ?< ?< ?@
Permissions ............................................................................................................. ))
iv
Practitioner Research and Evaluation Skills Training in Open and Distance Learning
Programme evaluation and its role in quality assurance
HANDBOOK
B%
1and ook overvie#

/elcome to the hand ook on Programme Evaluation and its role in Quality InsuranceF 2y name is 2erridy /ilson"Strydom- and + #ill e taking you through this hand ook. + have een involved in various evaluation and applied research pro4ects over the past !ive years and + hope this #ill make me a good guide !or this topic. During the #riting o! this hand ook + have #orked together #ith 6eil .utcher #ho you may have GmetD in the hand ook on Using Programme Monitoring in Research and Evaluation. +n case you have not yet #orked through that hand ook let me rie!ly introduce you. 6eil has spent the last ten years o! his li!e researching distance education programmes and institutions !or the South %!rican +nstitute !or Distance Education. %s such he has much experience to contri ute to this hand ook. %!ter #orking through this hand ook + hope that you #ill e a le to$ develop a process and the necessary tools to evaluate a distance education programme and demonstrate ho# this #ill lead to an improved educational experience !or learners in that programme.
.ecause o! my ackground and experience o! doing evaluation in the South %!rican context + #ill e dra#ing heavily on South %!rican examples in this hand ook. O! course- + #ill also make use o! the est o! international literature that + have managed to !ind on programme evaluation. +n this #ay- + am con!ident that together #e #ill e a le to go on a learning 4ourney that you #ill !ind enriching and interesting.
Learning outcomes
This hand ook has een structured to help you to e a le to do the !ollo#ing things y the time you are !inished$ 8 de!ine the purpose o! an evaluation process and prioritise the key questions that the evaluation #ill need to ans#er
1and ook .< E Programme evaluation and its role in quality assurance
; select appropriate methods !or gathering the necessary data !or a good quality and practically use!ul evaluation o! a distance education programme = identi!y the characteristics o! e!!ective evaluation processes and !actors that are most likely to impede success!ul implementation o! an evaluation < de!ine the reporting requirements o! an e!!ective evaluation process @ descri e the relationships et#een programme evaluation and quality assurance A plan an evaluation study. %t the end o! the hand ook- + #ill encourage you to return to this page to see #hether or not the hand ook has lived up to its promises. Hou #ill !ind that several o! the research concepts and methods that are relevant in the context o! evaluation have also een covered in other core modules. /here this is the case- #e #ill rie!ly summarise the issues rather than covering them in detail again.
1and ook organisation

The hand ook is structured into this introduction and !our units- as !ollo#s. Bnit 8$ /hat is programme evaluation and #hy is it importantC '< hrs( Bnit ;$ De!ining the purpose o! an evaluation 'A hrs( Bnit =$ Designing and implementing an e!!ective evaluation 'A hrs( Bnit <$ Evaluation and quality assurance- ensuring the impact o! evaluation '< hrs( 'These times exclude any pro4ect #ork that you choose to do.( Each unit is made up o! the !ollo#ing components$ an introductory paragraph or t#o that provide an overvie# o! the unit- its !ocus and outcomes one or more activities !or you to engage in- such as readings to complete and analyse- questions to ans#er- or pro lems to solve a commentary on these responses that takes you deeper into the topic y providing ne# in!ormation and suggesting !urther reading a unit summary !eed ack on your responses to the questions or pro lems posed in each activity.
Hou #ill need a out ;> hours to #ork through the !our units o! this hand ook. The
2
+ntroduction
time required !or the pro4ect tasks is really up to you- since it depends on the scope o! your pro4ect and the num er o! people involved in it. % time limit is suggested !or each step o! the pro4ect process- ut this can only e a guide. %ll the times set out in the schedule that !ollo#s are suggestions- ut Bnits 8 to < have een designed on the assumption that hand ook users #ill e a le to complete the core #ork '#ithout the pro4ect( in a out ;> hours.
1o# to use the materials

+ have tried not to assume that all readers #ill #ant to use these materials in the same #ay. There are three road options open to you depending on your time availa ility- current and !uture involvement in distance education research and evaluation- and access to a distance education programmes to use as a practical example. 8 Hou could simply read through the hand ook- and its accompanying reading Resources File. +! you do this- you #ill get a good introduction to the topic o! the hand ook- and to the various issues contained in the list o! outcomes a ove. O! course- #ithout completing any o! the activities- you #ill not achieve the same depth o! experience and insight that #ill come !rom completing them. So- you are advised only to !ollo# this route i! you need a quick overvie# o! key issues in using programme monitoring in research and evaluation. ; Hou could read through the hand ook and its accompanying reading Resources File- and only complete the activities. These are tasks that you can complete in the com!ort o! your o#n home or o!!ice- and #ill not require you to do any !ield research. ,or this hand ook- some o! the activities #ill e centred on t#o examples o! evaluation studies. Depending on your time constraints you might like to select one o! the studies that you est identi!y #ith- or that est suits your research needs- to complete the activities. The ene!it o! these activities is primarily that they #ill give you good opportunities to think through the key issues in the hand ook- and !ormulate your o#n opinions e!ore reading the material o! the hand ook units. %s #e all kno#many learners simply skip over these activities- and go straight on to reading the content o! the material. This is no pro lem i! you do not #ant to do this extra #ork. 1o#ever- these activities are designed to get you to re!lect on issues and !ormulate your o#n opinions- rather than 4ust relying on the hand ook to tell you the GrightD ans#er. +! you do choose to complete these activities- + strongly recommend that you complete them at the exact point at #hich they appear in the text- as their placement is quite deli erate. +! you read ahead- and then complete these activities- you #ill lose many o! their intended educational ene!its. = Hou could read through the hand ook and its accompanying resources- complete
the activities- and #ork through the pro*ect tas+s. These tasks #ill require you to go into the !ield to do research #ork #ith ODL practitioners. O viously- those #ho complete these pro4ect tasks are going to gain the most educationally out o! this hand ook- ut these ene!its come at the expense o! your time. There is no point in attempting these tasks unless you are serious a out pursuing them throughout the hand ook. + advise you to complete these pro4ect tasks i! you$ are keen to pursue a career in distance education research and evaluation 'or are already pursuing such a career path( are eginning to conduct your o#n research and evaluation- and !eel that the pro4ect tasks provide a good !rame#ork !or supporting you in this #ork can esta lish a good #orking relationship #ith a distance education programme manager some#here in your area have the time to do the !ield research thoroughly are completing this hand ook as part o! a structured learning experience.
2ore in!ormation on the pro4ect tasks is presented elo#- so you might like to read the next section e!ore making a !inal decision on #hether or not to do them.
The pro4ect tasks

Hou #ould e !orgiven i! you are no# some#hat concerned a out the pro4ect tasksas + think + might have made them sound a little intimidating. 1o#ever- it is in doing these tasks that you #ill really derive the greatest ene!it !rom this hand ook. + can introduce ideas and concepts- and suggest #ays !or you to think care!ully a out them- ut it is really only #hen you go out into the !ield and see #hat is happening in practice that these ideas #ill come alive. So- i! you have the time and you are serious a out research and evaluation- + #ould strongly encourage you to !ind #ays to do these pro4ect tasks. The pro4ect tasks #ill require you to !ind an existing ODL programme #ith #hich you can #ork. Over the course o! the hand ook the pro4ect tasks #ill assist you to$ develop an evaluation !rame#ork and implementation plan !or an evaluation o! a programme o! your choice decide on and develop appropriate methods !or gathering the data needed !or your evaluation descri e ho# this evaluation research can lead to an improvement in the quality o! the programme #ith #hich you are #orking summarise and descri e the quality assurance systems o! the programme chosen
+ntroduction
use a checklist !or revie#ing the impact o! your evaluation.
This is challenging- ut + can assure you that it #ill e #orth the e!!ort. +! you complete the pro4ect- you #ill have a !ull evaluation plan y the end o! the hand ook. Hou could then use this plan as a proposal to raise !unds to e a le to conduct the evaluation or- i! possi le- immediately egin conducting an evaluation o! your programme. To do the pro4ect tasks- there!ore- you #ill need to consider one o! the !ollo#ing options$ %re you involved in an ODL programme yoursel!C +! so- you could use that programme to complete the pro4ect tasks. Do you kno# anyone #ho is running or helping to run an ODL programmeC /ill he or she e #illing to let you evaluate the programmeC +! so- then you could ask that person to let you #ork #ith her. +! you donDt personally kno# someone #ho is involved in ODL- do you kno# o! an organisation or institution near #here you are living that is involved in delivering distance education programmesC +! so- then you might approach that institution to help you #ith your pro4ect tasks. +! there is no ODL delivery near you- then do you have access to the +nternetC +! so- then you might try to !ind someone running a distance education programme in another part o! the #orld. Hou might then approach someone online and ask him or her to supply you some in!ormation via e"mail. % good starting point #ould e the #e site o! the &ommon#ealth o! Learning E ###.col.org E that has connections to distance education providers all over the #orld.
+ hope you #ill strongly consider doing the pro4ect tasks. +t should e a !ascinating exercise- and #ill add enormously to the experience o! this hand ook.
The 1and ook as a #ork ook

This ook is designed !or you to use as a #ork ook- particularly #ith respect to completing the activities. Space has een provided #ithin this #ork ook !or you to store your ans#ers- and + encourage you to use this space. 2any o! you might !ind it very di!!icult to #rite in a ook- ut it is an ideal #ay to store your responses to the questions posed and to !orm a record o! your o#n thoughts and learning process. The reading that accompanies this hand ook has een divided into essential reading #hich must e done to achieve the learning outcomes set out a oveoptional reading that is linked to activities and allo#s you to extend your learning experience- and !urther reading- reading that you might like to !ollo# up in your o#n time i! a particular issue interests you.
,ood !or thought

%nother device that + have used is called G!ood !or thoughtD. 1ere- + have tried to insert some questions to get you to think more a out #hat #e are discussing. /ith G!ood !or thoughtD questions- you are encouraged at least to think rie!ly a out your responses to the questions posed e!ore proceeding. +! you #ant to- though- you might also consider keep some paper handy to 4ot do#n your thoughts.
Evaluation examples
+nstead o! using the pen portrait characters that you are pro a ly !amiliar #ith !rom other hand ooks and core modules- in this hand ook #e #ill dra# on t#o evaluation studies as examples to illustrate the key points eing made. Each o! these evaluations has een included in your Resources File and Bnit 8 has an activity during #hich you #ill have the opportunity to revie# them and to choose one that is particularly relevant to you to use !or later activities. + kno# that this might look like a lot o! reading to e done E and it is E ut + do recommend that you try to set aside the time to do this as it #ill enhance your learning experience throughout this hand ook. The t#o evaluation studies are rie!ly summarised elo#.
Evaluation 8
,entiman- %. ;>>= SOMDEL$ Somali Distance Education Literacy Programme (Macallin a Raddiya!- report prepared !or %!rica Educational Trust '%ET( y the +nternational Research ,oundation !or Open Learning '+R,OL(- &am ridge- B7$ +R,OL 'This #ill e re!erred to as Evaluation " in the rest o! this hand ook.( +n 2arch ;>>; the %!rica Educational Trust together #ith the ..& /orld Trust launched a ne# and innovative distance education literacy programme in Somaliland and Somalia. This programme is called SOMDEL # the Somali Distance Education Literacy Programme # Li$e S ills %&&roach. The programme !ocuses on the provision o! literacy- numeracy and li!e skills training to out"o!"school children and adults. SOMDEL involves the use o! radio- print and !ace"to"!ace teaching. +n order to assess the e!!ectiveness o! this programme the +nternational Research ,oundation !or Open Learning #as commissioned to undertake an external evaluation. The evaluation used the logical !rame#ork o! the programme as the asis on #hich achievements #ere measured. Iarious research methods #ere employed to gather the evaluation data- including examination and scrutiny o! statistical records and documents- !ocus group discussions- semi"structured intervie#s- case studiessurveys and participant o servation.
6
+ntroduction
Evaluation ;
2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policy( ma ers in Southern %$rica- Iancouver$ &ommon#ealth o! Learning 'This #ill e re!erred to as Evaluation ) in the rest o! this hand ook.( The &ommon#ealth o! Learning '&OL( contracted the evaluator to evaluate an online course !or policy"makers #ho #ork in ODL institutions in su "Saharan %!rica. This course set out to provide participants #ith$ a conceptual plat!orm and in"depth introduction to open learning and distance education concepts an understanding o! the policy development process an understanding o! the implications o! di!!erent policy choices in open learning and distance education.
The evaluator #as asked to determine #hether the online course met the participantsD needs and also #hether the delivery strategies 'online delivery supported y a one"#eek !ace"to"!ace #orkshop( #ere e!!ective. To ans#er these questions the evaluator used questionnaires- intervie#s- and analysis o! documentse"mail messages and learning activities.
Resources
The !ollo#ing resources are used in this hand*oo $
,esource -ame when re!erred to in our te.t Evaluation " /ocation
,entiman- %. ;>>= SOMDEL+ Somali Distance Education Literacy Programme (Macallin a Raddiya!- report prepared !or %!rica Educational Trust '%ET( y the +nternational Research ,oundation !or Open Learning '+R,OL(&am ridge$ +R,OL 2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policy(ma ers in Southern %$rica- Iancouver$ &ommon#ealth o! Learning 7achroo- 7. 8999 GDistance education programmes at college and university levels in 5ammu region$ an evaluative studyD Indian ,ournal o$ O&en Learning :- =$ ;9="=>@ 2ugridge- +. 8999 GJuality assurance in open and distance learningD Indian ,ournal o$ O&en Learning :- =$ ;8=";;> 2oore- 2. 8999 GEditorial E 2onitoring and evaluationD .he %merican ,ournal o$ Distance Education 8=- ;$ 8"@ /hyte- %. ;>>> %ssessing community telecentres+ guidelines $or researchers-Otta#a$ +nternational Development Research &entre- at http$))###.eldis.org)static)DO&?:A?.htm
Resources File
Evaluation )
Resources File
-achroo
Resources File
Mugridge
Resources File
Moore
Resources File
/hyte
htt&+001112eldis2org0static 0DO'34532htm
These resources are included as an appendix at the end o! the pu lication.
Status o! the readings

+n order to achieve the learning o 4ectives outlined a ove you #ill need to make sure to read the essential reading. /e #ill re!er to these evaluation studies throughout this hand ook. The optional readings #ill allo# you to !urther develop your understanding o! the concepts presented here. +! possi le- + #ould recommend that you try to read these. The !urther readings have een provided to assist you to explore issues that you !ind particularly interesting or use!ul !or your #ork.
Essential Reading
,entiman- %. ;>>= SOMDEL+ Somali Distance Education Literacy Programme
+ntroduction
(Macallin a Raddiya!- report prepared !or %!rica Educational Trust '%ET( y the +nternational Research ,oundation !or Open Learning '+R,OL(- &am ridge$ +R,OL 7achroo- 7. 8999 GDistance education programmes at college and university levels in 5ammu region$ an evaluative studyD Indian ,ournal o$ O&en Learning :- =$ ;9="=>@ 2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policy( ma ers in Southern %$rica- Iancouver$ &ommon#ealth o! Learning
Optional reading
2ugridge- +. 8999. Juality %ssurance in Open and Distance Learning. Indian ,ournal o$ O&en Learning- :'=(.
,urther reading
2oore- 2. 8999 GEditorial E 2onitoring and evaluationD .he %merican ,ournal o$ Distance Education 8=- ;$ 8"@ Rathore- 1. and Schuemer- R. 'eds.( 899: GEvaluation concepts and practice in selected distance education institutionsD- 6IFF &a&er 8>:- 1agen$ &entral +nstitute !or Distance Education Research- ,ernBniversitat- at http$))###.!ernuni" hagen.de)K+,,)pre!8>:.htm /hyte- %. ;>>> %ssessing community telecentres+ guidelines $or researchers Otta#a$ +nternational Development Research &entre- at http$))#e .idrc.ca)en)ev" ;:=>8";>8"8"DOLTOP+&.html + hope you are looking !or#ard to #orking through this hand ook. The 4ourney egins y considering #hat programme evaluation is and #hy this !orm o! applied social research is use!ul in the context o! distance education.
10
/hat is programme evaluation and #hy is it importantC
B6+T
Bnit overvie#
This unit !ocuses on helping you to develop a clear understanding o! #hat programme evaluation is and #hy it could e use!ul !or you. /e are going to egin y looking at #hat programme evaluation is. To do this- #e #ill consider a series o! de!initions o! evaluation and then look in more detail at #hat the concept o! a programme includes. Since this hand ook is part o! a series !ocused on research skills #e #ill rie!ly look at evaluation as a !orm o! research. The unit ends #ith a consideration o! the relationship et#een programme monitoring and evaluation as #ell as #hy evaluation is use!ul !or distance education programmes. 2any o! the concepts that you #ill learn here- although covered in the context o! distance education- can in !act e used !or designing evaluations o! a #ide range o! social interventions or programmes. So- + hope this unit 'and the #hole hand ook in !act( #ill e a use!ul research tool in many di!!erent research and evaluation contexts that you might !ind yoursel! in. +n this unit you #ill e encouraged to read the t#o evaluation studies rie!ly descri ed in the introduction. Each o! these studies has een chosen to highlight speci!ic issues that #e #ill cover as #e progress.
,eading 0 ,esources !ile
You might like to read the executive summaries of Evaluation 1 and Evaluation 2 in the Resources File be ore beginning the handbook to gain a sense o !hat the" cover# You should not s$end more than about 30 minutes doing this# %s a reminder& the t!o studies are'
11
(entiman& %# 2003 SOMDEL: Somali Distance Education Literacy Programme (Macallin a Raddiya!" re$ort $re$ared or % rica )ducational *rust +%)*, b" the -nternational .esearch (oundation or /$en 0earning +-.(/0,& 1ambridge& 23' -.(/0# *his is called Evaluation 1 in this handbook# 4agagula& 1# 2##2 Evaluation re$ort of t%e &ourse for Distance Education Policy'ma ers in Sout%ern (frica" 5ancouver' 1ommon!ealth o 0earning# *his is called Evaluation 2 in this handbook#
Bnit outcomes
/hen you have #orked through this unit- you should e a le to$ 8 state a de!inition o! programme evaluation ; explain programme evaluation !rom a research perspective = descri e the relationship et#een programme monitoring and evaluation < give reasons #hy conducting programme evaluation is important !or distance education programmes and institutions.
/hat is programme evaluationC

/hat do you understand y the concept o! programme evaluationC +s it a term you have come across e!oreC +n #hat contextsC .e!ore #e look at some !ormal de!initions o! programme evaluation- #rite do#n your response to the t#o short questions in %ctivity 8.
1ctivity 1 8@ mins
1 6o! !ould "ou de ine $rogramme evaluation7
12
2 8ive at least one reason !h" "ou think "ou might !ant to conduct a $rogramme evaluation at "our institution#
.he $eed*ac to this activity is in the $ollo1ing te7t
Programme evaluation E some de!initions

6o# lets see ho# your ideas compare to the de!initions given in the literature.
2ore de!initions
8Evaluation is the &rocess o$ collecting and0or using in$ormation $or the &ur&oses o$ determining the value and 1orth(1hileness o$ the su*9ect o$ the evaluation &rocess: '.irley M 2orel 899:(. 8Evaluation is the collection; analysis and inter&retation o$ in$ormation a*out any as&ect o$ a &rogramme o$ education or training; as &art o$ a recognised &rocess o$ 9udging its e$$ectiveness; its e$$iciency and any other outcomes it may have: 'Thorpe 899=(. 8Evaluation is the systematic &rocess o$ collecting; analysing and inter&reting in$ormation that ena*les 9udgements to *e made a*out the value o$ a &rogramme (o$ learning! and its e$$ectiveness and0or e$$iciency in achieving a set o$ outcomes: (Dolley "<<=!2 /e could perhaps sum up the a ove three de!initions quite simply using the #ords o! ,rancis Ru in here. Since #e seem to have een only talking a out evaluation in the de!initions a ove- you might e #ondering i! the de!inition o! programme in the concept o! programme evaluation is simply assumedC )valuation means 9to understand the value o something in order to do things better: +.ubin 1995,#
/ell- the ans#er could e yes or noF Iery o!ten the term programme is used loosely to re!er to any social intervention that is to e evaluated. This could then e- !or example- a community"se#ing pro4ect- a national literacy campaign- or a set o! courses #ithin a distance education institution. +t is ho#ever use!ul- and important- to look at the concept o! a programme. Hou #ill see as #e continue- that understanding #hat a programme is and consists o!- means that you are one step closer to designing your evaluation plan #ithout even eing a#are that you are doing soF
13
.a ie and 2outon ';>>8( present a use!ul conceptual model or map o! a programme ',igure 8(.
Figure " % conce&tual ma& o$ a &rogramme Source+ >a**ie; E2 ? Mouton; ,2 )@@"; &2A=A2 This diagram sho#s us #hat a programme GlooksD like. The details o! each o! the oxes #ill di!!er depending on the speci!ic programme. The example here is relevant to an educational programme. This diagram highlights the core !eatures that should e part o! a programme and sho#s the links et#een them.
1ctivity 2 <@ mins
1 -n the s$ace belo!& re$roduce this conce$tual model& using a $rogramme that "ou kno! o & or $erha$s& are a $art o & to com$lete the details o each o the blocks# *o do this& start !ith the block numbered one and the ill in the details o each block in numerical order#
14
2 6o! !ell did this model hel$ "ou to ma$ out "our exam$le o a distance education $rogramme7
.he $eed*ac to this activity is at the end o$ the unit
Evaluation as a !orm o! research

0iven that this hand ook is one resource in a series dealing #ith research methods #e should ask #hat the relationship is et#een evaluation and research more roadly. %re they one and the same thingC To help us egin this discussion- #e re!er to the #ords o! 2ichael Patton taken !rom a much"quoted ook on evaluation. 1e says$ 8/hen one e7amines and 9udges accom&lishments and e$$ectiveness; one is engaged in evaluation2 /hen this e7amination o$ e$$ectiveness is conducted systematically and em&irically through care$ul data collection and thought$ul analysis; one is engaged in evaluation research: (Patton "<<@!2 /hat does this say a out the relationship et#een evaluation and researchC /ell#e conduct in!ormal evaluations all the time. +magine making a decision a out #hich variety o! !ruit to uy #hen standing in the local !ruit and vegeta le shop. Hou #ould look at ho# !resh each o! the varieties is- think a out #hat you need the !ruit !or- #ho #ill e eating the !ruit- #hich is the est value !or money etc. 1ere you are conducting an in!ormal- elementary evaluation- assessing #hich variety o! !ruit #ill est meet your needs. .ut- is this researchC + am sure you #ould agree #ith me that this is not research- and again PattonDs #ords can guide us. 1e tells us that evaluation research must involve systematic and care!ul data collection and analysis in order to ans#er a speci!ic research question. This means that all the requirements !or conducting good research that you have een considering throughout these hand ooks should e applied in the context o! conducting an evaluation. /hile #e are not a le to cover all these details in this hand ook- #e #ill highlight the research requirements as #e go. The question o! #hether evaluation is di!!erent !rom research is one that is de ated in the literature and among evaluation theorists and practitioners. Some argue that there is little or no di!!erence et#een research and evaluation- and so use the term
15
evaluation research. Others argue that in !act evaluation is a di!!erent type o! activity !rom research- even though oth make use o! similar methods. Bnderstanding this de ate is important ecause your stance #ill a!!ect the #ay that you approach the design and implementation o! an evaluation. /hat do you thinkC Read through the extract elo# that outlines some possi le similarities and di!!erences et#een research and evaluation.
,eading
Similarities between research and evaluation Same methods, same skills .esearchers and evaluators use the same methods o social science in;uir" and re;uire the same sorts o skills& ;ualities and $rinci$les in the $ractitioners conducting an in;uir"# Evaluation and research share the same domain )valuation is $art o the domain o educational research and is not isolated rom general develo$ments and issues in educational research& its methodolog" and theoretical issues# Rational re-construction versus reality: a shared challenge <oth evaluation and research ace similar challenges in choosing bet!een descri$tion and $rescri$tion +the real and the ideal, in re$orting their indings# .e$orts are o ten rational re=constructions o the +sometimes di erent, realities o the research $rocess& $resented to it norms o 9ideal: $ractice +as described in textbooks, rather than accounts o the realities themselves# Increasing influence of governments and sponsors brings convergence .esources or educational research and large=scale evaluations are increasingl" determined b" the demands o governments or unders !ho seek solutions to centrall" de ined to$ics or ;uestions !hich are $roblem=oriented and $olic"= related# /ne result o this ma" be to make research more like evaluation in having ocus on $ractical outcomes and s$eci ic $roblems#
16
Differences between research and evaluation

Selecting uestions or problems for investigation % researcher is $robabl" more ree to select his or her o!n research ;uestions and to seek ans!ers to them in !hatever !a" he or she sees as a$$ro$riate $ro essionall"# %n evaluator o ten is commissioned to investigate a $re= determined ;uestion or to$ic# >ome $ractitioners vie! the di erence bet!een research and evaluation as l"ing in the $ur$ose o the t!o activities and the kinds o ;uestion asked# ?hile researchers ma" seek generalisation in their indings& and ask ;uestions relating to general $ro$ositions& evaluators ocus more on s$eci ic activities in $articular contexts +e#g#& 9?as this $articular $rogramme cost=e ective or this institution using these media7:,# ! different relationship between the research uestion and methodology used .esearch ;uestions are likel" to be sha$ed b" the $roblem=solving or in;uir" techni;ues or methods available and considered sound& so an initial research ;uestion ma" be amended in the light o !hat it is most $ossible to measure or ans!er# 0earning to ormulate ans!erable research ;uestions is $art o the training o researchers# <" contrast& the evaluator ma" have the research ;uestions $rovided to him or her and change in these ma" not be negotiable# ! different balance between disciplinary rigour and the rapid or timely production of usable findings for decision-makers )valuators ma" choose +i reluctantl" sometimes, a di erent balance rom 9$ure: researchers in a$$l"ing the critical canons o research $ractice because o the need to $rovide ra$id indings and ans!ers to $ractical $roblems# *he" ma" have to com$romise in order to $rovide indings or ans!ers that are 9good enough: or the $ur$ose in hand !hile holding to good $ractice as ar as $ossible in the conduct o the in;uir"# )valuations are o ten heavil" constrained b" time and the realities o ield conditions# ?hile researchers and evaluators& !orking on the same ;uestion& might carr" out the in;uir" in similar !a"s on some as$ects& the" might also carr" it out di erentl" in others# "he need to provide reports for non-specialist audiences *he orms o re$orting in evaluation have to meet the needs o a range o non= s$ecialist users or stakeholders& rather than a narro!er grou$ and the more s$ecialist interests o a research communit"#
17
! more political activity than research )valuation is inesca$abl" a $olitical activit" because o the vested interests o the various stakeholders and the di erent meanings the" $lace on the evaluation activit" and its indings# >ince evaluations $oint to!ards $ractical action& di erent stakeholders are likel" to be more immediatel" a ected b" the indings than b" a 9$urer: research stud"# ! broader set of competences and skills re uired )valuators o ten need to be com$etent in a !ide range o methods and techni;ues& es$eciall" i !orking alone& !hereas researchers ma" $re er& or are some!hat more likel" to s$ecialise in one or other a$$roach +;uantitative or ;ualitative,# ?here researchers combine methods o research& one a$$roach tends to lead and the other su$$orts +the" are not generall" used in e;ual measure,#
Robinson 2001, pp. 5-7
So- to summarise the point here E evaluation is a speci!ic !orm o! research that should !ollo# the same overall guidelines and protocols as any other research process. 1o#ever- !or various reasons listed a ove- an evaluation pro4ect is also di!!erent !rom a research pro4ect. %s an evaluator you need to e clear on oth these similarities and di!!erences. Evaluations are usually applied research- conducted !or someone or some organisation and involve making some !orm o! value 4udgements a out the programme in question. +n other #ords- evaluations should al#ays have a clearly de!ined purpose and target audience. Iery o!ten- evaluations have implications !or resource decisions- most commonly !unding decisions- and so evaluation research can e more sensitive than other !orms o! research.
18
The relationship et#een programme monitoring and evaluation

Hou may or may not have #orked through the hand ook on programme monitoringso 4ust in case- here are the de!initions o! monitoring presented in that hand ook. 8Monitoring is the continuous assessment o$ the intervention and its environment 1ith regard to the &lanned o*9ectives; results; activities; and means2 It ta es &lace at all levels o$ management and uses *oth $ormal re&orting and in$ormal communications2 Monitoring ena*les a sta eholder to revie1 &rogress and to &ro&ose action to *e ta en in order to achieve the o*9ectives2 Monitoring identi$ies actual or &otential successes or $ailures as early as &ossi*le and $acilitates timely ad9ustments to the o&erations: 2inistry !or ,oreign %!!airs 899:(. 8Monitoring+ recording and trac ing the &rogress o$ a &rogramme 1hile it is running; 1ith a vie1 to 9udging 1hether a &lan is on course and its o*9ectives are on the 1ay to *eing achieved2 In$ormation collected $or monitoring &ur&oses is also use$ul as in$ormation $or evaluation: (Dolley "<<=!2
,ood !or thought

8iven these de initions o monitoring& ho! do "ou think $rogramme monitoring $rocesses relate to $rogramme evaluation7
Programme monitoring in ODL

Programme monitoring re!ers to the process o! continually tracking ho# a programme is eing run. ,or an ODL programme- this includes- !or exampletracking student enrolment- su mission o! assignments- results o! assignmentstutorsD participation etc. 2onitoring should also e taking place at all levels o! a programme. ,or example- a lecturer #ould monitor the progress o! her class o! studentsN the course coordinator #ould monitor progress o! all students in a particular course as #ell as the lecturers and tutors etc. Evaluation- ho#ever- is speci!ically a out making a *udgement a out the quality o! a programme. This may e a 4udgement to determine #hether a programme should e continued or not- or perhaps to !ind #ays o! improving a programme. +n the previous section #e considered the !ollo#ing quotation$ 8/hen one e7amines and 9udges accom&lishments and e$$ectiveness; one is engaged in evaluation2 /hen this e7amination o$ e$$ectiveness is conducted systematically and em&irically through care$ul data collection and thought$ul analysis; one is engaged in evaluation research: (Patton "<<@!2 /hen talking a out evaluation o! distance education programmesprogramme
19
monitoring is one o! the central means o! systematic data collection. So- programme monitoring is one o! the tools !or use in programme evaluation. % good programme monitoring system should provide the researcher #ith much o! the data needed #hen conducting an evaluation. The challenge is to identi!y #hat in!ormation to look !or in order to ans#er your speci!ic evaluation question's(. Deciding on #hich in!ormation to gather #ill e a topic that runs throughout this hand ook. Hou might also like to have a look at the hand ook Bsing Programme 2onitoring in Research and Evaluation.
/hy should distance education practitioners conduct evaluationsC

So !ar #e have considered #hat programme evaluation is- ho# it relates to research more roadly and also the role that programme monitoring plays in evaluation. .ut #hy should you e interested in conducting evaluationsC
!ctivity # $ hour 1ctivity # 8 hour
You should use either o the t!o evaluation re$orts& Evaluation 1 or 2& in the Resources File or this activit"# @ote do!n "our res$onse to the ollo!ing ;uestions as "ou read through the ull re$ort# +6int' *he $ur$ose o the evaluation is usuall" stated earl" on in the re$ort,# 1 ?hat is the stated $ur$ose o the evaluation7
2 6o! is this evaluation use ul or the distance education $rogramme in ;uestion7
20
One o! the conclusions to dra# !rom these examples is that in most cases evaluations are a tool !or decision"making- and !or assessing and promoting the quality o! a programme or institution. /hile evaluations provide !indings that can help programme planners make decisions- it is also important to note that this does not necessarily make decision"making easier. Sometimes evaluation !indings #ill recommend that very di!!icult decisions e made- !or example that a programme is not #orking and should e discontinued. Evaluations have an important role to play in quality assurance processes. Evaluations are a out making 4udgements a out quality and value o! a programme. /e #ill look in much more detail at quality assurance in Bnit <. ,or no#- it is 4ust important that #e make this link et#een programme evaluation and quality assurance.
Bnit summary
This unit is titled /hat is &rogramme evaluation and 1hy is it im&ortantB2 /e egan y looking at de!initions o! programme evaluation- and this section has highlighted !or you some o! the purposes o! evaluation. +n the context o! ODL programmesprogramme evaluation is important to support decision"making and to enhance the quality o! the programme in question. To summarise- in this unit #e$ de!ined the concept o! programme evaluation considered evaluation as a speci!ic and applied !orm o! research noted that programme monitoring is an important data gathering tool !or programme evaluation egan to esta lish the links et#een programme evaluation and et#een and quality assurance.
Pro*ect tas+
-t is no! time to begin to think about "our $roAect or this handbook# 1 (irst& establish a !orking relationshi$ !ith a distance education $rogramme that "ou !ould be interested in evaluating# 2 *hen dra! a conce$tual ma$ +see (igure 1 on $age 11, o the $rogramme that "ou have chosen to evaluate# You !ill $robabl" ind it easiest to begin !ith <ox 1 and then !ork through each box numericall"#
21
You ma" need to a$$roach the leader o the $rogramme "ou have chosen or documentation or $erha$s a brie intervie! to ind the in ormation that "ou !ill need to hel$ "ou to develo$ "our s$eci ic model#
Re!erences
.a ie- E. and 2outon- 5. ;>>8 .he &ractice o$ social research- &ape To#n$ Ox!ord Bniversity Press in South %!rica .irley- 2. and 2orel- 6. 899: % &ractical guide to academic research- London$ 7ogan Page Dolley- 5. 899< Planning; monitoring and evaluating learning &rogrammes .uckingham$ Open Bniversity Press 2inistry !or ,oreign %!!airs. 899: Cuidelines $or &rogramme design; monitoring and evaluation- 1elsinki$ Department !or +nternational Development- at http$))glo al.!inland.!i)4ulkaisut)yleis)pdme)index.html Patton- 2. 899> Qualitative evaluation and research methods ';nd edition( London$ Sage Pu lications Ro inson- .. ;>>8 % &ractical guide to evaluation- a #orkshop hand ook produced !or the B6ES&O Learning !or Li!e Pro4ect in Blaan aatar$ 2ongolia Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!am Pu lications Thorpe- 2. 899= GEvaluating open and distance learningD- 1arlo#$ Longman- cited in D. Ro#ntree 899: G%ssessing the quality o! materials" ased teaching and learningD in %. Tait 'ed.( O&en Learning =$;
2eed3ac+ to selected activities
,eed ack to %ctivity ;

%t a !irst glance- the dou le arro#s tell us that each o! the parts making up a programme are inter"related and have an e!!ect on each other. ,ollo#ing the num ers a ove each lock- #e see a direct relationship et#een goals and the target group. This is an important point- since a programme is usually conceptualised and implemented to meet the needs o! a speci!ic target group. /ho the target group is #ill di!!er !rom programme to programme as #ill the goals. % good and #ell"de!ined programme #ould operationalise these goals into measura le indicators o! success. 1o#ever- you #ill see as you #ork #ith programmes that many do not reach this point- goals remain vague and one is not sure #hether they
22
have een achieved or not. This is one point at #hich programme evaluation is o! central importance. /e #ill return to this issue in later units. +n order to achieve the goals o! the programme several components #ill e needed. .ox < re!ers to the mechanisms and means o! implementation used to achieve the goals o! the programme- #hich- in this example- are di!!erent #ays o! delivering an educational programme. +n order to e a le to implement the programme- a management and a human resource ase #ill e needed. +n the context o! a distance education institution this #ill include institutional management structureseducators- tutors- materials developers- administrators etc. These are not the only people involved in a programmeN there are also various other stakeholders. Stakeholders are the people or organisations #ho have any interest- direct or indirect- in the success o! the programme. +n a distance education setting learners are an important group o! stakeholders. ,inally- all o! the parts making up a programme exist in context. There are several levels o! context that might in!luence a programme. ,or example- a distance education programme exists in the context o! the department and institution providing it- as #ell as in a local- national or international context. /hen designing an evaluation it is important to keep all o! these contexts in mind- as they could in!luence the !indings o! your evaluation. % programme evaluation- depending on its purpose- #ill consider all or some o! these components. Hou #ill see later that one approach to #riting an evaluation plan is to #ork out appropriate research question's( !or each o! the parts making up a programme.
,eed ack to %ctivity =

Hou #ill pro a ly have identi!ied several purposes and various #ays in #hich the reports #ould e use!ul to the organisation. These might include$
Purpose
to analyse the e!!ectiveness o! the programme including its reach o! the target audience to examine the impact o! the programme to decide #hether a !underDs money has een #ell spent to explore the strengths and #eaknesses o! the distance education approach used to make recommendations !or changes to the programme.
23
Bse!ul ecause
it provides a thorough appraisal o! the #hole programme y an external agent it in!orms the quality assurance process.
24
Evaluation conceptualisation$ de!ining the purpose o! an evaluation
B6+T
Bnit overvie#
The !irst unit o! this hand ook !ocused on setting out de!initions and clari!ying some important issues in the !ield o! programme evaluation and ho# this relates to ensuring quality #ithin distance education. +n this and the next unit you #ill #ork through some concepts and approaches that underlie programme evaluation. +n this unit #e #ill e concentrating on understanding the di!!erent purposes o! evaluationsho# #e ensure that the evaluation is use!ul !or those it is eing conducted !or- and also- ho# #e decide on #hat questions to ask #hen !ocusing our evaluation study. Each o! these is important !or planning an evaluation.
Bnit outcomes
/hen you have #orked through this unit- you should e a le to$ 8 de!ine the purposes and types o! evaluation ; state the di!!erence and relationship et#een !ormative and summative evaluation = identi!y the target audience o! an evaluation < !ormulate key evaluation questions @ descri e #hy consultation is important #hen conducting an evaluation.
Purposes and types o! evaluation

/e ended Bnit 8 noting the importance o! evaluation in supporting e!!orts to ensure quality in ODL programmes. 1ere #e #ill look in more detail at some o! the reasons
25
#hy people conduct evaluations- i.e. the purpose o! the evaluation. Bnderstanding this is central !or designing and conducting good evaluations. /e #ill then explore ho# the purpose o! an evaluation shapes ho# the evaluation is designed and conducted.
1ctivity 1 8@ mins
0ist at least three reasons !h" someone might !ant to conduct an evaluation# 1
The main purposes o! evaluations

People #orking in the !ield o! programme evaluation have attempted to summarise this #ide range o! reasons into categories so that it is easier to #rite a out and discuss evaluation purposes. /hile there is al#ays some di!!erence et#een the approaches adopted y di!!erent authors there is also a general acceptance o! the main purposes !or #hich evaluations are conducted. %lthough #e look at only t#o here- many di!!erent taxonomies o! evaluation have een proposed in the literature.
26
.a ie and 2outon ';>>8( use the #ork o! 2ichael Patton to de!ine three main purposes o! evaluations as$ 8 ; = to make 4udgements '4udgement"orientated( to make improvements 'improvement"orientated( to generate kno#ledge 'kno#ledge"orientated(.
Orientations o! evaluations
1 2 Budgement -m$rovement kno!ledge
Similarly ut #ith one addition- ,rancis Ru in '899@( de!ines !our categories$ 8 ; = < to improve per!ormance to make choices and decisions to learn lessons to increase accounta ility.
/e #ill look in more detail at .a ie and 2outonDs three categories. .ut e!ore #e do- #e should note that the last o! Ru inDs points E to increase accounta ility is also an important purpose o! evaluation- especially in the context o! quality assurance. Evaluation is one means o! assessing #hether people and institutions do #hat they have een assigned to do- and so provides a means o! checking sta!! per!ormanceas people #ill have to provide ans#ers !or their actions during an evaluation process. %s #e noted in Bnit 8- !or these kinds o! reasons evaluations can sometimes e challenging to conduct- and in such cases evaluators are rarely liked y everyoneF Some approaches to evaluation have explicitly sought to address this issue. ,or example participatory evaluation seeks to involve programme participants in all parts o! the evaluation and approaches evaluation more as learning opportunity than an opportunity !or making 4udgements. To return to our purposes o! evaluation E #e #ill look rie!ly at each o! the three purposes de!ined a ove. This #ill also provide us #ith an opportunity to look at the distinction et#een !ormative and summative evaluation- #hich has ecome a #idely accepted distinction #ithin evaluation methodology.
5udgement"oriented evaluations
Think o! a situation in #hich you might need to conduct an evaluation o! a programme that has provided teachers #ith computer training to assess #hether they are no# a le to use computers and #hether this has improved their teaching. +n this instance you #ould e conducting a 4udgement"oriented evaluation. This category o! evaluation research includes those studies that set out to determine the #orth- value or success o! a programme. .ut- ho# do you kno# i! the teachersD teaching has improvedC &entral to all
27
evaluations that set out to make 4udgements is the !ormulation o! criteria against #hich to 4udge the success o! the programme. Think ack to Bnit 8- #here #e looked at the structure o! a programme. Do you remem er that #e noted that a good programme #ould have clearly de!ined goals and that those goals #ould have een translated into measura le outcomes or critical success !actorsC /e also noted that- o!ten- programmes do not have clear goals and measura le outcomes. O!ten- it is only #hen evaluations egin that goals are made explicit and measura le outcomes de!ined. % core step in conducting a 4udgement"oriented evaluation is to de!ine these criteria and ho# they #ill e measured. Since #e are measuring #hether a programme meets speci!ied criteria- then #e must e conducting the evaluation a!ter the programme has een implemented or to#ards the end o! implementation- !or example- at the end o! a ne#ly introduced distance education course. This is #here #e meet up #ith the concept o! summative evaluation.
Summative evaluations
4ummative evaluation is an evaluation that takes place to make 4udgements a out the overall success or !ailure o! a programme- and is o!ten related to decisions a out #hether the programme should e continued or not. Summative evaluation usually takes place to#ards the end o! a programme or a!ter implementation.
Summative evaluation
Summative evaluation is an evaluation that takes $lace to make Audgements about the overall success or ailure o a $rogramme#
,unders o!ten use this kind o! evaluation to test #hether the programme they have !unded has achieved the outcomes it #as supposed to. The ans#er to this question may then determine #hether !urther !unding is granted or not. This kind o! evaluation is also conducted to identi!y the lessons that can e learned !rom the experience and applied to !uture pro4ects or programmes.
+mprovement"orientated evaluations
%s the name suggests- improvement"oriented evaluations are concerned #ith improving the programme #hile it is eing implemented rather than 4udging ho# success!ul it has een during implementation or a!ter completion. This kind o! evaluation is usually called a !ormative evaluation.
,ormative evaluation
Formative evaluation is concerned !ith im$roving a $rogramme !hile it is being im$lemented#
1ere the evaluation helps to G!ormD the programme y providing a means o! assessing the strengths and #eaknesses o! the programme- looking at ho# implementation takes place- considers the response o! programme participants etc.
28
The aim here is to identi!y pro lems as they occur so that they can e quickly corrected. Sometimes more su stantial changes o! approach or ad4ustment o! plans may e needed to improve the programme.
7no#ledge"orientated evaluations
5udgement and improvement"orientated evaluations are carried out in order to provide direct input into a speci!ic programme or to assess #hether or not the programme is success!ul. +n other #ords- oth !orms o! evaluation have an applied !ocus. +n contrast+nowledge5oriented evaluations are conducted to improve our understanding- to help uild theory and sometimes to in!orm policy"making processes.
7no#ledge"orientated evaluations
im$rove our understanding build theor" in orm $olic"=making
,or example- you might evaluate a range o! distance education programmes to assess #hich approach led to the est learning outcomes- the results o! #hich could in!orm the planning o! !uture programmes. This purpose !or evaluation is much less common than the t#o descri ed a ove and you are pro a ly more likely to make use o! !ormative and summative evaluationsor a com ination o! these- in your o#n evaluation #ork.
.alancing !ormative and summative evaluation

/e have de!ined summative evaluation as evaluation that is usually conducted to#ards the end o! a programme #ith the purpose o! 4udging its success or e!!ectiveness. ,ormative evaluation in contrast #as de!ined as evaluation conducted during a programme !or the purpose o! improving it. ,rom these de!initions you might have concluded that these t#o !orms o! evaluation stand in opposition to each other- that the di!!erent kinds o! evaluation are de!ined y purpose and point in programme development at #hich they are conducted.
7no#ledge"generating evaluations
%n exam$le o no)ledge'generating evaluation is included in the Resources File C see *ac%roo# -n this stud" the authors evaluated a range o distance learning higher education $rogrammes in order to make recommendations or success ul im$lementation o distance education $rogrammes at college and univerist" levels in the Bammu region# *he evaluations are used to generate kno!ledge about im$lementing distance higher education $rogrammes in a general sense& rather than making Audgements about or im$roving a s$eci ic $rogramme#
Rather- in practice these categories overlap. Summative evaluation can e conducted during a programme to measure achievement o! speci!ic o 4ectives at a point in time. Simultaneously- the evaluator may also e using !ormative evaluation
29
to evaluate and improve programme processes. These di!!erent kinds o! evaluations complement each other and using a com ination o! oth is likely to ensure a more thorough evaluation study. Each type serves a speci!ic purpose in the evaluation o! a programme. %lthough complementary- planning !or !ormative and summative evaluations usually get done at di!!erent times and may even e conducted y di!!erent evaluators depending on the speci!ic programme and circumstances.
1ctivity 2 => mins
*his activit" is based on !hichever o the t!o evaluation exam$les& Evaluations 1 or 2 in the Resources File" "ou have chosen to !ork on# -t !ill hel$ "ou think about the extent to !hich a given evaluation is ormative& summative& or a combination o both# 1 -s this a ormative or summative evaluation& or a combination7 )x$lain brie l" !h" "ou came to this conclusion#
+denti!ying an evaluationDs target audience

Evaluations are usually applied research E except in the case o! kno#ledge"oriented evaluations rie!ly descri ed a ove. +t !ollo#s that this applied research should e use!ul to some ody or some organisation. /hen designing an evaluation you need to clari!y #ho the target audience is. This #ill depend to some extent on #ho asked you to conduct the evaluation and the type o! evaluation eing conducted E and #hat the evaluation report #ill e used !or. +n oth o! our evaluation examples- #e have an evaluation consultant conducting an evaluation o! an ODL programme. +n each case the !unders o! the programme commissioned the evaluation and- as such- are its main target audience. The programme implementers #ould also e target audiences since they #ould e interested in ho# e!!ective their distance education programme #as and #ould like to kno# #hich areas should e improved. Others #ho may #ish to use similar approaches- or perhaps replicate a programme in another context- #ould also e interested in the evaluation !indings. .ecause o! the range o! audiences the
30
evaluation #ill need to consider the politics o! the situation. ,or example- the needs and interests o! !unders and pro4ect implementers may not e the same- and may in !act e contradictory i! the !unders are considering a #ithdra#al o! !unding ased on the evaluation results. .oth o! these examples highlight !or us the !act that evaluation studies and the report thereo!- most o!ten have a range o! target audiences. /hen designing your evaluation and #hen #riting your evaluation report you should ask yoursel! the !ollo#ing questions$ +s the audience's( !or the evaluation clearly speci!iedC Does the evaluation address the particular concerns o! the main target audienceC This is not an easy question to ans#er. One #ay o! arriving at an ans#er is through a process o! consultation- #hich #e #ill address elo#. 1o# should the evaluation report e #ritten so that it is appropriate and accessi le !or the target audiencesC
,ood !or thought

*hink back to the institution "ou have chosen or "our $roAect tasks# @o! think about ho! "ou might begin to ans!er these ;uestions in that context# You !ill return to this belo! !hen "ou go back to "our $roAect !ork#
Hou might like to look at the Re&orting on research and evaluation to in$luence and su&&ort change module- #hich deals in more detail #ith the construction o! reports and dissemination strategies.
,ormulating key questions

The core module- Planning research and evaluation- has covered ho# to develop research questions- and also the importance o! this !or the overall success o! the research pro4ect eing conducted. ,or a success!ul evaluation- the clear !ormulation o! your key question's( is !undamental. %t this point it use!ul to recall the point made y 7umar '8999 p. ?($ 8% research &ro*lem is li e the $oundation o$ a *uilding2 .he ty&e and design o$ the *uilding is de&endent u&on the $oundation2 I$ the $oundation is 1ell designed and strong; you can e7&ect the *uilding to *e also2 .he research &ro*lem serves as the $oundation o$ a research study+ i$ it is 1ell $ormulated you can e7&ect a good study to $ollo12: The same principles that you covered !or research questions in general also apply to evaluation research questions. +n addition- the purpose and type o! evaluation eing conducted #ill in!orm your decision a out the key question's(.
31
Evaluation types are a slightly di!!erent- although related- distinction to that #e made a ove et#een di!!erent kinds o! purposes !or evaluations. /ithin each o! the purpose categories it is possi le to conduct di!!erent types o! evaluations. /e #ill again dra# on .a ie and 2outon ';>>8( in presenting this section. These authors dra# on the !our types o! evaluation proposed y Posavac and &arery '899;( although it is important to note that di!!erent authors classi!y these types o! evaluation in other #ays too.
Type 8$ Evaluation o! need

1ere you #ould e interested in evaluating the needs o! an identi!ied target group. O!ten this is related to the provision o! a speci!ic service. ,or example- you may need to evaluate #hether students taking part in your distance education programme need additional peer support or not- e!ore implementing a ne# peer" support programme. This type o! evaluation can also e called a needs assessment.
Juestion examples
/hat training do primary teachers need on the ne# school curriculumC /hat income"generating activities do #omen in rural communities #ant to learn a outC /hat kind o! childcare in!ormation do young parents in rural areas needC
Type ;$ Evaluation o! process

+n this case the evaluation #ould !ocus on the design and implementation o! a programme. ,or example- you may need to evaluate ho# #ell your institutionDs current administration system #orks #ith respect to student registration- assignment su mission and return o! marks and examinations. Hou #ould most likely e interested in understanding the processes !ollo#ed in order to assess #hether this is the most e!!ective system or #hether improvements are needed.
Juestion examples
+s the programme eing implemented as designedC Does the programme serve the target populationC %re the necessary administrative and management procedures in place !or the programme to run e!!ectivelyC Do the materials reach the rural village to coincide #ith the supporting radio roadcastsC
32
Type =$ Evaluation o! outcome

Outcome evaluation is one o! the most common !orms o! evaluation. +t assesses the e!!ects o! the programme. This is usually done y comparing the e!!ects #ith the goals and measura le outcomes de!ined at the outset. To assess outcome an evaluator should try to collect aseline data that re!lects the situation e!ore the programme #as implemented. .y collecting the data a!ter and)or during the programme implementation- the evaluator can then assess #hether there have een any changes due to the programme. /hen considering programme outcomes or e!!ects the evaluator usually looks at oth intended and unintended e!!ects. Evaluators are usually interested in proving that some !orm o! positive change has occurred as a result o! the programme. To do this- it must e esta lished that some change or e!!ect has occurred- and also that this is a result o! the programme " and not other external !actor's(. +t is also important that #e distinguish et#een outcomes and outputs. These are terms that you have pro a ly seen used in a variety o! #ays. %n outcome is the end result or e!!ect o! a programme- #hile an output is a product o! a programme. ,or example- in the SO2DEL evaluation example- the programme aims to achieve the outcome o! providing literacy- numeracy and li!e skills !or out"o!"school children and adults in Somalia and Somaliland. +n striving to achieve this outcome- the programme includes several outputs- !or example- print materials- radio roadcastsand !ace"to"!ace teaching sessions.
Juestion examples
To #hat extent have intended outcomes een achievedC To #hat extent have literacy levels een improvedC /hat unintended outcomes have een achievedC 1as the programme achieved the speci!ied goalsC
Type <$ Evaluation o! e!!iciency

+n this kind o! evaluation E e!!iciency E the emphasis is on #hether the programme has achieved its outcomes in the most e!!icient and cost e!!ective manner. +n this case the costs o! the programme '!inancial and others such as time(- are measured against the ene!its o! the programme. ,or example- a pilot pro4ect might test the use o! in!ormation and communication technology '+&T( in supporting distance education in a developing country context. +t might involve setting up a centre #here ;> learners have access to their o#n computer- and a common television and video recorder to support a speci!ic distance education course. The pro4ect might sho# that learning outcomes improved
33
remarka ly due to the introduction o! +&Ts. One might then e tempted to conclude that this programme #as a success and should e replicated in other environments. 1o#ever- an evaluation o! e!!iciency may sho# that the costs o! providing this technology per learner #ould more than dou le standard learning costs. +n this case it #ould e necessary to assess #hether the improved learning outcomes 4usti!ies this additional cost and also #hether !unds are availa le to implement this on a larger scale.
Juestion examples
/ere !unds spent !or the intended purposeC Did the programme achieve success at an accepta le costC 1o# do the outcomes o! this programme compare to those o! a similar programme o! compara le costC
The range o! terminologies used in this area can e con!using. ,or this reasonsome use!ul explanations are provided in Ta le 8. 6a3le 1 4ome terminology in the evaluation o! e!!iciency
6erm &ost ene!it E.planation The relationship et#een the costs o! a programme and its ene!its- in terms o! the actual or estimated monetary value.This contrasts #ith cost"e!!ectiveness measurement- #hich may measure ene!its in terms o! di!!erent 'non"monetary( kinds o! outcomes. &ost ene!it results are sho#n as the ratio o! costs to ene!itsor as a !igure representing the net ene!it 'or net loss(.The technique allo#s comparisons to e made et#een programmes and policies that have quite di!!erent outcomes. &osts o! input are typically easier to measure than ene!its o! outcomes. G&ost ene!itD is o!ten used loosely in everyday language to re!er to the relationship et#een the cost 'not necessarily !inancial( o! doing something and the ene!its 'o! #hatever kind( gained. The relationship et#een the achievement o! o 4ectives o! a programme- course or pro4ect and their costs. % cost"e!!ective programme is one that can e sho#n to provide either the maximum gains !or a given level o! resources or a given level o! ene!its at the lo#est cost. &ost"e!!ectiveness analysis is most easily applied #hen there is one clearly identi!ia le outcome measure or gain 'this is not al#ays the case #ith education pro4ects(. %n education programme is cost e!!icient i! its outputs cost less per unit o! input#hen compared #ith alternatives. The measurement o! the resources used 'costs( to achieve stated goalsN the measurement or ratio o! output to input.To e e!!icient- a programme or institution needs to demonstrate that good use is eing made o! the resources availa le. E!!iciency is not an a solute measure ut a 4udgement a out relative #orth. The extent to #hich an education or training programme or intervention can demonstrate that its goals 'explicit ones( have een met.
&ost" e!!ectiveness
&ost e!!iciency E!!iciency
E!!ectiveness
Source$ Ro*inson ($orthcoming!
34
Summary
+n this section #e have looked at the evaluation o! need- the evaluation o! processthe evaluation o! outcome and the evaluation o! e!!iciency. %s #ith the distinction et#een !ormative and summative evaluation- there is likely to e overlap et#een these types o! evaluation #ithin any one evaluation study. Hou #ill very o!ten !ind that questions !or all o! the a ove types are asked in a comprehensive programme evaluation.
1ctivity # => mins
-n this activit" "ou !ill take another look at the evaluation exam$le !hich "ou used in %ctivit" 2# 1 ?hat t"$e o evaluation do "ou think this is7
2 @ote do!n !hat research ;uestions are being ans!ered#
&onsultation E a key to success

This !inal issue that you need to think a out in the design o! your evaluation is the relationship you 'as the evaluator( #ill have #ith those involved in the programme eing evaluated- as #ell as #ith any others #ho have an interest in the evaluation 'these are the stakeholders(. These could include !unders such as examination oards or accreditation authorities or pro!essional associations or employers or sponsors o! students !or example. There has een much de ate concerning the relationships et#een researchers and participants and their respective roles in evaluation research. Traditionallyevaluation research assumed that researchers stood ack !rom those eing researched- collected data !rom them- analysed it and #rote a report. 2any
35
evaluators no longer accept this and usually the evaluator and programme participants- programme leaders or !unders- must reach some clearly de!ined agreements a out shared expectations o! the purpose o! the evaluation and ho# it #ill e conducted. Bnless the programme is small and !ocused it is unlikely that the evaluator #ill e a le to consult all participants or stakeholders. +n this case- the consultation usually takes place #ith those requesting the evaluation research. ,or contract research- this is o!ten clearly stated in a #ritten contract agreement. There is an approach to evaluation called Gparticipatory evaluationD that is ased on the principle o! consultation at all levels. %s noted a ove- this approach to evaluation !ocuses on evaluation as a learning process rather than a process o! making 4udgements. Programme participants make the 4udgements- rather than the evaluator. The role o! the evaluator in participatory evaluation is as a !acilitator rather than as the Gexpert researcherD. The evaluator !acilitates an evaluation process- ut the !ocus o! the evaluation- ho# it takes place- #hen it takes place- the research questions and so on- are determined y the programme participants rather than the evaluator or !under. +t is particularly important !or researchers or evaluators to e a#are o! organisational dynamics and structures- as o!ten the person responsi le !or negotiating a contract or requesting that an evaluation e conducted is not directly involved in the micro management o! the programme eing evaluated. +t may also e the case that the evaluation is not #elcomed y the programme providers or y the institution. Throughout the evaluation- expectations o! oth the evaluator and programme participants need to e monitored and managed. This may include returning to the !ocus o! the evaluation and revie#ing #hether this still meets expectations- and i! not- #hat should e done a out this. 1o# this is done #ill depend on various !actorsincluding the kind o! programme eing evaluated- the relationship o! the evaluator to the programme 'e.g. an employee or contracted in !rom outside(- the scope o! the evaluation- the time!rame etc. ,or large programmes and complex evaluations #ith a range o! stakeholders- it might e use!ul to set up a steering group to ensure consultation. +t #ill e important that this group is representative o! the range o! stakeholders in the programme. There #ill e di!!erent kinds o! evaluation participants or stakeholders. Some #ill e the evaluation !unders- the programme !unders- those responsi le !or the planning and management o! the programmethose #ho #rite distance education materials- the learners- the assessors etc. Equally important to managing expectations !rom the outset o! evaluation researchis to esta lish agreement et#een researchers and evaluation participants a out ho# data #ill e collected and analysed.
36
+n this regard- 2iles and 1u erman '899<( suggest the !ollo#ing kinds o! questions !or oth parties to consider$ /hat are the time and human resource requirements involvedC /hat data collection techniques #ill e usedC /ill participants volunteer to take part in the researchC /ill participants and their in!ormation)responses e anonymousC /ho #ill plan and manage the studyC /ho #ill keep notes- collect and analyse dataC /ho #ill revie# the reports producedC +n #hat #ays #ill researchers and participants ene!it !rom the evaluation researchC
This section has highlighted the range o! issues that the evaluator #ill need to !ace in the course o! an evaluation. +t is essential that you negotiate these issues and relationships #ith care and !rom the outset o! the evaluation research process.
Bnit summary
To summarise- in this unit #e covered a lot o! groundF /e$ de!ined di!!erent purposes !or and types o! evaluations explored the di!!erence et#een !ormative and summative evaluation as #ell as the relationship et#een them highlighted the importance o! question !ormulation !ormulated evaluation questions discussed the importance o! de!ining the target audiences o! an evaluation discussed the importance o! consultation !or success!ul evaluations.
37
Pro*ect tas+
>o ar in "our $roAect "ou should have' identi ied a distance education $rogramme to !ork !ith dra!n a sketch o the $rogramme so "ou have a clear idea o !hat "ou are !orking !ith#
You are no! read" to begin the $re$aration o "our evaluation $lan# 2se the ollo!ing ;uestions to hel$ "ou do this' 1 ?hat is the $ur$ose o m" evaluation7 2 ?hat are m" research ;uestion+s,7 3 ?hat t"$e o evaluation !ill - be conducting7 4 ?ho is the target audience o m" evaluation7 5 ?ho !ill it be im$ortant to consult in designing this evaluation7 +You might like to arrange a time to brie l" meet !ith one or some o the $eo$le to ask !hat the" think about the evaluation "ou are $lanning#,
Re!erences
.a ie- E. and 2outon- 5. ;>>8 .he &ractice o$ social research- &ape To#n$ Ox!ord Bniversity Press in South %!rica 'pp ==?"==9( Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!am Pu lications 'pp =8( 7achroo- 7. 8999 GDistance education programmes at college and university levels in 5ammu region$ an evaluative studyD Indian ,ournal o$ O&en Learning :- =$;9="=>@ 7umar- R. 8999 Research methodology+ a ste&(*y(ste& guide $or *eginners - London$ Sage Posovac- E. and &arey- R. 899; Programme evaluation+ methods and case studies Engle#ood &li!!s$ Prentice 1all Ro inson- .. '!orthcoming( %chieving Duality in o&en and distance learning- Paris$ B6ES&O 2iles- 2. and 1u ermann- %. 899< Qualitative data analysis- London$ Sage 'pp <:(
38
,eed ack to %ctivity 8

Hou might have included some o! the !ollo#ing reasons !or evaluations in your list$ !or e!!ective management and planningto measure progress and)or e!!ectiveness to learn lessons !or !uture programmes ecause !unders made evaluation a condition !or giving money as part o! a research pro4ect ecause your institution has a policy a out conducting evaluations regularly even though sta!! arenDt al#ays clear a out #hy evaluations are done or ho# the results #ill e usedF

/e thought that these evaluations could e classi!ied as !ollo#s$
8 Somali distance education literacy programme

This evaluation aimed to 8analyse the e$$ectiveness o$ the SOMDEL &rogramme in terms o$ reaching its target *ene$iciariesE to e7amine the im&act o$ this three(1ay a&&roach 1hich incor&orates &rinted materials; radio *roadcasts and $ace(to($ace instructionE to discuss the strengths and 1ea nesses o$ this a&&roach 1ith a variety o$ sta eholdersE and to ma e recommendations o$ suggestions $or $uture &rogrammes: 'p. :(. % look at the research design o! this study #ill sho# that the research #as conducted to#ards the end o! the !irst set o! roadcasts !or this programme. /e see that some o! the aims o! the evaluation #ere to assess the impact and e!!ectiveness o! the programmes- i.e. summative evaluation approach. The evaluation also had the purpose o! learning lessons !rom this !irst set o! roadcasts !or improving the !uture implementation o! the programme. This aspect o! the evaluation can e descri ed as !ormative evaluation.
; &ourse !or distance education policy makers in Southern %!rica

The aim here #as to 8determine 1hether the online course met course &artici&ants:
39
needs and 1hether the delivery strategies 1ere e$$icient and e$$ective: 'p. @(. Thus- this study is a out making 4udgements a out the programme and as such is an example o! a summative evaluation. 1o#ever- a closer look at the methodology used #ill sho# that !ormative evaluation data #as also used in carrying out the summative evaluation.
,eed ack to %ctivity = Evaluation Example One$ SO2DEL

This example is an interesting one ecause instead o! !ormulating research questions- the evaluation has !ormulated aims and then makes use o! the pro4ect log!rame as the !rame#ork o! the evaluation. +n the introduction to this evaluation report #e read the !ollo#ing$ .he s&eci$ic aims o$ the evaluation 1ere to analyse the e$$ectiveness o$ the SOMDEL &rogramme in terms o$ reaching its target *ene$iciariesE to e7amine the im&act o$ this three(1ay a&&roach 1hich incor&orates &rinted materials; radio *roadcasts and $ace(to($ace instructionE to discuss the strengths and 1ea nesses o$ this a&&roach 1ith a variety o$ sta eholdersE and to ma e recommendations or suggestions $or $uture &rogrammes (&2 4!2 .ased on these aims #e can say that this evaluation is an outcome)impact evaluation and also an evaluation o! process. %lthough no speci!ic research questions are noted- do you see that #e could re!ormulate these aims as research questionsC &onsider the !ollo#ing possi le research questions$ 8 ; = Did the SO2DEL programme reach those it #as intended to reachC OEvaluation o! processP /hat #as the impact o! the three"#ay approach adoptedC OEvaluation o! outcomeP /hat are the strengths and #eaknesses o! the SO2DEL programmeC OEvaluation o! processP
Evaluation Example T#o$ &ourse !or Distance Education Policy 2akers

On page : o! the evaluation report #e !ind the terms o! re!erence !or this study. /e see that- #ith a !ocus on online delivery and the !ace"to"!ace #orkshop- the evaluator #as requested to determine 1hether the online course met the &artici&ants: needs (i2e2 o$
40
increasing their understanding and no1ledge o$ &olicy develo&ment &rocesses! and 1hether the delivery strategies 1ere e$$icient and e$$ective2 On the asis o! this #e can conclude that this evaluation is oth an evaluation o! process and also an evaluation o! e!!iciency. The research questions are indicated in the quotation. They are$ 8 ; Did the online course meet the participantsD needsC /ere the delivery strategies e!!icient and e!!ectiveC
41
42
Designing and implementing an e!!ective evaluation
B6+T
Bnit overvie#
+n this unit you #ill consider ho# to move !rom the research design phase that you reached in Bnit ; to the practical issues o! choosing research methods- collecting data- analysing data and #riting up the !inal evaluation report. %lthough #e #ill re!er to di!!erent kinds o! research methods in the context o! conducting an evaluation- #e #ill not cover these in detail here since you can read a out research methods in other hand ooks and modules in the PREST series and other Research 2ethodology re!erence ooks.. +n addition- the Resources File includes more detailed in!ormation. /e #ill also- very rie!ly- consider some important issues to keep in mind #hen udgeting !or an evaluation. %!ter completing this unit- you #ould have #orked through all the steps required !or planning a good evaluation study. .y the end o! this unit- i! you decided to do the pro4ect tasks- you should also have your o#n evaluation plan or proposal.
Bnit outcomes
/hen you have #orked through this unit- you should e a le to$ 8 ; = < @ list the planning questions you need to ans#er #hen designing an evaluation de!ine evaluation outputs and speci!y activities using the Logical ,rame#ork approach choose evaluation research methods descri e the logistical issues that need to e taken into account #hen conducting an evaluation outline the udgeting issues !or evaluations.
43
Evaluation planning- !orming a conceptual !rame#ork

+n Bnit ; #e spent some time thinking through some o! the theoretical and research design principles important !or conducting an evaluation. To summarise much o! #hat has already een covered and some o! #hat is yet to come in this unit- let us have a look at a set o! key questions that Ru in '899@ p. =>( notes should in!orm the development o! an evaluation design. Overall evaluation planning questions to ans#er$ /hat is the purpose o! the evaluation and !or #hom is it eing conductedC /hen in the pro4ect)programme li!ecycle #ill the evaluation e conductedC /hat are the scope- o 4ectives and key questions o! the evaluationC /ho holds responsi ility !or managing the evaluation- collecting data- #riting reports etcC 1o# #ill the evaluation e conducted- #hat procedures #ill e !ollo#edC /hat !inancial resources- human resources- time and other resources #ill e neededC 1o# #ill the evaluation !indings e presented to di!!erent audiencesC 1o# #ill !ollo#"up take placeC
+n addition to these questions that help #ith planning evaluation researchesta lishing a conceptual !rame#ork !or an evaluation also requires that the evaluator understands the programme to e evaluated- ho# it is designed and #hat its aims are. This in!ormation might e o tained !rom documentation and)or talking to those directly involved in the programme. +n many cases there #ill also e a theoretical !rame#ork underlying the design programme and ho# it is implemented. ,or example- i! the aim o! the programme has een to introduce ne# teaching and learning approaches #ithin an existing distance education programme- you #ill !ind a range o! literature and other resources that address the issues o! teaching and learning !or distance education. Perhaps the programme you are evaluating !ocuses on the use o! computers to enhance distance education. Hou may !ind a range o! literature on the use o! technology in distance education that could provide help!ul ackground !or your evaluation. These resources can e used to help you design your study. +n some instances- esta lishing the conceptual asis #ould also include exploring the context #ithin #hich the speci!ic programme takes place. ,or example- i! the programme #ere seeking to ring current teaching and learning methods in line #ith ne# national distance education policy- then it #ould e important !or the evaluator to understand the policy.
44
6ot only #ill articulating the conceptual !rame#ork o! the evaluation provide use!ul input #hen developing your research questions- ut you may also e a le to raise some generic issues out o! your !indings- instead o! programme speci!ic !indings only. ,or example- you might e a le to make some more general conclusions a out a ne# teaching and learning approach- or perhaps the use o! computers in distance education.
1ctivity 1 => mins
Evaluation 2 +see "our Resources File, contains a !rite u$ o the conce$tual rame!ork o the evaluation# You !ill ind this in cha$ters 2 +<ackground, and 3 +1once$tual (rame!ork, %s "ou read& ans!er the ollo!ing ;uestions' 1 Do "ou think the context o the $rogramme is thoroughl" ex$lored7
2 1an "ou identi " the literature that has been consulted to ormulate this conce$tual rame!ork7
.here is no $eed*ac to this activity
45
De!ining the evaluation delivera les and activities

+n Bnit ; #e looked in some detail at research question !ormulation. +n order to ans#er your evaluation questions you #ill need to de!ine the delivera les 'i.e. #hat you need to produce to ans#er your questions during the evaluation(- that are required !rom the evaluation process. +n some cases- the delivera les o! the evaluation are de!ined y the organisation requesting the research and may also e de!ined in your research contract or rie!. ',or an example o! this- have a look at %ppendix 8 'p. 9A( o! Evaluation ) in your Resources File. 1ere you #ill see the terms o! re!erence !or this evaluation pro4ect. 6ote that these terms o! re!erence speci!y exactly #hat the evaluation should cover- and so #hat the delivera les #ill e.( Once you have de!ined your delivera les- you then need to think #hat speci!ic research activities are required in order to produce each o! these. There are many di!!erent #ays o! doing this- depending on the requirements o! the speci!ic evaluation- your research ackground- and your approach to research. This process has een explored in depth in the core modules. 1ere #e #ill look at t#o methods o! evaluation planning.
!ctivity % $ hr #& m 1ctivity 2 8 hr => mins
2se Evaluation 2 rom "our Resources File or this activit"# 1 *urn to $age 96 o Evaluation 2# 2 .ead the terms o re erence given to the evaluator# 3 2sing onl" these terms o re erence and& imagining that "ou are the evaluator o this $roAect& com$lete the ollo!ing table'
/ist the delivera3les !or this evaluation /ist some o! the research activities needed

46
Log"!rame approach
+n the last activity you sa# a detailed example o! one #ay o! planning your evaluation study. Since in this series you have een introduced to the logical !rame#ork 'o!ten simply called a log !rame( !or planning research- #e #ill #ork through a log !rame example here. % log !rame can e used to plan a research pro4ect or the implementation o! a programme. Since this !ormat outlines the outputs and activities o! the programme 'note$ not the outputs and activities o! the evaluation( it can e used as the conceptual !rame#ork against #hich to evaluate ho# success!ul the programme is. %lthough people o!ten !ind log !rames time consuming and sometimes even intimidating to complete- they are a use!ul planning tool. +n general- log !rames summarise in a standard !ormat$ #hat a programme aims to achieve ho# the goals #ill e achieved #hat is needed to ensure success 'including assumptions as #ell as inputs( proposed #ays o! measuring progress potential pro lems that might e !aced.
Log !rames E #here to !ind out more

htt$'EE!!!#bond#org#ukE$ubsEi$!# htm Do!nload the resource guide& 0ogical (rame!ork %nal"sis# <ond is a 23=based net!ork o voluntar" organisations& !orking in the international develo$ment ield# *his means that these guidelines are !ritten or $eo$le living in the 23 !ho are involved in develo$ment !ork internationall"# *he guide does& ho!ever& include some use ul tables that $rovide de initions o all the log rame terms as !ell as guidelines on ho! to make use o a log rame or $lanning#
The !ollo#ing activity is designed !or you to assess ho# #ell you understand the role o! the di!!erent parts o! a log !rame.
1ctivity # => mins 2se Evaluation 2 rom "our Resources File or this activit"# *urn to $ages 11=14& !here "ou !ill ind an exam$le o a logical rame!ork being used or $lanning an evaluation# % ter reading through the logical rame!ork& ocus on onl" the irst column that describes the $rogramme being evaluated to ans!er the ollo!ing ;uestions# 1 6o! does the ormulation o the 9goal: and the 9$ur$ose: o this $rogramme di er7
47
2 ?hat does the out$uts section tell us about the $rogramme7
3 -n terms o $rogramme im$lementation& ho! are the activities related to the out$uts7
The other columns o! the log !rame

6o# look at the second and third columns o! this log !rame #here you #ill see ho# the evaluator has made use o! this tool !or the purposes o! her evaluation study. Read through the second column. Do you see that the evaluator has documented the extent o! achievement !or each aspect o! the programme as de!ined in the programme log !rameC This is #hat is documented in column ;. .ecause the log !rame outlines exactly #hat the programme should achieve- it is a use!ul tool against #hich the success o! the programme can e assessed. Read through the third column. /hat is the purpose o! this columnC /ell- this column provides additional details !rom the evaluation research to ack up #hat has een noted in the second column- #here the extent o! achievement is assessed. &olumn = also provides in!ormation that is help!ul !or making recommendations a out the !uture o! the programme. To sum up- #e looked at t#o #ays o! planning an evaluation study. Remem erthere are also many other #ays o! presenting your evaluation plan. 7no#ing ho# to use a log !rame is a very use!ul skill since many !unding agencies #ill request proposals in this !ormat.
&hoosing evaluation research methods

Programme evaluation usually requires that several aspects o! the programme itsel! are assessed- and that a range o! programme participants- their rolesresponsi ilities and activities are also researched in order to ans#er the evaluation research question. 'Hou sa# the complexity o! this in ,igure 8.( .ecause o! this complexity you #ill usually !ind that a range o! data gathering
48
methods- oth quantitative and qualitative are used in evaluation. The rationale !or using a variety o! methods and sources o! in!ormation is to enrich the quality o! the data and to ensure sensitivity to the !ull range o! issues and concerns that are likely to emerge during an evaluation study.
1ctivity % => mins
2se Evaluation 1 and Evaluation 2 rom "our Resources File or this activit"# 1 -n the tables belo!& list belo! the di erent kinds o research methods used# 2 @ote !hether these are ;ualitative or ;uantitative# 3 0ist one strength and !eakness o each#
Evaluation 8 2ethods Jualitative)quantitative Strengths)#eaknesses
Evaluation ;
2ethods
Jualitative)quantitative
Strengths)#eaknesses
49
Bse o! existing in!ormation

The speci!ic research questions eing asked are one in!luence on the choice o! research methods. This #ill also e a!!ected y the kind o! in!ormation that already exists #ithin the programme- and that #hich needs to e speci!ically collected !or the purposes o! the evaluation. .oth o! the evaluation examples that #e have looked at have made use o! programme documentation in their analyses. This #ill e in!ormation that already exists in the programme and includes- among others- proposals- programme reports- planning documents- memos- emailsudgets- etc. +! you think ack to Bnit 8 you #ill remem er that #e noted there that programme monitoring systems are an important tool !or evaluation. % good programme monitoring system #ill provide the evaluator #ith much o! the in!ormation needed- especially in!ormation related to the processes and systems #ithin the programme.
Sources o! in!ormation a out qualitative and quantitative methods

*he core modules o this series# Evaluation 1 C see .esources (ile# Evaluation 2 C see .esources (ile# +%yte" &%a$ter , C see htt$'EE!!!#eldis#orgEstaticED/1 7867#htm
,or example- you #ill e a le to !ind in!ormation a out the num er o! learners enrolled- num ers o! learners in each course- lists o! tutors- assignments sent outassignments received- marks and assessments etc. +n the context o! distance education programmes- a monitoring system should also provide you #ith in!ormation that you can use as indicators. ,or example- researchers sometimes use dropout rate as an indicator o! success o! a programme. Once you have esta lished!or example- the dropout rate o! students #ithin a particular course- you might then conduct individual intervie#s #ith students to determine their reasons !or #ithdra#ing !rom the course. This #ould provide you #ith in!ormation that the monitoring system could not provide. ,or example- a researcher- interested in studying the causes o! student dropoutintervie#ed y telephone all the students 'over 8>> practising teachers(- #ho had dropped out o! one course in a distance learning diploma programme that comprised o! !our courses 'Ro inson 89:<(. Reasons !or #ithdra#al #ere explored #ith each person during the intervie#. ,rom this data- the researcher #as a le to quanti!y the reasons- to explain them in terms o! personal"related and course"related reasonsand to compare them #ith another set o! data on the student dropout on another o! the courses in the same diploma programme.
50
2ultiple methods research

/e noted a ove that the complex evaluations o!ten require the use o! a range o! research methods. The use o! a com ination o! research methods can e termed multi5method or mi.ed5method approaches. These approaches can e more rigorous than single method assessments- i! each o! the research methods is used correctly 'Tshakkori and Teddlie 899:(. 1o#ever- one should not assume that using more than one method necessarily means a etter study. To e e!!ective- multiple methods research must$ e competently designed and conducted !or all o! the methods used e appropriate !or the research question 'collecting GmoreD data is not al#ays G etterD( and multiple methods tend to e more expensive to use than single methods !it together as a set o! methods in relation to the research question avoid spreading limited research resources too #idely and thinly in !inding ans#ers to the research question e conducted y researchers #ho have a su!!icient level o! competency in the various research methods involved.
+n addition- di!!erent research methods can e com ined in di!!erent #ays depending on the reason !or using multiple methods. ,or example- 1ammersley '899A( notes the !ollo#ing three #ays in #hich research methods can e com ined. 6riangulation E the use o! quantitative research to corro orate qualitative research !indings or vice versa. ,or example- to determine the success!ul outcome o! a distance learning programme the researcher may use assessment results and attendance patterns- together #ith intervie#s #ith learners to explore reasons !or success and)or !ailure. 2acilitation E #hen one research strategy is used to aid research that is using another research strategy. .e!ore conducting a survey- the researcher might conduct intervie#s to determine the est #ay o! asking the questions. 7omplementarity E #hen t#o research strategies are used to dovetail di!!erent aspects o! an investigationN !or instance- our example o! the evaluation o! the online course !or distance education policy makers. This included oth the evaluation o! the online component and the !ace"to"!ace #orkshop that supported it. Slightly di!!erent research approaches #ere used !or each- ut these complemented each other and oth #ere used to produce the !inal evaluation report o! the programme.
51
Triangulation
%re you !amiliar #ith the term triangulationC This is a term that has ecome increasingly commonplace in education and social science research methods so it is important !or us to explore it in a little more detail here. One o! the reasons !or using multiple methodsi! used appropriately as outlined a ove- is that this can help to achieve triangulation and so potentially enhance the rigour o! the study.
Triangulation
*riangulation is the use o a combination o di erent research methods to stud" the same thing in order to veri " the indings o each method#
Triangulation is a term adopted !rom surveying- #here surveyors take readings o! a geographical point !rom several angles to ensure and check the accuracy o! their conclusions. +t has een used in social science and educational research as a #ay o! increasing con!idence in the researcherDs !indings. 2ethodological triangulation re!ers to the use o! a com ination o! di!!erent research methods to study the same thing in order to veri!y the !indings o! each method. +n this #ay- the same programme- process- person etc is studied !rom more than one perspective as is highlighted in the !ollo#ing quotation$ 8Firstly the researcher needs to *e con$ident that the data generated are not sim&ly arte$acts o$ one s&eci$ic method o$ collection2 .his con$idence can only *e achieved 1hen di$$erent methods o$ data collection yield su*stantially the same results2 Furthermore; the more the methods contrast 1ith each other; the greater the researcher:s con$idence2 I$; $or e7am≤ the outcomes o$ a Duestionnaire survey corres&ond to those o$ an o*servational study o$ the same &henomena; the more the researcher 1ill *e con$ident a*out the $indings: ('ohen ? Manion "<<=!2 %ccording to Tashakkori and Teddlie '899:( it is use!ul to adopt a pragmatist research stance #here the research question is central and the most appropriate research methods are chosen to ans#er the speci!ic research question rather than !or reasons o! methodological pre!erences or theoretical #orldvie#s. 2ixed method approaches #ork !rom the assumption that all methods have particular strengths and all methods are !la#ed in some respects- thus using di!!erent methods allo#s the researcher to make maximum use o! the strengths o! each #hilst striving to overcome the #eaknesses o! each method. %long the same lines- and in the context o! evaluation speci!ically- 5enni!er 0reene ';>>>( notes that8>ecause &rogramme evaluation is ine7trica*ly intert1ined 1ith &olitics and values; and *ecause evaluators must navigate care$ully amid com&eting &olitical and value agendas; it is essential that evaluators have a diverse set o$ a&&roaches to hel& guide &ractice2:
52
.udgeting !or evaluations$ keeping expectations realistic

Research takes time and is costly. +t is essential that #e remain realistic a out #hat #e can and cannot do #hen planning an evaluation. ,or this reason- care!ul udgeting is an important part o! planning !or an evaluation study. The evaluation plan- o 4ectives- delivera les- and methods chosen must e realistically achieva le #ithin oth the time!rame o! the study and the availa le udget. +t is important at the outset to try to note any constraints or research limitations that might arise due to either time or !inancial limits. There are di!!erent #ays in #hich an evaluation might e !unded and this #ill a!!ect the udgeting required and the kind o! evaluation undertaken. ,or example- an in" house evaluation o! an ODL programme might e conducted y someone #orking #ithin the programme as an aspect o! his or her #ork. The udgeting !or such an evaluation #ould e quite di!!erent !rom one in #hich an external evaluator is contracted to do the evaluation and has to travel to another country !or this- as in the SO2DEL example. The kinds o! research methods to e used also a!!ect the udgeting. ,or example- conducting a !ocus group rather than many individual intervie#s can reduce costs ' ut #ill provide a di!!erent kind o! data(. Large"scale surveys- #hilst o!ten use!ul !or evaluations- are costly- unless linked to other distri ution channels o! an ODL programme- !or example including a survey #hen registration materials are set out. /hen udgeting !or an evaluation 'or any other research pro4ect( you need to think care!ully a out all the expenses that are likely to e incurred. This includes !inancial costs such as salaries- daily expenses- travel costs 'vehicle hire- petrol etc(- costs o! meetings and venues- costs o! materials- production o! research instrumentsreports- the costs o! gathering !eed ack !rom participants on the !inal results o! the evaluation- and the time allocation o! sta!! usually employed !or other tasks. Iery o!ten research udgets are ased on the num er o! days it is estimated that the research #ill take. There are many di!!erent #ays o! #orking out a udget. Hou may !ind that your institution has a standard #ay in #hich all udgets are calculated. % help!ul tip !or creating a udget is to map out your delivera les and activities in ta les as #e did a ove 'see the !eed ack to %ctivity ;(. ,or each o! the activities you can then #ork out ho# much time and other resources #ill e needed- to #hich a monetary value can then e attached. /hatever method you use- #hen putting together a udget you should al#ays e a#are that costs add up quickly. %lso- unanticipated costs 'such as di!!iculties #ith gaining access to the people #ho need to e intervie#ed- #hich means that research time ecomes longer( o!ten arise. This is particularly relevant to evaluation research since you #ill al#ays e #orking closely #ith an organisation or
53
organisation's(- programme's(- etc that have their o#n #ay o! !unctioning and processes o! doing things #hich might cause delays in your research process. Hour udget needs to make provision !or this.
Evaluation logistics
So !ar #e have looked at the theoretical- design and methodological issues that are important #hen planning an evaluation study. The previous section presented some practical in!ormation a out udgeting. +n this section #e very rie!ly note some o! the logistical issues and decisions that you might e required to make #hen planning and conducting an evaluation. O viously- each evaluation study #ill have speci!ic logistical requirements. The list elo# outlines some o! these. contact stakeholders E make contact #ith- and possi ly negotiate terms o! access to- the stakeholders in the evaluation pro4ect documentation E gain access to all the relevant documentation plan data collection E plan the most appropriate #ay o! collecting data !rom participants #ho- in a distance education setting- are dispersed. ,or example#ould it e etter to use a postal survey- send emails- or do telephone intervie#sC plan timing !or data collection E plan the most appropriate timing !or data collection- e.g. perhaps during a lock contact teaching session clari!y #ho #ill e responsi le !or collecting data and #hen. ,or example- you may request that a tutor send out a survey #ith a set o! assignment !eed ackto e returned #ith su mission o! the !ollo#ing assignment ensure research tools are availa le E e.g. questionnaires copied- and assign responsi ility !or doing this appoint #orkers E e.g.- researchers or !ield#orkers to assist #ith data gathering. Hou may need to prepare terms o! re!erence !or their #ork and dra# up a contract arrange transportation and accommodation !or researchers i! required communication E ensure good communication processes et#een all evaluation team mem ers and also #ith evaluation participants)stakeholders monitoring data collection E put in place systems !or monitoring and documenting data collection processes 'especially i! you have employed a team o! researchers !or the evaluation( other E add any other logistical issues you can think o! to this list.
54
,rom data gathering to analysis and reporting

Once you have collected your evaluation data you are hal! #ay there- ut still have t#o important research steps to complete. The !irst o! these is to analyse the data you have collected. Depending on your research design- you may or may not have egun analysing the data during the data collection processes. 1o# the data is analysed #ill also e dependent on the research methods you have used. ,or more in!ormation on ho# to analyse di!!erent kinds o! data you should have a look at core modules on qualitative and quantitative data collection and analysis. Once you have analysed your data- you egin to seriously !ocus on #riting your research report. Iery o!ten- parts o! the !inal report #ill have een #ritten as you have progressed #ith the evaluation and simply need to e dra#n together into a coherent #hole at this point. /e have noted e!ore the importance o! ensuring that the evaluation is designed and reported in a manner that meets the needs o! the target audience. Remem er- your results must re!lect the purpose o! the evaluation and should clearly ans#er the research question's( you de!ined #hen designing the study. ,or more in!ormation on report #riting you can re!er to the module on reporting on research and evaluation.
1ctivity " 8@ mins
2se Evaluation 1 and Evaluation 2 rom "our .esources (ile or this activit"# 2sing the contents $ages o these t!o evaluation exam$les as a guide& note do!n !hat "ou think is essential to include in an evaluation re$ort#
55
Bnit summary
+n this unit #e$ de!ined delivera les and activities o! an evaluation considered issues relevant #hen selection research methods !or evaluation outlined some udgetary considerations !or evaluation planning speci!ied possi le logistical issues that #ill need to e addressed outlined #hat must e considered #hen #riting an evaluation report.
'ro(ect task
Pro*ect tas+
>o ar in "our $roAect "ou have ans!ered the ollo!ing ;uestions' ?hat is the $ur$ose o this evaluation7 ?hat is m" research ;uestion+s,7 ?hat t"$e o evaluation !ill - be conducting7 ?ho is the target audience o m" evaluation7 ?ho "ou think it is im$ortant to consult in designing "our evaluation#
You !ill no! com$lete "our evaluation $lan in the $roAect task belo!# Dra!ing on all the $roAect tasks "ou have !orked through so ar "ou no! need to com$lete the ollo!ing ste$s in order to have a ull evaluation $lan that "ou can use to begin im$lementing "our evaluation or as a basis or raising research unding# - "ou liked the log rame a$$roach $resented in this unit& "ou might at this $oint like to begin !orking in this ormat# - "ou did not ind the log rame hel$ ul& then - suggest that "ou use the alternative a$$roach !e !orked through# 1 ?rite u$ the theoretical basis or "our $ro$osed evaluation# 2 De ine "our deliverables and activities# 3 /utline the research methods that "ou !ill be using# 4 0ook through the research tools included in "our reader to see i an" o these exam$les !ill be use ul or "our research# 5 ?ork out a $roAect budget#
56
6 0ist the logistical issues that "ou ex$ect to ace& and note ho! "ou !ould deal !ith them#
Re!erences
&ohen- L. and 2anion- L 899< Research methods in education- London$ 2ethuen 'pp ;=<( 0reene- 5. ;>>> GBnderstanding social programmes through evaluationD in Den3in- 6 and Lincoln- H. 'eds.( Fand*oo o$ Dualitative research ';nd edition(- London$ Sage 1ammersley- 2. 899A GThe relationship et#een qualitative and quantitative research$ paradigm loyalty versus methodological eclecticismD- in 5. Richardson 'ed.( Fand*oo o$ research methods $or &sychology and the social sciences - Leicester$ .ritish Psychological Society .ooks Ro inson- .. 89:< % study o$ student dro&(out on the .eaching o$ Reading Di&loma Programme- internal +nstitute !or Educational Technology document- 2ilton 7eynes$ B7 Open Bniversity Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!am Pu lications 'pp =>( Tashokkori- %. and Teddlie- &. 899: G2ixed methodology$ com ining qualitative and quantitative approachesD %&&lied Social Research Methods Series <A- London$ Sage /hyte- %. ;>>> %ssessing community telecentres+ guidelines $or researchers Otta#a$ +nternational Development Research &entre- at http$))#e .idrc.ca)en)ev" ;:=>8";>8"8"DOLTOP+&.html

There is a range o! #ays o! doing this activity. /e #ill look at one possi le set o! delivera les and activities elo#. Hou may have approached this slightly di!!erently #hich does not mean your ans#ers are incorrect. +! you have een #orking #ith Example ; throughout this hand ook #ill see that this approach also di!!ers slightly !rom that o! person #ho conducted the evaluation. /e have approached this task y de!ining three overall delivera les #ith speci!ic delivera les making up each one and then identi!ying activities !or each speci!ic delivera le #here appropriate. % ta le !ormat is help!ul #hen presenting activities. Bsing a ta le- one can also speci!y the deadline !or each o! the activities. /e have
57
not included deadlines in this example- ut have indicated #here you could do so. This then ecomes a use!ul tool !or you to keep track o! #here you are in your evaluation and #here you are going. Depending on the complexity o! the evaluation and your requirements- you can speci!y varying levels o! detail !or your research activities.
Delivera le 8 Evaluation o! the online course components$ a( Revie# o! course planning and management ( Revie# o! online materials
%ctions
Deadlines
Q collect planning and management documentation Q intervie# course planners)managers Q analyse data Q #rite up Q gain access to online materials Q develop materials revie# criteria Q revie# materials Q #rite up Q develop assessment criteria !or #e site and online system Q revie# #e site and online system Q 4oin discussion lists Q continual monitoring o! online discussion Q develop questionnaire !or participants Q administer questionnaire Q analyse !indings Q plan !ocus group discussion Q run !ocus group discussion Q analyse !ocus group data Q #rite up Q document support provided Q develop questionnaire !or participants Q administer questionnaire Q analyse !indings Q plan !ocus group Q run !ocus group Q analyse !ocus group data Q #rite up
c( %ssessment o! #e site and online system
d( Revie# o! support provided to participants
; Evaluation o! the !ace"to"!ace component a( %ssess #orkshop quality and use!ulness Q collect #orkshop planning documentation Q attend #orkshop Q develop rie! #orkshop participant evaluation !orm Q distri ute and collect evaluation !orms during #orkshop Q analyse data Q #rite up Q revie# #rite ups compiled during the evaluation Q produce dra!t report Q circulate report !or comments Q produce !inal evaluation report and su mit to &OL
= /rite up !inal evaluation report
58

%s you have seen earlier in this hand ook- the goal is more general than the purpose #hen used in the context o! a programmeDs log !rame. /e could say that the goal is the ultimate result o! the pro4ect #hile the purpose is the more speci!ic aim or o 4ective. The outputs descri e #hat can e expected- or #hat #ill e presented at the end o! the pro4ect in order to meet the stated goals and purpose. The activities descri e #hat #ill e done in order to e a le to achieve each o! the outputs that have een speci!ied.
,eed ack to %ctivity <

Hou should have come up #ith ta les that look something like the ones elo#. Please note that this is not a complete coverage o! the strengths and #eaknesses o! each research method- ut rather an illustration o! the use!ulness o! di!!erent methods !or di!!erent purposes. One strength and #eakness !or each method is presented in the context o! evaluation. Hou may have listed others. Hou might like to re!er to the core modules on qualitative and quantitative methods !or more detail.
Evaluation 8
8ethods Revie# o! documents and statistical records ,ocus group discussions &ualitative9 &uantitative Jualitative and)or quantitative depending on content Jualitative 4trengths Provides an overvie# o! programme ,acilitates participation and provides in!ormation that may not have een !oreseen during planning %llo#s in"depth exploration o! speci!ic issues #ith individual particants Wea+nesses Documents may present the ideal- not #hat actually happens Sometimes di!!icult to !ind a time to ring participants together E especially !or distance education programmes #here participants are dispersed Time consuming and costly i! sample is large
+ntervie#s
Jualitative and)or quantitative depending on the kinds o! questions asked. Bsually more qualitative Jualitative
&ase studies
Provides detail and depth o! analysis
Time consuming
59
Surveys
Juantitative
Provide statistical and aggregate data !or standardised questions
&an miss important in!ormation and details ecause questions are standardised and may not cover all issues o! importance ResearcherDs presence can in!luence the situation
Participant o servation
Jualitative
Provides a !irst hand vie# o! the phenomenon under study 'e. g. radio roadcast session(
Evaluation ;
8ethods Juestionnaires &ualitative9 &uantitative Juantitative 4trengths &ollect standard in!ormation !rom all participants %llo# the in"depth exploration o! speci!ic issues #ith individual particants Wea+nesses 6o opportunity to !ollo#"up responses !or more detail Time consuming and costly i! sample is large
+ntervie#s '!ormal and in!ormal(
Jualitative and)or quantitative depending on the kinds o! questions asked. +n!ormal #ill e qualitative Jualitative and)or quantitative depending on content Jualitative
%nalysis o! documents Participant o servation 'through email discussions(
Provides an overvie# o! programme Provides a !irst hand vie# o! the phenomenon under study 'e.g. mem er o! online discussion and #orkshop(
Documents may present the ideal- not #hat actually happens ResearcherDs presence can in!luence the situation under o servation
,eed ack to %ctivity @

Evaluation reports- like other research reports- #ill have di!!erent structures and #ill include di!!erent sections depending on the nature o! the research conducted and also the authorDs pre!erences and #riting style. 1o#ever- there are a !e# sections that should usually e included in an evaluation report. These include$ Executive summary +ntroduction to the evaluation and ackground o! the programme eing evaluated
60
Research design- o 4ectives- questions- methodology and limitations Results)!indings 'usually ordered into di!!erent sections( Discussion o! results)!indings &onclusion Re!erences
61
62
Juality assurance- quality control and other vie#s
B6+T
Bnit overvie#
Thus !ar #e have covered #hat programme evaluation is and #hy it is important in the context o! distance education- as #ell as #orked through the development o! an evaluation plan. /e have on several occasions noted that programme evaluation is used !or decision"making and !or ensuring quality. This unit expands on #hat you learnt in units one to three y clari!ying #hat #e mean y quality assurance and ho# programme evaluation is related to processes o! ensuring quality distance education. This unit thus !ocuses on ho# evaluations can e used in an applied #ayso ensuring the impact o! the !indings.
Bnit outcomes
/hen you have #orked through this unit- you should e a le to$ 8 de!ine quality assurance ; descri e the di!!erence et#een evaluation and quality assurance = explain ho# to prioritise evaluation !indings !or action < explain the role o! evaluation in supporting quality assurance.
/hat is quality assuranceC

+s quality assurance a term you are !amiliar #ithC Perhaps you have een part o! quality assurance processes at your institutionC +n this section- together- #e #ill esta lish a common #orking de!inition o! #hat #e mean y quality assurance. This is important ecause quality assurance is a term that is sometimes used to re!er to a very speci!ic and clearly de!ined process and at
63
other times is used quite loosely to include any e!!orts to#ards improvement o! a course- programme or institution.
1ctivity 1 <@ mins
1 6o! !ould "ou de ine ;ualit" assurance in the context o distance education7
2 Discuss !ith a colleague !hat the" think ;ualit" assurance means# Do "ou agree !ith "our colleague:s ideas7 <rie l" summarise "our discussion belo!#
.he $eed*ac to this activity is in the $ollo1ing te7t
Juality assurance and quality control and other vie#s

.e!ore #e present a de!inition o! quality assurance it is help!ul to consider #hat quality assurance is not. This may seem like a strange place to egin- ut ecause o! the di!!erent #ays in #hich the term quality assurance is used con!usion can easily result. +n addition- speci!ying #hat something is not is also use!ul !or clari!ying #hat it is. ,irst o! all- quality assurance and quality control are not the same thing. Juality control is a out checking a!ter #ork has een completed to see i! it meets speci!ied standards- and products that do not meet these standards are discarded. Juality assurance is a out anticipating pro lems that might occur in reaching speci!ied standards in order to avoid them.
Juality control
Fualit" control is about checking after !ork has been com$leted to see i it meets s$eci ied standards# Groducts that do not meet theses standards are discarded#
64
Limitations o! quality control

There are t#o main limitations to quality control as compared to quality assurance. The !irst is that quality control does not !ocus on error avoidance as a strategy- so it can result in #asted e!!ort and materials and re"#ork costs. The second is that the responsi ility !or assuring quality ecomes that o! a !e# inspectors rather than all those #ho are doing the #ork- !or example a team o! materials #riters. .ecause o! this- quality control is o!ten regarded as an in!erior alternative to quality assurance.
Juality assurance
Fualit" assurance is about anticipating $roblems that might occur in reaching s$eci ied standards in order to avoid them#
1o#ever quality control is a necessary ' ut not y itsel!- su!!icient( part o! a quality assurance system and procedures. Think o!- !or example- the control o! quality at given points during ODL materials production and the re4ection o! dra!t materials that !ail to meet the speci!ication provided as part o! a quality assurance system 'Ro inson- !orthcoming(. Secondly- quality assurance and total quality management 'TJ2( E another term you may have come across in your dealings #ith quality issues E are not the same thing. /hile quality assurance involves a !ocus on processes and procedures to ensure that things go as planned to#ards the desired outcometotal quality management is a roader approach to management- underpinned y a particular set o! values. TJ2 includes quality assurance- quality control and also monitoringso- quality assurance is one part o! total quality management.
Total quality management 'TJ2(

*F4 encom$asses ;ualit" assurances $lus a set o values about $eo$le:s res$onsibilities or ;ualit"#
2ugridge '8999( sums up these distinctions in his lecture on quality assurance that #as presented as a paper in the Indian ,ournal o$ O&en and Distance Education. This article is included as an optional reading in your reader. 1e states the !ollo#ing$ 8Quality assurance can sim&ly *e de$ined as measures ta en to avoid $aults; as the set o$ activities underta en to ensure that standards are; $irst; s&eci$ied clearly and second; reached consistently $or all activities in 1hich an institution engages2 .he second e7&ression; Duality control; is de$ined as measures ta en to correct $aults; as a retros&ective activity; eliminating or im&roving $aulty &roducts or services2 >oth o$ these overla&&ing $unctions; along 1ith the monitoring o$ the &rocedures themselves; together ma e u& Duality management2:
65
1ctivity 2 8@ mins -n this activit" "ou !ill consider one more de inition o ;ualit" assurance' -.uality assurance is -a set of activities t%at an organisation underta es to ensure t%at standards are s$ecified and reac%ed consistently for a $roduct or service/0 +.obinson 1994,# 6o! does this de inition com$are to !hat "ou and "our colleague said above7 ?rite do!n belo! a de inition that "ou !ould like to use as "our !orking de inition o ;ualit" assurance in this handbook and also in "our !ork environment#
.here is no $eed*ac to this activity
Di!!erentiating et#een evaluation and quality assurance

/e #ill no# see i! #e can de!ine the di!!erence et#een evaluation and quality assurance. +n Bnit 8 #e said that programme monitoring is a tool to e used #hen conducting a programme evaluation. This #as the conclusion #e came to- since programme monitoring is one #ay o! gathering the data that #ill e needed #hen conducting an evaluation. /e could apply the same logic #hen #e think a out the relationship et#een programme evaluation and quality assurance. Lets quickly remind ourselves o! ho# #e de!ined programme evaluation. /e summarised the various de!initions o! evaluation using a simple quotation !rom a ook on evaluation y Ru in. %ccording to Ru in '899@(- evaluation means$ 8to understand the value o$ something in order to do things *etter2: %re you starting to see the links to quality assurance hereC
66
/e noted a ove that .ernadette Ro inson '899<( de!ined quality assurance as$$ 8a set o$ activities that an organisation underta es to ensure that standards are s&eci$ied and reached consistently $or a &roduct or service2: Do you agree #ith me that programme evaluation #ould e one o! these activitiesC +! you do- then you have understood the relationship et#een programme evaluation and quality assurance. +n order to con!irm ho# evaluation and quality assurance are related #e #ill look at a commonly used diagram ',igure ;( that illustrates the strategic and operational processes that are common components o! institutional management systems. Each o! these processes underlies e!!orts to ensure quality #ithin an organisation.
Figure ) 'ore com&onents o$ institutional management systems that su&&ort Duality
,igure ; sho#s !our key management processes essential !or e!!ective programmesincluding distance education programmes. %s #e sa# in ,igure 8- planning in!orms implementation. The implementation o! the programme needs to e evaluated against clearly de!ined outcomes- to see #hether it is success!ul. This evaluation then provides !eed ack to e used !or revie#- #hich in!orms decisions a out ho# the programme or institution can e improved. The diagram a ove there!ore emphasises the importance o! e!!ective evaluation research and the crucial role it plays in a quality assurance process.
67
Did you also notice that the arro#s linking each o! the processes go in oth !or#ards and ack#ards directionsC This sho#s us that quality assurance processes are unlikely to take place in a linear !ashion ut that it might e necessary to move ack to planning as lessons are learnt in the course o! an evaluation process !or example. .ased on #hat you have learnt a out the di!!erent kinds and purposes o! evaluation in the previous units- you #ill kno# that the manner in #hich evaluation relates to these management and quality assurance processes also depends on the kind o! evaluation eing conducted. To summarise #e orro# the #ords o! Rathore M Schuemer '899:($ 8Evaluation Gis notH an end in itsel$Ievaluation should rather &rovide in$ormation 1hich can *e used to maintain or im&rove the Duality o$ &roducts and &rocesses2: This quote de!ines !or us the ideal- that evaluation provides practical in!ormation that can e used !or improvements in a programme or institution. 1o#ever- one o! the key messages that emerge !rom Rathore M SchuemerDs ook is the di!!iculty o! getting institutions to use evaluation !indings. /e #ill pick up on this issue in the !ollo#ing section.
Evaluation and quality assurance E sources

>ee .ekkedal& *# 1998 HFualit" assessment and evaluation' basic $hiloso$hies& conce$ts and $ractices at @3-& @or!a"H& in 6# .athore and .# >chuemer +eds#&, )valuation conce$ts and $ractice in selected distance education institutions& 6agen' (ern2niversitat& at htt$'EE!!!# ernuni= hagen#deEI-((Erekked#htm +*he ull .athore and >chuemer book is also available at at htt$'EE!!!# ernuni= hagen#deEI-((E108cont#htm, .ekkedal ex$lores in more detail +and $rovides some additional $ers$ectives rom @or!a", on the issues o evaluation and ;ualit" in a distance education environment
&onnecting evaluation to programme planning and quality assurance processes

So !ar most o! #hat #e have discussed has een largely a stract and theoretical. /e have looked at ho# evaluation !its #ith other programme or institutional management systems and have also sho#n that programme evaluation is one important activity !or ensuring quality. 5ust to highlight once more the important role o! evaluation in ensuring quality o! distance education provision- #e consider the #ords o! 2ichael 2oore '8999($ 8One o$ the $e1 generaliJations that can *e made a*out any distance education &rogramme # 1hatever the communications media used and the content level # is that a good monitoring and evaluation system is li ely to lead to a success$ul &rogram; and a &oor system is li ely to lead to $ailure2:
68
The aim o! this section is to help you to ring these concepts to li!e. Lets egin #ith an example that highlights ho# evaluation can e used in the esta lishment o! a quality assurance system and also as an important planning tool !or a distance education institution.
7ase 4tudy )sing evaluation to inform uality assurance planning: an e*ample *echnikon >outhern % rica +*>%, is a large higher education distance education $rovider based in >outh % rica# -n res$onse to external $olic" demands and internal institutional trans ormation needs& in 2000 the senior management o *>% began !ork on a ;ualit" assurance revie!& $lanning and re=design c"cle at the institution# *he core o this $rocess !as a large=scale evaluation o a re$resentative range o course materials and some s$eci ic courses too# *he evaluation $roAect !as called the Evaluation &a$acity 1uilding Pro2ect (3S(4S(5DE 2##2!/ *>% !orked together !ith the >outh % rica -nstitute or Distance )ducation +>%-D), on this evaluation# *he aims o this evaluation $roAect !ere to' Develo$ cost e ective ;ualit" course revie! and design $rocesses# (acilitate an organic intervention that inAects trans ormative energies and $rocesses into *>%# )nsure that residual ca$acit" remains at the end o the $roAect#
-n meeting these aims& the evaluation included the ollo!ing our ke" activities' 1 4a$$ing C the collation o in ormation about instructional o erings +-/s,& i#e# it ma$$ed the current state o a airs# 2 (irst=level evaluation o materials and courses C getting a brie but in ormed sense o the general trends and overall ;ualit" o an -/ b" examining the course!are& and teaching and learning strategies 3 -n=de$th evaluation o materials and courses C adding de$th and ;uestioning insights gained through the irst=level evaluation 4 .edevelo$ment C tr"ing to $ut recommendations into $ractice b" re!orking identi ied units o the courses evaluated# *his $roAect !as run over a t!o="ear $eriod# ?hile di iculties !ere ex$erienced C some resisted the $rogramme and others tried to $artici$ate but ound the time re;uired challenging C the ollo!ing achievements and deliverables resulted#
69
Develo$ment of customised 3S( evaluation instruments and course design su$$ort materials/ *his included a range o ma$$ing and evaluation instruments to use or evaluating course materials& a *>% curriculum redesign model to su$$ort sta in redesigning their $rogrammes to meet the ;ualit" criteria& and a course design su$$ort $ackage# Re$orts of evaluations" $ro2ect activities and $rocesses/ )valuation re$orts o 80 di erent -/s !ere $re$ared# *hese !ere used to conduct a trend anal"sis# %n in=de$th evaluation re$ort !as $roduced based on ten courses# (inall"& our consolidated re$orts documenting the $rocess !ere also $roduced# 1uilding ca$acity/ %t the end o the 2="ear $eriod& ten sta across divisions had been trained as evaluators !ith s$eci ic skills in using the evaluation instruments develo$ed# (ive o this ten !ere additionall" trained as acilitators to acilitate this evaluation ca$acit" building $rocess more !idel" across the institution# (dditional unintended outcomes/ /ne o the main incidental outcomes o the $roAect !as that it hel$ed to bring various initiatives& such as access management and academic develo$ment across the institution together& and hel$ed to develo$ acce$tance o !orking across disci$linar" boundaries#
The role o! evaluation in programme planning and quality assurance systems

This example highlights the role that evaluation can play in oth programme planning and the development o! quality assurance systems. 1o#ever- it is unusual in that the evaluation #as speci!ically designed to support the quality assurance processes. /hat a out evaluations that are done independently o! such a processC This is pro a ly the kind o! situation you #ill !ind yoursel! in as you egin your evaluation pro4ect. 1o# can you try to ensure that your results #ill e use!ulC There are a range o! #ays in #hich evaluation results can e used. These include- ut are not limited to$ improve management and organisation o! the programme help in decision"making y providing an indication o! actions needed to improve the programme- #here training might e needed- #here additional research #ould e help!ul improve planning y providing in!ormation a out past per!ormance and the results achieved provide in!ormation that might e use!ul !or ringing more resources into the programme. ,or example an evaluation might sho# that a distance education programme is understa!!ed and so not meeting its o 4ectives. This in!ormation
70
might support the e!!orts o! the programme to lo mem er
y !or an additional sta!!
improve accounta ility y sho#ing that resources have een e!!ectively used.
1ctivity # 8@ mins ?hat actors do "ou think could in luence the success ul im$act o an evaluation stud" on the $rogramme being evaluated7
+mplementing recommendations
Let us look at a !e# o! the recommendations made in Evaluation ". Hou might like to quickly read through these e!ore continuing.
,eading 0 ,esources !ile
*he recommendations can be ound on $ages 27 and 28 o )valuation 1# .ecommendation 2 states 9a strateg" needs to be devised to ex$and >/4D)0 urther to target t!o distinct grou$s:# *hese grou$s are children o school age !ho live in remote rural areas& and male "outh !ho have missed out on education# *r" to think in $ractical terms !hat it means to ex$and this $rogramme that makes use o $rint& radio and ace=to= ace teaching to remote rural areas# Do "ou see that this is ;uite something to achieve7 - "ou look care ull" at ho! this recommendation is $resented ho!ever& "ou !ill see that the maAor obstacles are also identi ied# *hese include inaccessible communities& lack o trans$ort and the lack o ;uali ied teachers# *his sho!s the reader that the researcher is a!are o the obstacles and challenges o the $rogramme# 6ave a look at >ection 3 o these recommendations& Programme $ointers# Do "ou see an" recommendations in this list that "ou think are unrealistic7 - think
71
that most o these recommendations look achievable# /ne $oint !orth noting though& this is a long list o 9to=dos: or those running the $rogramme# -t is also use ul to $rioritise the recommendations that "ou make as this !ill make it easier or "our indings to be im$lemented#
1ere is a possi le checklist that you can apply to an evaluation to provide an indication o! the impact is it likely to have on the programme evaluated. The more questions to #hich you can ans#er yes- the more likely that your evaluation !indings #ill have an impact on the programme eing evaluated.
&hecklist !or revie#ing the likely impact o! an evaluation

+s the evaluation #ritten !or a target audience at the programme level #ho could use the !indingsC %re the evaluation questions relevant to the programmeC %re the !indings clearly presentedC %re clear and practical recommendations madeC %re these recommendations prioritisedC /ere there processes o! consultation #ith programme sta!! during the evaluation to assess relevanceC +s the report presented in a reada le #ayC ',or example- a reasona le lengthsimple language- etc.( Does the report contain an executive summaryC %re there strategies to make the results o! the evaluation availa le to a range o! stakeholdersC %re there possi ilities !or !ollo#"up to discuss and clari!y the !indings and recommendationsC '% #orkshop or a meeting !or example.(
72
Bnit summary
To summarise- in this unit #e$ de!ined quality assurance in the context o! distance education have seen that evaluation is an important component o! quality assurance processes considered an example o! ho# programme evaluation plays a role in institutional planning have developed a checklist !or assessing the likely impact our evaluation #ill have on the programme eing evaluated.
Pro*ect tas+
You are no! read" to return to "our $roAect task# -n the irst three units "ou $re$ared an evaluation $lan# *his inal $roAect task is designed to hel$ "ou to link this evaluation $lan to the enhancement o ;ualit" o the $rogramme "ou are evaluating and to assess the im$act that "our evaluation might have on the $rogramme# 1 Gre$are a brie overvie! o the current ;ualit" assurance s"stems in $lace at the institutionE$rogramme "ou have chosen# 2 -n !hat !a"s do "ou think "our evaluation stud" could bene it this s"stem7 3 You might ind that the institution "ou have chosen does not have a ;ualit" assurance s"stem# - so& then consider ho! conducting an evaluation could be used to begin setting in $lace a ;ualit" assurance s"stem# 4 ?hen "ou have conducted "our evaluation& use the checklist $rovided above to revie! "our evaluation# +*his is likel" to be an activit" that "ou come back to a ter some time,#
73
&onclusion
Hou have no# #orked through the hand ook on Programme Evaluation and its Role in Quality %ssurance. /e hope that you have !ound this a !ruit!ul exercise. +n the introduction #e outlined the six learning o 4ectives !or this hand ook- and + copy them elo#. 1ave a look through each and i! you !eel unclear on any o! themthen you might like to return to the relevant section o! the hand ook to re!resh your memory. 8 De!ine the purpose o! an evaluation process and prioritise the key questions that the evaluation #ill need to ans#er. ; Select appropriate methods !or gathering the necessary data !or a good quality and practically use!ul evaluation o! a distance education programme. = +denti!y the characteristics o! e!!ective evaluation processes and !actors that are most likely to impede success!ul implementation o! an evaluation. < De!ine the reporting requirements o! an e!!ective evaluation process. @ Descri e the relationships et#een programme evaluation and quality assurance. A Plan an evaluation study. +! you !eel happy that you have achieved each o! the a ove outcomes- then you are ready to go out and conduct your evaluation. 0ood luckF
Re!erences
2oore- 2. 8999 GEditorial$ 2onitoring and evaluationD .he %merican ,ournal o$ Distance Education 8=- ;$8"@ 2ugridge- +. 8999 GJuality assurance in open and distance educationD Indian ,ournal o$ O&en Learning :- =$ ;8=";;> Rathore- 1. and Schuemer- R. 'eds.( 899: GEvaluation concepts and practice in selected distance education institutionsD- 6IFF &a&er 8>:- 1agen$ &entral +nstitute !or Distance Education Research- ,ernBniversitat- at http$))###.!ernuni" hagen.de)K+,,)pre!8>:.htm Ro inson- .. 899< G%ssuring quality in open and distance learningD in ,. 'Lock#ood ed.( Materials &roduction in o&en and distance learning - London$ Paul &hapman 'p 8:?( Ro inson- .. '!orthcoming( %chieving Duality in o&en and distance learning$ Paris$ B6ES&O Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers - Ox!ord$ Ox!am
74
Pu lications 'pp 8=( TS%)S%+DE. ;>>; Evaluation ca&acity *uilding &ro9ect &art "+ summary re&ort ,lorida$ Technikon Southern %!rica

.eing realistic a out ho# to make use o! your evaluation !indings is an important skill to learn. %s researchers- it is sometimes easy to assume that the recommendations made on the asis o! a research processes are easier to implement than they in !act are. Trying to keep the recommendations #e make reasona le and practical is one important #ay o! enhancing the chance that decision"makers and programme implementers #ill use the !indings. % collection o! suggestions or recommendations that seem impossi le to achieve are likely to send those #ho are struggling #ith the daily realities o! a programme runningF
75
76
Permissions
The pu lishers- editors and authors o! this hand ook are very grate!ul to the !ollo#ing copyright holders and authors !or permission to include extracts !rom their #ork. /e are particularly inde ted to those pu lishers and individuals #ho supported the pro4ect y #aiving copyright !ees. /e have made every e!!ort to track do#n copyright holders. +! you consider #e have used material #hich is your copyright #ithout ackno#ledgement- please accept out apologies and let &OL kno# so the correction can e put into the next edition.
Bnit 8
2ichael .rophy o! the %!rican Educational Trust '###.a!ricaeducationaltrust.mcmail.com( and the +nternational Research ,oundation !or Open Learning '###.col.org)ir!ol( !or use o! ,entiman- %. ;>>= SOMDEL+ Somali Distance Education Literacy Programme (Macallin a Raddiya! - a report prepared !or %!rica Educational Trust '%ET( y the +nternational Research ,oundation !or Open Learning '+R,OL(- &am ridge$ +R,OL &ommon#ealth o! Learning '###.col.org( and Pro!essor &isco 2agagula !or permission to reproduce- on a non"commercial asis- 2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policy(ma ers in Southern %$rica Iancouver$ &ommon#ealth o! Learning The +ndian 5ournal o! Open Learning and the +ndira 0andhi 6ational Open Bniversity !or permission to reproduce 2ugridge- +. 8999 GJuality assurance in open and distance learningD Indian ,ournal o$ O&en Learning :- =$ ;8=";;> Pro!essor 2ichael 2oore- Editor- The %merican 5ournal o! Distance Education !or the reproduction o! 2oore- 2. 8999 GEditorial E 2onitoring and evaluationD .he %merican ,ournal o$ Distance Education 8=- ;$ 8"@ Bli .ernath o! the &entral +nstitute !or Distance Education Research !or permission to reproduce and link to Rathore- 1. and Schuemer- R. 'eds.( 899: GEvaluation concepts and practice in selected distance education institutionsD- 6IFF &a&er 8>:1agen$ &entral +nstitute !or Distance Education Research- ,ernBniversitat- at http$))###.!ernuni"hagen.de)K+,,)pre!8>:.htm The +nternational Development Research &entre '+DR&( !or permission to use and link to /hyte- %. ;>>> %ssessing community telecentres+ guidelines $or researchers Otta#a$ +nternational Development Research &entre- at http$))#e .idrc.ca)en)ev" ;:=>8";>8"8"DOLTOP+&.html The Ox!ord Bniversity Press- South %!rica !or permission to use a diagram 'p =<=( !rom .a ie- E. and 2outon- 5. ;>>8 .he &ractice o$ social research- &ape To#n$ Ox!ord Bniversity Press in South %!rica
77
Pro!essor .ernadette Ro inson- Bniversity o! 6ottingham !or permission to use Ro inson- .. ;>>8 % &ractical guide to evaluation- a #orkshop hand ook produced !or the B6ES&O Learning !or Li!e Pro4ect in Blaan aatar$ 2ongolia 7ogan Page !or permission to quote !rom p 8;@ o! .irley- 2. and 2orel- 6. 899: % &ractical guide to academic research- London$ 7ogan Page The Open Bniversity Press !or permission to use ; quotations !rom Dolley- 5. 899< Planning; monitoring and evaluating learning &rogrammes - .uckingham$ Open Bniversity Press The 2inistry !or ,oreign %!!airs- ,inland !or permission to use and link to 2inistry !or ,oreign %!!airs. 899: Cuidelines $or &rogramme design; monitoring and evaluation 1elsinki$ Department !or +nternational Development- at http$))glo al.!inland.!i)4ulkaisut)yleis)pdme)index.html Sage Pu lications to use a quote !rom Patton- 2. 899> Qualitative evaluation and research methods ';nd edition( London$ Sage Pu lications Pitman Pu lishing !or permission to quote !rom Thorpe- 2. 899= GEvaluating open and distance learningD- 1arlo#$ Longman- cited in D. Ro#ntree 899: G%ssessing the quality o! materials" ased teaching and learningD in %. Tait 'ed.( O&en Learning =$;
Bnit ;
The +ndian 5ournal o! Open Learning and Pro!essor 7. 7achroo !or permission to reproduce 7achroo- 7. 8999 GDistance education programmes at college and university levels in 5ammu region$ an evaluative studyD Indian ,ournal o$ O&en Learning :- =$ ;9="=>@ Sage Pu lications !or permission to use a quote !rom 7umar- R. 8999 Research methodology+ a ste&(*y(ste& guide $or *eginners - London$ Sage Sage Pu lications !or permission to use a quote !rom pp <: !rom 2iles- 2. and 1u ermann- %. 899< Qualitative data analysis- London$ Sage Pro!essor .ernadette Ro inson- 6ottingham Bniversity !or permission to reproduce a ta le !rom Ro inson- .. '!orthcoming( %chieving Duality in o&en and distance learning$ Paris$ B6ES&O
Bnit =
&ommon#ealth o! Learning '###.col.org( and Pro!essor &isco 2agagula !or permission to reproduce- on a non"commercial asis- 2agagula- &. ;>>; Evaluation re&ort o$ the 'ourse $or Distance Education Policy(ma ers in Southern %$rica Iancouver$ &ommon#ealth o! Learning
78
Permissions
.O6D B7 !or permission to reproduce the link http$))###. ond.org.uk)pu s)ip#.htm to their Logical ,rame#ork %nalysis 0uidance 6otes <. .O6D is a net#orker !or over ;:> B7" ased voluntary organisations #orking in international development and development education. Sage Pu lications !or permission to use a quotation !rom 0reene- 5. ;>>> GBnderstanding social programmes through evaluationD in Den3in- 6 and Lincoln- H. 'eds.( Fand*oo o$ Dualitative research ';nd edition(- London$ Sage Taylor and ,rancis !or permission to use a quote !rom pp ;=< !rom &ohen- L. and 2anion- L 899< Research methods in education- London$ 2ethuen
Bnit <
The +ndian 5ournal o! Open Learning and the +ndira 0andhi 6ational Open Bniversity !or permission to reproduce 2ugridge- +. 8999 GJuality assurance in open and distance learningD Indian ,ournal o$ O&en Learning :- =$ ;8=";;> Taylor and ,rancis !or permission to quote t#o paragraphs !rom p 8:? !rom Ro inson- .. 899< G%ssuring quality in open and distance learningD in ,. 'Lock#ood ed.( Materials &roduction in o&en and distance learning - London$ Paul &hapman Ox!am !or a quotation !rom pp => Ru in- ,. 899@ % *asic guide to evaluation $or develo&ment 1or ers- Ox!ord$ Ox!am Pu lications. Details o! all Ox!amDs ooks at ###.ox!am.org.uk)pu lications Bli .ernath o! the &entral +nstitute !or Distance Education Research !or permission to reproduce and link to Rathore- 1. and Schuemer- R. 'eds.( 899: GEvaluation concepts and practice in selected distance education institutionsD- 6IFF &a&er 8>:1agen$ &entral +nstitute !or Distance Education Research- ,ernBniversitat- at http$))###.!ernuni"hagen.de)K+,,)pre!8>:.htm Torstein Rekkedal and 67+- 6or#ay !or permission to link to and use as a reading Rekkedal- T. 899: GJuality assessment and evaluation$ asic philosophies- concepts and practices at 67+- 6or#ayD- in 1. Rathore and R. Schuemer 'eds.-( Evaluation conce&ts and &ractice in selected distance education institutions - 1agen$ ,ernBniversitat- at http$))###.!ernuni"hagen.de)K+,,)rekked.htm 'The !ull Rathore and Schuemer ook is also availa le at at http$))###.!ernuni" hagen.de)K+,,)8>:cont.htm( Pro!essor 2ichael 2oore- Editor- The %merican 5ournal o! Distance Education !or the reproduction o! 2oore- 2. 8999 GEditorial E 2onitoring and evaluationD .he %merican ,ournal o$ Distance Education 8=- ;$ 8"@ Technikon Southern %!rican and the South %!rican +nstitute !or Distance Education !or permission to use TS%)S%+DE. ;>>; Evaluation ca&acity *uilding &ro9ect &art "+
79
summary re&ort- ,lorida$ Technikon Southern %!rica
80

PREST Module Program

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PREST Module Program

Uploaded by

Copyright:

Available Formats

PREST

Programme evaluation and its role in quality assurance

The PREST core team

%dditional mem ers

Programme evaluation and its role in quality assurance

1and ook overvie#

1and ook organisation

1o# to use the materials

The pro4ect tasks

use a checklist !or revie#ing the impact o! your evaluation.

The 1and ook as a #ork ook

,ood !or thought

These resources are included as an appendix at the end o! the pu lication.

Status o! the readings

/hat is programme evaluation and #hy is it importantC

,eading 0 ,esources !ile

/hat is programme evaluationC

1 6o! !ould "ou de ine $rogramme evaluation7

.he $eed*ac to this activity is in the $ollo1ing te7t

Programme evaluation E some de!initions

1ctivity 2 <@ mins

.he $eed*ac to this activity is at the end o$ the unit

Evaluation as a !orm o! research

Differences between research and evaluation

The relationship et#een programme monitoring and evaluation

,ood !or thought

Programme monitoring in ODL

/hy should distance education practitioners conduct evaluationsC

2 6o! is this evaluation use ul or the distance education $rogramme in ;uestion7

.he $eed*ac to this activity is at the end o$ the unit

2eed3ac+ to selected activities

,eed ack to %ctivity ;

,eed ack to %ctivity =

Evaluation conceptualisation$ de!ining the purpose o! an evaluation

Purposes and types o! evaluation

.he $eed*ac to this activity is at the end o$ the unit

The main purposes o! evaluations

.alancing !ormative and summative evaluation

1ctivity 2 => mins

.he $eed*ac to this activity is at the end o$ the unit

+denti!ying an evaluationDs target audience

,ood !or thought

,ormulating key questions

Type 8$ Evaluation o! need

Type ;$ Evaluation o! process

Type =$ Evaluation o! outcome

Type <$ Evaluation o! e!!iciency

&ost e!!iciency E!!iciency

Source$ Ro*inson ($orthcoming!

1ctivity # => mins

2 @ote do!n !hat research ;uestions are being ans!ered#

.he $eed*ac to this activity is at the end o$ the unit

&onsultation E a key to success

2eed3ac+ to selected activities

,eed ack to %ctivity 8

,eed ack to %ctivity ;

8 Somali distance education literacy programme

; &ourse !or distance education policy makers in Southern %!rica

,eed ack to %ctivity = Evaluation Example One$ SO2DEL

Evaluation Example T#o$ &ourse !or Distance Education Policy 2akers

Designing and implementing an e!!ective evaluation

Evaluation planning- !orming a conceptual !rame#ork

1ctivity 1 => mins

.here is no $eed*ac to this activity