Selection and use of indicators

for monitoring and evalution

 Introduction
• Project:
– putting at inputs at disposal to generate
capacity
– generated capacity results in outputs (more
production, more education, …)
– outputs generate effects at the level the
targeted population (higher income, better
nutrition, better health, …)
– effects generate impacts at social and
economic level within a community
 Monitoring and evaluation
• To verify whether project goals have been
reached
– Monitoring is a method to measure the progresss
of a project and a tool for the daily management.
Answers on the question whether inputs are used
as expected
– Evaluation is a method to measure the effects and
impact of a project and looks whehter project
reaches targets and whether it does that efficiently
• Importance of both depends on the kind of
project:
– for standard applications monitoring is most
important as we know already effects and impact
from other applications
– new procedures demand also a good evaluation
 Monitoring
• Following progress of a project
– Physical and financial progress
– delivery of inputs to end-users
– possibly reactions of end users (e.g. adoption
rate)
• Monitoring system needs to be simple and
precise with a minimum of parameters.
Emphasis on detection of possible
derogations of original plan
 Evaluation
• Needs to determine whether objectives in
terms of outputs, effects and impacts have
or wll be reached:
– evaluation of results
– lessons for future similar project
• Evaluation emphasizes the effects in time
and has therefore higher data needs than
monitoring (extra data collection after
project)
• Difficulty of showing causality of effects
with project efforts
 The notion of indicators
• Indicators produces quantitative
information for helping communication,
negotiation, analysis and decision
• Evaluation = indicators measuring the
success of an intervention
• Ad-hoc indicators for specific goals,
monitoring indicators to follow-up the
planning
 Relevance, efficiency,
effectiveness and performance
• Relevance indicators: relates the programme
objectives to the needs: e.g. number of trainee
places
• Effectiveness indicators: relate what is obtained
to what was expected: tow values of same
output (e.g. effective trainees/expected trainees)
• Efficiency indicators: relate what is obtained to
the resources mobilised (output/input: trainees/
1000 euro input)
• Performance indicators: relates effectiveness
and efficiency of interventions
 Performance

Society,
Outreach
Needs, Economy,
Problems, Environment
issues
Outcomes

Programme
Objectives Inputs Outputs

Effectiveness

Evaluation Relevance

Efficency
INDICATORS
 Measurement of indicators
• Direct indicators
– Have a direct link with the result, output or impact
(e.g. higher yields)
• Indirect indicators
– Have no direct relation with results, but we suppose a
link among the indirect indicator and that is hard to
determine direct indicator (e.g. number of people in
better jobs as a result of an education programme)
• early pointers (monitoring)
• quantitative and qualitative indicators
 Good indicators
• An indicator must be:
– Relevant
– Simple
– Informative: different from data !
– Quantitative: different from a descriptor
(qualitative measurement) or a score (=
relative measurement)
– Indicate something (either in space, either in
time)
 Elementary, derived and
compound indicators
• Elementary indicator : basic information that can
be used to calculate other indicators (e.g.
income, unemployment, temperature, ..)
• Derived indicator: ratio between two elementary
indicator: e.g. unemployment rate
• Compound indicator: weighted sum of derived
indicators (e.g. quality of life index,
environmental impact index, …)
• Derived and compound indicators can be broken
down (e.g. unemployment rate per category, …)
 Specific and generic indicators
• Specific indicator: used for a specific intervention
and not used for comparison (e.g. number of
trainees of certain group)
• Generic indicator: an indicator used for
comparison between similar interventions (e.g.
number of trainees per 100 inhabitants)
• Key indicators: indicators allowing comparison
among different kinds of interventions (e.g. GNP,
cost per job created, …)
 Programme and context
indicators
• Context indicators apply to an entire territory,
population or category of population. It says
something about total impact/effect of the global
changes taking place (intervention + external
changes)
• Programme indicator: only concern the part or
category being reached by the intervention
• Difficult to use: e.g. overall employment rate
when there is an intervention to create jobs, but
at the same time increase in population
 Level of indicators
• Resource indicators: provide information on the
level of resources used: e.g. annual budget,
propostion of budget used, ..
• Output indicators: measures the product of
activities: km of roads build, capacity of
purification plant, number of trainees, …
• Result indicators: advantages and
disadvantages of a programme or intervention
for addressees or recipients (time saved by use
of road, number of jobs per 100 trainees, …)
• Impact indicators: consequences of
interventions for all concerned people
 Impact indicators
• First category: grouping of consequences for all
addressees or recepients (e.g. increase of
traffic, jobs created, …)
• Second category : effects for non-addressed or
non-recepients: e.g. total new jobs in the area
(includes multiplier effects)
• Impacts are not easily measurable and need to
be measured from time tot time e.g. through
surveys about opinions, changes taken place, ..:
difference between gross and net effects
(because of substitution and displacement
effects)
 Performance indicators
• Are forcing people from the planning stage to
think about critical factors
• Performance accounts may help in allocation
of inputs
• allow to indicate deteriorations in progress
• measuring the reaching of goals not only
what input use
• may help in communication about results
• may be used for benchmarking of projects
(standards)
• Element of quality management
 Usefullness of a system of
indicators
Indicator system
Produces desired
information
Reflects objectives and
makes them more
Information is legible
Correctly interpreted

Information is
used Information is
dessiminated
Better
decisions

Improved Citizens can form better

performance an opinion
 Sources of information
Evaluation

Secondary data Primary data

Monitoring Series of
Literature
system statistics

Surveys,
Research,
Management Statistical Case studies
former
documents institutes evaluations

Reality
Sustainability indicators
 Farm employment
Use of pesticides 100 Farm productivity
Farmland birds 90 Farm income
80
Nature grasslands 70 Farmers health
60
Tree rows 50 Direct energy use
40
Risk of soil erosion 30 Indirect energy use Farm economics
20 Resources
10
Nutrients
Heavy metals 0 Water use
Manure handling
Soil
Carbon content Waste water treatment Nature

pH N use

Type of animal house P use

Manure processing N balance

Residual soil N Concentrate use
Injection of manure