IMPACT ASSESSMENT & PROJECT MANAGEMENT CYCLE (PMC)

IMPORTANT NOTE:

Most of this section will be familiar to DFID staff as it is a paraphrase of

information contained in Office Instructions, with added commentary on the
practical application of those instructions in the context of Enterprise
Development interventions. The purpose of summarising the main points
here is to enable NGO's, other partner organisations and consultants working
with DFID to understand DFID's progammme management procedures, and
to place Impact Assessment in the context of these procedures. Further
information can be obtained from the Office Instructions which are now
reproduced on the DFID website, and on CD (address given on website)

CONTENTS:

Summary
1 Introduction
2 Project Header Sheet
3 Project Concept Note
4 Stakeholder Analysis
5 Logical Framework
6 Project Submission: Project Memorandum
7 Monitoring requirements
8 Reporting requirements
9 Review procedures
10 Sample Logical Framework: TEEM (Traidcraft Exchange), Malawi

SUMMARY:

Most of this section will be familiar to DFID staff as it is a paraphrase of

information contained in Office Instructions, with added commentary on the
practical application of those instructions in the context of Enterprise
Development interventions. The purpose of summarising the main points
here is to enable NGO's, other partner organisations and consultants working
with DFID to understand DFID's progammme management procedures, and
to place Impact Assessment in the context of these procedures.

The paper goes through the PMC in chronological order, looking at how
impact assessment should be built into each stage of the cycle, beginning
with appraisal and completion of the Project Header Sheet, through
development of the logical framework, project memorandum, to Project
Completion Report.

This text has been prepared by Tertia Gavin and Caroline Pinder
1 INTRODUCTION:

Enterprise development is about a series of different interventions, involving a

wide range of possible partners, over varying timescales, but all with the
common underlying goal of contributing to the elimination of poverty.
Interventions may range from macro to micro level, with different implications
for enterprise development impact assessment at each level.

Working at different levels may involve DFID in partnerships with different

types of organisations, ranging from multilaterals to other bilateral donors,
international NGOs, banks, professional organisations and local NGOs.
Whatever the nature of the partnership, there are some common underlying
principles for success.

The key factor is to develop a good working relationship with the partner
organisations, based on trust and good communication. Different methods will
be required depending on the nature of the partner, for example, some
partners may feel comfortable using logframes as a tool for exploring and
agreeing on important programme issues. Other partners, may find this
formalised approach difficult to work with, and may prefer methods commonly
used within their own organisations. Good communication, involving a
listening role by DFID, is essential from the beginning to ensure that partners
do not feel that an outside agenda is being imposed on them.

Channels of communication will again vary depending on the nature of the

partner and the type of intervention. It is vital that ways are found to ensure
honest and open communication that will allow an atmosphere of lesson
learning and problem solving. Frequently during interventions, unexpected
outcomes arise (both positive and negative) and it is important that partners
feel comfortable raising these issues with DFID as and when they become
evident, so that their implications for the programme can be considered in
time for review.
Table 1: Levels of Intervention

Key areas for Implications for

Level intervention Goals Impact Assessment
Macro
Economic policy General legal and DFID usually only one
Government / reform; trade policy; fiscal framework player among several
administration regulatory of conditions multi-laterals and bi-
environment, eg: oriented to laterals. Has an
CDF, PRSPs1 enterprise and influencing role, so IA
competition more difficult to
control

Meso
Organisational and Targeted policy DFID sometimes one
Business Institutional implementation, of several donors;
associations; development, eg eg job creation in other times sole or
trade unions; strengthening of labour intensive major donor. Need to
trade/industrial representative sectors, income take partners'
sectors;quangos agencies as part of safety nets and agendas into
national & inter- support for civil re-training consideration and to
national NGOs society programmes develop coherent IA
framework with
partners

Micro
Starting and Job and self- DFID often sole or
SMEs; local expanding individual employment main donor, and able
NGOs/service SMEs; financial and creation; income to set agenda for IA.
providers non-financial support generation Eg: micro finance,
services BDS institutions

1
CDF = Comprehensive Development Framework; PRSP = Poverty Reduction Strategy Paper. For
more information refer to the World Bank website.
FIGURE 1. PROGRAMME CYCLE:
AREAS WHERE IMPACT ASSESSMENT NEEDS CONSIDERATION
IDENTIFICATION Look at EDIAIS for lessons from other programmes

*PROJECT HEADER SHEET (PIMS, POM, PAM)

• Decision on markings give indication of which
TSP checklists to consider(eg gender, poverty
etc)
*CONCEPT NOTE
• Indicate consideration of level and methodology
of IA

DESIGN *STAKEHOLDER ANALYSIS

• Which stakeholders will be involved in proposed
IA? Are these the appropriate stakeholders?
What methodologies are best for securing their
involvement?

*LOGFRAME
• Agree purpose of IA and indicators
• Agree methodologies and mix to fit
• Agree on baseline data required, & how this will
be collected
• Agree frequency/timing of IA

APPRAISAL *PROJECT MEMORANDUM (& Technical Annexes)

• Ensure IA reflected throughout document
including budget
• Collate multi-disciplinary tech annexes (in
particular, economic, social and environmental);
negotiate and agree IA strategy for project with
multi-disciplinary Advisers/team members

APPROVAL *MONITORING ANNEX

• Detail proposals as agreed at earlier stages, ie
purpose, methods, frequency etc

IMPLEMENTATION *MONITORING
AND MONITORING . DFID staff monitoring visits at least once a year
. Scoring for PRISM

*REPORTING
• internally written reports by project staff required
at least once a year for projects over £250 000
reporting against logframe indicators, activities
and budget
• End-of-project report will include IA findings
REVIEW / IMPACT
ASSESSMENT *REVIEWING
• Mid-term review should include a mid-term IA
• Output-to-purpose review held 6mths before end
of project; should include full-term IA
• If extension likely review IA strategy & include in
new PM(Project Memorandum)
The outputs, or what happens on the ground, is clearly the most important
part of programme cycle management. To ensure that the desired outputs
occur, thorough preparation is essential, and this will be reflected in the
project documentation. The following notes are intended as a guide for project
preparation and subsequent monitoring and reviewing. It is not intended to
suggest that documentation and paperwork are the most important aspects of
programme management!

2 PROJECT HEADER SHEET

This requires the Enterprise Development Adviser to classify ('score' or 'mark')

the proposed intervention in three ways2:

2.1 Policy Information Marker System (PIMS)

PIMS is an indicator of initial intent but scope exists within the system to
change markers during the course of project implementation. The markers
provide a policy relevant framework against which project performance can be
judged. However, the main value of PIMs in in providing a measure of the
extent to which projects are being targeted to pursue key policy areas, and it
is therefore most relevant at the pipeline and commitment stages of projects.

Most enterprise development interventions will fall within the PIMS framework
for "Policies and actions which promote sustainable livelihoods".

2.2 Policy Objective Marker (POM)

There are however, three categories of PIMS objectives of which "Policies

and actions which promote sustainable livelihoods", as mentioned above at
2.1, is likely to be the one into which will fall most enterprise development
interventions. The other two objectives are "Better education, health and
opportunities for poor people" and "Protection and better management of the
natural and physical environment".

Classification under just one of these headings is not easy. Sustainable

poverty elimination requires a dynamic balance between the promotion of
each of these three objectives, and projects often contribute to more than one
objective. Each of the three broad objectives is therefore broken down
further, into POMs.

Within the broad category "Policies and actions which promote sustainable
livelihoods" the objectives (POMs) most likely to apply to enterprise
development programmes are:

Sound social and economic policies

Direct assistance to the private sector
Access of poor people to land, resources and markets

2
This section is drawn from "PIMS" produced by DFID Statistics Dept, March 1998
Business partnership
Training and skills development

2.3 Poverty Aim Marker (PAM)

The PAM system comprises three categories for the way in which poverty
elimination is assessed:

• enabling actions which under-pin pro-poor economic growth and the

broader policies and context for poverty reduction and elimination, and will
lead to social, environmental and/or economic benefits for poor people
• inclusive actions (eg sectoral development programmes) which aim to
benefit broad-based population groups, including poor people, but also
address issues of equity and barriers to participation or access of poor
people
• actions focused predominantly on the rights, interests and needs of poor
people

In many cases projects will fall into more than one PAM category, but
Advisers are required to indicate the predominant means by which individual
projects target poverty elimination.

3 PROJECT CONCEPT NOTE

The requirements for the Project Concept Note (PCN) are set out in Office
Instructions. Many of the important points relating to consideration of impact
assessment in the PCN are discussed below at Section 5, the Logical
Framework, which must be submitted with the PCN.

Critically at this early stage, some consideration is required of baseline data

against which subsequent impact can be measured. Frequently, this process
is neglected which has significant negative repercussions for future impact
assessments, as is evident in several of the case studies (eg MDA in South
Africa, AGENT in Zimbabwe – forthcoming link). There is an increasing trend
with 'process' type projects to have a series of phases, which often start with a
feasibility study (for example, Training for Enterprise and Exports in Malawi
(TEEM), a Traidcraft project). This often provides an opportunity to collect
information about baseline conditions, and make an early exploration of
indicators with future stakeholders.
4 STAKEHOLDER ANALYSIS

A separate Technical Note describes how to carry out a Stakeholder Analysis

(SHA). To briefly summarise, an SHA should be carried out at the design
stage of every project and programme with a view to:

• Identifying key stakeholders, their interests and potential degree of

influence on the project
• Identifying areas of potential conflicting interests, with a view to minimising
these and/or looking at ways for ensuring everyone has something to 'win'
from working together
• Drawing out key risks and assumptions which will need to be taken into
account in the project design
• Identifying possible areas of Impact Assessment: likely indicators based
on stakeholder's perceptions and expectations of impact
5 LOGICAL FRAMEWORK

Box 1:

From: “CARE INTERNATIONAL SOUTH WEST AFRICA REGIONAL

MANAGEMENT UNIT (SWARMU) :Developing Design of Monitoring &
Evaluation (DM&E) Capacities “A major weakness is the ability of project staff to
utilise their logframe for designing a coherent and integrated, overall information
system, where a manageable and limited number of feasible information activities are
planned, which together will ensure that effective effect and impact level monitoring
will occur. It is typical for projects to end up collecting too much rather than too
little information. Frequently though, much of this information is not relevant to
monitoring the results and impacts for which the project is accountable, and that
which is, is not collected sufficiently reliably or regularly. By restricting the
number, but improving the quality and reliability of their major information
gathering activities, projects will much improve their information systems.”

The logical framework is a key component of the approval process for DFID
funding. It can also be a helpful tool in working through issues between
stakeholders - whatever the nature of intervention – from a micro-level project
to a less tangible, macro-level intervention aimed at, for example, influencing
international trade relations or government economic policy. Although the
logframe can sometimes be used to shape discussions, some partners may
find the logframe approach difficult to work with, and other methods will need
to be used to explore the issues with these stakeholders.

Development of the Project Framework should involve primary beneficiaries

(ie the poor people who will be the ultimate users of the services provided by
the intervention) as well the intermediary partner organisations responsible for
delivering those services. The Project Cycle Management (PCM) approach
emphasises the importance of participation of stakeholders in a "meeting of
minds" to agree the project purpose. At the start those stakeholders who
should be involved in the development of a framework must be identified,
while recognising that, in turn, these stakeholders may identify others who
should contribute to the process. The PCM approach can be used most easily
between donor and recipient (partner) institutions but the broad elements of
the approach should also be used with all other interested parties. (See the
technical notes on Stakeholder Analysis and Participatory Methods for more
information about how to involve stakeholders in a participatory way)

In designing the Project Framework the principal stakeholders need to be

involved in:

• agreeing objectives, outputs and activities

• determining and weighting risks (assumptions)
• identifying indicators for measuring progress.

Different stakeholders may well select different indicators. Minimally there

should be some DFID-driven indicators, some from the main implementing
agency and some from the primary stakeholders: in other words the range of
indicators should reflect the various critical expectations of all the key
stakeholders. The experience of several NGOs has shown that participatory
processes using people's own indicators can generate quantifiable and time-
bound indicators which relate to outputs relatively easily.

Box 2:

From: Central Region Infrastructure Maintenance Programme (CRIMP

Malawi).
A baseline study was carried out early in 2000. Using CARE's Household
Livelihood Security (HLS) framework, the study aimed to accurately identify
the percentage of households in different wealth/livelihood categories, and to
develop an information system that is responsive to the needs of CARE,
donors, community and partners, and can be used by communities to monitor
change over time.

The study was conducted in 36 villages across the five Traditional Authorities
over a period of three weeks by a team of 24 researchers. The researchers
were oriented in the research protocol over a one-week period proceeded by
pre-testing of the tools in the community before the start of data collection. A
total of 709 households were interviewed over the five TAs. In addition social
maps across the TAs incorporated basic data on 2,032 households. As a
result of the pre-test, a standard set of indicators was developed for the
differentiation of households into livelihood categories. These are:

• Land
• Food Security
• Assets
• Income Sources
• Education/skills/capabilities

See case study for illustration of how these indicators are used.

Although indicators of purpose and goal are more difficult to develop in a

participatory fashion, it is possible; and in fact debates with other stakeholders
on this issue have often started to show up differences in objectives not only
between beneficiary stakeholders but between them and the implementing
agency, leading to a changed project purpose. Clearly it is better for these
differences to be aired at the start so that accommodation can be made for
the full range of partners' views, rather than allow them to be a potential
source of conflict at a later stage.

The extra time spent in developing key indicators of achievement with

stakeholders is an important investment in the success of the project, leading
to a sense of greater shared commitment among the various stakeholders.
Local intermediary organisations should be identified who would be
responsible to the implementing agency for much of the monitoring, with DFID
providing support and technical assistance as necessary.
The project framework should be seen by all parties, as a working document
that can be adapted as necessary during the life of the intervention. Many
partner organisations do not recognise the flexibility of the logframe, and
regard it more as a restrictive device against which their performance will be
judged. It is important that a clear understanding is reached that the logframe
is an active part of management and can be changed as appropriate
throughout the intervention to reflect changing circumstances. Even with the
most carefully designed intervention there is always the possibility of
unexpected outcomes both positive and negative. These need to be identified
and recognised throughout an intervention, and unexpected positive
outcomes reinforced, and methods sought to reduce negative impacts.

Table 2: Summary of keypoints for consideration of Impact Assessment

in the Logical Framework

OVI MoV Assumptions

Goal and purpose: • Are target • Who is • Do these include
• Are these beneficiaries responsible for scope for
shared and mutually agreed collecting negative and
understood by all upon? information? positive
stakeholders? • What is purpose • Is there a clear unexpected
of IA? timetable outcomes
• What are • Has necessary
indicators? baseline data
• Have these been been collected?
selected in • Has monitoring
consultation with and IA been
stakeholders? budgeted for?
• Will they provide • How will info
necessary feed back into
information for project?
purpose of IA?
• Are they
consistent with
other partners
requirements?

Outputs
• What are the • Have indicators • Have local • What are the
expected been agreed with intermediaries risks if any,
impacts? stakeholders? been identified to which might
• Do they include a carry out IA? impact on the
mix of DFID, • Has a timetable achievement of
partner- for IA been outputs?
organisation and agreed? • Does
stakeholder • Will this provide achievement of
indicators? the necessary impacts/outputs
information? depend on any
• How will info particular instit-
feed back into utional or other
project? factors?
Activities
• Do these include • Is collection of • Have sufficient • Do inputs and
necessary inputs baseline data an funds been activities depend
for IA? identified allowed for on any particular
 activity? baseline study, factors?
• Has setting up MIS,
implementation ongoing
of MIS been collection and
included? review of data?
• What
arrangements
are made for
ongoing
collection &
review of data
6 PROJECT SUBMISSION

The required format for project submissions is set out in Office Instructions
Volume II D6 (forthcoming link to DFID site with OIs). This format is required
for all types of DFID funded intervention (other than those being jointly funded
through Civil Society Department). Consideration of IA is required in a number
of different sections of the Project Submission (also often referred to as
Project Memorandum). It should not be thought of as merely a subsection of
the document under the heading monitoring; rather it should be integrated
throughout.

Here is a checklist of IA requirements within the text of the submission:

6.1 Summary and recommendation

• What will be the outputs (ie expected impacts)?

• Who will benefit? Has a comprehensive stakeholder analysis been
done?
• Project budget: Does the budget provide scope for effective IA
activities?

6.2 Background

• What has been done before by DFID and by others? Check

EDIAIS database for relevant IAs within DFID/elsewhere
• What else needs to be done prior to implementation? eg
- What baseline data needs to be collected to assist with
meaningful IA during project?
- Who is responsible for baseline data collection? Is there a
workplan setting out tasks, personnel and deadlines?

6.3 Project Approach

• What are the expected benefits? What impact is

the project hoping to achieve?
• Who are the key stakeholders and how has
participation been handled?
• What are the options, if any, for review and
change during implementation?
- Are there appropriate indicators?
- Are there realistic means of measuring these
indicators?
- Is there a clear plan setting out who is
responsible, how
information should be collected, who should
compile and
analyse it, and how often it should be
reported?
- Where possible are these built into
organisations' MIS?
 - Is training required for partner staff to
ensure effective use of
MIS?
- Who is responsible for collecting information
and reporting on
them?
- Are there plans for regular stakeholder
meetings for lesson
learning and forward planning?

6.4 Evaluation

• What lessons of best practice are being applied? Consult EDIAIS

database for relevant projects
• How will the indicators for Goal, Purpose and Output levels provide
the basis on which to judge success?
• Do indicators reflect careful consideration of mix of IA
methodologies needed? (refer to Core Text for more information)
• Will indicators assist internal learning throughout the life of the
project as well as being a means of judging project success at it's
completion?

6.5 Implementation

i) Management arrangements
• What will be the role of key stakeholders?
• How will their participation be handled?
• How will they be involved in IA?

ii) Timing
• Is collection of baseline data allowed for before project start
• How often during the project will external IA exercises be
required?

iii) Inputs
• Are additional inputs associated with IA specified?
• Is facilitation required for regular stakeholder meetings?
• Is training required for MIS systems?
• Are regular monitoring visits set out?
• Have funds been budgeted for training, facilitation and other
needs identified? Has sufficient time been allowed for staff
to do monitoring? Or will there be a special appointee to
carry out M&E work?

iv) Monitoring
• Who gathers what information and when, for
chosen indicators, at each level of the
logical framework?
 • What management arrangements will ensure that
action is taken on the results of the
monitoring?
• Who reports to whom and when?
• What arrangements will be made for Output to
Purpose Reviews?

v) Technical Annexes
• MIS may need to be covered in some technical annexes (eg
micro-finance projects usually have a technical annex
detailing the types of ratios and other financial information
that should be build into MIS from the start)

Box 3:

Central Region Infrastructure Maintenance Programme (CRIMP), Malawi

Integration of Monitoring Work into Staff Time

The project has a Monitoring, Research, Evaluation and Documentation Co-

ordinator who is responsible for the collection of data, analysis and reporting
on a quarterly basis. Sometimes the project hires part time research
assistants to help with the collection of livelihood monitoring data. This is
funded from the Monitoring and Evaluation budget.

The Monitoring, Research, Evaluation and Documentation Co-ordinator has

the primary responsibility of ensuring that appropriate information is collected
systematically and used to inform programme activities. Staff implementing
the project activities contribute to the monitoring process through reports from
which qualitative information is drawn. The qualitative information from the
reports, to a large extent helps to explain the quantitative information collected
during the scheduled (quarterly) monitoring exercises. The qualitative
information is stored separately to the quantitative, but in a way that is
accessible to any user – on the server hard drive, and filed in hard copy form.

The findings from the monitoring exercises continue to guide and inform the
implementation processes adopted by the project. The information collected is
also be used to inform the design of the second phase of the project.

6.6 Monitoring Annex

This should set out:

• Purpose of IA
• Mix of IA methodologies, and how they will be integrated
• What the partner-organisation is expected to produce as part of
internal MIS
• Capacity building required to ensure effective MIS/monitoring
 • Necessary external IA inputs, (who? how often?)
• Management processes to ensure IA findings feedback promptly
into project
• Dissemination of findings beyond project

Box 4: EXAMPLE OF DETAILED MONITORING ANNEX

Annex F of the TEEM (Training for Enterprise and Export Malawi) Project
Memorandum lays out the broad areas of impact which will be investigated
and disaggregated through the life of the project. It creates an innovative set
of research questions in pursuing the impact at the following levels:

• impact of business development services offered by the Business

Service Centre
• impact of Trading Company on SME production
• impact of the project as a whole on poverty.

Each of these levels require careful methodologies not only to ensure impact
is correctly measured and attributed, but also that valuable lessons are fed
back into the project to enable adaptation and innovation. The methodologies
should also ensure that valuable information is integrated from TEEM’s
internal MIS, and that these are correctly designed to provide complementary
data.. The implementation of these studies should be externally tendered and
managed by DFID in conjunction with TX and the local TEEM institutions.
7 MONITORING

This is undertaken by DFID staff (including field managers) together with

project management and focuses on progress to achieving outputs and
purpose. It is often viewed by stakeholders as a policing exercise and can
cause suspicion and distrust. With sensitive handling, however, monitoring
should be an opportunity to assess project progress against various levels,
and more importantly give the opportunity of face to face contact with key
stakeholders which can be an important opportunity for lesson learning.

Monitoring visits can help assist a good relationship between DFID staff and
local stakeholders. With the improvement of channels such as e-mail, ongoing
communication should be encouraged to help identify and deal with problems
and queries as they arise during project implementation. This can help to
foster a sense of partnership between colleagues, as opposed to an “us-and-
them” mentality. This is more likely to be the case if implenting agency staff
have been involved from the start so that they do not feel that they are
undergoing job performance assessments.

The Project Memorandum must include the plans for

monitoring. This will usually include visits by DFID
staff to project sites and regular meetings with project
management, as well as arrangements for monitoring the
project's progress through perusal of the reports
provided by project staff. It is preferable that
monitoring visits are undertaken with the recipient's
project management staff. Also, as far as possible, the
same DFID staff or consultants should undertake
monitoring for each project, to ensure continuity and to
allow the building of relationships between project
personnel and DFID.

Monitoring visits are not only time consuming for DFID

staff, but also require a substantial investment of time
by project management and often a variety of
stakeholders. It is therefore particularly important that
when planning monitoring visits, while having to fit them
into the schedules of DFID staff, consideration is also
shown to the project staff to allow them the necessary
time to plan for the visit. It also underlines the
importance of ensuring that the monitoring visit is seen
as a useful exercise for all participants, rather than as
an onerous chore imposed by donors. This can be
facilitated by taking time in advance to find out whether
there are any particular issues that stakeholders wish to
raise and by ensuring that sufficient time is allowed for
a thorough debriefing and discussion of issues before
leaving the project. It is also important to ensure that
monitoring reports are issued swiftly with clear
indications of what actions are required by whom.
Stakeholders, including project staff can feel let down
after monitoring visits, if expectations are raised and
no action subsequently taken.
Conventional periodic review missions often involve a large DFID inter-
disciplinary team and representatives from a range of recipient institutions
joined, in co-financed projects, by representatives from other donors.
Meetings often concentrate on senior bureaucrats in urban centres, which
means that the viewpoints of beneficiaries, representatives of local-level
institutions and of field workers are not taken into account. Because such
review missions tend to be large, hierarchical and formal, even field visits fail
to establish any real communication with junior officials and primary
stakeholders.

Strategies to overcome these problems include:

• designing review missions as a series of shorter, more varied, visits, rather

than an all-inclusive single visit: one component of such a review would be a
preliminary participatory field visit, using participatory appraisal techniques;

• avoiding large and, for some stakeholders, intimidating meetings where

lower level staff or women beneficiaries may be afraid to speak out. Smaller,
more informal discussions, perhaps on the lines of a focus group, may help to
elucidate more "truthful" opinions.
8 REPORTING

As set out in Office Instructions, these are usually completed by non-DFID

staff (eg contractors, consultants, project staff) and compare progress of
project implementation with what was planned at the project output and
activity level. The report should also cover any unexpected outcomes that
have arisen as a result of the intervention, as well as progress towards the
planned outcomes.

The project management must have a work plan agreed with

DFID and the recipient which is:

• developed from the work breakdown activity chart in the

Project
Memorandum;
• regularly reported against and updated.

The work plan content should:

• relate to the Logical Framework

• include activities needed to gather data against which
to assess progress in relation to agreed
targets/indicators in the logical framework, and
reporting frequency.

The format will require reporting against the logical

framework and the work plans to review progress against
activities and outputs (and purpose where appropriate).
Report formats should be thought through at project
design stage, when considering the purpose of IA, the
information required and who is responsible for gathering
it. (Office Instructions Vol II G1 Annex 1 provides a
recommended format).

If projects are co-funded with other donors a common

format and timetable of reporting should be agreed if at
all possible, to minimise the administration for the
partner organisation. Preparing multiple reports on the
same project for different donors with different timing
and format requirements can contribute to reporting being
seen as more of a burden and less of a helpful learning
process.

Reports should :

• give full reasons if progress is not proceeding as

planned
• identify any unexpected outcomes (positive or
negative),
• check that the assumptions in the logical framework
remain valid,
• highlight recommended action points by DFID, recipient
or others
• identify responsibility for follow up with a target
date for action.
Reports should be sent to the DFID project officer, who
is responsible for arranging circulation within DFID to
all those who should see the report, and for ensuring
relevant follow up action is taken.
9 REVIEWING

Most projects' monitoring plans will include arrangements

for an Output to Purpose Review (OPR) and an MTR (Mid-
term review.) The aim of this is to take a strategic
overview, particularly assessing progress towards
achieving the project Purpose and Goal. OPRs should be
undertaken jointly with the recipient, and should involve
key project stakeholders. The purpose is to take a
strategic overview, particularly assessing progress
towards achievement of the project's objectives (Purpose
and Goal) and checking on whether the assumptions remain
valid, particularly in the light of experience of any
unexpected and /or negative outcomes. In the past, mid-
term reviews may have, in part, covered this ground but
they sometimes lacked a clear statement of the project's
impact and overall achievements against objectives.

OPRs may also identify important lessons for DFID's

future project management systems, both for the project
under review and for new projects. The review will also
provide an initial assessment of progress against the
PIMS markers (see back to section 2 on project header
sheet). Meaningful reviews, involving primary
stakeholders, require long term planning and may include
a number of stages for information gathering and
assessment.
The main DFID project team should participate in the
final stage of the OPR so that the widest inter-
disciplinary view and assessment can be established. The
same principles and caveats apply in planning and
implementing review missions as applied above with
monitoring visits.

Where external consultants are appointed to undertake the

OPR, it is particularly important that they understand
the DFID approach, and that they have the capacity to
undertake follow on work which may arise from the OPR.
Guidance on commissioning IA work is given in the Core
Text. Frequently, OPRs may recommend some extension to
the intervention to ensure that the project purpose is
achieved. Successful OPRs will require the Review Team to
establish a good rapport with the participants, and this
should be built on in subsequent work by use of the same
external personnel. Frequent changes of consultant for
different stages of work, not only takes up time as each
person has to learn about the organisation from scratch,
but also undermines the sense of partnership and
continuity with the recipient organisation.
Box 5: Example of Staff Participation in Impact Assessment:

MDA Impact Assessment, South Africa Feb 2000 - extract from Case Study
(For full report click here)
As important as the findings themselves was the involvement of staff at all the
Centres, with a view to developing their capacity and understanding of clients’
needs in order to inform future strategic planning across the Centres and
organisation. The assessment was therefore a learning experience in terms
of MDA’s institutional capacity as well as its programme outputs. This meant
the research was perhaps not as refined or objective as it might have been if it
had been carried out by external, professional or academic evaluators; on the
other hand, the learning gained by the trainers is likely to extend their
empathy towards clients, and enable them to offer more appropriate advice
and solutions to their business problems in the future.

Research methodology:
The role of the Consultant for this assignment was to facilitate the design and
implementation of a monitoring and evaluation system which would enable an
impact assessment to be carried out. As far as was practical in the time
available, a process approach was used, beginning with a workshop for the
training staff based at MDA’s Head Office in Johannesburg, to identify
indicators and design the structure of the system. At the workshop, staff also
designed the questionnaire which would be the main assessment tool, and
put in place the logistics for carrying out the survey. Head Office training staff
then became team leaders of the trainers based at each of the local Centres,
taking on responsibility for co-ordinating the field research to be undertaken at
the Centre allocated to them. This included making arrangements for
interviewing clients, supervising the trainers, compiling and analysing the
results, and writing up the report to be presented to a workshop a few days
following completion of the field research. The Consultant then summarised
the contributions from the teams, and compiled the final report.

A mix of different methods was used to collect

information:
• Desk research
• Survey
• Focus groups
• Case studies
• Validation

Lack of baseline data

The principal problem with this first impact assessment was the absence of
any baseline information against which to compare the organisation’s and
clients’ progress at the end of the project. The report recommended the
adoption of an MIS which will enable easy collection and analysis of such
information for future reports. Future impact assessment timings were also
recommended, and the suggested timings are scheduled to coincide with
other management reporting requirements such as reporting to the board.
This is intended to streamline all information processing and to minimise extra
work.
Dissemination of findings and Follow on work

The outcomes of the IA were disseminated to all staff through workshops

where all MDA Units and District Centres (DCs) were represented. The
representatives from the DCs reported back in their areas.

It was a logical progression to follow the IA with an organisational review. The

same consultant was contracted for this. This exercise was effectively an
internal IA look at the impact of MDA units on the DC’s. The organizational
review showed anomalies in the structure of MDA and that some services
could better be delivered through re-positioning them within the structure of
MDA.

Following the IA and organisational review a national strategic planning

workshop took place in April 2000 where the findings of the IA were discussed
in the context of institutional development. At the end of MDA’s process of
strategic planning with all staff, centre managers and centre staff
representatives, certain key themes and strategic priorities emerged. These
were presented to stakeholders at MDA’s AGM.

This is a good illustration of how IA findings were used as a dynamic tool for
influencing and changing the organisation from within, rather than being a
paper exercise where the recommendations remain unacted upon in a dusty
report.
 10: Sample Logical Framework: TEEM (Training for Enterprise and
Export in Malawi), Traidcraft Exchange, Malawi (for more information
please refer to the Case Study notes and annexes)
NARRATIVE
GOAL
The alleviation of poverty,
through equitable trade, in
Malawi
PURPOSE
To improve the livelihoods
of poor people in Malawi
through the expansion of
market-led local
businesses and exports
OUTPUTS
1. Business Service Centre
sustainably delivering
market-led business
counselling to SMEs.
2. Sustainable Trading
Company operating and
exporting in Malawi, with
growing annual turnover.
3. Products developed in
conjunction with the
business counselling
service, being sold through
the Trading Company, and
linkages made between
larger and smaller
producers
OBJECTIVELY VERIFIABLE MEANS OF RISKS AND
INDICATORS (OVIs) VERIFICATION (MOVs) ASSUMPTIONS
Reduction by 10,000 of people 1. Baseline study That monitoring
living under one US dollar per 2. External impact and evaluation
day in Malawi assessment systems can
3. Income, Consumption measure change
and Expenditure (ICE) in income levels
survey for participants
4. Project documents &
records
5. Trading Company
transactions
1 2,000 wage earning 1. External impact That equitable
positions created or assessment data & trade with and
sustained by EOP in the project client records within Malawi is
manufacturing and 2. External impact possible, given
smallholder sectors assessment data & current level of
2 Change in Employment project client records poverty
Practices (wage levels, use 3. Trading Company
of child labour & security) in transactions and
the majority participating household ICE survey
businesses*
3 20% increase in
participants’ cash sales
1.Business Service Centre cost Business Service Centre Business Service
recovery 40% local costs by quarterly and annual Centre client
end of project based upon a reports to board base are willing
clear fee structure to grow from
2.Business Service Centre informal to
client base 360 over project life formal sector
3.6 Business Service Centre
business counsellors in place
and working to the satisfaction
of their client base
1 Trading Company break-even Trading Company That a suitable
by Year 2 of project (Annex E) quarterly and annual client base exists
2 Sales follow path outlined in reports to board for the Business
Annexes P & A, or as indicated Service Centre &
in the business plan during the Trading
Phase 1. Company
1. 80% of the Business Service MTR & EOP Reports Political,
Centre clients state an Project Monitoring economic and
improvement in business Records social stability is
turnover by the mid term Training records maintained
review (MTR), & EOP SME Records and ICE
2. 80% of clients have received survey
product related training and Impact survey
counselling by MTR & EOP.
NARRATIVE
4. A contribution to the
development of a BDS
industry for SMEs in
Malawi.
5. Fair trade / ethical
criteria and standards
operating in all
intermediary organisations
within the project
6. Monitoring, evaluation
and research system
which feeds back lessons /
insights to improve project
performance
OBJECTIVELY VERIFIABLE MEANS OF RISKS AND
INDICATORS (OVIs) VERIFICATION (MOVs) ASSUMPTIONS
3. 50% of businesses supply
larger businesses by EOP
1. 2 business counsellors 1 & 2. Project and training All students
completed training and Records successfully
employed elsewhere as 3. Minutes of annual completed
training / business consultations training course
counsellors 4. Produced
2. 6 business counsellors recommendations by
completed training MTR & EOP
successfully and employed
within the Business Service
Centre
3. Project active in
disseminating lessons with
other sector players and
MCCI annually
4. Project drafts
recommendations by MTR &
EOP on changes to
regulatory environment and
BDS in Malawi
1. Ethical criteria targeting 1. Ethical standards report
poverty (based on CDC / IFAT Inception Review
guidelines)*, and which meet MTR & EOP Reports
the needs of stakeholders, Survey of participants
operating in 80% of businesses ICE
by EOP.
2. Gender strategy established 2. Gender Strategy Report
and operating effectively by end Inception Review
of inception period, with MTR & EOP Reports
measured impact by EoP. Survey of participants
ICE
3. Children Labour strategy 3. Child Labour Strategy
operational by end of inception Report
period, with national consensus Inception Review
with stakeholder interests by MTR & EOP Reports
EoP Survey of participants
ICE
4. HIV/AIDS policy adopted in 4. HIV/AIDS Strategy
80% of participating Paper
businesses, and is beneficial to Inception Review
employees by end PY2. Survey of participants
ICE survey
1. Internal M&E system 1. Inception Review
established by inception review M&E system manuals
2. External Poverty impact 2. IA Inception Report
monitoring system established Completed External
by end of PY1 Baseline Survey
IA progress and final
reports
3. Action Research programme 3. Action Research Plan
established by end of inception Strategy Papers
NARRATIVE
ACTIVITIES
Milestones achieved
during the inception period
of 18 months
Business Service Centre
set-up and operational by
Month 6 (Annex E)
Trading Company set up
and operational by Month
9 (Annex E)
Contractual link between
the Business Service
Centre and Trading
Company established by
Months 9 - 12 (Annex E)
TX works together with the
Business Service Centre to
deliver a market-led
business counselling
service.
Impact Assessment
tendered and contracted
by DFID, and agreed with
TX
Action Research
programme deigned and
initiated
OBJECTIVELY VERIFIABLE MEANS OF RISKS AND
INDICATORS (OVIs) VERIFICATION (MOVs) ASSUMPTIONS
period and co-ordinated using PSC Papers
Project Steering Committee
(PSC)
PROJECT BUDGET of: TX Visit reports It is possible to
£2,657,645 (Annex G for Quarterly TX reports to create a Trading
breakdown) DFID & PSC Company in
See attached action plan Quarterly financial reports Malawi within the
for detailed activities (Annex E) to boards given time frame
Inception Review in Month18 Annual, MTR & EOP
DFID M&E visits evaluations
Progress, as a minimum, to TX Visit reports 1. That the
meet objectives set out in Quarterly TX reports to project can
Annex E (Figure 5 ). DFID & PSC attract and retain
Quarterly financial reports staff of a suitable
Annual dissemination to boards calibre
workshops operational by Annual, MTR & EOP 2. That the
Month 12, and thereafter evaluations Business Service
annually. Centre clients
are willing to
contribute to the
cost of inputs
Progress, as a minimum, to TX Visit reports That an ethical
meet objectives set out in Quarterly TX reports to Trading
Annex E (Figure 6). DFID & PSC Company can
Quarterly financial reports be set up in
to boards Malawi
Annual, MTR & EOP
evaluations
MoU or legal contract by month MoU That a workable
12 between two parts of TEEM. Quarterly TX reports to relationship
DFID & PSC develops
Quarterly financial reports between the
to boards Business
Annual, MTR & EOP Service Centre
evaluations & the Trading
Company
Products design based on client Client Surveys That the GoM is
surveys and fee collection Fee collection Manuals willing & able to
systems established which Quarterly TX reports to create an SME
maximise sustainability by end DFID & PSC friendly
of inception phase. Quarterly financial reports environment
to boards
Annual, MTR & EOP
evaluations
Tenders accepted by Month 5, Tender documents That contractors
and initial baseline work started Inception and baseline are available to
by Month 7. reports carry out the
envisaged work
Activities laid out in Annex G Action Research Strategy That action
(Table 8) and Annex E planned PSC minutes research is
and time-tabled by Month 18. Tender documents appropriate to
the needs of the
project, and
consultants can
be engaged.