Evaluation systems and evaluation

practice: towards a framework for

research evaluation
Maria Nedeva, MIoIR
the University of Manchester
Warsaw, December 3, 2010
Outline of presentation
Science/innovation systems and evaluation
systems
Evaluation framework
Evaluation practice
Broad conditions for evaluation
What to watch for
Science/innovation systems
Science/innovation systems
Research spaces and fields
Research
organisation(s)
FO1
FO2
FO3
KC1
KC2
KC3
KP2
KP1
KP3
KEY:
KC = knowledge community
KP = knowledge pool
FO = funding organisation
KCs interactions with
different research spaces
Research space
Research field
Research fields and policy spaces
Research space is a policy space
Relationships between regulators, funders
and performers of research
Funders can be layered e.g. Government
organisations, executive agencies etc.
National Evaluation System:
definition
Research evaluation systems are organised sets of
procedures for assessing the merit of research
undertaken in publicly funded organisation that are
implemented on a regular basis, usually by state or
state-delegated agencies...[NESs vary] in terms of
their frequency, formalisation, standardisation and
transparency.
R. Whitley, 2007
(The Changing Governance of the Sciences)
National Evaluation System
Permeates research spaces and fields
Institutionalised: rules, responsibilities,
procedures, consequences
Repeated
National Evaluation System:
organisation
Who commissions
Who organises
Who carries out
Who applies the results
Example the RAE in the UK
Organisation of science and
evaluation systems
Answers depend on the organisation of the
research space (national)
As a rule of the thumb:
Political organisations commission (Ministries, Govnt
Departments)
Intermediaries (agencies) organise
Multiple actors carry out (contractors, panels, experts, in-
house specialists)
Political organisations and agencies apply
Organisation of science and
evaluation systems
NESs are conceived, developed and
governed in the Research Space
They can target social phenomena in
research spaces and research fields
Generally evaluations are about performance
management: of funders, research
organisations, research teams, researchers
and knowledge
Evaluation as management practice
Information
Judgement
Action
Information
About what?
Object
Types of information
How collected?
Methodologies
Sources
Who collects?
In-house
Outsourced
Judgement
Criteria
Standards
Multiple criteria
Layered
Depend on dominant ideological beliefs
Action
Funding decisions
Regulation
Change
Closure
Evaluation practice
Distortions
Indicator based
Peer review based
Evaluation methodology(ies)
Broad conditions of evaluation
Fit between the overall and localised
evaluation practices
Time-lines
Sharing of practice
Experience
Comparison
Relying on others
Pitfalls
Over- evaluation
Distortions of
the science system
knowledge
management
Assumptions
Thanks for listening! Any questions?
Maria.Nedeva@mbs.ac.uk