You are on page 1of 33

THE ART OF EVIDENCE Sudarno Sumarto

BASED POLICY MAKING IN The National Team for the


Acceleration of Poverty
INDONESIAN SOCIAL POLICY Reduction (TNP2K) &
The SMERU Research
Institute
KSI, 21 April 2016
5 “INSIDER” LESSONS ON EVIDENCE BASED POLICY MAKING

① How Priorities Were Made


② The Supply And Demand Of Evidence
③ When Difficult Choices Have To Be Made
④ Experience From Social Assistance Reform In
Indonesia
⑤ Insights On Evidence Based Policy Making
HOW PRIORITIES
WERE MADE
ACTORS BEHIND PRIORITY
SETTING
Individuals
e.g., the President or a
minister are able to get key
issues on the agenda by
virtue of their public office.

Priority
The International Setting The Public
Community
through media,
e.g., UN through the advocacy, policy briefs,
SDGs, ASEAN through shapes social
the AEC Blueprint, perceptions that signal
demands commitments issues that are
from countries to reach currently important.
specific goals.
TRADITIONAL RESEARCH IS
NOT THE ONLY TYPE OF
EVIDENCE
Blusukan
is also
evidence!
Systematic
Review

Randomized
Control Trial

Cohort Studies

Case Control Studies

Case Reports & Case Series

Anecdotal & Expert Opinion


RESEARCH HAS BEEN A PART OF THE
GOI PRIORITY SETTING PROCESS

External Parties

Internal Government
Research
Evidence

Ideology
Skills to & Values Political
Use Situation
Evidence
However policy making
Bureaucracy
Resources is a non-linear process.
Policy Estimation Research is often a
Makers small input as many
additional factors
Lobbyists Time influence the decisions
& Pressure Pressure of policy makers.
Groups
Habits &
Tradition Competing
Alternatives
YES, EVIDENCE MATTERS.
But, policy actors can look at the same
evidence and come up with different
conclusions based on their own ability,
ideology, and interest.

A person from an academic


background may view education as
key to alleviating poverty, while a
businessman may think strengthening
small enterprises is the best approach.
THE SUPPLY AND
DEMAND OF
EVIDENCE
WHICH SHOULD COME FIRST?
THE SUPPLY OR THE DEMAND
PROBLEMS WITH SUPPLY (1)
Do we outsource too much?

• International assistance may not be sustainable.


• Indonesian investment in the knowledge sector is
very low at around 0.05% of GDP (Singapore 2%,
Brazil 1.02%, Malaysia 0.64%).*
• In-house research capacity is low due to lack of
trainings.
• Government tenders eliminate universities and
NGOs from the knowledge provision services
market.
• Highly qualified graduates choose not to work in
policy sectors due to lack of career advancement
incentives.

* Source: KSI presentation, 2010


PROBLEMS WITH SUPPLY (2)
Correct Whom to trust?
assumptions
Sufficient Accurate
sample method
Many research are available out there,
often with conflicting results, produced by
competing parties, who equally think their
Credible Sound Environment research leads to a better world.
data Research that can be
Evidence generalized

What to trust?

Research must be handled with care. Sound


evidence requires the right instruments,
accurate method, and representative
sample.
PROBLEMS WITH SUPPLY (3)
How to ‘sell’ the evidence?

Common problems of research:

1. Provide recommendations that are


too futuristic or politically mission-
RESEARCH impossible.
RECOMMENDATION
2. Fail to capture the big picture by
focusing only on a specific subject.
3. Speak different “language” by not
understanding specific questions and
the types of information for which
policy officials care about, or method
of presentation that wins buy-ins.
REAL PROBLEM TO BE 4. Results take time to produce.
SOLVED
PROBLEMS WITH DEMAND (1)
 Capacities need to exist at multiple levels.

 At organizational level there needs to be


structural capacity – i.e. adequate internet
bandwidth, access to relevant academic
journals or data.

 At an individual level, policy makers need


to be ‘research and evidence-literate’ – i.e.
know what research evidence is, where to
find it, how to appraise it, and how to make
policy decisions based on it.
PROBLEMS WITH DEMAND (2)
 Evidence based policy vs. policy based evidence
 Too much emphasis on evidence may backfire and lead policy makers to
manipulate data especially when facing negative-incentives or pressures.

A classic example from education data in Kenya.

A donor abolished school fees and rewarded Kenyan authorities financially


for putting students through primary school. According to administrative
data, there was a rapid increase in enrollment following the abolishment of
fees. However the Demographic and Health Survey (DHS) showed
enrollment rates that were flat over time. This discrepancy manifested itself
after the program.

Why? Schools got more funding if they reported more pupils.

Sometimes investment must also be made in data-collection itself. Blind


Don’t turn Mr. faith on external data may backfire especially when knowledge production is
Data into evil part of a complex political economy.
WHEN DIFFICULT
CHOICES HAVE TO BE
MADE
POLICYMAKING ULTIMATELY IS AN
INHERENTLY
POLITICAL PROCESS

Actions are often taken based on what’s popular in the short term not on what
works in the medium and long-term.
NEVERTHELESS POLITICS IS NOT
THE ENEMY OF EVIDENCE BASED
POLICY MAKING
• Dismissing politics is impossible
as politics is the environmental
context in which policy is made.
Resources • Policy reforms require additional
Support
skills and capital beyond
Political Humility
Intelligence Patience
evidence production.
Analytical Networking
Skills
Persistence

Policy Politics
EXPERIENCE FROM
SOCIAL ASSISTANCE
REFORM IN
INDONESIA
TNP2K BRIDGES RESEARCH AND POLICY
# Research:
 Piloting and testing new approaches
 Monitoring and evaluating program effectiveness
 Collaborating with national and international experts
# Policy Reform:
 Getting “buy-in” for evidence based reform initiatives
 Supporting agencies responsible for implementing reform

Example:
Priority 1: Improving targeting performance of social programs
Priority 2: Improving benefits with the use of social protection cards
WHY TARGETING BECOMES A
PRIORITY?
Benefit Coverage by Decile Share of Benefits Received by Decile
100 25
Non-target Target Non-target
Target
80 20
Percentage Receiving Benefits

Percentage Receiving Benefits


60 15

40 10

20 5

0 0
1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10
Household Per Capita Consumption Decile Household Per Capita Consumption Decile

UCT Rice Health UCT Rice Health


WHAT ARE THE POSSIBLE
OPTIONS FOR TARGETING?

1 Proxy Means Test (PMT)

2 Community-based Targeting

3 Self-targeting
EVIDENCE SHOWS …

PMT is the most accurate But communities are better


method in identifying the extreme
poor
NATIONAL REGISTRY FOR SOCIAL
PROGRAMS WAS BORN
 New and improved
proxy-means testing
identified the poor
more accurately
 Expanded to cover
about 25 million
households,
classified in the
poorest 40% of the
population
 Using community-
based meetings to
update targeting
lists to address
exclusion errors
IMPROVING RASKIN BENEFITS
THROUGH RCT*
 572 villages (378 received cards, 194 did not) within 6 Districts
in Central Java, South Sumatera, Lampung).
 Card: randomly varied in terms of design (coupon or no
coupon), content (price or no price), distribution (to all
beneficiaries or the poorest 10%).
 Socialization: from 378 villages; 186 received standard
socialization, 192 received enhanced socialization.
 Delivery: from 378 villages; 191 did biddings (anyone can
apply to be Raskin implementer), 96 pursued monitoring
(community meeting on Raskin distribution), 285 were made
as control.
* Research conducted by TNP2K in collaboration with the Abdul Latif Jameel Poverty Action Lab
Raskin card with coupon and price

Raskin card without


coupon, with price

Bidding Monitoring
Results*
Raskin Card + Raskin Card + Monitoring Bidding
standard soc. enhanced soc. treatment treatment

Total cost per village


(million IDR) 2,5 9,5 2,9 4,7

Change in monthly subsidy Rp. 6.597 Rp. 10.863


per eligible household (no change) (no change)
(23%) (38%)
Change in % eligible
households purchasing (no change) (no change) (no change) (no change)
Raskin
Change in price for eligible - Rp 55 / kg - Rp 93 / kg
households (no change) (no change)
(8%) (14%)
Change in quantity for
eligible households 1,11 kg 1,86 kg (no change) (no change)

Note limited economies of scale in experiment may drive up bidding cost as


compared to large roll-out
* For more information see the papers:
• ”Tangible Information and Citizen Empowerment: Identification Cards and Food Subsidy Programs in Indonesia”
• ”Contracting out the Last-Mile of Service Delivery: Subsidized Food Distribution in Indonesia”
SOCIAL PROTECTION CARD WAS
INTRODUCED

 Cards were delivered to the


bottom 25% households in the
national registry (covering
around 70 million individuals)

 Cards give access to:


 Subsidized rice allocations
 Scholarships for the poor
 Unconditional cash
transfers
INSIGHTS ON
EVIDENCE BASED
POLICY MAKING
COLLABORATION IS KEY

Evaluation Implementation
Government
(Central/Local)

Evaluation Findings
Replications/
Scale ups
based on
Multi/Bi Lateral Evidence
Institutions & CSOs
Donors

Research
Institutions Ongoing monitoring & Evaluation
FOR POLICY MAKERS
1 Master basic analytical skills.
2 Be selective with your evidence.
3 Be creative in addressing practical challenges (such as
time constraints, political bargain).
4 Carefully consider the program’s scalability.
5 Avoid a one-size-fits-all mindset.
6 Invest in in-house research capacities.
FOR KNOWLEDGE
PRODUCERS
1 Build a reputation by always providing reliable research.
2 Be humble; being smart is not enough.
3 Be politically sensitive by knowing the ‘Indonesian way’.
4 Ask the right questions and understand the big picture.
5 Include government early on, if possible from the design phase, to help
identify analytical needs and implementation challenges.
6 Frequent coordinations with all stakeholders, especially high ranking
policy makers.
7 Wet apetitite for policy input by making research easy to understand
and relevant.
8 Incorporate government internal knowledge/data.
THANK YOU, AND
LET’S GET TO WORK!

You might also like