2010 REGION 5 Parent

Technical Assistance
Conference

 June 3 -4 , 2010
 Salt Lake City
 You are here!
 Are you tuned in?
 Are You Tuned In?
Asking the Key Questions:
Alignment in Evaluation of Project
Outcomes

Susan M. Moore
Clinical Professor
University of Colorado, Boulder
Proposition

 Your plan for your program

 “is dynamic and can respond to changes in context, focus,
and purpose over time and

 must be supported by a process that brings stakeholders

together regularly to ask the necessary critical questions.”
You can’t see the forest for the trees…

 www.youtube.com/watch?v=vJG698U2Mvo
 What is your RFP saying?
WHO This priority supports Parent Training and Information Centers
(PTIs) in the areas to be served by the centers that will:
provide parents of children with disabilities, including low- What
income parents, parents of limited English proficient children,
and parents with disabilities, with the training and
information they need to enable them to participate
cooperatively and effectively in helping their children with Why
disabilities to—
(a) Meet developmental and functional goals, and challenging
academic achievement goals that have been established for all
children; and
(b) Be prepared to lead productive, independent adult lives, to
the maximum extent possible.
Outcomes
Logic Models
Applicant must include in its application—

(a) A logic model that depicts, at a

minimum, the goals, activities, outputs,
and outcomes of the proposed project. A
logic model communicates how a
project will achieve its outcomes and
provides a framework for both the
formative and summative evaluations of
the project;

Note: The following Web site provides more information on logic

models and lists multiple online resources:
http://www.cdc.gov/eval/resources.htm.
Logic Model
 Using a logic model for planning a sequence of
change strategies or activities that
cumulatively would achieve the desired multi-level
outcomes
 Ongoing and cyclical evaluation and monitoring of
the accomplishment of benchmarks identified for
activities in the plans, to allow mid-course
corrections and fine tuning
 Evaluating the effectiveness of the plan in making
the intended improvements at the state, services,
and family and child levels ww.nectac.org
What is a logic model?...A Framework

 “If …then” …causal sequence

 Graphic display
 Based upon a situations /priorities
 Critical components: resources, activities, initial outcomes,
intermediate outcomes and intended results/impact
 a framework for both the
formative and summative evaluations
of the project;
Definition of terms…

 Formative evaluation is an:

Analysis of how program operates,

implementation and potential barriers…
undertaken to inform and improve the
program along the way …conducted
throughout program
 Killion, J.,(2008)
Definition of terms…

 Summative evaluation is an

 Analysis to determine outcomes of the program

and overall impacts…typically at end of program
to share overall results with funder

 Killion, J.,(2008)
What are you evaluating ?

Caution: Evaluation of a series of events is not necessarily program

evaluation… you need a comprehensive plan

E.g.. Workshops might be a staff development event or activity versus

A comprehensive, ongoing plan might involve the workshops,
ongoing coaching and follow-up, learning labs etc…all designed to
develop educator knowledge, attitudes and skills
Why are you evaluating your
programs?

 To determine merit, worth, and impact…overall

effectiveness, expansion, continuation or
discontinuation or compare with other programs

 To facilitate improvements through analysis of

strengths, progress, identification of benefits and
barriers

 and
Why

 To generate knowledge and identify

patterns of best practice or principles of
effectiveness and shape language and
thinking about practices

 Tobe accountable in terms of fiscal

responsibility, compliance with intended
purposes, accountability for resources
Theory of Change…

 Your program is based

upon a theory of
change
 Example:
Teacher Education Implementation Performance
Theory of Change

Evaluate Observe

Implementation Plan
Another example…systemic

 If Then
Local
Results are
improved
Improved for children
Personnel Development Practices and families

State
The underlying logic of the model is that for
results to improve for children and families,
practice needs to be research-based, of high
quality and appropriate
The How… First A Clear Purpose

 Pull in your stakeholders and create a shared understanding of how your

plan will reach its goals;

 Determine which goals are realistic, given available resources and priorities

 Chart the progress of your work and intended outcomes, which can guide
the evaluation and improvement of your strategies

 Communicate what you want to achieve and how it will get there, including
specific areas of focus
 The How…Steps

1. Define the Goals That Will Shape Your Strategy

2. Identify Inputs

3. Specify Activities Your Program Will Implement

4. Define the Desired Outcomes with Timeline

5. Select Performance Measures to Track Progress

Developing Logic Models
Overall Questions
 Do the inputs, outputs, and outcomes link together in a
sequence to achieve the desired result? Alignment
 Is the logic model truly logical? If …then
 Do the steps that turn inputs into outputs into outcomes seem
sensible and logical? If…then
 Can the program, as described in this logic model, be
implemented with available resources?
 Is what you intend to do possible, given your resources? If not,
what will be done?
 What might be unintended or negative outcomes?
 Does the one-page graphic communicate well?
Ask Questions to Guide Your Plan

Step 1. Define the Goals that Will Shape

Your Strategy Based Upon an
Identified Situation or Priority

Q: What are the goals that will help your

program decide how it will implement and
assess its family engagement efforts
Ask Questions…

 Step 2: Identify Inputs

 Q: What inputs— such as collaborators,

leadership, resources, and infrastructure
—need to be in place in order for your
program to reach its goals?
Ask Questions

 Step 3 Specify Activities Your Program

will Implement

 Q: What will your program do to promote

family engagement? What activities will
the staff perform? What services will you
provide and how often and who will
participate?
Ask Questions

 Step 4 Define the Desired Outcomes

 Q: What changes do you expect as a

result of your projects family engagement
efforts? What will family members, school
and district staff, and/or students believe,
know, or do differently?
When Defining Outcomes, Think…

Alignment …Logical Sequence and Realistic

Time Frame… Short term, Interim, Long term

 Short-term outcomes: Increased awareness /knowledge of the

importance of family engagement; increased awareness and
understanding of rights to and opportunities for family engagement
 Interim outcomes: Improved home–school communication and
family–school staff relationships; better home environment and
parenting to support learning; improved school culture
 Long-term outcomes: Improved student outcomes?
Stay tuned in…
 Make Outcomes SMART …
 While goals express the big-picture vision for what you
hope to accomplish, outcomes should be:
 Specific,
 Measurable,
 Action-oriented,
 Realistic, and
 Timed.
 The “smarter” your outcomes are, the easier it will be to
manage performance and assess progress along the
way….transparency
√ Outcomes Check

 Is each outcome truly an “outcome”?

 Are the outcomes written as change statements?
Will things increase, decrease or stay the same?
 Are the outcomes linked as progressive steps
towards a long-term goal?
 Are the outcomes meaningful and relevant?
 Are the outcomes realistic and attainable?

_________________________________________
Ask Questions

 Step 5: Select Performance Measures to Track

Progress

 Q: How will you assess your program’s family

engagement activities and progress towards
desired outcomes?
Performance Measures

 Measures of effort describe whether and to what extent

activities were implemented as you had intended. These
measures describe whether the participants were
supported as you had planned. Such measures can be
calculated by counting the number of attendees at an
event, or materials that your program produced and
disseminated. Measures of effort can also describe what
participants thought of the activities and information you
offered through participant satisfaction surveys.
Performance Measures
 Measures of effect/impact convey whether or not you are
meeting your desired outcomes. Unlike measures of effort,
which demonstrate what activities your plan implemented and
how it implemented them, measures of effect help track
whether your activities have made a difference.
 For example, an evaluation of a project’s professional
development program for parent liaisons might include
measures of effort, such as how many liaisons attended a
training session about sponsoring a family-friendly
walkthrough of their schools, and measures of effect, such
as what percentage of participating schools showed
improvements on their level of family-friendliness.
Types of Evaluation: Align with Logic
Model http://www.uwex.edu/ces/pdande

“Needs/Asset Assessment”
Ask:
What are the needs, priorities, characteristics,
strengths of target population?
What are potential barriers/facilitators ?
What is most appropriate to do?
Process Evaluation

Ask:
How is program implemented?
Are activities delivered as intended?
Is fidelity an issue? Are participants being
reached as intended?
What are participant’s reactions?
Outcome Evaluation

Ask:
 To what extent are desired changes
occurring?
 Who is benefitting/not benefitting? How?

 What seems to work? Not work? What are

unintended outcomes?
Impact Evaluation

 To what extent can changes be attributed to

the program?
 What are net effects?
 What are final consequences?
 Is program worth resources it costs?
 Are impacts program sustainable?
Evaluation Methodologies

Evaluation of the implementation of the plan

using multiple data collection methods,
including

 Telephone interviews,
 Surveys and questionnaires
 Client focus groups
 Case studies
Summative

 Focus Questions

Collect Data

 Analyze & Interpret

Report
Evaluation Methodologies

 What have you used or currently use to

collect information at a formative level about
the process as well as a summative level
about your outcomes?
Overall Q’s to Ask Yourself
Is each listed outcome truly an 'outcome'? Does the logic model clearly
separate outcomes from outputs, or are the distinctions blurred?
Does the highest-level outcome represent a meaningful benefit of value to the
public? Does it have inherent value? Can it be associated with the program?
Is the model truly logical? Do the relationships among the program elements
make sense? Are the casual relationships supported? Three ways to check:
Starting at inputs, ask “why?” at each level: why do we need these inputs?
Why do we need to conduct these activities?
 Starting at the impact level, and working backward, ask “how?” How are we
going to
produce these outcomes? The items immediately preceding an outcome
should show
“how.”
More Qs to Ask Yourselves
Sometimes components are necessary but not sufficient. Ask yourself,
“What else?”
For example, achieving healthy one-year-olds requires not only
achieving a healthy birth but also achieving proper care during the
baby's first year. Asking 'what else?‘ helps spot leaps of faith.
Are the resources realistic? Is what you intend to do even possible
given your resources?
How valid are the assumptions? Are they based on experience and
research, or are they best guesses?
Does the logic model reflect the opinions and support of key
stakeholders? Were any stakeholders left out?

EVALTALK, on-line, 7/9/98; Freddolino, P. 1998, Michigan

Adapted from Mike Hendricks,
Safe and Drug-Free Schools and Communities;
Full Circle…

 “An effective logic model is dynamic and can

respond to changes in context, focus, and
purpose over time.
 To retain its relevance, any logic model must
be supported by a process that brings
stakeholders together regularly to ask the
necessary critical questions.”
 Harvard Research Project
Resources & References

http://www.nectac.org
http://www.cdc.gov/eval/resources.htm.
http://www.uwex.edu/ces/pdande

Westmoreland, Lopez, & Rosenberg, (2009)

How to Develop a Logic Model for District wide Family
Engagement Strategies, Harvard Family Research
Project http://www.hfrp.org
Killion, J., (2008) Assessing Impact, 2nd Edition, Thousand
Oaks: CA, Corwin Press.