Community-Based

Evaluation Methods
& Data Analysis

Amy Carroll
April 22, 2005
 Different Types of Data
 Opinions, priorities
 Aspirations,
motivations
 Level of awareness,
knowledge, attitudes or
beliefs
 Behaviors, practices
 Assets, skills
 Networks, associations
 Needs, fears, problems,
concerns
 Demographic
 Services or resources
provided
 Resident utilization of
services or resources
provided
 Numbers or rates of
disease, illness,
disability, injuries
 Sales transactions,
purchases
 Policies
 Pictures, other visuals

Quantitative vs. Qualitative
Data
Quantitative:
 Collected in the form of
numbers or percentages
 Closed-ended questions
 Answers who?, what?,
when? and where?
 Can demonstrate cause
and effect
 Can “represent” a
population
 Cannot collect new
ideas or responses, only
those considered ahead
of time
Qualitative:
 Collected in the form of
words, concepts,
themes, or categories
 Open-ended questions
 Answers how?, why?
 Can provide richer,
more in-depth data
 Can provide data in a
respondent’s own
words
 Can explore new ideas
in a dynamic and
unstructured way
 Common Community-Based
Evaluation Methods
• Secondary data
• Activity tracking
• Client tracking
• Mapping/ Photovoice
• Surveys
• Focus groups
• Key informant interviews
Which of these methods
collect quantitative data?

Which collect qualitative

data?
Secondary Data
• County health department
• Vital statistics  Birth & death certificates
• City or county government
• Schools (including school nurse records)
• Hospital or clinic records
• Local health promotion, health ed or advocacy
agencies
• Private foundations which fund community
programs Universities or local community
colleges
• Local surveys or research efforts going on in
your community??
• Data you already collect!
 Activity Tracking
• Tracking and documenting program
activity implementation
• Tracking and documenting number of
people reached
• Process evaluation method, but
important for outcome evaluation
• May include: weekly/monthly program
implementation log, sign-in sheets at
classes or health fairs, counts of
materials/ brochures distributed, etc.
Client Tracking

• Tracking and documenting encounters

with clients; describing the encounter
and client characteristics
• More than just counting
• Can be used for process and outcome
evaluation
• May include: outreach encounter form,
home visit form, clinic intake form, etc.
Mapping/ Photovoice
• Data can also include maps and photos → don’t
just have to be numbers or words
• Maps and photos can provide visual imagery to
give meaning to quantitative data
• They can show graphically what cannot be
described with numbers or words, such as the
conditions of a neighborhood or program
facilities, or a community’s proximity to park
space or health services.
• Make sure your maps and photos are clear and
illustrate the points you would like to make
Surveys
• Surveys are composed of carefully
crafted questions on a topic or issue for
the purpose of revealing information
about community residents or services
they utilize.
• The most commonly used survey
methods are:
1. Mailed Surveys
2. Hand-out Surveys
3. Face to Face Surveys
4. Telephone Interview Surveys
 Planning A Survey

• Determine your purpose

• Determine if the survey data collection
is appropriate
• Select your target audience
• Select a survey method
• Design survey questionnaire
• Pilot test survey
• Publicize and distribute surveys
 Determine the Data You
Need
• Draft your research questions – what
you want to learn from these
interviews.
• Identify the data you will need to
collect in order to answer these
questions.
• The type of data you are looking for
will help you identify the best persons
in the community to survey, and the
best survey questions to ask them.
 Target Population

The following are two steps in

determining your survey target
population.
1) Define your target population --
identify the group of individuals from
your community from who you want to
collect information (single mothers,
high school students, the uninsured,
etc.)
2) Identify the geographic area of your
target population.--Where is your group
of interest located? What are the
 Select A Survey Method

Self Administered: These surveys are filled

out by respondents themselves without the
assistance of trained interviewers.
A. Mail Surveys
B. Drop-off Surveys

Administered by Interviewer: These surveys

are filled out with the assistance of trained
interviewers.
A. Face to Face Surveys
B. Telephone Surveys
 Design Survey
Questionnaire
• Your survey questions should be in direct
relationship with what you want to learn from your
target population.
• Asking unrelated questions will interrupt the flow of
the survey and tire out your respondents.
• Be brief and strategic when designing questions!
• Open-ended questions are great for answering
“why” questions, but only include one or two.
• Motivating people to fill out your survey will help
you collect the data you need – this can include a
benefits statement or incentives such as entry into
a lottery or drawing.
 Pilot Test Survey

• It is always a good idea to pilot test

the survey with a small group of
people from your target population.
• The purpose for piloting the survey is
to get feedback on the survey question
and structure before implementing it.
• This will ensure you identify potentially
confusing, offensive, leading, or loaded
questions from your survey.
 Publicize and Distribute
Surveys
 Advertise your survey a couple of weeks
before distribution in the community.
 Publicizing what you are trying to do and
how the information collected will benefit
the community may help increase your
response rate.
 Create a survey distribution timeline and
stick to it - postponing survey distribution
date may limit the time you have set
aside for data analysis.
 Advantages and
Disadvantages of Surveys
Advantages Disadvantages
 Data can be collected  Survey instrument must
from a lot of be carefully constructed
respondents easier than to avoid leading
any other method questions, and to make
 Can get a large enough sure the appropriate
sample that can be responses are available
representative of the  Response rates can be
larger population low for self-administered
 Findings can be surveys, especially
generalized to the larger mailed ones
population  Response will be low if
 Can cover a lot of topics survey is too long
 Can easily compare
Focus Groups

Focus group are a qualitative

research method designed to
learn more about how people
think, feel, or make decisions
(attitudes, perceptions, opinions,
experiences) through focused
discussions.
Focus Groups (cont.)

 A series of discussions involving 8-12 people,

selected to share their perceptions of a
defined topic
 Stimulates participants to share their
opinions openly in a group discussion
 Captures rich data in participants’ own words
 Flexible to capture new ideas and issues
 Can be difficult to summarize and interpret
results across groups
 Need to be creative when recruiting busy
people
 Planning a Focus Group

• Determine the data you need

• Determine and select your target
audience
• Develop a focus group discussion
guide
• Schedule focus groups
• Recruit focus group participants
• Recruit moderator and note-taker
• Plan and conduct focus groups
 Determine the Data You
Need
• Draft your research questions – what
you want to learn from these focus
groups.
• Identify the data you will need to
collect in order to answer these
questions.
• The type of data you are looking for
will help you identify the best persons
to recruit for your focus groups, as well
as the best questions to develop for
 Target Audience

• To whom do you want to talk?

• What types of experiences and
perspectives you are hoping to capture
in your discussion?
• Design and use a screening tool to
recruit participants: ask questions that
qualify people to participate.
• It is important that you include
participants who will feel comfortable
talking to each other.
• Participants should not know each
 How Many Focus Groups?

• Conduct 2-3 focus groups for each sub-

population of interest (i.e. gender, age,
race/ethnicity, language, etc.)
• 2-3 ensures that the perspectives and
experience you capture are more
representative of the larger population
and not just by “chance”.
• Length: 1.5 hrs.-2 hrs.
• Focus Group Size: 6-12 participants
 Focus Group Discussion
Guide
• Discussion guide: An outlined script to
help guide the moderator and focus the
discussion
• Organization:
Introduction
Purpose
Ground rules
Focus group questions
Closing statements
 Discussion Guide Questions
• Introductory questions: getting people to
think about the general topic.
• Transition questions: questions that get
participants to logically transition between the
introduction topics and the ideas contained in
the key questions.
• Key questions: 2-5 questions important to
getting the information you have set out to
collect.
• Ending questions: questions that provide
closure and reflection.
• Summary question: note-taker summarizes
the major themes heard throughout the
discussion in a short oral summary (2-3
minutes), then will ask the participants if the
summary covered all the major points.
 Scheduling Focus Groups

• Location: The focus groups should

take place in a convenient and
accessible location. It is also important
to select a neutral environment.
• Time of day: most convenient for
your target audience. Set time earlier
than intend to start.
• Incentives: Food or refreshments,
money or gifts, drawings, child care,
transportation, anything that is
meaningful to those community
members.
 Conducting & Moderating

• The discussion guide will help guide the

moderator with the questions you want
answered.
• Moderator Responsibilities: setting up
ground rules, keeping the discussion focused
and on track, drawing out comments from
quiet participants, and asking for clarification
or additional information from participants
when needed.
• Note-taker Responsibilities: takes
comprehensive notes, operates audio
equipment (tape-recorder), keeps track of
time, handles environmental conditions and
 Skills of the Moderator

Extremely important to select moderators based

on skills and experience:
• Comfortable/familiar with leading meetings or
groups
• Able to exercise mild control over the group
• Able to guide conversation back on target
• Maintain group enthusiasm and interest
• Curious about the topic
• Respects participants and their comments
• Creates and maintains a comfortable
environment
• Adequate background knowledge
• Good oral and written skills
Advantages and
Disadvantages of Focus
Groups
Advantages Disadvantages
 Flexible  May be challenging to
 Captures rich, in- recruit participants
depth data  Need to schedule at
 Immediate results least 2-3 focus groups
to capture diversity
 Encourages and
stimulates individuals  Difficult to generalize
to share more openly results to the larger
population because of
 Data can be combined
small numbers of
with quantitative data
participants
to provide a complete
picture about an issue  Difficult to compare
results across groups
Key Informant
Interviews
• Key informant interviews are qualitative, in-
depth interviews with key informants
• Key informants are individuals who know
what is going on in the community
(community leaders, professionals,
residents, advocates, elected officials,
doctors, etc.)
• The following are two common techniques
used to conduct key informant interviews:
1. Telephone interviews
2. Face-to face interviews
 Planning Key Informant
Interviews
• Determine the data you need
• Determine target population and
brainstorm possible key informants
• Select key informants
• Select interview technique
• Develop an interview guide
• Determine documentation method
• Select designated interviewer(s)
• Conduct key informant interviews
• Compile and organize key informant
interview data
 Determine the Data You
Need
• Draft your research questions – what
you want to learn from these
interviews.
• Identify the information you will need
to collect to answer these questions.
• The type of data you are looking for
will help you identify the best persons
in the community to interview, and the
best questions to ask them.
 Target Audience and Key
Informants
• Determine your target population (program
clientele, racial/ethnic minority groups,
adolescents, women, the elderly, HIV+
persons, etc.)
• Brainstorm and create a list of possible key
informants who are knowledgeable and
closely involved with your target population.
• Select key informants:
• Who have first-hand knowledge about the target
population, the program, and the issues you are
trying to investigate.
• Who represent different experiences, backgrounds,
and viewpoints.
 Select Interview Technique

Telephone interviews (15-25 min):

• May be the most convenient and least time
intensive way to interview busy key informants.
• Disadvantage: not having the personalized
interaction otherwise possible through a face-to-
face interview.

Face-to-face interviews (20-30 min):

• Most frequently used format. But time intensive.
• Advantages: provides a free-exchange of ideas
and lends itself to ask more complex questions.
 Develop Interview Guide

Introduction- Introduce yourself and your

purpose and/or benefits statement.
Key questions- Draft 5-10 questions
important to getting the information you
have set out to collect.
Probing questions – Probing questions
encourage participants to reflect more
deeply on the meaning of their comments.
Closing question—Provide an opportunity for
the key informant to give any additional
information or comments.
Summary—Quickly summarize the major
 Determine Documentation
Method
 Note-taking: manually write the key
informant’s comments while
conducting the interview, on a copy of
the interview guide.
 Tape-recording: use a tape recorder
to document what key informants say.
This approach may allow the
interviewer to freely engage in the
conversation without worrying about
note-taking.
 Conduct Interviews

• Thank respondents for their time and

stress the importance of the interview.
• Interviewers should listen carefully for
recurring and new opinions or beliefs.
• In order to compare with other
interviews and identify emerging
themes, it is important to get answers
from every person interviewed on key
questions.
Advantages and
Disadvantages of Key
Informant Interviews
Advantages Disadvantages
 Rich data can be  Selecting the “right” key
gathered relatively informants may be difficult
cheaply & easily so they represent diverse
 Allows interviewer to backgrounds and viewpoints
establish rapport with the  May be challenging to reach
respondent, clarify and schedule interviews with
questions, and draw out busy and/or hard-to-reach
responses respondents
 Allows for discussion of  Difficult to generalize results
topic without group to the larger population
dynamic of focus groups unless interviewing many
 Provides an opportunity key informants
to build relationships
with important
 Things to Consider Before
You Select Your Methods
• What data collection method(s) would best
collect the type of data you need from your
target population
• What data collection method(s) would help
you to best answer your research
question(s)?
• Think of your available resources: time,
staffing, skills, funding, computer technology
• Prior experience
• Potential benefits
• Potential challenges
 How to Understand
the Data You Collect
(Data Analysis)
Why It’s a Good Idea to Plan
Ahead for Data Analyses
 To determine if the data you collect
are practical for analyses
 To review the appropriateness of your
chosen data collection method(s)
 To inform the data collection
instrument you develop
 To decide if you will need help with
analyses
1: Check the Data

 Make sure it’s all there

 Make sure it makes sense
 Catch any mistakes that happened while
filling out the survey or entering the data
 Perform data quality checks throughout your
data collection process
 Common mistakes:
– A missing or incomplete response
– An impossible response
– A response unrelated to the question
– A response that contradicts an earlier response
2: Go Back to the
Research Questions
 Be sure to analyze only the data that will
help you answer your research question(s)
 Don’t get side-tracked into analyzing other
pieces of data with your limited time and
resources
 It is normal to collect more data than you will
need for your evaluation
 You can return to any other interesting data
after completing your evaluation
3: Reduce the Amount of
Data
 Save your data in its original form so
that you can return to it if you need to
 Look at it in small chunks at a time
 Eliminate any irrelevant data from
your analysis
 Create summary documents to help
you “eyeball” the data more easily to
look for patterns or themes:
– Numeric spreadsheet
– Qualitative summary document
 4: Analyze the Data
Quantitative Data Analysis: Looking
for Patterns in the Data
 Calculate averages
– Used with “continuous” data: infinite number
of values
– Add numbered responses
– Divide by number of responses
– 2 + 4 + 6 + 10 = 22 22/4 = 5.5
 Count frequencies
– Used with “discrete” data: set number of
response categories
– Count number of responses
– Number “yes”, number “no”
– Number of choice “A”, Number of choice “B”,
 4: Analyze the Data
Quantitative Data Analysis: Looking
for Patterns in the Data
 Calculate proportions
– Is standardized
– Equation: % = # of responses/ total # of
respondents X 100
– Example: 45 females/ 100 respondents X 100
= 45% of the respondents were female
 Calculate rates
– Similar to proportions; also standardized
– Easier to compare
– Equation: Rate = freq of event in pop/ total pop
X 100,000
– Example: 15 cases of lung cancer/ population of
3,500 X 100,000 = 428 persons per 100,000
have lung cancer
 4: Analyze the Data
Quantitative Data Analysis: Looking
for Patterns in the Data
 Compare averages, frequencies, proportions,
rates
 Compare data from different populations:
– County vs. County
– County vs. State
– State vs. US
 Compare data from different segments of a
population:
– Male vs. female
– African American, white, Latino, Asian, American
Indian, etc.
– Children, teens, adults, seniors
– Clientele of your program vs. those who are not
 4: Analyze the Data
Quantitative Data Analysis: Looking
for Patterns in the Data

 Present the data in different ways to see

additional patterns and relationships:
– Chart/ table – shows averages, counts, proportions,
or rates side-by-side
– Pie graph – demonstrates percentages of the whole
– Bar graph – compares quantities
– Line graph – shows trends over time

 Determine your findings

– Interesting results?
– Interesting patterns or relationships?
– These are your findings
 Exercise 5.4A
(Using Step 5 of Performing a
Community Assessment)

 Look at sample survey and

sample spreadsheet in Appendix
C
 Answer questions on page 51 and
52
 Step 4: Analyze the Data
Qualitative Data Analysis: Looking
for Themes in the Data
 Qualitative data analysis can be deceptively
trickier
– Qualitative data is by nature “bigger”
– Analysis requires more analytical thinking and
interpretation
 Qualitative data is more open to “bias”
– Numbers don’t lie
– Analysts bring their values, assumptions and
opinions
– Analysts may think they “know” how people feel
 Include at least 2, preferably 3 people in
each stage of data analysis to avoid bias
 Create as structured of a process as possible
 Step 4: Analyze the Data
Qualitative Data Analysis: Looking
for Themes in the Data

1. Read through all of the data at least twice

– Stay “close” or “grounded” in the data
2. Create categories
– Focus group or interview questions
– Or specific assessment health issues
3. List themes that emerge from discussion
– Sub-categories
– Ideas
4. Rank order according to frequency of
appearance
5. Summarize discussion around each theme
6. Determine your findings
 Exercise 5.4B
(Using Step 5 of Performing a
Community Assessment)

 Look at sample focus group

discussion transcript in Appendix
C
 Answer questions on page 56
5: Verify Findings

 You now have:

– Some patterns and relationships you found in your
quantitative data
– Some themes and categories you found in your
qualitative data
 Based on the results you tallied and summarized,
pull out the main findings of each method you
used
 Verify these findings by re-tallying and re-
summarizing the data to make sure you get the
same results
 You can also verify your findings by comparing
the results reached by 2 analysts independently
6: Interpret Findings and
Draw Conclusions
 Determine what interpretations can be drawn from
each finding
 Invite different perspectives from different Partners and
community members, as this will make your
conclusions stronger
 Are the results similar to what you expected? If not,
discuss why you think they are different
 Brainstorm alternative explanations for your results to
make sure you have considered all possibilities
 Make sure the conclusions answer the original
assessment questions
 Draw conclusions and recommendations that can be
shared with external audiences
 Exercise 5.6
(Using Step 5 of Performing a
Community Assessment)

 Look at the secondary data tables

in Appendix C
 Answer questions on page 59