You are on page 1of 21

A JOURNAL REVIEW:

Evaluating the Policy-Practice


Gap in Transitional Housing
Program: An Innovation in
Process Evaluation
Niani Coker
Suborna Bhattacharjee
Anabel Castaneda
Farhan Danish
Aleka Arredondo
Jerry Aneke

HSCI 615
GAPP Five Domain Theory
(Gap Assessment of Policy Program) Explain factors of homelessness
By: Zlotnick, Robertson and Lahiff
Learning Objectives

Objective 1: Objective 2:
● Exposure to current ● Understand the uses of
process evaluation GAPP to address the
methods discrepancy of social
services being provided to
● Critique the Article
the homeless population
Abstract

● Recent analysis of processes evaluation has not properly


valued qualitative data while collecting data (interviews,
document reviews and content analysis)

● This study will use the GAPP tool as an outline to integrate


qualitative methods while accessing program guidelines
Intro

● Evaluation is important for improving human services


programs

● The article uses a theory called the five domain theory,


which explains the five factors causing homelessness, to
suggest a new form of process evaluation called the Gap
Assessment of Policy and Practice tool, which will be used
to find the gap between a program’s intended and actual
delivery of service in the five domains stated in the five
domain theory.
Process Evaluation
Why Was This Process Evaluation Conducted?
● Improve the effectiveness of homeless assistance programs
● Aligning program practices with goals
● Increase program impact by encouraging frequent/efficient self-
assessment

Comparing Process Evaluation Tools


● Traditional “Five Domain Theory”
○ the application of this theory has been confined to understanding the homeless
condition using quantitative analysis
● Gap Assessment of Policy and Practice (GAPP) tool
○ diagnostic tool that assesses the discrepancy between a program’s intended
and actual service delivery in each of the theory’s five domains
Process Evaluation
Elements of Process Evaluation that Appear
● To whom did you direct program efforts?
○ The target population for Integrated Transitions = adult men who are sober, employed at
least 30 hours a week, and who have displayed a willingness to equip themselves with
the tools necessary to live independently. (Excludes individuals convicted of violent
crimes or crimes against children.)
● Why was your program implemented?
○ Transitional Housing Programs which aims to facilitate homeless individuals’ exit from
homelessness
● What are the barriers/facilitators to implementation?
○ Dysfunction encompasses any barrier that impedes ability to function in broader society.
In the context of homelessness, impaired function can refer to physical disabilities,
mental illness, and/or substance abuse
● When did your program activities take place?
○ 3 month study period, 120 hours of participant observations
● Where did your program activities take place?
○ Shelter Now’s emergency shelter, Haven - California
Study Participants
Participants
● Interviews with program staff and previous clients
● Program Staff:
○ Shelter Now’s Director of Social Services and Director of
Administration & Social Services (2)
○ Haven’s Lead Case Manager and Resident Manager (2)
● Previous clients, due to IRB requirements:
● individuals employed at the Haven, working on a volunteer basis,
or, still using emergency services (e.g., emergency shelters).
● 7 individuals fit these criteria at the time of the study, 5 of whom
agreed to be interviewed.
Methodology
Case Study Approach
● Conducted observations of:
○ program operations
○ interviews with program staff and previous
clients
○ content analysis of client files
○ review of organizational materials (ie. policies)
● 3-month study period, 120 hours of participant
observations
Methodology
Interviews
● Questions Addressed:
○ Program activities, implementation, and their
perceived effectiveness
● Semi-structured interviews provided the flexibility to
pose follow-up questions
● 8 face-to-face interviews with knowledgeable staff
members and previous clients were completed
Methodology

Interview Questions Covered 3 Topics:


1. Background Information
○ Clients were asked about their homeless history and staff members
were asked about their roles at the Haven
2. Program Services/Delivery
○ Clients and staff were asked about the case management, recovery
support, and spiritual and emotional growth opportunities at
Integrated Transitions (services listed in program policies)
3. Previous Concepts and Operations of the Five Domains
○ Staff and clients described how Integrated Transitions addressed
each domain and gave examples
Methodology
Organizational Document Review
● 70 client files at the Haven’s case management offices
○ Info on client’s activities/status while in the program
○ Reviewed info about clients’ progress and verified
with interview responses
○ Report how often clients are seen by case managers
○ Reviewed written policies
Methodology
Measuring Five Domains
This study was interested in how Integrated Transitions addresses each domain in their service delivery.
Five Domains
1. Impaired function. staff/previous clients interviewed about program’s efforts to address
substance abuse and physical/mental health needs
2. Disaffiliation. staff/previous clients asked about the program’s ability to strengthen social
networks and support systems
3. Cultural identification. Staff/previous clients asked how the program provided life skills
training (e.g., budgeting, goal setting, planning ahead, and maintaining a household) and
exposure to positive role models
4. Human capital. Clients/staff asked to identify provided opportunities to increase clients’
educational training to lead to degree attainment and/or job skills.
5. Economic resources. Review of client files and program policies and clients/staff were asked
how program helped clients obtain housing (e.g., other transitional shelters, rental
apartments, and affordable housing). Interviews focused on housing due to evidence
suggesting that providing stable housing environments is one of the best ways to secure
successful homeless exits
Methodology
Data Coding for Analysis
● Performed a content analysis of program policies and client files to
determine how Integrated Transitions addressed five domains
○ Derived themes related to each area of focus, program activities,
and services
● Completed open coding of interview transcripts
○ Techniques from Glaser’s constant comparative method
○ Codes were used to conduct a focused coding of interviews, field
notes, and primary documents
○ All data were analyzed
○ Data coding and analysis took approx. 3-months
Findings
Traditional Evaluation vs. GAPP tool
● Purpose of study: introduce the GAPP tool as a low-cost evaluation tool
to be used by homeless/human service providers for more frequent ‘‘in-
house’’ process evaluations
● Both evaluations produced consistent results
○ Traditional evaluation are more detailed/analytic vs. GAPP tool’s
findings.
○ The GAPP tool forces evaluator to list all policies/practices that
address each domain’s areas of focus and exposes general patterns
in program operations that are easy to identify and resolve
● Programs either ALIGNED, POLICY HIGHER THAN PRACTICE, or
POLICY LOWER THAN PRACTICE
Findings
Lessons Learned
● Traditional vs. GAPP tool evaluations - both methods are in agreement
● Each method provides unique information about the program
○ The GAPP tool uncovers trends in alignment between program
policies, or intentions, and program practices
■ 24 man-hours over 2-3 days
○ The traditional evaluation provides data to understand why trends
are occurring
■ 120 hours over the course of 3 months
● Advocated for the use of both the traditional (when economically
feasible) and GAPP tool evaluations (in between traditional evaluations)
Findings

Generalizability
● A benefit of the case study method is the potential for
establishing analytic generalizability

● The GAPP tool has analytic generalizability


○ 2 broad frameworks: process evaluation and
five domain theory
Strengths
● Article attempts to address all factors that contribute to
homeless exits both Structural (job changes, and housing
markets)and Individual (personal problems and life
situations).
● Utilizes the framework of process evaluation and explains
causal domains for homelessness.
● Focus is solely towards program goals and priorities.
● Integrates the use of qualitative data(interviews, document
reviews, data analysis) in process evaluation.
● Empowers program administrators by addressing various
barriers (lack of trust, social incompetence, knowledge).
Limitations
● Study reports, preliminary success for the GAPP tool in a
single THP evaluation
● GAPP score - a gap only communicates a misalignment
○ Administrators/staff must refer to the justification to
identify discrepancies
Critique
● Lack of specificity to make accurate prediction.
● Limited novelty.
● GAPP tool produce similar result compared to traditional.
● A five domain theory was used which take time to produce
and do not provide benefits commensurate with the effort.
Reference

Perez, J. L., & Yerena, A. (2016). Evaluating the Policy–Practice Gap in


a
Transitional Housing Program. American Journal of Evaluation,
109821401561773. doi: 10.1177/1098214015617730

You might also like