Elps431 Final Report Pie

Running head: PROGRAM EVALUATION NORTHWESTERN 1
Program Evaluation for the Peer Inclusion Educators Program at Northwestern University
Anthony Sis and Lorrena Johnson
Loyola University Chicago

PROGRAM EVALUATION NORTHWESTERN 2
Table of Contents
INTRODUCTION ...................................................................................................................................................... 3
PROGRAM HISTORY .............................................................................................................................................. 3
PROGRAM OVERVIEW .......................................................................................................................................... 4
PROGRAM CONTEXT ............................................................................................................................................. 5
PROGRAM DESCRIPTION .................................................................................................................................... 6
SELECTION OF PEER FACILITATORS ............................................................................................................... 7
PROGRAM PURPOSE ............................................................................................................................................. 8
PROGRAM STAKEHOLDERS ............................................................................................................................... 9
LOGIC MODEL OVERVIEW ................................................................................................................................ 10
PREVIOUS EVALUATION................................................................................................................................... 11
GENERAL EVALUATION APPROACH ............................................................................................................. 12
QUANTITATIVE APPROACH ............................................................................................................................ 13
PARTICIPANTS ............................................................................................................................ 13
SURVEY DESIGN ........................................................................................................................ 14
INSTRUMENT DESCRIPTION ........................................................................................................ 15
PILOT TESTING........................................................................................................................... 17
IMPLEMENTATION/ADMINISTRATION ......................................................................................... 17
STATISTICAL ANALYSIS/ RESULTS PRESENTATION .................................................................... 18
QUALITATIVE APPROACH ............................................................................................................................... 19
QUALITATIVE RATIONALE ......................................................................................................... 19
ANALYTICAL APPROACH ........................................................................................................... 20
FOCUS GROUP PLAN .................................................................................................................. 21
FOCUS GROUP IMPLEMENTATION .............................................................................................. 23
FOCUS GROUP ANALYSIS ........................................................................................................... 23
LIMITATIONS .............................................................................................................................. 24
RESULT PRESENTATION ............................................................................................................. 25
NEXT STEPS ............................................................................................................................... 25
REFERENCES ......................................................................................................................................................... 26
APPENDICES ......................................................................................................................................................... 27
APPENDIX A: PIE PROGRAM LOGIC MODEL .............................................................................. 27
APPENDIX B: PREVIOUS EVALUATION ....................................................................................... 30
APPENDIX C: ABBREVIATED SOCIAL JUSTICE KEY CONCEPTS .................................................. 33
APPENDIX D: INTERGROUP RELATIONS (IGR) LANGUAGE ACTIVITY ........................................ 34
APPENDIX E: AGENDA FOR INTERSECTIONAL LANGUAGE ACTIVITY ......................................... 36
APPENDIX F (LANGUAGE ACTIVITY SURVEY) ........................................................................... 37
APPENDIX G (SOCIAL IDENTITY SURVEY) ................................................................................. 41
APPENDIX H (INTERSECTIONALITY SURVEY) ............................................................................. 45
APPENDIX I: SURVEY ITEM MATRIX .......................................................................................... 48
APPENDIX J: FOCUS GROUP PROTOCOL FOR THE LANGUAGE ACTIVITY .................................... 50
APPENDIX K: FOCUS GROUP PROTOCOL FOR SOCIAL IDENTITY ................................................ 53
APPENDIX L: FOCUS GROUP PROTOCOL FOR INTERSECTIONALITY ............................................ 56
APPENDIX M: CONSENT FORM................................................................................................... 59
APPENDIX N: PROTOCOL INVITATIONS ...................................................................................... 61
APPENDIX O: PROTOCOL INVITATIONS (CONT.) ......................................................................... 62
Introduction
Northwestern University is an academic driven institution. The mission statement for the
university states that Northwestern University is committed to excellent teaching, innovative
research, and the personal and intellectual growth of its students in a diverse academic
community (About: Northwestern university, 2016). Within the university, there is the office of
Campus Inclusion and Community (CIC). In this office, there are three major departments that
promote the mission of CIC, which is to to work collaboratively with the university community
to cultivate inclusive learning environments through the intentional engagement with difference
(Campus Inclusion & Community, 2016, p. 4). These three departments are Multicultural
Student Affairs (MSA), Student Enrichment Services (SES), and Social Justice Education (SJE).
An evaluation and assessment will be created for one particular program within CIC, Peer
Inclusion Educators (PIE), which is housed under the Department of Social Justice Education.
Program History
Campus Inclusion and Community (CIC) began as an office in the Division of Student
Affairs five years ago. In 2012, Executive Director Lesley Ann Brown-Henderson was brought
in by the Vice President of Student Affairs to create this office. This office was created due to
student activists that demanded and protested for a university led effort to create a more
equitable and just campus culture (Campus Diversity & Inclusion, 2016, p. 15). CIC was
initially a one-woman office led by Lesley Ann Brown-Henderson. She began recruiting
students who would be brave enough to engage in dialogue surrounding topics of social justice.
This program became known as Sustained Dialogue. In 2014, Campus Inclusion and
Community was restructured due to the sizeable growth of the Sustained Dialogue program,
which is how the Social Justice Education Department was established. Noor Ali was selected to
be the Assistant Director of the Social Justice Education Department and the Peer Inclusion
Educators Program was created to help alleviate the administrative tasks within CIC and the
Sustained Dialogue program.
Peer Inclusion Educators has only been established for two years. During this time, one
graduate assistant was hired to oversee another program within SJE, but they additionally
volunteered to also be a Peer Inclusion Educator. After each workshop the participants are given
a survey that asks the students if they felt the workshop was beneficial, if the material they were
learning was new, and they are also asked to evaluate the facilitators. To date, no formal
assessment has been completed based off all of the surveys that were collected thus far.
Program Overview
There are several conceptual frameworks that guide the work of CIC, SJE, and PIE.
CICs strategic theme is to advance social justice. CIC has claimed to, work authentically and
ethically to create an inclusive and socially just learning environment for all members of the
Northwestern University community. The model of CIC is adapted from the tri-sector model of
cultural practice by Jenkins & Walton (2008). The tri-sector model is commonly used in student
affairs it provides a framework based on three components education, enrichment, and
engagement. CICs structured its office by using the tri-sector model effectively. The Social
Justice Education department was created for the purposes of education described in the tri-
sector model above. The Social Justice Education department creates co-curricular educational
opportunities in partnership with the student community to foster conversations across difference
and support actions that create social change on campus (p. 9). The Student Enrichment Services
department was created for the purpose of enrichment described in the tri-sector model. The
Student Enrichment Services department works with low-income first generation students to
enhance their academic success, personal development, and professional growth (p. 9). The
Multicultural Student Affairs department was created for the purpose of engagement described in
the tri-sector model. Multicultural Student Affairs enriches the cultural experience of
Northwestern through leadership and education programming providing opportunities for
community engagement (p. 8). This formative assessment will focus specifically on SJE and an
evaluation and assessment will focus primarily on just one of the programs that SJE offers which
is PIE. SJE and its other programs are highlighted in the next paragraph.
SJE operates under a director, two assistant directors, and one graduate assistant. The
specific programming that SJE is responsible for is Sustained Dialogue, Analyzing Whiteness,
Step Up Bystander Intervention, and Peer Inclusion Educators. SJEs programming is mostly
funded through private grants and donors. The Assistant Director is hoping to hire a graduate
assistant whose primary focus will be to work on the enhancement of the PIE program. The
funds will be used to put a three-year plan in motion. In previous years, the department did not
receive any funding for Peer Inclusion Educators and all of the peer facilitators were working on
a voluntarily basis. The peer facilitators still work on a voluntary basis to date, but this is
expected to change over the next three years. The effectiveness of voluntary peer educators will
be included in our formal assessment and evaluation of the PIE program.
Program Context
Peer Inclusion Educators (PIE) is, a program started by students, for students. PIE
dedicates its workshops to focus on social justice, social identities, and dialogue surrounding the
Northwestern community (Social justice education: Peer inclusion educators, 2016). Training is
held for all students who would like to become a peer facilitator. During this training, the
Assistant Director trains the peer facilitators to understand their social identities, meaning
gender, race, ethnicity, sex, religion, socioeconomic status, etc; students are able to explore their
identities and begin to understand what identities give them power and privilege in the United
States and what identities are oppressed. The workshops also put an emphasis on the
intersectionality of each participant privileged and oppressed identities. This is completed by
having the facilitators complete assignments such as the social identity grid, followed by
discussions about social justice terms such as target, agent, inclusion, collusion, privilege,
prejudice, discrimination, and oppression. After the facilitators spend time reflecting on their
own identities they then complete mock facilitations to prepare them to engage in dialogue with
student groups on Northwesterns campus.
Once facilitators have completed their training student groups will have access to
complete requests for one time workshops by visiting the SJE website, which has a separate link
for programming requests under the CICs website. Students can then design their workshop by
requesting activities such as the language activity, social identity grid, or both. The language
activity looks at terms such as Target, Agent, Inclusion, Collusion, Privilege, Prejudice,
Discrimination, Oppression, and Social Justice. The group is then asked to come up with a
definition of each term collectively to begin the process of dialogue. The Social Identity Grid is
an activity that lists social identities such as ethnicity, ability, gender, sex, and race. The idea
with these activities is to have students think about their most salient identities and the identities
that they think about the least. For most people, the identities that you think about least are often
the identities that give you power and privilege. When students can identify parts of their
identities that gives them power and privilege, a rich dialogue is able to happen that plays a
crucial role to the relevance and importance of a program such as PIE to exist on a campus like
Northwestern.
Program Description
The Peer Inclusion Educators (PIE) program has three different workshop topics for
which student groups such as residence assistants, peer mentors, sororities, and fraternities can
request. These workshop topics include: general inclusion facilitation, social justice 101, and
inclusive language. Student groups also have the privilege of designing their workshops to fit
the needs of their group. For example, the general inclusion facilitation workshop outlines the
basics of inclusion and allows for the opportunity for your group to brainstorm ways to be even
more inclusive (Social justice education: Peer inclusion educators, 2016). The social justice 101
workshop focuses on tailoring conversations about social justice by having the group engage in
an intimate dialogue. An intimate dialogue is able to happen because workshops will only take
place with a group no larger than 20 students. Peer Inclusion Educators need to be able to
engage in conversations with group. Often times when a group is too large deep conversations
are unable to happen. The assistant director found that it is easier for facilitators to control the
space in an intimate setting (Social justice education: Peer inclusion educators, 2016). The final
workshop option is inclusive language. This facilitation workshop focuses on implementing
activities such as the language activity by putting those terms within the context of the culture
specifically at Northwestern University. Student groups have the opportunity to create a
workshop engaging all three of these topics, or only one topic, depending on the needs of the
person or group requesting a PIE workshop. All participants are expected to engage in dialogue
and come up with ways in which the language that they use on a daily basis can be more
inclusive by using a critical lens to analyze the meaning of social justice terms. A critical lens is
a tool to analyze works of literature. During the language activity facilitators are asking the
group to define social justice terms and then put them into the context of their lives as a
Northwestern student which is what is meant by using a critical lens when engaging in the
language activity.
Selection of Peer Facilitators

Peer Facilitators are chosen through a selective application process. The Assistant
Director of SJE, Noor Ali, has created a live application on the website
http://www.northwestern.edu/socialjustice. The questions on the website include why an
applicant would like to become a peer inclusion educator, their facilitation experience,
challenges they anticipate in this position, and how the applicant brings fun into the program.
Once the facilitators have been selected, they are required to attend two-day training from 9 AM-
5 PM on Northwesterns campus.
The trainings typically launch in November for new recruits. During the beginning of the
quarter (September through October), previously trained peer facilitators are encouraged to sign
up for the workshop requests that have been submitted to SJE. Noor Ali sends an email to all
facilitators with a description of the group that has completed a request along with dates and
times of the request. Peer facilitators then sign up for workshops that will work around their
schedule. The peer facilitators also have monthly meetings. During the monthly meetings, the
facilitators discuss what challenges they have faced and the group provides suggestions on how
they can improve.
Program Purpose
PIE has four learning outcomes (Social justice education: Peer inclusion educators
(2016), which include:
Educate campus peers on how to best allow for the growth of an inclusive community.
Allow for reflection on ones self , backgrounds, and biases.
Identify issues on campus and address them through active and passive programming.
Encourage dialogue around issues of difference and to develop competent mechanisms
for addressing these issues.
This program has been identified as an introductory sister program to the Sustained Dialogues
program, which encourages in-depth conversations once a week for 90 minutes throughout the
quarter.
Program Stakeholders
There are direct and indirect primary stakeholders in the PIE program which includes the
university at large, students, the vice president of student affairs, the director and assistant
director of SJE and community partners. The students of Northwestern University are
stakeholders because the office of SJE was created by students for students. Students actively
seek out workshops. Student facilitators are trained so they are learning life skills that they can
take with them and sharing knowledge with their peers. Assessment on the effectiveness of PIE
is determined by the participants (students, faculty, and staff) of the workshops as they are given
an evaluation after each workshop. The evaluation measures the effectiveness of the content and
measures the effectiveness of the facilitators. Faculty and staff seek out workshops as well
which would make the university at large indirect primary stakeholders. After each workshop all
participants complete surveys and their feedback is critical to how PIE will continue to operate.
PIE also trains a number of students to become active facilitators of PIE. The students
engagement with this program is what keeps it running. The vice president of student affairs is a
primary direct stakeholder as they are the chief personnel officer of the university and they
directly advise the president of all non-academic activities on campus including PIE (About:
Northwestern university, 2016). The director of SJE is a primary direct stakeholder. The
director is responsible for communicating with alumni and possible donors to fund PIE. The
assistant director of PIE is a direct stakeholder. The assistant director trains all student
facilitators and works collaboratively with the SJE staff to make sure PIE is being run
effectively. This includes recruiting student facilitators, training all participants, and facilitating
workshops.
Logic Model Overview

The PIE program logic model (See Appendix A) focuses on the operations of each of the
three PIE workshops. According to the W.K. Kellogg Foundation (2004), a logic model is a
systematic and visual representation of the relationships and resources provided to create the
intended outcomes of a workshop. We start assessing the logic model by looking at the
following themes: situations, inputs, outputs (activities and participation), followed by outcomes
and impact (short-term, medium-term, and long-term outcomes).
The situation begins with an acknowledgment that there has been no consistent data
produced to assess the effectiveness of each PIE workshop. As previously mentioned, thousands
of people have participated in a PIE workshop but due to capacity and other restraints, the need
for consistent assessments has not been present until now. The presence of a new direction in
SJE not only contributes to the situation, but is also identified as an input in order to operate the
program. The input category also includes two assistant directors, one graduate assistant, one
director for CIC, and the volunteer student facilitators. Another contributing input is a newly-
awarded $15,000 grant for five years for the continuation of the PIE program, which can
influence the effectiveness and implementation of assessments throughout all three PIE
workshops.
Activities in the outputs category are mostly attributed to student facilitator training and
continued dialogues about their experiences as PIE facilitators. Their facilitation expands not to
just training students but also student leaders in various departments such as residence life and
student activities. By offering spaces where the student facilitators can be vocal about their
experiences, SJE is able to check-in with them and enhance their facilitation skills by providing
opportunities for self-reflection and growth. It is noteworthy to mention that not all of the
participants are undergraduate students, but also graduate/professional students, faculty, and/or
staff. We did not identify any major significance in creating two different assessments for both
populations, so one comprehensive assessment will be used for both undergraduate and
graduate/professional students, faculty, and staff who attend a PIE workshop.
The three workshops that are currently offered (Language, Social Identity, and
Inclusivity) all have newly developed learning outcomes. The short-term, medium-term, and
long-term goals of each workshop vary primarily by their topics. These outcomes were
developed based off of the curriculum of each workshop. Although there is some overlap with
terminology, it became very evident during this process that the learning outcomes for each of
the three workshops were slightly different and needed to be addressed through individual
assessments.
Additionally, the three workshop templates are often combined throughout the year,
depending upon the request of the student group or organization. For example, a student
organization can request a workshop on language and social identity, or they can request one
workshop incorporating all three workshops. In these situations, students will receive more than
one assessment immediately after the workshop to assess the effectiveness of these outcomes.
Previous Evaluation
Currently, SJE has no data regarding effectiveness of the workshop content. There is one
evaluation (see Appendix B) that was to be distributed after each PIE workshop. Due to a low
staff in SJE last year which only consisted of an assistant director and one graduate assistant at
the time no data was able to effectively be assesses. PIE is also a fairly new program that has
only been around less than five years. Now that SJE has hired a Director more attention will be
given to the PIE program and assessments are planning to happen with the completion of this
evaluation. The new staff of SJE has hired an undergraduate student who will focus on
collecting data and inputting the survey items into SPSS for the 2017-2018 academic school
years.
The number of workshops and participants who attended a PIE workshop skyrocketed
between 2014-2015 and 2015-2016. According to the assistant director, there were a total of 25
workshops that reached 538 participants in 2014-2015, while in 2015-2016 there were 79
workshops that reached a total of 2,518 participants. It is unclear why there was such a drastic
shift in the amount of workshops and participants within one year but it is clear that an
assessment and evaluation is needed.
General Evaluation Approach

Completed assessments for PIE are very limited. As previously mentioned, there is only
one assessment in their database that was distributed over the course of one year. Since the
initial onset of the PIE program, no formal assessment has been consistently created for the
student facilitators or the participants. Given the high number of participants during the 2015-
2016 academic years, we decided to focus our general evaluation approach to assessing the
participants of each workshop and evaluating the effectiveness of the learning outcomes to each
workshop topic.
Our general evaluation approach will be primarily outcomes based. An evaluation is
considered to be formative if the primary purpose is to provide information for program
improvement (Fitzpatrick et al). The evaluations we created will focus on the effectiveness of
the PIE workshop and contain elements that address ways in which the workshop can be
improved for future purposes on the topics of general inclusion facilitation, social justice 101,
and inclusive language as well as providing critical feedback to the student facilitators so they
can improve their skills.

The learning outcomes associated to PIE are inclusive to all three major topics that are
offered through the PIE program. For this reason, we created an assessment that can effectively
assess all three of the major workshop topics offered by PIE facilitators. We would like to
effectively measure to what extent the goals that PIE is intended to create are achieved during
these one-time workshops. Ideally, after each workshop the participants will be given a survey
asking questions related to the effectiveness of each workshop based off of content and the
facilitators engagement.
As evaluators we decided a formative assessment on PIE would be beneficial to the data
we are trying to collect, analyze, and assess. All data that is collected during this assessment will
be continuously discussed with the stakeholders, specifically in regards to this process, and
throughout this implementation of this evaluation. This evaluation is a processed based
assessment that will analyze short-term, medium-term, and long-term outcomes based on survey
items that participants will complete.
Quantitative Approach
Participants
Participants of PIE workshops vary across the campus of Northwestern University. PIE
facilitators do not go out and recruit student participants. Instead, student groups can request a
PIE workshop (Social justice education: Peer inclusion educators, 2016). Student groups are
defined as any student organization on campus which includes but is not limited to; resident
halls, fraternities, sororities, peer advisors, student government, etc. Many student affairs staff
members will request PIE workshops for their student groups for which they supervise. Staff
members also participate in the PIE workshops so the data will include some surveys completed
by staff members. All of the questions on the PIE workshops post-surveys does include questions
that would be targeted for a staff member or student within the Northwestern community (see
Appendices F-H).
Survey Design
An indirect assessment will be conducted to evaluate the short-term, medium-term, and
long-term outcomes of each of the three workshops: the Language Activity, the Social Identity
Activity, and the Intersectionality Activity. A longitudinal approach will be employed because
an initial, six month, and one year post-survey related to the workshop that they have
participated in will be distributed to participants after each workshop. The groups of students
who participate in PIE workshops are learning social justice key terms right after participating in
a PIE workshop. The three workshops also ask the participants to reflect on the knowledge they
have just learned and how it can apply to spaces outside of the workshop. All of the students are
given individual post-surveys after each workshop, which assesses their knowledge of the social
justice key terms they have learned before participating in a PIE workshop(s) and their
knowledge of the same social justice key terms they learn after participating in the workshop(s).
Our method includes convenience sampling because every participant will complete an
assessment after each workshop (Banta & Palmoba, 2015). Our approach will also include a
longitudinal method of sampling because we ask the participants to complete an initial, six
month, and one year post-survey related to the workshop that they have completed. The survey
response process is complex! This involves comprehension, retrieval, judgment, and response
processes (Tourangeau et al., 2000). We are expecting a lot from the participants by assessing
their comprehension of the material that was presented to them after each workshop. We then
must retrieve the surveys and assess the responses hoping they will provide feedback on how we
can improve the workshops. Not only are we asking for feedback to improve the workshops but
also we are asking the participants to assess how well the facilitators were in each workshop.
The process is complex and we must take this process seriously to ensure funding will continue
for the program.
Instrument Description
PIE has three different workshops; Language Activity, Social Identity Activity, and the
Intersectionality Activity. All three workshops have slightly different surveys that are
administered to each participant after the workshop (see Appendix F-I). The Language Activity
post-survey includes a total of 23 questions (see Appendix F). The first 18 questions are
measured using the Likert Scale (strongly agree, agree, neutral, disagree, and strongly disagree).
The first 12 questions of the survey assess their knowledge of social justice key terms before and
after the workshop. The first six questions gauges their knowledge with the following terms
before participating in the Language Activity workshop these social justice key terms include;
target/agent identities, privilege, power, oppression, discrimination, and collusion. The next six
questions of the survey gauges their knowledge of the social justice key terms mentioned above
after the workshop. The third section of the survey asks participants how they would apply the
social justice key terms learned during this workshop at Northwestern University. This means
defining what discrimination may look like within their community at Northwestern. The fourth
section of the survey ask participants to measure the skills of the facilitators by asking them a
series of questions that measures the environment provided, approachability, timeliness, and
engagement during the workshop. The last section of the survey asks for demographic
information including age, citizenship/nation of origin, class standing, gender, and race.
The Social Identity Activity post-survey includes a total number of 24 questions (see
Appendix G). The first 19 questions are measured using the Likert Scale (strongly agree, agree,
neutral, disagree, and strongly disagree). The first ten questions of the survey assess their
knowledge of social justice key terms before and after the workshop. The first five questions
gauges their knowledge of social justice key terms before the workshop which includes;
target/agent identities, privileged identities as it relates to their own social identity, salient
identity, defining their own salient identities, and how their salient identities are connected to the
terms privilege and oppression. The next five questions assess their understanding of the social
justice key terms mentioned above after the workshop. The third section asks participants to
apply social justice terms discussed during this workshop at Northwestern University. This
means to define what it would look like for them as a student to understand their salient identities
and how it may give them power and privilege within the community at their school. The fourth
The Intersectionality Activity post survey includes a total of 20 questions (see Appendix
H). The first 15 questions are measured using the Likert Scale (strongly agree, agree, neutral,
disagree, and strongly disagree). The first ten questions of the survey assess their knowledge of
social justice key terms before and after the workshop. The first five questions gauges their
knowledge with the following terms before participating in the Intersectionality workshop key
terms include; privilege, power, oppression, intersectionality, and solidarity. The next five
questions of the survey gauges their knowledge of the social justice key terms mentioned above
after the workshop. The third section of the survey asks participants how they would apply the
social justice key terms learned during this workshop at Northwestern University. This means
defining what solidarity may look like within their community at Northwestern. The fourth
For all of the post workshop surveys, we are collecting unique ID markers such as their
Northwestern Student ID so that we can make sure to follow up with them during our
administration of our six-month and one year post-surveys. Although all surveys are confidential
they are not anonymous because each participant indicates their student ID number on all of the
surveys. All surveys that students will complete will ask the same questions from the initial
post-surveys (see Appendices F-I).
Pilot Testing
We will pilot test the first two student groups who request a PIE workshop during the fall
quarter. We will ask the two student groups if they could participate in a series of all three
workshops. After each workshop we will immediately collect the initial post-surveys. We will
begin to measure how effective all three of the workshops were by benchmarking and comparing
the before and after survey responses of the social justice terms reviewed in each workshop. We
will assess and evaluate if the facilitators were engaged, on time, created a brave space, were
approachable, and promoted full participation throughout each workshop. All surveys will be
analyzed using SPSS. We will also provide feedback to all participating facilitators so that we
can make sure the workshops are improving. The purpose of pilot testing is to make sure our
survey items are clear, to see how long the surveys take to complete, and to see if the
flow/instructions are clear to the respondents.
Implementation/Administration
Administrations of the post-surveys for each workshop will be handled by the facilitators
present in the space. All facilitators are expected to administer these surveys during the last 10-
15 minutes of their workshop. The expectation for the student groups should have been set
before they participated in the workshop. The facilitators are expected to let all participants know
that a post- workshop survey needs to be completed during the last 10-15 minutes of the
facilitation. The facilitators are then expected to collect the surveys from all participants. All
facilitators will give the surveys to the assistant director, Noor Ali who will then give us the
surveys. We will work in collaboration with the Student Assessment Center to evaluate the post-
surveys for all three workshops. We will only assess the post- workshop surveys from all of the
requested workshops from the fall, winter, and spring quarters. The dates during this time are
September 18, 2017- June 11, 2018. Assessing the data collected during this time will help us
provide immediate feedback to the PIE team so we can improve the quality of the workshops
based on our short term outcomes for the program.
Statistical Analysis/ Results Presentation

All survey data from all three workshops will be organized, described, and collected for
statistical analysis. All information will be coded and analyzed using SPSS. The categorical
demographics asked in all three surveys such as the race and student class standing are nominal
variables which can help provide an in depth evaluation about who are participating in the
workshops. The findings will help us gauge who are our target audience. This will help us
analyze if there is only certain student groups, nationalities, and age ranges requesting
workshops.
As previously mentioned, almost all of the questions on all three workshop surveys
provide interval data with the use of the Likert scale which helps measure the dependent
outcome variables. The information can be treated continuously and can provide useful
information because of the running of means as a descriptive phase for each survey. The mean
scores can be used to gauge the knowledge of social justice key terms before and after the
workshops. The mean scores can also measure the effectiveness of facilitators who are not paid
but are volunteering their time. The mean scores can also measure social justice application of
the social justice key terms reviewed in each workshop before and after student groups and staff
participate (see Appendices F-I). An ANOVA will be used to compare these three means with
each other to see if they significantly differ from one another. This will be helpful as it will help
us determine between which workshops are the most impactful to our participants and if not how
we can improve the content so they will have a better understanding of social justice concepts.
We will also want to conduct comparable analyses of all three workshops to see which workshop
students found to be more beneficial or if participants found all workshops to be equally
beneficial.
Next we will want to analyze how the responses change over time. This can be completed
by using a sample t-test that will provide us with inferential statistics that can gauge if participant
responses will change over time. We will compare results from the initial post-survey, to the six-
month survey, and the one-year survey. This is a longitudinal method that will help us analyze if
the PIE workshops are creating change for the betterment of the campus at Northwestern
University.
Qualitative Approach
Qualitative Rationale
The qualitative approach of the evaluation of the PIE program will include focus groups.
Focus groups are implemented to understand participants diverse experiences throughout the
three workshops that PIE offers, such as the Language Activity, Social Identity Activity, and the
Intersectionality Activity, to help provide a deeper meaning of the data collected during the
quantitative analysis (Schuh, 2009). The purpose of the focus groups is to use the findings to
better explain the extent to which student outcomes are met. The outcomes of the workshops are
assessing the effectiveness of the content that the participants are engaged with as well as the
effectiveness of the student facilitators ability to engage students with the content. We also
want to provide students with a space to provide a deeper meaning of the social justice key terms
they have learned. We hope participants will provide reflections and feedback on the facilitators
abilities to provide a brave space (a space where students felt they could be vulnerable and safe
to do so), their timeliness, and the effort each facilitator put into the workshops in which they
have participated as well. The focus group protocols provide probes and questions that ask the
participants why they signed up for the workshops. We want to assess if students are signing up
because they were forced to do so or if they genuinely were interested. The in depth responses
will help us analyze this question. Another probe seeks to understand the type of student that is
signing up for workshops by assessing which student organization or group they are a part of and
how they got involved to participate in the workshops. The last probe seeks to understand what
information they have learned from the workshops based on the conversation that will take place
because of the goals for each workshop (see Appendices J-L).
The program stakeholders desire to launch the focus groups for each PIE workshop
because no evaluation or assessment has been complete to date. The stakeholders are anxious to
collect data and receive feedback on each individual workshop so improvements can be made for
the following 2018-2019 school year
Analytical Approach
Participants who completed the cross-sectional surveys (see Quantitative section) will all
have the opportunity to participate in a focus group. The last question of each survey asked
participants if they would be interested in participating in a focus group for the workshop(s) they
have participated in. The cross-sectional survey asked participants for their Northwestern ID
number. Based on their response their name will be looked up in the system. An email will be
sent to all participants that indicated they would like to participate in a focus group (s) (please
see Appendices N-O). Program evaluators will use maximum variation sampling because of the
expansive and diverse nature of workshop participants. This strategy recognizes general themes
or patterns where different opinions exist (Schuh, 2009). This is important because if you
deliberately try to interview a very different selection of people, their aggregate answers can be
close to the whole population.
Convenience sampling was used throughout the quantitative surveys but evaluators
recognize the importance of varying sampling when analyzing qualitative data. Depending on
the variance of participants for the focus groups, we are reaching out to 30 participants all of
whom will be selected randomly based on their responses. We are hoping to have 5-10
participants sign up for a focus group for one of the following workshops that they have
participated. We understand that some participants may have experience with multiple
workshops but we will ask that they only participate in one of the focus groups of their choice.
Initially an email invitation will be sent out on April 11, 2018 (see Appendix N) to all
participants that expressed interest in participating in the focus groups. Participants will have two
weeks to decide if they would like to participate. A reminder email will be sent out on April 25,
2018 (see Appendix O). Once participants confirm that they will be attending a confirmation
email will be sent out immediately after their response. The emails will confirm the date, time,
and location of where each focus group will take place. We will also make sure to highlight that
participants are only to choose one focus group for one workshop that they have participated in
even if they want to participate in multiple workshops.
Focus Group Plan

There are three different focus group protocols for each workshop. According to the
focus group protocol for the Language Activity (see Appendix J) participants will begin with an
introduction from the graduate assistant, assistant director, or director of SJE who begins the
focus group with a simple thank you to the participants for dedicating their time to the activity.
Participants are then explained that the focus group activity will take 60 minutes. It is then
explained to participants that the information gathered today will be shared with SJE staff and be
used to improve the quality of PIE language workshop for future students. The feedback that the
participants provide will be recorded by note takers and a tape recorder so we can be sure to
accurately capture everyones responses. The facilitators make sure to reiterate that the
information will be private and they will not be specifically identified as the participants
afterwards. The facilitators of the focus group will then pass out a consent form to every
participant (see Appendix M). It is explained that the consent forms will protect their
confidentiality while participating in the workshops. After, the consent forms have been
collected the introductory question will be asked to begin the dialogue amongst the focus group.
The first question simply asks participants why they signed up for the workshop. Probes are then
asked so that participants can engage in the dialogue in a deeper way, providing more nuance
related to their experiences. Each probe tries to get participants to answer the question in a way
that reflects the goals of each workshop. This process will be completed in all three focus group
workshops. See Appendices J-L to see the learning outcomes and goals for each focus group
based on which workshop participants attended. After the focus group is complete, facilitators
will ask the participants for any feedback or recommendations to improve the workshops. This
question is not limited to the content of workshop but also the ability of the facilitators for the
workshop in which they participated. Again, this process is completed for each individual
workshop with different participants in each focus group.

Focus Group Implementation

The focus groups will be implemented in late April 2017, which is approximately six
months after student/staff have participated in a PIE workshop during the fall quarter. A pilot test
will not be completed on the focus groups beforehand due to a small timeframe and budget costs.
The office of SJE also has a small staff that may not be able to prepare for a pilot focus group
and plan for the focus groups that will be used to assess the workshops. The graduate assistant,
assistant director and director of SJE will be facilitating the focus groups. The assistant director
(Noor Ali) and Director (Robert Brown) are key stakeholders in this evaluation. It will be great
for them to hear firsthand what the participants have learned and to hear feedback about the PIE
facilitators that they select for these roles. The focus groups will be conducted on campus for the
convenience of the student/staff. Light refreshments will be provided in each focus group as
incentives. We recognize that the office of SJE only have three staff members. More staff
members from CIC will be asked in advance to hold focus groups on campus on the same dates
in different rooms across campus.
Focus Group Analysis

Focus group analysis will occur through a process of transcription, coding schemes, and
data analysis. The three focus group sessions will be transcribed. Once the transcripts are
finalized, the evaluators will listen to the audio recordings, review the notes taken by the
moderator and assistant moderators, and read the transcripts. The two evaluators will code each
transcript separately along with the assistance of the Student Assessment Center. A priori codes
and any additional ones that emerged through the coding process, the program goals and learning
outcomes could be explained to stakeholders. Inductive coding will be used as we take from the
findings to theorize the themes. We feel findings will emerge as we are assessing the surveys.
The cutting and sorting technique will be used when analyzing data because it allows the
evaluators to analyze responses to create themes amongst the responses. This method is useful
because it allow us to recognize and pull sub-themes while reading through the evaluations to
pull themes that seem to be important. It is also useful to cut and sort through themes while
laying them in envelopes to create neat piles of the information. The data will also be
triangulated by using the quantitative and qualitative results in order to gain more holistic
interpretations of participants responses from the cross sectional surveys and the focus groups.
Limitations
Due to the structure of CIC and SJE, all of the moderators are very familiar with the PIE
program and the learning outcomes for each workshop. This knowledge that the moderators have
of the program can spark biases when asking participants what they have learned during the
workshop. We will have to recruit a volunteer moderator since there are only two evaluators to
facilitate the third focus group. We also recognize the power dynamics in the room from the
professional staff to students. We know the students may be intimidated by staff so it is even
more important that the evaluators are present in the rooms to ensure the space is as comfortable
as it can be for participants.
Another limitation is acknowledged in our ability to create an effective long-term
outcomes-based assessment. We envisioned this assessment to be distributed via email to all of
the participants of a PIE workshop. The questions for these assessments would be created in
collaboration with the stakeholders and will be influenced by the responses gathered from the
short-term and medium-term focus groups. The structure would be very similar to the
assessment distributed immediately after participating in a PIE workshop. Due to this intentional
collaborative effort on behalf of the stakeholders and SJE staff, in addition to not being
redundant on the types of questions we asked in the final post-year assessment, we did not move
forward with creating what a long-term outcomes-based assessment could look like for each of
the PIE workshops.
Result Presentation
Qualitative data will be presented to key stakeholders in several ways. The responses
from the focus group will be presented as themes to provide feedback on the workshops and the
facilitators abilities. The themes will report, negative, neutral, or positive responses. The themes
will be collected by drawing from direct quotations from the transcription of the notes and
recordings of the focus groups. Bar graphs can also be used to show the major themes, for
example the frequency that each theme emerged in the focus groups.
Next Steps
The PIE program at Northwestern University is a relatively new program. It has been in
existence for five years. The staff of SJE has never had the capacity to complete a formal
assessment. This evaluation and assessment will be very beneficial especially for a relatively
new program. We want to provide feedback on how PIE can improve by analyzing the cross
sectional surveys for the quantitative data and the responses of the focus groups for the
qualitative data. Our next steps are to ensure the SJE staff will receive results and feedback
efficiently so the PIE program can begin implementing changes for the 2018-2019 academic
years.
References
About: Northwestern university. (2016). Retrieved from http://www.northwestern.edu/.
Banta, T.W., &Palomba, C.A. (2015). Assessment essentials: Planning, implementing, and
improving assessment in higher education (2nd ed.). San Francisco, CA: Jossey-Bass
Campus Inclusion & Community. (2016). Division, Campus & Departmental
Information.Unpublished internal document.
Fitzpatrick, J., Sanders, J., &Worthen, B. (2011). Program evaluation: Alternative approaches
and practical guidelines (4th ed.) New York, NY: Longman.
Jacobs, J. (Ed.). (2015, August 24). University halts reorganization of Black House.
Jenkins, T. S. (2010). Viewing cultural practice through a lens of innovation and intentionality.
Culture centers in higher education: Perspectives on identity, theory, and practice, 137-
156.
Multicultural Center space.The Daily Northwestern. Retrieved from
http://dailynorthwestern.com/.
Schuh, J.H. & Associates (2009). Assessment methods for student affairs. San Francisco, CA:
Jossey-Bass.
Social justice education: Peer inclusion educators. (2016). Retrieved from
http://www.northwestern.edu/socialjustice/programs-and-events/peer-inclusion-
educators.
Tourangeau, R., Rips, L. J., &Rasinski, K. (2000). The psychology of survey
response.Cambridge University Press.
W.K. Kellogg Foundation. (2004). Using logic models to bring together planning, evaluation,
and action: Logic model development guide. Battle Creek, MI: Author. (pp. 1-48).
Appendices
Appendix A: PIE Program Logic Model
OUTPUTS
SITUATION INPUTS
Activities Participation
No consistent 2 Assistant Train student Undergraduate

assessment to Directors (only 1 facilitators who and Graduate
measure the directly oversees volunteer to Students
participants PIE) participate in PIE
No data that Host monthly Student Leaders
demonstrates the 1 Graduate meetings for
effectiveness of Assistant facilitators to Faculty and staff
these workshops. extend dialogue at Northwestern
The Assistant 1 Director of SJE on identity, power, University
Director who and privilege
oversees PIE and 1 Director of CIC Conduct Multiple campus
SJE is now workshops partners including
working with a Volunteer student surrounded on Residence Life,
new director of facilitators who three main Greek Life, and
SJE. get trained for two themes: Language, Athletics
days (9 AM- 5 Social Identity,
PM) before the and Full-time Staff
academic year Intersectionality from SJE and CIC
starts Present to campus and one Graduate
partners such as Assistant from
Budget from a Residence Life to SJE
newly received expand the scope
private donor for of their workshops
$15,000/per year
for the next 5
years until 2021
OUTCOMES + IMPACT
Short-term Medium-term Long-term
(Learning/Awareness) (Action) (Conditions)
Language: Language: Language:
Participants will be able to Participants will become Participants will feel
define key social justice more familiar with these compelled to engage in
terms terms and concepts more dialogues and
Participants will be able to Participants will recognize discussion of these terms
apply key social justice when certain terms and and concepts
terms to concepts related to concepts are at-play on Participants will apply
power, privilege, and campus these terms and concepts
oppression through self-reflection and
Participants will be able to Social Identity: interactions with other
identify examples of how Participants will apply students on campus
these terms and concepts these key terms and
are experienced on campus concepts to their own social Social Identity:
identities Participants will be able to
Social Identity: Participants will not clearly articulate their ideas
Participants are able to reinforce notions of power, and thoughts on social
reflect on how they view privilege, and oppression justice in relation to their
key terms and concepts on campus social identities
related to social justice Participants will become
relate to their own and Intersectionality: empowered to engage in
others social identities Participants will apply the difficult conversations
Participants can define the definition of target and agent related to social identities
meaning of salient identities to the greater campus on campus
identities community
Participants can Participants will actively engage Intersectionality:
acknowledge how salient in self-reflection in order to Participants will use self-
identities may be deconstruct notions of power, reflection of their target and
intertwined with privilege privilege, and oppression on agent identities as tools to
and oppression in a group campus combat power, privilege, and
setting oppression on campus
Participants will seek further
Intersectionality: dialogues and discussion on
Participants will be able to topics and concepts related to
define both target and agent social justice on campus,
identities, including their particularly through SJE
own.
Participants will gain
knowledge of how
privilege, power, and
oppression intersect with
their social identities
Participants will be able to
define solidarity
Participants will be able to
identify practices that
exhibit solidarity with
social identities other than
their own
Assumptions:
All participants will students come to the program willing to discuss their identities and
awareness of other social identities
All participants students are willing to work through their discomfort in talking about
identities and difference
All participants are going to respect each others learning process throughout the
workshop
All participants will ask questions when they seek clarification on a particular idea or
topic
All participants will be able to clearly articulate their views and opinions on topics related
to social justice and/or their social identities
All participants will engage in the three core topics though some action item post-
workshop
Environment:
New SJE director may want to continue, or reshape, the PIE program
The effect of current events such as #BlackLivesMatter on student engagement around
social justice issues and critical self-reflection
Prevalence of bias incidents on campus among students, faculty, and staff
An academically-driven institution environment may impact the quantity and quality of
engagement among participants in a PIE workshop
(Adapted from The University of Arizona, College of Agriculture & Life Sciences: https://extension.arizona.edu/evaluation/content/logic-
model-worksheets)
Appendix B: Previous Evaluation

SJE Workshop Evaluation
Workshop topic: Program/Organization:
Facilitator(s): Date:
Your feedback is an important element in our ability to evaluate the effectiveness of our workshops and to develop
sessions that meet your needs and expectations. Please complete this evaluation and return it at the end of the
workshop.
RATINGS:
KNOWLEDGE:
Vaguely Completely
1. Before this session, I felt like I understood the workshops content:
1 2 3 4 5
Vaguely Completely
2.After this session, I feel like I understand the workshops content:
1 2 3 4 5
Disagree Agree
3. The content presented increased my level of self awareness:
1 2 3 4 5
4. This workshop increased my understanding of perspectives other than Disagree Agree
my own: 1 2 3 4 5
Comments:
ENVIRONMENT:
Disagree Agree
5. I feel this was a safe environment for my participation:
1 2 3 4 5
Disagree Agree
6. This workshop allowed me to step outside my comfort zone:
1 2 3 4 5
Comments:
APPLICATION:
7. There is an increase in my ability to apply the workshop content to my Disagree Agree
daily life: 1 2 3 45
8. There is an increase in my desire to continue conversations related to Disagree Agree
thecontent presented: 1 2 3 4 5
9. Because of this workshop, I am more comfortable interacting with people Disagree Agree
of different identities from my own: 1 2 3 4 5
Disagree Agree
10. I would be interested in attending future related sessions:
1 2 3 4 5
Comments:
FACILITATORS:
11. The facilitators were effective in their delivery of the workshops Disagree Agree
content: 1 2 3 4 5
Comments:
OVERALL:
Not Beneficial Beneficial
12. My attendance at the workshop was:
1 2 3 4 5
FEEDBACK:
13. Have you attended CIC Workshops before? (Circle one) Yes No
If yes, how many and what topics?
14.Which component of the workshop was most beneficial to you? Why?
15.Which component of the workshop was least beneficial to you? Why?
16.What information are you still unclear about with regard to the content presented?
17. What topics would you suggest for additional sessions?
18.Other comments:
DEMOGRAPHICS:
Age: ____________ Citizenship/Nation of Origin: ________________________________________
Class Standing:First year SophomoreJunior Senior Graduate/Professional Student

Staff Faculty Member Not already listed: ________________________
Gender: Woman Man Transgender Not already listed _______________________________

Race (check all that apply):Native American /Alaskan Native Black or African American
Asian/Asian American Native Hawaiian/Pacific Islander
Latino(a) White/Caucasian
Middle Eastern/North African
Not already listed: ______________________________
YOUR FEEDBACK IS VERY MUCH APPRECIATED. THANK YOU!

Please return this evaluation to the facilitators upon completion.
Appendix C: Abbreviated Social Justice Key Concepts
Racism: The systematic subordination of members of targeted racial groups who have
relatively little social power in the United States (Blacks, Latino/as, Native Americans, and
Asians), by the members of the agent racial group who have relatively more social power
(Whites). This subordination is supported by the actions of individuals, cultural norms and
values, and the institutional structures and practices of society.
Discriminatory Behaviors: The differential allocation of goods, resources, and services,

and the limitation of access to full participation in society based on individual membership
in a particular social group.
Social Justice: Includes a vision of society in which the distribution of resources is

equitable and all members are empowered and are physically and psychologically safe and
secure.
Power: Access to resources that enhance one's chances of getting what one needs in order
to lead a safe, comfortable, and productive life.
Prejudice: A set of negative personal beliefs about a social group that leads individuals to
prejudge people from that group or the group in general, regardless of individual
differences among members of that group.
Privilege: Access to something of value solely because of ones agent social identity
membership.
Individual Oppression: Actions that reflect prejudice against a social group.

Societal Oppression: Policies and individual behaviors within institutions that have a
differential and/or harmful impact on target groups, whether or not they are so intended.
Cultural Oppression: A set of social norms, economic and political arrangements, roles,
rituals, and arts that reflect and reinforce the power of a single agent group or belief
system.
Collusion: Thinking, feeling, and acting in ways that accept the dominant groups ideology
about ones own target group and accepting a definition of oneself that is hurtful and
limiting. This involves demonstrating the devaluation of ones own group and of oneself as
a member of that group. Note: In contrast to an internalized ism, collusion involves the
target group member actively supporting oppression, as opposed to a passive
internalization.
Appendix D: Intergroup Relations (IGR) Language Activity
Purpose: To introduce key IGR concepts to a group of participants. This activity also serves
to gauge the participants levels of understanding regarding these concepts.
Concepts Used: Agent, Target, Privilege, Power, Oppression, Discrimination, and Collusion
Set Up: Write each individual concept at the top of large, separate sheets of paper. Place the
sheets of paper together dyadically as follows: AgentTarget, PrivilegePower,
OppressionDiscrimination. Collusion is not paired with any other concept.
Activity:
Part One: Each participant is to place a check mark on the individual sheets as a
reflection of how well they understand each concept. The sheet represents a
continuum of understanding, with one side representing fully knowledgeable about
said concept and the other side representing not-so-knowledgeable about said
concept. The use of a check mark means that persons do not have to publicly
express their knowledge or lack thereof. Participants are free to discuss where they
placed their checks if so desired.
Part Two: After check marks have been made on all sheets, the facilitator(s)
explains that the purpose is build clarity on what these concepts mean. They note
that there are not clear definitions, but that the conversation should bring to light
some key aspects of each concept.
The facilitator then begins a discussion around each concept, starting with either
agent or target, and then adding the other, seeing what interpretations are held by
participants. Follow-up questions probe for examples and distinctions between
other words. Clarification is also sought if participant uses another concept word
within their interpretation, to see if they can explain it in more simple words.
Steadily, the facilitator adds each new word. After each word, the facilitator may
choose to point out the explanation on the key concept list, highlighting again that
these are not meant to be definitions.
Challenges:
Facilitator needs to have a good sense of what each concept is.
It is challenging to affirm participants when they offer interpretations that are very
different from what IGR views to be the more fitting conceptualization, while
pushing them to understand the IGR perspective.
It is helpful for the facilitator to have examples that help illustrate these concepts
readily available, in case the group doesnt generate enough.
This can be a lengthy exercise, and therefore challenging to keep peoples focus for
the entirety. The duration completely depends on the number of group and prior
understanding of each group member.
Reminding participants that there is no clean definition is challenging because their

instinct is to search for an answer. These conceptualizations are far more complex
than that.
It is challenging to pull out the key points from what participants are saying,
because a lot of times (unless they have practiced articulating these
conceptualizations) they dont have concise thoughts.
Appendix E: Agenda for Intersectional Language Activity
Agenda Facilitator to lead

Welcome- Introductions of facilitators
Icebreaker- (check and see which ones were done before)

Rock, paper, scissors challenge
Look up, Look down
Shake it off
Review Guidelines
Define Intersectionality
Review Language
Power
Privilege
Oppression
Target and Agent
Intersectionality Activity
Divide into small groups of 4 (10 groups) ~ 3:50-3:55pm
Choose an identity you are most comfortable with (aka
salient) and share with group. Do you hold privilege in this
identity? Are you oppressed? How? ~3:55-4:03 (6 min per
round, 1.5 min per person)
3 Other identities (6 min round
Share out- 10min- 4:20-4:30pm (10)

How was this for you?
What surprised you?
Which identity was the easiest for you to interact with?
Which identity was the most difficult for you to interact
with?
What commonalities did you find?
Back into small groups
Solidarity- 10min 4:30-4:40 (10)
How does this impact how you might interact with one
another? How do you relate with others?
How does this help you to relate with others who have
different identities than you hold? (We have different
identities but we all are oppressed and hold privilege----
how does this allow us to move past individual identities?)
How can you show solidarity/create coalitions with these
things in mind?
4:40-4:55pm (15 min)
Come back to large group debrief
Closure- One word Whip
Appendix F (Language Activity Survey)
Peer Inclusion Educators Post Workshop Survey

Workshop: Language Activity
Name of Peer Facilitator(s):
Date:
Northwestern Student ID Number: 000
Please complete the following post-workshop survey. Please complete the following survey
about your experiences in todays PIE workshop. All of your answers are confidential but we
are asking that you provide your Northwestern Student ID Number so that we can follow up with
a 6-month post survey of the content that you have learned in our workshop. Your participation
is appreciated. If you have any questions please feel free to contact the assistant director of the
Social Justice Education Department Noor Ali, noor.ali@northwestern.edu. After you have
completed this survey the Peer Facilitators will collect them. Thank you!
Social Justice Key Terms Knowledge
The following questions are centered on knowledge of the social justice terms you have learned
and or been introduced to by engaging in this workshop. Please rate your overall agreement on
a scale from Strongly Agree to Strongly Disagree.
1. As a result of participating in this workshop, I can provide a deeper definition of

target/agent identities.
Strongly Agree Agree Neutral Disagree Strongly Disagree

privilege.
3. As a result of participating in this workshop, I can provide a deeper definition of power.

oppression.

discrimination.

collusion.
Social Justice Key Terms Application

The following questions are centered on the application of the social justice terms you have
learned and or been introduced to by engaging in this workshop. Please rate your overall
agreement on a scale from Strongly Agree to Strongly Disagree.
1. I can identify target/agent identities in various settings at Northwestern University.
2. I can identify privilege in various settings at Northwestern University.
3. I can identify power in various settings at Northwestern University.
4. I can identify oppression in various settings at Northwestern University.
5. I can recognize when others are discriminated against in various settings at Northwestern
University.
6. I can recognize when I am discriminated against in various settings at Northwestern

University.
7. I can differentiate between individual and systematic discrimination in various settings at

Northwestern University.
8. I can identify collusion in various settings at Northwestern University.

Peer Inclusion Educators

The following questions are centered on the facilitation skills of the peer facilitators who
completed this workshop. Please rate your overall agreement on a scale from Strongly Agree to
Strongly Disagree.
1. The facilitators provided ground rules prior to the start of the workshop to engage in
dialogue around social justice key terms.
2. The facilitators encouraged vulnerability on behalf of the participants throughout the

workshop.
3. The facilitators shared their own experiences with the language activity throughout the
workshop.
4. The facilitators promoted verbal participation throughout the workshop via large and
small-group discussions.
5. The facilitators were on time and set up the space prior to the workshop.
6. The facilitators were positive throughout the workshop.
7. The facilitators were engaging throughout the workshop.
8. The facilitators were open to questions throughout the workshop.

DEMOGRAPHICS:
Class Standing: First-Year Undergraduate Sophomore Junior Senior Graduate/Professional Student

Race (check all that apply): Native American/Alaskan Native Black or African American
Latino(a) or Latinx White/Caucasian

Appendix G (Social Identity Survey)

Workshop: Social Identity
Name(s) of Peer Facilitators:
Date:
a 6-month post survey of the content that you have learned in our workshop. Your participation
is appreciated. If you have any questions please feel free to contact the assistant director of the
Social Identity Key Terms Knowledge
and or been introduced as it relates to your social identities by engaging in this workshop.
Please rate your overall agreement on a scale from Strongly Agree to Strongly Disagree.

target/agent identities.
2. As a result of participating in this workshop, I can provide a deeper definition of privilege

as it relates to my social identity.
3. As a result of participating in this workshop, I can provide a deeper definition of salient

identities.
4. As a result of participating in this workshop, I can provide a deeper definition of my own

salient identities.
5. As a result of participating in this workshop, I can provide a deeper definition of how my

salient identities are connected to the terms privilege and oppression.

Social Identity Key Terms Application

The following questions are centered on the application of the social identity key terms you have
1. I can identify target/agent identities in various settings at Northwestern University.
2. I can identify privilege in various settings at Northwestern University.
3. I can identify the term salient identity in relation to the campus community at
4. I can identify my salient identities as being a part of a privileged or oppressed group at


Strongly Disagree.

workshop.
workshop.

DEMOGRAPHICS:

YOUR
YOURFEEDBACK
FEEDBACKIS
ISVERY
VERYMUCH
MUCHAPPRECIATED.
APPRECIATED.THANK
THANKYOU!
YOU!

Appendix H (Intersectionality Survey)

Workshop: Intersectionality Activity
Name(s) of Peer Facilitators:
Date:
a 6 month post survey of the content that you have learned in our workshop. Your participation is
appreciated. If you have any questions please feel free to contact the assistant director of the
Social Justice Key Terms Knowledge
and or been introduced to by engaging in this workshop. Please rate your overall agreement on
a scale from Strongly Agree to Strongly Disagree.

privilege.
2. As a result of participating in this workshop, I can provide a deeper definition of power.

oppression.

intersectionality.

solidarity.

Social Justice Key Terms Application

The following questions are centered on the application of the social justice terms you have
1. I can identify my own privilege(s) in various settings at Northwestern University.
2. I can identify my own power in various settings at Northwestern University.
3. I can identify oppression in regards to my social identities in various settings at

4. I can identify intersectionality in regards to my own social identities in various settings at

5. I can identify solidarity in various settings at Northwestern University.

Strongly Disagree.

workshop.
workshop.
DEMOGRAPHICS:



Appendix I: Survey Item Matrix

Appendix J: Focus Group Protocol for the Language Activity
Post-PIE Workshop Attendees
Overarching Objective of the Language Workshop Focus Group:

What effect, if any, does the language workshop have on its participants immediately after the
workshop? What effect do the facilitators have on the participants immediately after the
workshop?
Learning Outcomes:
Participants will feel compelled to interact with these terms and concepts in other spaces
on campus
Participants will find groups or organizations on campus that engage with social justice
concepts
Participants will initiate dialogues and discussion of these terms and concepts on campus
Participants will partake in an ongoing self-reflection regarding social justice key
concepts with other students on campus
Focus Group Script

Introduction:
Thank you for joining us today to participate in this focus group. My name is (Graduate
Assistant, Assistant Director, or Director of SJE). Our goal through this focus group is to hear
about your experiences in PIEs language activity workshop and how those experiences have
related to your experience at Northwestern University over the last six months. By taking time
to speak with us about your experiences on campus and those you had during your language
workshop, we hope to add a greater depth to the feedback that was provided during the post-
workshop surveys. We greatly appreciate your participation thus far. This will be the last time
that we ask for your involvement in our evaluation.
The discussion should take no longer than 60 minutes. Please note that you are free to skip any
questions that are raised if at any point you feel uncomfortable. Please feel free to keep eating
snacks, refill on drinks, and use the restroom, etc., throughout our discussion.
The information gathered today will be shared with SJE staff and used to improve the quality of
PIE language workshop for future students. The feedback that you provide during this
discussion will be recorded by [state name(s) of notetaker(s)] and a tape recorder so we can be
sure to accurately capture everyones responses. This information will be confidential and will
not be specifically identified to you afterwards. Of course, there are no right or wrong answers,
and wed like to hear from everyone if possible.
At this time I ask that you review our interview consent form to make sure that you understand
the details of what will occur today. By signing the form you consent to participate in this
interview. Once we have everyones consent forms, we will begin the discussion.
Introductory Question:
To begin, wed like to hear about your reasons for completing this particular PIE
workshop
Probe: What were some of the reasons that led you to sign up for a language
workshop through PIE?
Probe: Was it a mandatory part of your training for a student group/organization?
Probe: Did you sign up yourself through a student group or organization?
Goal Area 1: Participants are able to describe a deeper meaning of the following social justice
terms: target/agent identities, privilege, power, oppression, discrimination, and collusion.
Question 1.1: Since completing the workshop, in what ways have you been able to
reflect on how the terms target/agent identities, privilege, power, oppression,
discrimination, and/or collusion are present on campus?
o Probe: When have you found it easier to do this?
o Probe: Are there particular situations when this is more difficult? For what
reasons?
o Probe: Is there a strategy or technique you learned during the language
workshop that has proven useful in addressing these situations?
Question 1.2: Can you identify examples of how these terms have appeared in various
settings through your own personal experience?

Question 1.3: How did you address the situation/context that you identified when one
or more of the terms were present on campus?
o Probe: What prompted you to address the situation as you did?
o Probe: How did you feel after you did this?
Goal Area 2: Participants can recognize systems of privilege at Northwestern University and
can identify power relations in various settings on campus.
Question 2.1: Can you describe whether youve been able to recognize systematic
privilege on Northwesterns campus since your participation in the workshop?
o Probe: What do you think contributed toward your ability to recognize
systematic privilege at NU?
o Probe: How did you feel when you recognized this type of privilege at NU?
o Probe: Did you process or reflect on that experience with anyone? If so, how
was that?
o Probe: Were any of the factors associated with one of your privileged identity
groups?
o Probe: If you could change anything about the how you addressed the
situation, what would it be and why?
Goal Area 3: Participants can recognize systems of oppression at Northwestern University and
can identify when themselves or students different from them have been discriminated against
based on the system in place.
Question 3.1: Can you describe whether youve been able to recognize systematic
oppression on Northwesterns campus since your participation in the workshop?
o Probe: How did you identify the situation/context with a system of

oppression?
o Probe: Since the PIE language workshop, have you proactively fostered a
space, or spaces, for others to be vulnerable and share their personal stories?
o Probe: Since the PIE language workshop, have you proactively fostered a
group or student organization that engages with one or more social justice
term?
Goal Area 4: Participants are able to provide critical feedback on the facilitators abilities to
provide a brave space for students to be vulnerable and share their personal stories, timeliness,
and effort into providing a learning environment.
Questions 4.1: The idea of a brave space is something that comes up in our area a lot.
Based of your experience on campus, how would you define brave space?
o Probe: Based on our definition of brave space, did the facilitator(s) provide
this during your workshop?
o Probe: Did the facilitator or facilitators provide a space where you were able
to have a voice in the conversation?
o Probe: What did the facilitator(s) do to create this space?
o Probe: Was there anything different the facilitator(s) could have done to create
a brave space for you?
Question 4.2: Were there any positive actions or statements made by the facilitator, or
facilitators, that still resonate with you?
Question 4.3: Were there any negative actions or statements made by the facilitator,
or facilitators, that you still remember from the workshop?
Closing Segment: Improvement and Recommendations
Before we end, we would like to take this opportunity to ask you if there are any improvements
or changes that you would make to the language workshop.
Is there anything else you would like to add before we end our discussion? If not, then this
concludes our time together. We appreciate you sharing your feedback with us today. The
success of the PIE program is dependent upon the valuable feedback that you all have shared
during todays discussion. Thank you for taking the time today to participate in this discussion!
Appendix K: Focus Group Protocol for Social Identity
Overarching Objective of the Social Identity Workshop Focus Group:

What effect, if any, does the social identity workshop have on its participants immediately after
the workshop? What effect do the facilitators have on the participants immediately after the
workshop?
Learning Outcomes:
Participants will apply key social justice terms and concepts to their own social identities
Participants will be self-aware of their salient identities in regards to power, privilege,
and oppression on campus
Participants will be able to clearly articulate their social identities in the context of power,
privilege, and oppression
Participants will be able to engage in difficult conversations related to social identities on
campus
Focus Group Script

Introduction:
Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,
Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your
experiences in PIEs social identity workshop and how those experiences have related to your
experience at Northwestern University over the last six months. By taking time to speak with us
about your experiences on campus and those you had during your language workshop, we hope
to add a greater depth to the feedback that was provided during the post-workshop surveys. We
greatly appreciate your participation thus far. This will be the last time that we ask for your
involvement in our evaluation.
The information gathered today will be shared with SJE staff and be used to improve the quality
of PIE language workshop for future students. The feedback that you provide during this
sure to accurately capture everyones responses. This information will be private and will not be
specifically identified to you afterwards. Of course, there are no right or wrong answers, and
wed like to hear from everyone if possible.
workshop
Probe: What were some of the reasons that led you to sign up for a social identity
workshop through PIE?
Probe: What were some meaningful takeaways from the workshop?
Goal Area 1: Participants are able to define what identities are oppressed and what identities
gives them power and privilege in our US society.
Question 1.1: Have you been able to reflect on your own oppressed and privileged
identities?
Question 1.2: Have you been able to reflect on how your oppressed and privileged
identities are present on campus?
o Probe: Have you felt ostracized due to your oppressed and/or privileged identities
through various settings on campus?
Goal Area 2: Participants are able to recognize and state their salient identities, meaning the
identities that they identify with the most within US context such as their race, ethnicity, religion,
sexual orientation, gender, etc.
Question 2.1: Can you elaborate on how you describe your salient identities to others
in relation to power, privilege, and oppression?
o Probe: Did participation in the social identity workshop influence or change how
you describe your salient identities to others?
o Probe: What are some of the challenges that have come with the self-reflection of
your salient identities?
o Probe: Did the facilitator, or facilitators, provide a space where you were able
or changes that you would make to the social identity workshop.
Appendix L: Focus Group Protocol for Intersectionality
Overarching Objectives of the Intersectionality Workshop Focus Group:

What effect, if any, does the intersectionality workshop have on its participants immediately
after the workshop? What effect do the facilitators have on the participants immediately after the
workshop?
Learning Outcomes:
Participants will apply the definition of target and agent identities to the greater campus
community
Participants will engage in self-reflection by being exposed to notions of power,
privilege, and oppression on campus
Participants will use self-reflection of their target and agent identities as tools to combat
power, privilege, and oppression on campus
Participants will seek further dialogues and discussion on topics and concepts related to
social justice on campus, particularly through SJE
Interview Protocol
Introduction:
Thank you for joining us today to participate in this interview. My name is (Graduate Assistant,
Assistant Director, or Director of SJE). Our goal through this focus group is to hear about your
experiences in PIEs intersectionality workshop and how those experiences have related to your
experience at Northwestern University over the last six months. By taking time to speak with us
about your experiences on campus and those you had during your language workshop, we hope
to add a greater depth to the feedback that was provided during the post-workshop surveys. We
greatly appreciate your participation thus far. This will be the last time that we ask for your
involvement in our evaluation.
The information gathered today will be shared with SJE staff and be used to improve the quality
of PIE language workshop for future students. The feedback that you provide during this
sure to accurately capture everyones responses. This information will be private and will not be
specifically identified to you afterwards. Of course, there are no right or wrong answers, and
wed like to hear from everyone if possible.
workshop
Probe: What were some of the reasons that led you to sign up for an
intersectionality workshop through PIE?
Probe: What were some meaningful takeaways from the workshop?
Goal Area 1: Participants are able to define the intersectionality of their target/ agent identities,
meaning defining what identities are oppressed and what identities gives them power and
privilege in our US society.
Question 1.1: Since completing the PIE workshop six months ago, have you been
able to reflect on your own target/agent identities in relation to the definition of
intersectionality?
o Probe: Did participation in the intersectionalty workshop influence or change
how you describe your target/agent identities to others?
o Probe: What are some of the challenges that have come with the self-
reflection of your target/agent identities in relation to intersectionality?
Goal Area 2: Participants are able to recognize that their target identities do not outweigh the
power of their privilege identities, but that all of their identities intersect together.
Question 2.1: Over the last six months, have you used your privileged identities as a
tool to combat systems of oppression on campus?
o Probe: Are there times when this has been easier to do?
o Probe: What have been some of the challenges in utilizing your privileged
identities to raise awareness about the marginalization of target identity
groups?
Goal Area 3: Participants are able to provide detailed descriptions of what solidarity would look
like amongst their peers at Northwestern University.
Question 3.1: How have you demonstrated allyship with other marginalized identity
groups on campus?
o Probe: Were you able to better define allyship as a result of the intersectionality
workshop?
o Probe: What spaces on campus, if any, have allowed you to practice allyship?
Question 3.2: What are some techniques or strategies related to solidarity that you
have implemented since your participation in the intersectionality workshop?
o Probe: Did you learn this from the intersectionality workshop? Or from a
different experience?
o Probe: Did the facilitator, or facilitators, provide a space where you were able
or changes that you would make to the intersectionality workshop.
Appendix M: Consent Form
CONSENT TO PARTICIPATE IN PEER INCLUSION EDUCATORS (PIE) IMPACT

FOCUS GROUP
Project Title: Peer Inclusion Educators (PIE) Impact Focus Group
Evaluators: Anthony Sis and Lorrena Johnson
Introduction:
You are being asked to participate in a focus group regarding your experience outside of the Peer
Inclusion Educators (PIE) workshop. Because you completed a post-workshop survey and
expressed interest in following up regarding your experience on campus, we are interested in
hearing more about your thoughts and experience.
Please read this form carefully and ask any questions you may have as you decide to participate
in this interview.
Purpose:
The purpose of this interview is to gather information about students post-PIE workshop
experience at Northwestern University to better understand the effectiveness of the program.
The information generated from this interview will be used to improve the content of the three
PIE workshops offered (Social Justice Key Concepts, Social Identity, Intersectionality).
Risks/Benefits:
There are no foreseeable risks involved in participating in this research beyond those
experienced in everyday life. A key benefits will be the potential improvement of the PIE
program for future participants.
Confidentiality:
There will be no identifying information attached to your responses in the interview. A report
will be compiled of basic themes that emerge from the interviews. The information will be
shared with the Social Justice Education (SJE) staff and staff from Campus Inclusion and
Community (CIC).
Voluntary Participation:
Participation in this interview is voluntary. If you do not want to take part, simply decline
participation. If at any time you feel uncomfortable answering a question, you may remove
yourself and/or decide not to answer the question.
Contacts/Questions:
If you have any questions about the interview, please contact Noor Ali at 847-491-8063 or
Noor.Ali@Northwestern.edumailto:Noor.Ali@Northwestern.edu
mailto:Noor.Ali@Northwestern.edu
Statement of Consent:
Your signature below indicates that you have read and understood the information provided in
this document, have had an opportunity to ask questions, and agree to participate in this
interview.
You will be given a copy of this form for your records.
_______________________________________________ ___________________
Participants Signature Date
_______________________________________________ ___________________
Evaluators Signature Date
Appendix N: Protocol Invitations
Invitation request if at least ten people have not expressed interest in focus group
Re: Invitation to Join a PIE Impact Research Discussion
Thank you for attending a PIE workshop this academic year. Were interested in learning about
the outcomes of the PIE workshop you attended on (Language Activity/ Social Identity/
Intersectionality). Because of your participation, we are looking to hear back from you about
your experience outside of PIE workshop on campus. We hope that you will want to provide
your thoughts.
The focus groups will be conducted over the next few weeks by a graduate assistant, assistant
director, or director of Social Justice Education (SJE) with additional support from Center for
Inclusion and Community (CIC) staff. The format of the focus group is to include no more than
ten previous student participants and should take no longer than ninety minutes to complete.
They will be scheduled from 12:00 1:00 P.M. or from 4:00- 5:00 P.M., depending on the
availability of the participants. Light snacks and beverages will be provided.
Please let me know if you are interested in joining this interview by replying to this invitation. I
will work with those who have agreed to participate to schedule a time to meet for the focus
group over the next few weeks.
Thank you in advance for agreeing to participate in our interview. If you have any questions,
please call or email me.
Noor A. Ali, M.S.W. (she, her, hers)

Assistant Director, Social Justice Education
Campus Inclusion and Community
Noor.Ali@Northwestern.edu
847-491-8063
Scott Hall 046 (Lower Level)

601 University Place
Evanston, IL, 60208
Appendix O: Protocol Invitations (cont.)
Confirmation Email
Subj: Confirmation: PIE Impact Research Discussion on [insert date/time]
Hello [insert name of previous participant],
This email confirms that you have agreed to attend the Peer Inclusion Educators (PIE) impact
research focus group on [date/time/location]. Light snacks and beverages will be provided.
Please reply to this email if you are no longer able to attend. We look forward to your
participation!

847-491-8063

Evanston, IL, 60208
Reminder Email
Subj: Reminder: PIE Impact Research Discussion on [insert date/time]
Hello [insert name of previous participant],
This email is a reminder that you have agreed to attend the Peer Inclusion Educators (PIE)
impact research focus group on [date/time/location]. Light snacks and beverages will be
provided. Please reply to this email if you are no longer able to attend. We look forward to your
participation!

847-491-8063

Evanston, IL, 60208

Elps431 Final Report Pie

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Elps431 Final Report Pie

Uploaded by

Copyright:

Available Formats

Running head: PROGRAM EVALUATION NORTHWESTERN 1

Anthony Sis and Lorrena Johnson

Loyola University Chicago

university states that Northwestern University is committed to excellent teaching, innovative

Sustained Dialogue program.

affairs it provides a framework based on three components education, enrichment, and

Northwestern through leadership and education programming providing opportunities for

be included in our formal assessment and evaluation of the PIE program.

intersectionality of each participant privileged and oppressed identities. This is completed by

student groups on Northwesterns campus.

workshop option is inclusive language. This facilitation workshop focuses on implementing

specifically at Northwestern University. Student groups have the opportunity to create a

Selection of Peer Facilitators

http://www.northwestern.edu/socialjustice. The questions on the website include why an

they can improve.

(2016), which include:

Allow for reflection on ones self , backgrounds, and biases.

Encourage dialogue around issues of difference and to develop competent mechanisms

for addressing these issues.

Logic Model Overview

and impact (short-term, medium-term, and long-term outcomes).

graduate/professional students, faculty, and staff who attend a PIE workshop.

assessment and evaluation is needed.

General Evaluation Approach

Our general evaluation approach will be primarily outcomes based. An evaluation is

considered to be formative if the primary purpose is to provide information for program

can improve their skills.

As evaluators we decided a formative assessment on PIE would be beneficial to the data

throughout this implementation of this evaluation. This evaluation is a processed based

items that participants will complete.

for the program.

post-surveys (see Appendices F-I).

flow/instructions are clear to the respondents.

based on our short term outcomes for the program.

Statistical Analysis/ Results Presentation

students found to be more beneficial or if participants found all workshops to be equally

because of the goals for each workshop (see Appendices J-L).

the following 2018-2019 school year

close to the whole population.

even if they want to participate in multiple workshops.

Focus Group Plan

workshop with different participants in each focus group.

Focus Group Implementation

in different rooms across campus.

Focus Group Analysis

as it can be for participants.

Another limitation is acknowledged in our ability to create an effective long-term

outcomes-based assessment. We envisioned this assessment to be distributed via email to all of

the PIE workshops.

About: Northwestern university. (2016). Retrieved from http://www.northwestern.edu/.

Campus Inclusion & Community. (2016). Division, Campus & Departmental

Information.Unpublished internal document.

and practical guidelines (4th ed.) New York, NY: Longman.

Multicultural Center space.The Daily Northwestern. Retrieved from

Social justice education: Peer inclusion educators. (2016). Retrieved from

Tourangeau, R., Rips, L. J., &Rasinski, K. (2000). The psychology of survey

response.Cambridge University Press.

No consistent 2 Assistant Train student Undergraduate

Appendix B: Previous Evaluation

Workshop topic: Program/Organization:

If yes, how many and what topics?

14.Which component of the workshop was most beneficial to you? Why?

15.Which component of the workshop was least beneficial to you? Why?

Age: Citizenship/Nation of Origin: ____________________________

Age: Citizenship/Nation of Origin: ____________________________