Principles of Effective

Evaluation
Prof Keithia Wilson
Academic Leader Student
Success & Retention

Importance of
Evaluation

Results in evidence-based practice

Helps us to identify the most effective
& strategic interventions (make a
difference or add value)
Leads to sustainable & enjoyable
practice
Provides accountability to School &
Group leadership
Contributes to scholarship of the FYE

Kirkpatrick (1994)
Model of Evaluation Overview
4 levels of outcome evaluation
1.
2.
3.
4.

Level 1 Reaction
Level 2 Learning
Level 3 Behaviour
Level 4 Results

Level 1 - Reaction
Measures participants reactions to the training program,
including:

reactions to the overall program (outcomes) e.g.,

* To what extent did you find the Orientation program
useful?

reactions to specific components of the program

(processes) e.g.,
* What aspects of Orientation did you MOST
APPRECIATE and found USEFUL?
* What aspects of Orientation did you LEAST
APPRECIATE and feel is MOST IN NEED OF
IMPROVEMENT?

Level 1 - Reaction

Important to include closed-ended items

including rating scales e.g.,
How effective was the Orientation program overall in helping
you to make a transition to university?
not at all effective
very effective
1
2
3
4
5
6
7
How useful was the initial information session on the
predictors of success in first year?
not at all useful
very useful
1
2
3
4
5
6
7

As well as open-ended items

What else would you have liked included in the
Orientation program today?

Level 1 - Reaction
Some FYE examples
Starting@Griffith Survey questions on
Orientation (retrospective student
survey)
School specific Orientation Program
evaluation at the end of O-Day & the
Common Time (ongoing Orientation
Program) student surveys, focus groups
Peer Mentoring Program final evaluation
by mentors & mentees

Level 2 - Learning
Measures what participants have learned from
involvement in the program
What is measured needs to relate to what
was covered in the program viz. learning
objectives
Typically covers knowledge, skills, or
attitudes
Needs to include both rating scales & openended questions
Can include self-report & tests of actual
knowledge

Level 2 - Learning
Some examples
1.
Student self-report
* I feel confident that I know what to do to be successful
academically
this semester
not at all confident
very confident
1
2
3
4
5
6
7
* What have you learned about the transitions you will need to
make as
a commencing student in order to be successful?
2. Student knowledge
* I know the name of my First Year Advisor
yes
no (circle one)
* I know the difference between a lecture and a tutorial (circle one)
Strongly Disagree | Disagree | Neutral | Agree | Strongly Agree

Level 2 - Learning
Some FYE examples
Starting@Griffith Survey data evaluates
learning outcomes in terms of knowledge &
attitudes (e.g., knowing where to go to get
help; perceptions of good teaching)
Outcome evaluation for Peer Mentees (What
are your 3 most important learnings from your
participation in the Peer Mentoring Program?)
First-Assessment-First-Feedback amber alert
intervention (What have you learned as a
result of your participation?)

Level 3 - Behaviour
Measures the transfer of knowledge, skills &
attitudes from the training context to in vivo
or real-life contexts (the classroom & study).
Evaluate both before & after the program if
practical
Use survey, focus groups, interviews with
students, mentors, staff (convenors,
sessionals)
Starting@Griffith Survey (Did you attend
Orientation? How much time are you
spending on study?)

Level 3 - Behaviour
Some FYE examples
Starting@Griffith survey data
Attendance at E-Day, O-Day &
Common Time/ASP Program
Attendance at lectures & tutes
Assessment submission rates
in courses

Level 4 - Results
Measures return-on-investment, or the
extent to which the program has produced
results.
Some FYE examples include
Hard student outcomes such as * pass rates in courses for assessment items
* GPA in semester 1, 2 & FY overall
* student retention into year 2
Soft student outcomes such as
* student satisfaction

Evaluation
Key question What difference do our FYO&E
interventions (e.g., Orientation, Peer
Mentoring, Attendance, First Assessment
Intervention etc.) make overall?
Program/Intervention1 Reactions2
Learning 3 Behaviour 4 Overall
Productivity/Result
Program/InterventionFeelKnowDowhat
is the result or difference overall?