5 Levels of Data for

Measuring Educational Impact

The 5 Essentials to Understanding What Works and Why

As I’ve worked with educational organizations exploring different avenues with technology, one of the struggles
I’ve observed again and again is figuring out how to measure the success of adoption. This is true with any new
initiative, whether you’re in K-12 or higher education. But it seems to be especially true with technology.

The challenges are many. If the end goal is to impact student achievement (and really, whose initiative isn’t?),
how can you measure the relationship between integration and student growth or student achievement or
changes in teacher practice? In my previous district, the myriad initiatives happening concurrently with our
technology initiatives made correlating to success a difficult endeavor, as was identifying what types of data
would be most revealing and helpful about our rollout.

Measuring Educational Impact with the 5 Levels of Data

Thomas Guskey’s article Gauge Impact With 5 Levels of Data appears as a “thought piece” in Learning Forward’s
Reach the Highest Standard in Professional Learning: Data. This is one of the published guides for each of their 9
Professional Learning Standards.

Like all work from Learning Forward, the focus is on professional learning. In my perspective, if you are
implementing technology, you are basically exploring professional change and growth. And that means looking
at professional learning and how you might measure the impact that it is making in the classroom.

schoology.com 1 800-749-2590
 Each of the levels described in Guskey’s article addresses a different level of data that “represent an adaptation of
an evaluation model developed by Kirkpatrick (1959, 1998),” which was primarily used in business and industry.
And while each of the areas builds upon the other, all make up a larger data picture when it comes to measuring
impact of professional growth.

5 Levels of Data

Level 1: Level 2: Level 3: Level 4: Level 5:

Participant Participant Organizational New Knowledge Student Learning
Reaction Learning Support/Change /Skills Outcomes

PD Survey Analytics: Analytics: Teachers Analytics:

Skill Assessment
Organization and Groups Student Use

Experience Classroom
Skill Demonstration Structured Interviews Teacher Portfolios
Questionnaires Assessments

Walk-throughs
Reflection Follow-up Surveys Focus Groups
or Observations

Adapted from “Gauge Impact with 5 Levels of Data” by Thomas Guskey in Journal of Staff Development, February 2016 (Vol. 37, #1, p. 32-37)

The above chart is something I recently shared with a district who was just starting their implementation of
Schoology’s LMS. For those familiar with Thinking Maps, this Tree Map classifies the different ways you can collect
data for each level from Guskey’s article (which I customized for their implementation).

What’s most important to recognize about this graphic, though, is the backwards planning indicated by the
left-pointing arrows in the graphic. “The most effective professional learning planning,” explains Guskey, “begins
with clear specification of the student learning outcomes to be achieved and the sources of data that best reflect
those outcomes. With those goals articulated, school leaders and teachers then work backward.”

Putting Backwards Planning and the 5 Levels of Data into Practice

If you are implementing something new and want a way to gather data along the way, using these different levels
as you plan your implementation could be very helpful. Too often, we give surveys (Level 1 Data) after a workshop
or PD experience but don’t add the other layers on top. Or, we gather feedback at the end of a process, only to
realize that we could have prevented missteps along the way by looking at the feedback provided by different
layers of data.

schoology.com 2 800-749-2590
 Were I to have the opportunity to redo our own implementation of Schoology’s LMS at my former district, I would
have planned our data measurement around these levels over a multi-year period. In retrospect, I could have:

1. Created specific learning outcomes that we expected to see in classrooms as a result of our LMS implementation
before rolling it out. We had a general idea, but having outcomes identified would have made our work much
more intentional. (Level 5)

2. Identified analytics that would have been most helpful to explore in the system, as well as defining
a walk-through protocol that could capture the application of the knowledge and skills for using the
LMS. (Level 4)

3. Created interview questions and follow-up surveys that we could share with leadership at buildings
where we saw high and low adoption, as well as looking at the user analytics of the school. (Level 3)

4. Designed a process for gathering reflections and demonstrations of learning addressed in the PD
experience (and solidifying how we would provide regular feedback to teachers and leaders). (Level 2)

5. Construct PD participant surveys that would not serve as the final measure of learning but as the first
step in the ladder. (Level 1)

Measuring Impact is as Much About Where You’ve Been

as it is Where You’re Going

It’s critically important to have some sort of process to measure impact when trying to shift the behaviors, beliefs,
and practices in the classroom. What I especially like about Guskey’s approach is its clear path to both defining
goals and evaluating against them, not as a punitive measure for teachers but an enlightening measure for
leadership and future implementations.

Having a clear picture of how the different threads of data weave together the larger fabric can help us be both
intentional about how we gauge impact and also be in a better position to replicate success and avoid pitfalls in
adoption.

For more information on this process, check out the sister article to this one, The Vital Importance of Vision: How
to Lead Lasting Change in Education.

Watch this short video walkthrough to see an LMS About the Author

designed to help you measure and magnify the impact

of your initiatives.

Kellie Ady
WATCH THE VIDEO Director of Instructional Strategy
Schoology

schoology.com 3 800-749-2590