Practical Tools to Measure Organizational Effectiveness

Mark Morgan & Michael Abrams - 7/28/06 CLOs should have a vested interest in ensuring that the impact of learning can be measured effectively and efficiently. This requires an infrastructure linking training to skills, skills to behaviors and behaviors to results. We re all familiar with the classic questions we need to answer to measure the impact of training. Donald Kirkpatrick laid them out for us in the mid-1970s, and they haven t changed since. Level 1: Did the trainees like the training? Level 2: Did the trainees learn what we wanted them to learn? Level 3: Did the trainees put new skills learned to use? Level 4: Did the training have the desired impact on organizational results? Most organizations have a good grip on measuring at Levels 1 and 2. But evaluating at Levels 3 and 4 is hard, almost impossible, without organizational buy-in, so if your efforts are falling somewhere short of optimal, it might not be (entirely) your fault. But it s still your job, and you ll need the infrastructure in place to support it. We ve all heard it, and we all know it in our bones: Training and development efforts serve the overarching strategic goals of the organization. Resources dedicated to these efforts must produce a return that compares favorably to the return to be obtained from other uses. Yet when we attempt to measure the effectiveness of our efforts, we often act as if the organization ends at the boundaries of our function. We ve done the training. The trainees learned what we wanted them to learn. We re effective. All too often, evaluation stops here, at Level 2, which is rather unfortunate because it s at Level 4 that the training function demonstrates the return it produces. This means that the CLO has a vested interest in ensuring both that the impact of training carries through to Level 4, and that the impact can be measured practically, effectively and efficiently. Doing so requires an appropriate infrastructure linking training to skills, skills to behaviors and behaviors to results. The good news is this: Your organization might already have much of the infrastructure in place. Prerequisites for Measurement Before you can begin to measure the impact of training, you must have clearly defined goals. Precisely what are you hoping to achieve? At Level 1, that s relatively easy, because the goals don t change: Regardless of the specifics of the training, you want people to enjoy it, you want them to be engaged and you want them to feel that the time and effort expended was worthwhile. A simple subjective post-training questionnaire suffices to measure this. At Level 2, things get slightly more complicated. We need to define, before developing the training, what it is that we want to accomplish. We want to enhance the skills and knowledge that underpin performance. Doing this requires defining the competencies we want to develop, defining appropriate

measures of those skills, and either conducting pre- and post-testing or (even better) the use of an untrained (but equivalent) control group. Measuring the impact of training at Level 3 is still more complicated. Now we need to show that our trainees actually used the competencies we targeted. These are real-world behaviors that often occur over an extended period of time and are often out of our direct line of sight. We have total reliance on others to assess the extent to which the behaviors were put to use. Moreover, we need to have pre-training measures for comparison with our post-training measures. One way to accomplish this is to perform a one-off piece of research for each effort: develop a measurement tool specific to the competencies we hope to develop in training, identify (as best we can) those people that will be involved in future training, distribute the tool to those who are in a position to rate the extent to which those who will be involved in the training currently demonstrate those competencies (for our pre-training measure), follow up to ensure that ratings are actually performed, collect the ratings, conduct the training, re-distribute the evaluation tool, follow up again to ensure that ratings are actually performed, collect the post-training ratings and compare the pre-post test. Alternatively, we can ensure that our training is designed to impact critical competencies, conduct the training on those who need development in specifically those competencies, and compare routine measurements of how much those competencies are demonstrated taken preand post-training. There s really no comparison, is there? Assessing the impact of our efforts at Level 4 is no more complicated (from our perspective) than assessing it at Level 3. We ve developed the critical competencies, the competencies have been put to use, so presumably results improved. But this is where having identified the right competencies is crucially important. No one denies that behaviors impact results, but which ones? Identifying the critical competencies often falls to human resources, but it should be clear that the learning function has both a vested interest in the effectiveness of the competency identification process and a role to play in specifying the competencies themselves. Obviously, the competencies need to be correctly identified. Each one needs to be causally linked to the level of performance of some strategically important task. For learning s purposes, however, performance on these tasks needs to map to a metric number of closed sales, percent of projects completed on schedule that can tell us whether competencies are actually linked to performance. This is where the learning officer s need to evaluate the impact of training has an additional positive consequence. It serves to enforce quantitative rigor on the entire competency assessment scheme. If competencies have not been defined appropriately, learning will know, and it will be in learning s interest to see that the problem is corrected.

Developing the Level 3-4 Measurement Infrastructure The measurement infrastructure relies on a competency assessment that has certain distinguishing characteristics: y It must begin with identification of the strategic needs of the organization. y The strategic purpose of each role within the organization must be defined. y A job charter must be developed outlining accountabilities for each role. y Competencies that underpin the ability to accomplish identified accountabilities must be correctly identified. y Metrics that allow measurement of the relationship between the demonstration of competencies and the accomplishment of accountabilities must be in place. y Competency assessment must be a routine part of organizational life. y Training must be mapped to competencies, and the purpose of training should be the development of precisely those competencies that are weak. Does this describe the situation in your organization? If so, then you re in luck many important pieces of the measurement infrastructure are already in place (though they might still require some modification to allow their use). If not, then you have some work to do, both as an advocate and as an active player in bringing about change. First, you ll need to build an understanding within the organization of the way that training, competencies, accountabilities, roles and organizational strategy are related to one another. Then you ll need to convince key people that there is a reason to get on board with the idea. The convincing needs to go on at the senior level: All the C s need to be on board, as well as the head of human resources, wherever he or she sits in the organization. Once you ve got buy-in, the real work begins. Defining roles within the organization can be time-consuming and difficult work, and specifying appropriate accountabilities, competencies that support the successful discharge of those accountabilities and the appropriate metric for each is even more so. Mapping each competency to an appropriate training or development module (to say nothing of actually developing those modules), is even more work. And even then, you aren t done. Competency evaluation is related to performance evaluation, but it isn t the same thing. The learning function will need to take the lead in ensuring that everyone, evaluator and evaluated alike, understands the point of the exercise, grasps its function within the organization and knows how to execute the evaluation effectively. It is a lot of work, but it can be done, and in the long run it will improve the effectiveness of training and development of personnel decisions and of the organization as a whole. And that s why we re here, isn t it? Using the Measurement Infrastructure Now that you ve put the infrastructure in place, you have a very powerful tool for identifying development needs. Routine development reviews allow managers to pinpoint those competencies that are weak and in need of development in order for each individual to perform at the highest possible level.

This gets you away from the sheep dip method of training, dragging everyone who might possibly use the skills (all those in a given department, or all those in a particular role, perhaps) into the training and development program and allows you to focus on those who actually need the help. In addition, having a hard-wired method for identifying training and development needs in various competencies allows you to prioritize these needs and to focus curriculum development efforts on those areas in which they will have the greatest impact. With the infrastructure in place, you have immediate access to the development needs of the entire organization. The payoff to the organization can be enormous. Measurement at Level 3 could not be easier. You have pre-training measures in place for everyone. They are the scores on the development review that triggered engagement in the development activity in the first place. Post-training measures are also hard-wired into the system. Because competency evaluation is routine, you simply collect the post-training scores after the next review. Comparison of the pre- and post-training scores provides a useful measure of the Level 3 impact of your efforts. Comparison of the change in scores for those who underwent training to the change for an equivalent group who did not is the gold standard. Assessment of the impact at Level 4 also falls into place with little difficulty. Assessment of the attainment of organizationally important outcomes (accountabilities) associated with the competencies that were developed in training is also part of the routine, and these assessments should show: y That having well-developed competencies really does lead to achieving the accountabilities. y That improving those areas that are weak (through training and development activities) really does translate to improved performance. Returning to the importance of correctly specifying the competencies, it should be clear that Level 4 measurement will only show an impact if, in fact, there is a causal link between the specified competencies and achievement of the accountabilities. What if the competencies aren t correctly specified? Fortunately, the system is robust in the face of this potential problem, because learning has its eye on this outcome. We know when competencies improve, and we re looking for the change in performance. If we don t get it, we know where the link has been broken: It must be that the competencies are not correctly specified. At that point, we need to take action, because hiring and development are based on the presumption that the competencies are causally related to the outcomes, and if they aren t then we re hiring the wrong people and training them the wrong way. But consider the alternative situation, in which we don t have the infrastructure in place, and in which we don t have a way to assess the Level 3 and Level 4 impacts of training with any accuracy. In this case, we still aren t necessarily hiring the best people for the job, and we still aren t providing the appropriate training. We just don t know it, and so we can t fix it.

For those of us charged with maximizing the value of human capital to the organization, could there be a worse outcome? Measurement at Level 4 is Crucial If the point hasn t come across clearly enough yet, let us reiterate that accurate assessment of the impact of training and development efforts at Level 4 is the most important piece of the whole puzzle. It is there that you see the true impact of your efforts to increase the organization s stock of human capital, and it is there that learning justifies its budget. Too often, the learning function s value proposition is founded on faith, with the result that its budget is set using two parts guesswork, one part voodoo. Everyone knows it, and this makes it a favorite target when revenues fall short of expectations. Development of a measurement infrastructure that allows assessment at Level 4 is not, therefore, merely a functional necessity. It is also learning s insurance policy.
Mark Morgan is a business analyst and Michael N. Abrams is managing partner of Numerof & Associates Inc. (NAI), a strategic management consulting firm located in St. Louis. They can be reached at mmorgan@clomedia.com.