Learning Measurement: The Key To Success
So, why is learning measurement so important?
Most people have probably heard the old adage “measure twice, cut once”. I’ve been thinking about that a lot lately as I have a carpenter in my house working on a remodel. It’s a carpenter’s mantra that can be applied to almost every type of work and, like most, I’ve had to learn the hard way on project after project to always take a few extra breaths to measure before I start. What is so amazing is that such a simple common-sense approach is often far from the norm in many organizations when it comes to Learning and Development; not because of a lack of desire, but because tools and systems haven’t made it easy to access and correlate data in order to measure.
The first step is to figure out what to measure. A carpenter wouldn’t count the number of nails that he has driven to determine if a project is a success, so should we count the number of courses or even the volume of informal training that is consumed to determine success? Shouldn’t the only determination of success be a remodeled house on time and budget or a workforce that achieves its business goals?
Make no mistake, artificial intelligence is coming and will help us implement measurement in ways that are hard to imagine now. Our efforts in trying to correlate causation will be greatly enhanced by xAPI and AI. But even when we have an IBM Watson’s computing power in our reach, we will still need to decide ourselves what really makes sense to measure, and the answer will be different for every organization.
The Learning Measurement Process
In the Towards Maturity 2015 Benchmark™ research conducted with over 600 Learning and Development leaders and reported in Embracing Change, it’s considered how Learning and Development leaders are delivering business and staff impact and training efficiency, together with the tactics they are using to deliver results.
The graph below, taken from the report, shows that those using technology to record learning activity and performance improvement are performing far better than the others.
Full report available here.
The results of those that involve line management in the learning process also show that, across 500 companies surveyed, there is a x3 improved effect. So who doesn't want to make their learning programs three times more effective?
A broad strategy of learning measurement and the evaluation of learning success should be considered for every project or program you want to run; an approach that applies equally to a traditional learning event, a blend or learning architecture, or even what might be called a learning campaign.
Put simply, we need to know and agree how we are going to measure the following:
1. Has The Learning Landed?
Has the learning reached its intended audience? And, more importantly, did the learner connect with it? It’s not enough to know that a learner has opened a piece of learning or attended a workshop. How did they react? Did they engage? We need to know whether the channel was successful in reaching the learner in the way that was intended, regardless of any lasting learning change it may or may not have delivered.
2. Have Learner Behaviors Changed?
In the first instance, has the learning been effective in driving a change in the learner? Did they learn anything? Secondly and more importantly, have they applied the learning successfully through new or different behaviors? To be really useful, this behavioral change needs to be seen or measured at the moment of application, and in the real circumstances in which it’s needed.
3. What Is The Business Impact?
To what extent has the learning made the impact on the business that was being aimed for? This can be measured at a team level or group level, but most usefully it will be reflective of an organizational shift and improvement, which is why it is often so challenging to measure unless it is associated with an existing KPI. A key issue here is whether we are able to track the changes back to the learning in a meaningful way.
A final further step, associated with business impact, comes right back to the ROI question:
4. How Might The Organizational Change Have Resulted In Increased Financial Return?
Asking this question can open up a number of important chains of thought and discussion at the start of any project.
Starting With The Results
At the heart of any measurement strategy –whether measuring engagement, behavior shift or business results– must be the desire to view a shift from one point to the next.
Despite this, learning measurement strategies are often put in place only after the learning has been designed, or even after it has been delivered. Not only does this miss the opportunity for clean results, it also fails to take advantage of the most important moment in the design process.
Even on the smallest project, the discussion and agreement by a group of stakeholders of the measurable objectives of a program at the very start will feed critically into the early analysis and strategic design phases.
Learning is changing. Learning is now often delivered as a complex blend including informal, networked, or continuous learning strategies, but it’s worth pointing out that nearly all learning is based on an event or a series of events – a learning journey. For this reason, measurement and evaluation are always a series of snapshots, and where these can be focused around specific events or steps in a journey, they can give us a clearer view of where our successes are.
A learning measurement process needs to be simple if it’s going to be used and applied across a range of programs. As an example, one approach is shown in the diagram below.
The world is full of bad data science. We see examples of it everywhere with the drawing of wrong conclusions and misreading the data. Look at the recent U.S. election and referendum in the U.K. The experts were wrong in their predictions, not because they didn't consider enormous amounts of data, but because the science applied to that data was not sound. The point here is that intelligent data gathering is required to get this right, as is an awareness that landscapes are constantly changing and we must therefore constantly evolve what we measure and how we measure.
Want to learn more? Download a free LEO Insight, “Impact, Results, and Measuring Learning Success”.
