Custom eLearning Solution: How To Measure And Analyze Its ROI
“That was the best program we’ve ever had!” Ken went on to use words like “creative”, “engaging” and “relevant”.
It feels great to have a win like this, so go ahead and celebrate… but proceed with caution. A participant’s excitement about an eLearning experience is certainly important, but it may not affect the bottom line like you were hoping. If three months from now, Ken’s coach hasn’t noticed a difference in Ken’s confidence and independence in using the content from your custom eLearning, your win may fall short.
1. Measure Success In 5 Levels
So, what does success look like? Consider a full picture evaluation in 5 levels, which is an expansion of Kirkpatrick’s famous model. With ROI, you look at quantifiable data, but with 5 levels of evaluation, you consider both quantitative and qualitative data. Learning can have a profound impact on culture, job satisfaction and employee confidence, which are all valuable assets. While they may not immediately affect business metrics, they improve company health and safeguard against increased turnover in the long run. So, in addition to monitoring business performance data, it’s a good idea to track qualitative data, too.
With 5 levels of evaluation, you’ll see exactly where learning excelled, and your celebration won’t be cut short. Let’s take a closer look. Below you’ll see each of the levels with a few questions about what can be measured in that level.
Level 1: Learner Experience
- How did learners feel about the opportunity?
- Did the experience match learner expectations based on internal marketing?
- Was the material relevant, interesting, and useful?
Level 2: Learning
- For each learning objective, did participants gain the knowledge, skills, and values identified?
- Did learners achieve targeted assessment scores?
- Were assessment strategies effective in measuring results?
Level 3: Application
- Do learners know how to apply what they learned?
- How many times did they successfully make application during the first 30 days?
- What hindered them?
- What supported them?
Level 4: Business Impact
- How did the implementation of the program affect business metrics such as sales, productivity, operating costs, errors, job engagement, employee retention, job promotion or other targeted measures?
- What cost savings have you tracked?
- What is the reduced skills gap or increased employee proficiency?
Level 5: Return On Learning
- Do the monetary benefits of the improvement in business impact and culture measures outweigh the cost of the learning program?
- What other benefits are important to include? (For example, job satisfaction, collaboration, creativity, ease of use, relevance to the role, and customization)
2. Create A Data Collection Strategy
“But when we’ve evaluated programs in the past, we couldn’t seem to move beyond anecdotal evidence. We just didn’t see the validation we wanted.”
The success of an eLearning module depends not only on whom you ask but when. If you wait to start planning your analysis until after you’ve rolled out your program, you may not have access to the data you need. As early in scoping the project as possible, determine what numbers are required to document success. How is your learner currently performing? What gains do you hope to reach? What metrics will verify the improvement? Being able to compare before-and-after data is critical to tracking ROI.
That’s why you should use a data collection strategy with every large project. As part of your scoping process, create objectives for each of the 5 levels. At this beginning stage, determine:
- What success looks like
- Which existing measurements to use
- What new information to collect
- How to collect it
- From whom
- When
Data comes in many forms, so what kind of data should you collect? An in-course questionnaire or quiz, skills-based testing following eLearning, a performance contract, and business metrics monitoring can be the designated tools from which to figure ROI. The most relevant data is often dictated by the content and the constraints of the project. And timing is important, too. Some application won’t be felt until one or two months (possibly longer) after the learning opportunity. So, it’s important to decide when data would best be gathered.
If results are important to you, then a data collection strategy is the way to go. It will guide you as you gather and sort the right kind of data to verify your success.
3. Tell A Great Story
“I was pretty happy with the results of our learning initiative, but that’s what we expected. Why take the time to make a big deal out of it? We’re all busy.”
Did I say, “Tell a story?” I should have said, “Throw a party!”. As strange as it seems, this is another area where companies may fall short. They develop an engaging eLearning program, track both quantitative and qualitative results, only to prepare a report that gets filed before anyone even reads it. No sharing in the success. No boost to morale. No friendly competition between divisions. No incentive to work hard and achieve new goals. No inspiration.
Of course, you want all these amazing results from effective eLearning, so along with your ROI and quantifiable results, be sure to tell a good story. Your story should show the obstacles encountered, (including mental hurdles) and how they were overcome to gain results. It should inspire others and give them something to take away, showcasing the successes without compromising authenticity. And when it comes to custom eLearning, there is usually something to celebrate, even if the ROI was less than desirable. Here’s what to look for:
- Team Building Stories
Prompting significant conversation among learners - Flexible Execution Stories
No travel costs, unlimited learners, personalized pace, and schedule - Customized Learning Stories
Core values reinforced, custom scenarios and problem-solving, stratified entry for learners - Standardized Content Stories
A unified message that eliminates confusion, company-wide vocabulary and process - Business Impact Stories
Changes in productivity, error reduction, job engagement, promotion rate
To celebrate well, you’ll want to do it right away and make it public. Plan to spread the news as soon as the ROI results are compiled, and your story is created. The kind of celebration should be in scope with the size of the eLearning program: the bigger the program, the bigger the celebration (and don’t skimp here). Celebrations of learning help build momentum and inspire new creativity.
Evaluating in 5 levels, building and completing your data collection, and telling a great story will provide a full analysis of your next eLearning solution. From beginning to end, you will know how well your custom eLearning succeeded and the impact it made. Using 5 levels of evaluation and both qualitative and quantitative data could be a lifesaver as you plan your next awesome eLearning opportunities. We all have an innate drive for curiosity and learning. Learning sets us apart, inspires us, and shapes us into better versions of ourselves. Now that’s a story worth celebrating!
Are you an L&D professional looking forward to adopting great corporate training practices which employees will find more than useful? Download our eBook Providing An Amazing Custom eLearning Solution For Corporate Training to discover when companies should invest in custom eLearning, the factors to determine the appropriate level of interactivity needed, the stages of the custom eLearning development process, and much more...
There are many ways to dive deep into custom eLearning development processes as well as examine all the data. Want to learn how to truly measure and analyze the ROI of your custom eLearning? Learn more about it by checking the webinar Developing Custom eLearning For Corporate Training. This webinar will be led by experts who have led some of the largest eLearning projects in the world.
