How Effective Are Your Learning Experiences?
Companies investing in online learning often don’t have the tools or practices to effectively evaluate their learning experiences. They tend to focus on shallow metrics such as course completion or the number of questions answered correctly in activities. While students may complete a course, there’s no telling if there’s behavior change, which should be the goal of the learning experience, and without the full picture of the impact of the learning experience, there’s no knowing its return on investment.
Successful learning experiences should increase skills and competencies in employees and teams, as well as contribute to an organizational learning culture, all while increasing revenue and decreasing costs. Measuring completion rates and final grades, particularly in self-directed online courses, will only result in poorly perceived adoption rates and in all likelihood, waning support from budget holders.
So, how do you measure the impact of learning?
This is where we can draw on the principles of monitoring and evaluation and a few well-known learning evaluation models.
Monitoring And Evaluation
Monitoring and evaluation is an approach predominantly used to measure the effectiveness of programs or initiatives in the development sector with the intention of having maximum impact. The principles of this approach can also be translated to measuring online learning experiences. Here are a number of ways you can apply M&E principles to learning:
- Set clear objectives and goals for what success looks like and which criteria to measure the program’s progress from beginning to end.
- Collect and analyze data at the beginning of the program to serve as your baseline. From then, regularly collect data relevant to the program’s goals and that indicates the level of its effectiveness.
- Upon analysis of the data, make changes and iterate throughout the program implementation to improve the program’s impact.
- At the end of the program, compare your findings with the data collected at the beginning of the program to determine whether there is a change in behaviors or competencies and what can be improved going forward.
With a monitoring and evaluation mindset, you’ll be able to more effectively implement and design criteria for evaluating learning effectiveness.
What Are Learning Evaluations?
Learning evaluations are the criteria you will analyze to determine the success of your learning experiences. There are a few theorists who have developed different learning evaluation models, which suggest key evaluation criteria relating to the micro-level (the learning experience itself) as well as the macro-level (its impact on your business and even society). For example, Kirkpatrick’s model founded by training expert Donald Kirkpatrick indicates that a learning experience should be evaluated across the following four levels: reaction, learning, behavior, results.
Roger Kaufman, an expert in evaluation and needs assessment, expanded upon Kirkpatrick’s model to include an evaluation of a learning experiences’ societal impact in what is termed Kaufman’s Five Levels of Evaluation.
What To Evaluate?
Knowing your general evaluation criteria introduced above is a good start, but you also need to identify the specific metrics that indicate whether learning was successful or not.
Begin with tracking the achievement of the learning objectives. At the start of the learning design process, you should have decided on objectives and competencies that you would like learners to develop through the learning experience. These not only help to clarify learning and motivate learners toward specific goals but are also useful for measuring whether the learning experience achieved the objectives or not.
Other metrics for evaluation include:
- Conduct assessments
Assess your employees’ capabilities and competencies before and after the training through simple baseline surveys and knowledge checks to determine whether there has been any development. - Track engagement
Tracking course completion, the grades achieved on individual assessments, the course as a whole, as well as learners' engagement on the course through clicks, time spent, contribution to discussion forums, and liking or sharing of content will help determine the success of your course. - Measure results
Observing an increase or decrease in grades and determining what prompted this change will assist in understanding whether the content and your learners' experience of the content has been impactful and whether there are knowledge gaps based on their understanding or on course design. - Analyze data
Analyzing data from the leaderboards as well as course levels, badges, and rewards achieved in gamified learning experiences also contributes to the measurement of the impact of a course.
Using digital data, through xAPI, is also a good indicator to note how learners interact with your course and other online platforms in your organization. xAPI, or experience application program interface, allows you to track and view how learners use your course, what they click, how much time is spent on which activities or content, what was revisited, and so forth. This data is invaluable when analyzing the success of individual components of your course. In the case of informal workplace learning where employees are largely self-led in their learning, xAPI can also track engagement with different browser tabs and applications, giving you a sense of the employees' commitment to continual learning.
Conclusion
Ultimately, having an understanding of the various learning evaluation models and identifying useful metrics is the first step toward the effective evaluation of your online learning experiences. Coupling this with understanding your learning analytics will result in more enhanced and accurate learning return on investment and more improved learning experiences for your learners.
