Everything L&D Leaders Need To Know About Learning Data Analytics
John Cleave holds a Master's in Education and a PhD in Computer Science from Northwestern University. He has over 30 years of experience designing and developing learning solutions for a variety of clients, from Fortune 500 companies to nonprofits and academia. His areas of expertise include simulations, learning platforms, and learning analytics strategy.
Today, we're discussing learning data analytics in L&D. For more insight from John, catch SweetRush's hot new webinar Learning Analytics Improves The L&D Craft. Don't Just Design, Develop, And Deploy…Discover!
Most of us in L&D recognize the value of learning analytics data to help evaluate and shape learning design strategy, yet many don't do much of it. Why is that?
While many organizations are doing learning analytics, I believe five reasons factor into why it isn't practiced more extensively.
- It takes effort to create assessments (especially skill assessments), and when time and budget are tight, many would rather invest in the training itself than in measurement.
- Training is often developed via a waterfall method (analyze needs, design/develop training, deploy). Work then stops, the team moves on to the next step, and learning analytics offers little value since it isn't useful in going forward.
- An L&D leader may worry that, if learning analytics determines a course has no impact, it may reflect badly on them—no data means no judgment.
- Some L&D leaders and teams lack confidence in their skills to evaluate learning impact and so neglect to do it.
- It can be difficult to assess people's skills and knowledge, and a flawed analysis may cause more harm than good.
As a result of one or more of these factors, less data is collected on training impact than is possible; hence, little is typically learned. In my opinion, that's a loss, as we have only so many tries at the brass ring.
It's difficult to measure changes of learner behavior on the job or the impact of training (Kirkpatrick's Levels 3 and 4). Do you have ideas for how learning impact can be measured?
Gathering data on people's ability to perform on the job is tricky: manager assessments have some validity but miss things, and monitoring people's behavior on the job can be intrusive.
One way to address this challenge is to identify indicators that tell you whether learners are acquiring the knowledge and skills being targeted by training, then gather data on those indicators within the learning experience (LX) itself. For example, if you're creating sales training, indicators might include asking good questions, recognizing customer needs, formulating an optimal solution, presenting a product well, and addressing objections.
Once indicators are defined, you can then devise ways to capture learning analytics data around each indicator in situ, such as measuring salespeople's ability to ask questions, to identify needs, to devise solutions, and so on.
Plenty of data can be captured within a course, such as including a simulation or case study analysis to see what questions learners ask, how well they surface customer needs, and the solutions they recommend based on those needs. This data is a good guess at what people are able to do when they're back on the job, and can be more accurate than manager reports or self-reporting.
What do organizations often get wrong when it comes to learning analytics?
Probably the single biggest mistake is to not do it—that is, not to gather learning analytics data beyond whether some learner population has completed training. But when an organization does decide to measure impact, I think the biggest mistake is measuring the wrong thing. Many assessments focus on recall; that is, a learner's ability to regurgitate what they were told, because those are the easiest kinds of assessments to develop, and AI makes this surprisingly easy to do.
The problem with that is twofold:
- First, people who can recall information may not be able to apply it in real-world situations.
- Second, people who perform well on the job may not be able to recall information they were taught.
In either case, a person's performance on an assessment may have no correlation to their ability to perform in the real world, so the assessment has no predictive or diagnostic validity.
The only tool we have for collecting learning analytics data is our LMS. What can we do with it?
Though limited, LMSs do provide learning analytics data, and hence can offer insight. Virtually all LMSs record course status (Not Started, In Progress, and Completed) and score (if a course reports it). Both status and score can be useful. You need to be able to track status if you want to make any claim about a course's impact. For it to have had an effect, people had to have taken it.
Moreover, if there are optional courses on the LMS (ones that learners can enroll in on their own), then enrollments and status (how many complete it) can tell you which ones learners value. If demand is high, you may choose to create additional courses under the same topic.
Scores likewise can be informative if the assessments that create them are designed well. Resist the temptation to require people to keep re-answering every question until they achieve a passing score; instead, allow a lower passing score (say, 30%), then record the score as a general indicator of how well learners mastered the material.
Additionally, most LMSs allow you to conduct a comparative analysis of different populations, so you can compare scores by role, region, department, years of experience, and other characteristics, which may yield insight into who knows what.
Finally, some LMSs report on SCORM's cmi.interactions data, which Rise, Storyline, and other authoring tools will send to the LMS. This data provides you with insights on how learners answered individual questions in the eLearning, so you can, for example, identify which questions were most frequently answered correctly or incorrectly, helping you uncover a knowledge or skill gap to address.
How has the advent of AI in L&D impacted learning analytics?
AI feeds on data, so learning analytics data can be invaluable to it. For example, AI can adapt training content to individual learners, so knowing what learners have mastered or are struggling with can go into AI's decision-making of what to cover and how. Learning analytics data can also be provided alongside demographic information about individual learners (their role, responsibilities, and so on), which could also factor into AI's treatment of a topic.
Another area where AI could impact learning analytics is in evaluating learners' work products: have learners create a report or presentation, then feed it to AI to conduct an evaluation against a rubric.
AI is also great for pattern matching, so it may be able to glean insight from learning analytics data as to what learners have completed, how they answered questions, what they did in a simulation, and so on. For example, we've fed large numbers of xAPI statements into AI and asked it to build summary statistics, uncover interesting patterns of behavior, identify common mistakes, and other items we're curious about.
What do you think is the future of learning analytics, and how might it evolve and change in the coming years?
I believe that, in the future, learning analytics will be (a) collected more routinely, because it's getting easier to do, and AI is data-hungry; and (b) more powerful, since AI allows us to analyze volumes of data and detect patterns.
For example, AI allows us to generate simulations much more easily than in the past, and simulations offer a rich window into how learners are able to perform in real-world situations. If AI can simplify the process of distilling and interpreting the data, more L&D leaders will no doubt make use of it. And because knowing the impact of a learning experience helps us make the next one better, the increased use of learning analytics can only be a good thing for the L&D industry as a whole.
I need to convince my supervisor that it's worth investing time and effort to collect and analyze learning analytics data. What arguments can I use to get them on board?
Since it makes intuitive sense to measure the impact of a learning experience, I would probe more deeply into why your supervisor is resistant. As we've mentioned, one reason may be fear: if a course is produced and no impact from it can be shown, it might make a supervisor look bad.
If fear underlies their resistance, you can encourage them to use learning analytics formatively—that is, to collect data along the way in order to make adjustments, recognizing that L&D is a process. For example, one might do a pilot, then use the learning analytics data that results from it to make improvements to the course design.
Another reason for a supervisor's resistance can be due to concern that spending time on an assessment will take time away from building the content itself. In that case, you can argue that the assessment and the content go hand-in-hand—and, in fact, developing an assessment first can provide a True North for learning design.
A third reason for reluctance may be due to not having strong analytic and quantitative skills. In this case, you can promise to help them make sense of the data, or perhaps recommend some courses that can improve their ability to understand and interpret the data. You may even want to conduct an analysis on your own to demonstrate the value of the data.
One piece of advice: specify how you are going to use learning analytics data whenever you recommend collecting it. The value of its use can often justify the effort needed to collect and analyze it.
Wrapping Up
Thanks so much to John Cleave for sharing his expertise on fully leveraging the often untapped potential of learning data analytics. Are you ready to get serious about learning analytics? Reach out to the experts at SweetRush and start measuring what your training actually changes today.
