One Take On A Formula For Adaptive Learning ROI
If you aren’t an adaptive learning zealot, there’s probably a burning question in your mind, how do we know that adaptive learning actually works? This is a valid question and worth exploring. It’s well and good to adopt adaptive learning, but if it doesn’t improve performance any better than what is currently being done, is it worth the trouble?
Show Me The Money
Let’s say we need managers to be better at coaching, so they can cultivate their team in order to improve three KPIs: increase productivity, decrease turnover, and heighten employee engagement.
We decide to build training to make them better at coaching. We have 2 learning designs to consider—one adaptive, and one nonadaptive.
Let’s look at this quantitatively. What ROI do we get from each option?
But obviously, measuring these is tricky. What are the numeric values of B1 and B2, and c1 and c2, to make these kinds of analyses work?
Unless we’re willing to do both options and run a random-sampled treatment-and-control experiment (and spend c1 + c2 doing it), we’re going to have to select one of them on faith, build and deploy it, then try to guess what would have happened if we had chosen the other.
So we choose adaptive learning design, spend c1, gain B1, then guess at (B2 – c2). Or we go with nonadaptive learning design, spend c2, gain B2, and guess at (B1 – c1).
Before we calculate this, let’s look at B1 and B2 more deeply. If we don’t see quantitative improvements in our KPIs (retention, productivity, or engagement) as a result of either option, we might then conclude B1 and B2 are essentially zero. But can we blame either option? Yes, it may well have been due to weak learning design. But maybe it was due to other factors, such as, oh, I don’t know:
- Inadequate measurement (how do you quantify engagement, exactly?)
- Organizational pressures that work against behavior change (for example, managers have no time to coach because they’re too busy with day-to-day duties)
- Attitudinal resistance in our coach or coachee (either of which might be thinking, Things are good now. I don’t have time for this. Get me out of here.)
Dozens of factors influence employee behavior and training is only one of them, regardless of whether we went adaptive or not.
ROI From A Cost Standpoint
Suppose, as said above, both B1 and B2 are zero. It then comes down to costs, c1 and c2, whichever is lower wins. So you add up the total hours to build and deploy each option, times the labor rate or vendor cost to do so, and derive c1 and c2.
But in order to come up with the real value of c1 and c2, you need to factor in not only the cost to develop and deploy the training, but also the cost to consume it.
Let’s envision what might happen if we don’t adopt an adaptive learning approach. Returning to our coaching example, we develop a traditional 2-hour course, either ILT or eLearning, or a combination, to teach 100 managers how to better coach their employees. This is a one-time experience, 2 hours total, that all 100 managers are mandated to complete. The audience includes young managers fresh out of MBA school and experienced ones who have been managing people for years.
It’s likely that some of the learners won’t get it, regardless of the cleverness of our treatment: they can’t connect the content to their concerns, life experiences, or daily decision-making. And many won’t get it because they think they already know how to coach, are busy, and don’t pay attention.
Chances are, the course won’t make much impact on either group, and those managers will return to their desks behaving exactly as they did before the training.
Let’s attach numbers. Say 15% of the group doesn’t get it because they can’t relate it to their experience, and 15% doesn’t get it because they think they know it already and don’t invest. In that case, 60 hours (2 hours * 30 managers) were wasted. This 1.5 person-weeks of wasted time, at managers’ labor rate, needs to factor into our calculations of c2.
Now, imagine that instead, we use an adaptive learning design. We choose to build a course that has a mandatory 15-minute path through content, with pull detours (links, PDFs, explanations, "More" buttons, and so on) seeded at strategic points along the way that allow learners to go more deeply into areas of interest.
Yeah, people may not get it, for the same reasons above. But if 30 managers don’t gain value, they will likely spend the minimum time in the course, 15 minutes as per the design, so only 7.5 hours total (30 * 15 minutes) were wasted, one person-day versus a person-week and a half with the nonadaptive approach.
Plus, for the 70 managers who do get it, there are opportunities to go much more deeply into topics, which can factor into B1.
Add In The Value Of Analytics
By adopting an adaptive approach, and faithfully measuring learner choices (say, via xAPI), we have an opportunity to learn more about our audience and what they are interested in and care about, which we can use to make improvements to our L&D over time. This insight can’t be gained from a nonadaptive approach such as lecture or page-turner, since learners have few choices that express their mental state except the desire to move on to the next mandated screen, slide, or interaction.
If we’ve deployed the adaptive eLearning course described earlier, on the other hand, that has a 15-minute mandatory path and 3 hours of optional tangents via pull detours, learners may spend anywhere from 0.25 to 3.25 hours in the course. Or more if they’re revisiting stuff, less if they skipped class. We can now look at the average total amount of time people are in the course, and the standard deviation, to gain insight.
If the average is just 16 or 17 minutes, it tells us that none of our detours (roadside dinosaur parks and the world’s largest quartz ring, or the like) are interesting to our audience. Or maybe they didn’t see value in the subject as a whole and so paid as little attention as was allowed.
But suppose the average is 40 minutes. Or 2 hours. Or a sizable segment of the population spent that long. This means there’s something there. A significant number of managers are pursuing detours, which gives us data to digest.
First, we can look at which detours were the most popular. Why? They may represent unanticipated areas to provide additional content to managers.
Second, we can find clusters in how managers behave, which corresponds to their demographic: for example, younger managers spend more time in the course than older managers, or managers in some departments went here, while others went there. Sifting through this data yields clues, and these clues can help us improve the quality of our training.
And, most adaptive learning designs make it easier for people to come and go than do nonadaptive designs, and if they are coming and going, we have another data point to analyze. What are they looking at when they come back? This too can tell us something about our audience, and where to add more training in future iterations, that nonadaptive approaches can’t provide.
Adaptive learning gives learners choice, choice is data, and data has value, which needs to be factored into B1.
And Maybe Adaptive Learning Does Teach Better
Because adaptive learning design allows learners to get what they want and need, and not have to slog through stuff they don’t, it likely leads to greater retention and understanding. If only from reduction of cognitive load, since they can spend their time and energy on the stuff that connects and not on stuff that doesn’t. This argues that B1, in terms of learning, is higher than B2. If c1 and c2 are close, then there’s greater ROI in adaptive learning.
But proving this is a real challenge because of numerous influences.
And Don't Discount Joy
Even if there is no greater learning from adaptive design (and as said, plenty of examples suggest there is), there’s another consideration, that’s hard to quantify for ROI calculation, but important. Humans intrinsically value agency. That is, the ability to control their environment—in the case of training, what they see and experience, based on what actions they take. Being forced to do something is sometimes necessary but often unpleasant.
Adaptive learning offers greater agency to learners, which all else being equal heightens their joy. If B1 and B2 are close, increased joy warrants doing an adaptive approach—all else being equal, it’s better to make learners happy than not. To put this another way, if in doubt, offer learners choice.
At the same time, no one likes to be lost. Absolute freedom promotes confusion: we must ultimately guide the learner through a journey because self-exploration leads to the kid stuck in the maze whom security guards rescue. Pure discovery learning is frustrating. Instead, offer sliding guidance that gives learners freedom while they still systematically progress to mastery.
This puts ownership for learning into the hands of the learners, making them responsible for their own growth and development by what they do and what they learn. Like throwing a teenager the keys to the convertible. This shows respect and trust. That, in turn, may shift their attitude and compel them to pay more attention, and possibly pursue optional learning in other areas, a value that can be added into our calculation of B1.
Final Calculation
So, determining the ROI of an adaptive learning design should include the following calculations:
- Less time wasted in training
- Greater insight into the mind of the learner
- Greater potential learning gains
- Greater joy
This makes a strong case for adopting adaptive learning wherever feasible. Download the eBook Hats Off To Adaptive Learning: Tailoring Corporate Training For Each Learner, and get to discover valuable insight into adaptive learning in corporate learning.
