Learning Analytics: xAPI Data Traps

Where To Start With Learning Data?

"We have xAPI now and we can track anything. What learning data are you interested in?"

If the answer you got is less enthusiastic than you expected, read this article. If you're just about to explore xAPI and data analytics, read on as well. While you might be excited about breaking out from the old SCORM limitations to track anything in a course using the Experience API (xAPI), this question may not lead to the best outcome with stakeholders, even if they say they are data-driven. Why do you think this is? What's wrong with this seemingly good, open-ended data question?

Avoid Trap #1: Speak The Language Of Your Audience

Some say data is the language of information. Speaking the language of your audience is key. For example, the phrase "data-driven" may be somewhat misleading for some. It's actually not the data that drives decisions. It's the insights gained from the data that people use to make decisions. Therefore, it's not the language itself that matters, it's the relevant information you deliver that matters, the actionable insights. However, what's relevant and actionable may be different for different stakeholders.

What do your stakeholders want to know? In L&D, when we say "stakeholders," we often mean business stakeholders. In this case, anyone who has an interest in gaining actionable insights from the data is a stakeholder, including:

1. Learners
2. Managers
3. Senior business leaders/operations
4. HR/legal
5. L&D leadership
6. Instructional Designers, learning experience designers, graphic designers
7. Delivery team

And the list goes on. It is a good idea to make a list of all the roles (or specific names, if they are influential) and their potential interest as part of your data strategy document. This list may be as simple as the role and the questions or business problems they are interested in. If you are familiar with the concept of a project charter, this is a similar approach.

Why is a stakeholder charter important? Because a senior business leader couldn't care less about the number of clicks on a slide or the details of your supporting learning objectives. However, they might be eager to learn about when they can schedule agents on the phone based on the proficiency level they reach. Your data strategy document should list your stakeholders, their roles and influence on the project, and the insights or questions they might be most likely interested in.

"We have xAPI now and we can track anything. What learning data are you interested in?"

Back to our original dilemma. This question may work with your L&D team as stakeholders to learn about how users progress through a course, but it may not resonate with a business leader. The problem with this question is that your business stakeholders a) may not know even know or care what "learning" data is; b) may not think beyond courses; and c) may interpret "API" as an IT issue (sorry, the word "Experience API" can be more confusing than it should be).

• Speak the language of the business
  What business stakeholders do care about is their own business metrics. If they don't see the connection between business metrics and learning, collecting and analyzing data may just sound like extra work that slows down the project. How do you work with stakeholders if they don't tell you what data to track?

Avoid Trap #2: Be Curious, Start With The Problem

To speak the language of your business stakeholders, you need to understand how the business works. Using one of the data literacy hats—curiosity—ask them what they and their managers care about. Ask them what decisions they or their managers will make today. Ask them about information gaps in those decisions. Ask them what they know, and what they don't know. What would help them make more data-driven or data-informed decisions? Don't start with data. Especially not "learning" data! Start with the problem, the opportunity, and the questions they already care about.

For example, ask them about the assumptions they operate on today. Assumptions are a great place to start. Why? Because each assumption has an associated risk. And reducing risk by providing actionable insights from data is valuable. Once they are interested in the why and the what, you can then go backwards from there to get to the how (but still, you may not want to mention xAPI too many times).

Avoid Trap #3: Do Not Boil The Ocean, Solve Something Specific

In my last twenty years in the industry, I've seen a similar pattern over and over again for successful buy-ins: instead of boiling the ocean (like fixing the complete selling process), solve a small, very specific, but relevant, problem for someone who's already willing to try new things. This is how to get a champion on your side. Then, with your champion, build a case study using data storytelling. Show the results to others. Spread the word.

But! Not as a solution for everything (which often threatens the status quo and may result in defensive reactions), but rather as an example of a solution for the specific problem you solved. The trick is that you shifted the reaction from "this would never work here..." to "hmm... it clearly solved that problem. We have somewhat similar issues. What would it take to try it?"

In the corporate world, there are lots of bright people with lots of bright and innovative ideas. The question is if you can operationalize the idea and for what cost. Speed matters. A solution you can try tomorrow with a 70% chance of success will most likely beat the 90% chance solution 6 months down the road. Why? Because you can iterate fast based on the data you're getting, instead of waiting for six months and relying on assumptions. Data can "expire" like food on the shelf.

Avoid Trap #4: The Two-Edged Sword

Data is a two-edged sword. Bad data, even with the right decisions, can lead to incorrect conclusions and serious consequences. Not using data properly can also be limiting for L&D. I know it sounds counterintuitive because we do want data. However, over-indexing narrowly on what data to track (which is often some number), instead of what questions to answer or what opportunities to explore, can lead to multiple traps under the two-edged sword:

1. Focusing on the trees while missing the forest
2. Oversimplifying complex problems
3. Discarding anything beyond what you know you can track
4. Tracking everything under the sun
5. Chasing band-aids instead of finding the cure
6. Using only learning data

We will go through each of these potential traps when using data for learning analytics:

1. Focusing On Trees While Missing The Forest

If you start out with data, you may limit your focus to what you do today (scores, completions, clicks, time spent, etc.), what you have access to today (LMS, course content, etc.), and what you've always done (pull reports and build dashboards to show what happened). In other words, you are biased by your past limitations. At the other extreme, past limitations have an interesting side effect: going wild. It's like growing up with your parents forbidding chocolate in the house. When you leave the house the first thing you do is to eat as much chocolate as you can. Just because you can.

xAPI lets you track anything. No more perceived SCORM limitations. All the chocolate you want! You can track every single slide, every single mouse movement, and every single "next" button click if you want to. Now, add multiple designers, maybe even multiple teams across the organization, joining the all-you-can-eat chocolate party. It is going to be a mess in the learning record store. Well, inconsistent at minimum. Finding insights will be a challenge. So, before you dive into the chocolate factory, let's consider what you want to do with the data!

• What decisions are you making?
• What actions are you going to take?
• If none, why are you increasing the noise?
• Are you limiting your thinking to descriptive analysis (what happened)? Or, are you expanding to diagnostic (why), and predictive (what will happen and when) analyses?
• What additional value could you provide your stakeholders? For example, your level 1 survey has very little value for the business. Would it be more practical for your stakeholders to get information about what challenges they might see during learning transfer?

2. Oversimplifying Complex Problems

In reality, business problems often require multiple data points to answer a single question. If you're not thinking on a higher level, your xAPI statements may not tell a full story. Data will not make anyone's mind change. The story needs to be told in a way that is relevant, interesting, insightful, and practical.

For example, you may be wondering if those expensive live-actor videos make a difference in how managers handle difficult conversations. You will not have a single data point to answer this question. So, start with the question or the problem, and then figure out what multiple data points may help tell the story at the end. There's nothing more frustrating than collecting tens of thousands of statements in a program and still being unable to answer your stakeholder's question (which, of course, they came up with after the program).

Oversimplifying also happens when a complex notion such as empathy or customer service skills is reduced to a number on the quality card. We need to increase the number! Don't forget that data is never a reality. It is a limited snapshot of the full picture. Don't just focus on a single number to move the KPIs. Consider what the number represents. When a business KPI number itself becomes the target, it can go really wrong (see the Wells Fargo case where employees opened up over 1.5 million accounts illegally to meet the KPIs) [1].

3. "We Can't Track That"

The third problem is that you may discard a data point just because, currently, you can't track it.

Operations: "Could we track how much time supervisors spend on retraining new hire agents on things they should know after the training, and in what topics?"

What happens after the new hire training is out of your control. Your immediate reaction is that you can't track this data. Instead of discarding the data point, explore the question: how much impact would that mean for the business? What data-informed decisions could the business or L&D take if somehow we could answer this question? What would be the effort required to make this happen?

4. "We Can Track Anything!"

The fourth problem is the opposite: it is now easy to track data, so you may forget to use critical thinking about why you need to track that data in the first place. Let's say, your SME wants to know if someone is multitasking by going to a new tab or a new browser window while playing the video. Technically, you can track everything. Even eye-tracking, if you have the equipment. But, is that really the solution? That you track if people are not engaged? So you can punish them, maybe?

Instead, why don't we dig into why people are multitasking in the first place? What makes the content boring, irrelevant, or disengaging? Maybe they already know that stuff? Track that! Maybe they don't see the relevance? Ask them first. Track that! Maybe they don't see how they can apply it on the job? Maybe they have other projects to get done and no time for training? Find the root cause, and then track the data you need.

• There is one important element of data tracking: compliance. GDPR, for example, has clear rules on what you can track and how.

5. Chasing The Band-Aids (Instead Of Finding The Cure)

Sometimes, using critical thinking can help you find the cure rather than producing band-aids:

Operations: "Supervisors are struggling with the 10-minute coaching model. We need to send them to a refresher course on that. Can we track the time they spend on each practice activity in the course? I don't think they took it seriously last time."

And you swing in action: the new xAPI statements can track all kinds of data about the practice exercises. All supervisors go through the refresher. Operation is pleased with the dashboard. Except, there is no impact on the job. Supervisors are still not using the 10-minute coaching techniques they learned.

Turns out, the problem wasn't the lack of skill in applying the coaching techniques in the first place. It was that supervisors spent way too much time answering questions that new hires should've known and retraining them on things they'd forgotten. There was no time left for coaching. Yes, you can track anything but don't forget about root cause analysis before you start tracking things. What would you do differently in this situation?

Operations: "Supervisors are struggling with coaching. We need to send them to a refresher course. Can we track the time they spend on each practice in the course? I don't think they took it seriously last time."

How do we know they're struggling? What data do we have that indicates that? What KPIs are they held accountable for? And what would be the measure of success? What incentives are in place?

"Well, they're not spending enough time on coaching. Quality scores, observations, and feedback from direct reports."

If they're not spending enough time on coaching, what are they spending their time on?

"Looks like answering questions that agents should already know and retraining them on things they forgot. We don't have exact numbers, but it is bad. Senior leadership wants results ASAP."

So, assuming we can lower the amount of time supervisors spend answering questions agents should know and somehow reduce the time spent on retraining them, would that increase the time supervisors have for coaching?

• If yes, let's figure out how to track a couple of data points:
  • How many questions are supervisors answering that should not be questions?
  • How much time are they spending on these questions?
  • How many times does the answer require retraining?
  • How much time do they spend on retraining?
  • What are the topics or tasks they're spending their time on?

You may not track any of these data points today. But, in order to show impact, you need a baseline, you need an intervention, and you need the results. Now you can figure out how to solve the issues, what needs to change in the new hire program, what needs to change in the transition, etc.

6. Using Only Learning Data

Finally, do not limit your thinking to learning data! Learning without application is called scrap learning. We are not just responsible for proper learning objectives. Our responsibility does not end with the LMS or delivering a program. We are responsible for people's skills, their ability, and their motivation to perform on the job. If we're only focusing on the formal learning in our data strategy, we're missing the mark. xAPI data can come from anywhere: from formal learning, observations, workshops, projects, AR/VR/XR devices, and even from AI.

Avoid Trap #5: You're Not Alone, So Build Your Skills, Build Your Network

The point of data literacy is that everyone in the organization speaks the same language and thinks the same way about data. Having one team filled with data scientists is a bottleneck. Find people who already use data in your organization. Join the Learning Guild's xAPI Cohort group [2]. Experiment with AI to help: ChatGPT and Claud 2 are now not only capable of providing tips or guidance but also suggest statistical analysis, run those for you, and even create visualization along the way. For example, I asked ChatGPT and Claud 2 to give me some advice on this topic, and then tested their analytical capabilities on my dataset. I'll share the results in my next article here on elearningindustry.com.

Interested in more about learning analytics and xAPI? I strongly suggest Megan Torrance's book, Data & Analytics for Instructional Designers [3].

References:

[1] Wells Fargo Fined $185 Million for Fraudulently Opening Accounts

[2] Learn xAPI by Doing xAPI

[3] Data & Analytics for Instructional Designers