Overview: An L&D team often gets thousands of ratings, multiple-choice survey question results, and open-text comments across their portfolio of courses and programs. Making sense of the numbers are easier: counting, averaging, and visualizing the results can be automated.
Summarise this page with your favorite AI assistant

When to Use Humans, Machine Learning, and GenAI?

Actionable Course Evaluation Insights: What's the Best Approach to Find Them?

An L&D team often gets thousands of ratings, multiple-choice survey question results, and open-text comments across their portfolio of courses and programs. Making sense of the numbers are easier: counting, averaging, and visualizing the results can be automated. However, the open-text comments are more difficult to handle. Ten years ago I did by hand. Just five years ago, at Amazon, when I had to analyze hundreds of thousands of comments I used Python and sophisticated natural language processing (NLP) libraries (machine learning). Still, the results required a lot of manual tweaking just to get the sentiment score right. For example, sarcasm was completely missed. And I can tell you I saw similar comments like:

"It was so exciting I had to put on music not to fall asleep." Or “I loved this mindless clicking”

Fast forward today: when I work with survey and evaluation data, I have a choice: human analysis only, machine learning only, GenAI only, or hybrid, combining all. In fact, you don't even need a data professional like me anymore. You can just throw in all comments in your favorite LLM and ask for a summary and themes. It is not a technical problem anymore, it is a tradeoff question: accuracy vs. speed, transparency vs. scale, and the big one: level of deterministic results (aka do I get the same answer every time?).

The human-only, manual effort is not an option today. So, I decided to put the course evaluation methodologies to the test with ML and GenAI approaches. This two-part article is about an experiment I've done to answer the questions:

1.      Is machine learning (ML), GenAI, or a combination of the two a more practical way L&D should approach survey data analysis.

2.      What are the strengths and weaknesses of each methods?

3.      When should we you use each?

I’m not going into the technical details but if you’re interested, connect with me for the sample data, code, prompts, and run details. And finally, if you're thinking:

Well, I'm not a technical or data person...

Any decent LMS or EdTech solution nowadays will offer you analytics. You don't need the technical details but it's good to ask how they approach their recommendation engine or data insights generation. Don't get fooled by the shallow "AI" answer.

The problem: what are we solving for?

ML, GenAI, or Hybrid. All three methods return results: summary, themes, insights, and even verbatim quotes. But how do we know the findings are valid? Reliable? And fair? How do you know if they don’t fall into the traps mentioned above (sarcasm, false alarms, SCREAMING single loud voice, mixed-messages, overgeneralized issue, non-English responses, etc.) With real data, you never know what the right answer was, so you can't tell a good analysis from a confident one. That's why I decided to build a synthetic data set for the test.

What’s a synthetic data set?

Generated data that mimics the real world with more control over what hidden stories it includes. In other words, I started with the end in mind. Creating a data set that included realistic but planned themes and traps (like sarcasm), so I can then test the end result of each method against it using the same rubric.

How do you create such data set? I’ll show in part 2. For now, I just give you the highlights of what it entails.

The Hidden Story

A fictional semiconductor company, Helix Microsystems: 1,200 learners, 10 courses (self-paced, classroom and virtual) over six months. Everything a real L&D team would have: course and session records, enrollments and completions, assessment scores, learner profiles, and 1,386 survey responses (1,380 after removing duplicates).

Simulated Courses in the Data Set

The course evaluation for these come from employees of the company. They are represented through 6 personas and a baseline. P0 is the baseline. These responders will be generated to be the generic noise. The rest, P1-P6 represent a certain response pattern we're hoping to find through the analysis:

  • night-shift technicians who can't attend live sessions;
  • new managers who rate a course highly but can't yet apply it;
  • experts sent to beginner courses ("not worth my time, but recommend it for new hires");
  • early-career enthusiasts who rate themselves experts and score lowest on the test;
  • remote salespeople clicking through a compliance course;
  • finance planners trained on a tool their team doesn't have yet.

Note: anything resonates? The point was here to include known personas for the test but these are realistic issues.

So, we have courses and we have responders associated with a persona. With these data, we can introduce the hidden themes.

Course-related themes:

  • Five course or delivery problems: one weak facilitator, lab failures in one region, outdated screenshots, a course that is fun but shallow, and a policy question about GenAI tools that appears in only one survey question.

People-related traps:

  • sarcasm ("Loved spending my Friday clicking next 87 times");
  • comments that praise and complain at once;
  • non-English comments;
  • facilitator names and contact details;
  • one long, vivid rant from a single person;
  • duplicate submissions and learners who gave every question the same answer ("straight-lining");
  • the groups least likely to answer the survey being the ones with the biggest problems.

So, we have courses, employee personas to respond, and hidden stories in the responses. The last missing piece is: where are these responses coming from? I didn't spend my month pretending to be a persona to respond to surveys. I provided the survey questions to Claude and through iterations with me, Claude created a document that served as the "simulated survey" to collect the data.

One note on the survey questions. I did not use the traditional smile sheet questions like satisfaction with the content, room temperature, organized content, etc. I used survey questions that provide more actionable data about the application of learning. Many of these questions are inspired by Will Thalheimer's Distinctive Questioning method, described in this book, https://www.amazon.com/Performance-Focused-Learner-Surveys-Distinctive-Effectiveness/dp/1941577032 .

Putting it all together: a Python script using all this data above, created the final data set that included the given courses, the response personas, all the themes and traps, and enough general noise to make the whole thing random. Note, the code is generic, so you can rerun it with a different "seed" to make a new data set that includes all the themes above but with a different noise. It also created the keys to evaluate the results. Those keys were kept in a "sealed" folder until the evaluation.

What's the verdict?

I'll skip to the end to see what I learned. For those who just wanted to know what the end-results was, you're done your reading. For those who are interested in learning about the process and evaluation, I'll get into those in the second part.

  • ML only (classic pipeline) - I used Python libraries to process the data (for geeks: tokenization, lowercasing, lemmatization, stopwords such as "course", "class", "instructor", "module," TF-IDF for vectorization, and NMF for topic modeling)
  • AI only (using GenAI for everything) - I used Claude and Gemini latest models. I ran the analysis 3 times to test reliability (for geeks: Temperature is set to 0 to reduce variation but even then it is not deterministic; provided role and context in prompt with access to the files in Google Collab; context-window chunking; requested JSON output)
  • Hybrid method - I used the strength of both ML and AI with a human in the loop. (for geeks: LLM drafts a codebook from a sample -> humans refine and approve it -> then comments are classified one by one by LLM so counts come from code, not from the model's estimate; human-in-the-loop checkpoints: validating labels, spot-checking assignments, verifying quotes.)

Results of Comparing Course Evaluation Analytics

Same data is available online in an Observable notebook .

Scoring

Scoring was against the same rubric in all cases based on an answer key generated along with the hidden stories. Maximum score was 40. Beyond the scoring, I was also looking at specific findings, pattern matching, and data privacy issues.

One note on the "Facilitator names in quotes" measure. There is very clear instruction in the prompt NOT to use facilitator names for direct quoting. Relying on GenAI solely with instructions tend to fail. I had the same experience using real data. No matter how many times and how the instructions explained to replace a name with "Facilitator," time to time GenAI failed to follow them.

While the prompt instructed GenAI to find patterns (ignoring one time rants, for example), I also made sure that critical comments would be raised no matter how weak the signal is (privacy concerns, behavior issues, accessibility comments, etc.).

Finally, here's the comparison between ML-only and the Hybrid method with exact themes and patterns:

Heatmap Showing Pattern Findings

On the left is the result of ML-only result. This approach is using words and frequency of similar words to match patterns. On the right: the hybrid approach where GenAI created the themes first before pattern matching. The darker a cell the more prominent is the finding. What we are hoping to see is connecting the right issue with the right planted theme.

For example: look at the C02 with Fac-02 line. This was a planted theme with one facilitator (ID Fac-02) who did not perform well in course C02. Ideally, what we would like to see from the analysis is a theme about facilitation, so we can act on it.

The classic ML results: darkest cell belongs to "stuff / minor stuff" which is a very generic noise. Not helpful. There's also some signal pointing to "real / scenarios." Again, this is not about the course, it's supposed to be about the facilitation.

Hybrid approach: very clearly indicates "Rushed pacing..." and "Facilitator unprepared" issues. This is a strong enough signal to look into that one course facilitated by that one facilitator. A minor signal even points to "One-way lecture format." By itself, it could mean the learning design but with the stronger problem pointing at facilitation (and only facilitation for that one facilitator), it is more likely about the facilitator's performance.

Conclusion

When to use what method? It depends. And that's why we still need humans. Based on your  wn situation, you may go with the traditional ML approach, GenAI-only, or the hybrid method.

First, ask fundamental questions:

  1. What decisions are we making on these findings? What are the consequences?
  2. Is this a one-time analysis? Or, are we re-running it periodically?
  3. Who's your audience? What matters for them? What is the tradeoffs you can afford? Speed, accuracy, nuance, exact numbers, etc.
  • Use classical NLP when: results feed ongoing trend tracking, decisions need an audit trail, data cannot leave your environment, volume is high. It is also performing well when the data is consistent as in everyone calls the same thing the same, since it is word-based.
  • Use GenAI when: you need a fast read on one course, nuance matters more than counts, the audience is non-technical. Works well when the data includes mixed messages, sarcasm, and other language traps, but you need to trust and verify mathematical procedures. GenAI tends to be "lazy" sometimes without letting you know (there was one run when Gemini used only a portion of the data it deemed representative enough without telling me).
  • Use hybrid when: results will inform investment or employee-related decisions, be reported to leadership for data-informed decision making, or generally the consequences of the decisions are hard to reverse or risky.

In the second part of the article I'll show the process and details behind the experiment. I also ran a side-quest to see if GenAI would cheat (by using the answer key to achieve high scores) under competitive circumstances.

Originally published at: www.linkedin.com

About the author

Join our newsletter Sign up for tips and Elearning insights, delivered straight to your inbox
5 / 5
It's my daily dose of e-learning insights! Martin, from Adobe

Related articles