Is GenAI good at course evaluation analysis?
Using Machine Learning and GenAI to Analyze Course Evaluation Survey Data
In the previous article, I showed the result of analyzing course evaluations using machine learning, GenAI, and the hybrid approach. The data wasn't real, but it was realistic. Think of it as realistic simulation. I planted themes, traps, and insights in the simulated data set so I could validate the different analytical approaches. The conclusion was that the most effective version is the hybrid method, which uses both the classic machine learning and GenAI together. This article is about how to run your own experiment analyzing course evaluations.

In order to get a result similar to this chart by testing your GenAI or hybrid analysis, you will need a synthetic (simulated) dataset. The vertical elements are the planted stories in the simulated data. The horizontal findings are the results of the analysis (themes GenAI identified). You can see the hybrid approach (using both machine learning and GenAI) did a good job identifying themes, labelling them (horizontal items), and then pointing to the location in the data (vertical themes) as evidence.
What will you need to run your own process?
- General knowledge about your current course evaluation process, including survey questions, typical answers, and challenges with analysis. In my experiment, I created a document with the survey questions, explanations, and desired options. Then, I just fed that to Claude as a base for the survey simulation. If you do the same, give a short, distinctive id for each question so both Claude and you share the same language rather than trying to refer to the whole question everytime. For example, the question about worthwhile investment of your time could have worth_time as an id.
- Access to GenAI such as Claude, Copilot, or Gemini to create the initial simulated data set. If this is for work, make sure you're using approved tools.
- If you want to use the most effective hybrid (machine learning, GenAI, and human in the loop) approach, you will need Python installed on your computer or access to Google Collab where you can run Python.
- If you just want to repurpose the idea itself and build it completely on your own with a GenAI tool, I'll share a prompt-only approach below.
Prompt-only approach
If you just want the idea itself, the synthetic, simulated course evaluation data to build and evolve your analysis, start with the following prompt using your approved GenAI tool. Replace [Highlighted] text with your own instructions to customize it for yourself.
Prompt 1: Crate the dataset
I want to test how well different methods analyze course evaluation surveys, using made-up data where I know the right answers. Work with me step by step and stop for my approval after each step.
Step 1: Build a synthetic dataset.
- A fictional company with about [1,000 learners and 8 to 10 courses (self-paced, classroom, virtual)] over one year.
- Files: courses, sessions (with facilitators and regions), learners (business unit, level, tenure, region, shift, work arrangement), enrollments (completion, time spent, assessment scores) and survey responses.
- Survey questions: [paste your questions, upload your survey document, or use: would you recommend it; was it a worthwhile use of your time (1-5); how ready are you to apply it; manager support; plus 3 or 4 open-text questions].
- Before generating, propose 5 or 6 learner groups (personas) and 4 or 5 course problems to plant, each showing up only in specific courses, in both ratings and comments.
- Add realistic traps: [sarcasm, comments that praise and complain at once, non-English comments, names and contact details, one long rant from one person, duplicate submissions, people giving every question the same answer, and the groups with the biggest problems being least likely to respond].
- Write comments in varied, natural wording, not repeated templates.
- Put the planted story in a separate answer key file. Never use or refer to the answer key while analyzing the data later.
This first part of the prompt will start the process. You will need to upload or provide the survey questions you're using. Ideally, your questions are not traditional Likert-scale smile sheets and they include open-text questions. The data you collect determines the quality of the insights you can find. Once the initial dataset is created, you can move on to ask for the scoring rubric.
Prompt 2: Create scoring
Step 2: Write a scoring checklist (points for finding each pattern with the right group and cause, for handling each trap, and minus points for unsupported claims).
With Prompt 2, you have an answer key. Make sure you ask for downloadable versions of these documents, so you can keep a copy. Next, prompt for the analysis.
Prompt 3: Analysis
Step 3: Analyze the data three ways, each in a fresh chat or session without the answer key:
A) AI only: you read the data and report findings.
B) Classic statistics and text analytics only, in code.
C) Hybrid: code computes every number and group comparison; you read and label every comment against a theme list I approve; your findings may only use numbers the code produced, and code checks every number and quote you cite. Remove names before reading comments.
Step 4: Score all three against the answer key, show a comparison table, and list what each method got wrong.
Throughout: rank findings by their link to learners applying what they learned, not by satisfaction alone, and weigh comment themes by how many people and courses raise them, not by how strongly a single comment is worded.
These prompts will you get started. You may need to tweak the process and files. GenAI uses Python in the background to calculate things, however, without Python installed on your machine, you will not be able to rerun the code, so you will need to rely on GenAI.
Step-by-step process for the full analysis
What if you want more control over the process? If you do have Python on your machine or you have access to Gemini Collab, you have more control over the process.
Building the synthetic data set
I worked with Claude to create the simulated data set for the experiment so I can test different methods on something that I know. I wanted to build this as a repeatable activity that produces the planned story and traps but creates the “noise” randomly. Therefore, asked Claude to build a set of Python script that I can independently run over and over again.
1. Step one: building a company with a known story
My experiment: A fictional semiconductor company, Helix Microsystems: 1,200 learners, 10 courses (self-paced, classroom and virtual) over six months. Everything a real L&D team would have: course and session records, enrollments and completions, assessment scores, learner profiles, and 1,386 survey responses (1,380 after removing duplicates).
Let’s start with the courses. If you want more control over the process, create your own list, similar to this. You can tweak the type of courses or programs you want to include and their length. AI will use this as base information to create the simulated data set.

2. Teach AI your survey questions
GenAI will create user personas who will “submit” course evaluations. But first, GenAI needs to understand the potential survey questions you’re using for evaluation. I loaded a set of questions with explanation of why we’re using them, how we evaluate them, and what type of insight they generate.
For example, I explained in the document that the recommendation question and worthwhile time spent question may seem similar, but they provide different insights. The recommendation (with a follow-up question on open-text suggestion) calculates the percentage of desired responses as “Recommend with minor changes” or “Recommend as is.” The result of this question is always high (90%-100%). The Worthwhile time spent question uses Likert 1-5 and calculates the top-2 box (4 and 5). This is a reflection of an individual experience and varies 60% - 95%. Use insights you already have to make the simulation more realistic.
Once AI learned about the questions, and it has the list of courses, the next step is to prompt for personas.
Step 3: Planting the hidden story
Personas will represent person-based issues. You can start with my list as an example and adjust it, or you can use your own. Make a list of common problems and ask AI to brainstorm more from industry.
- night-shift technicians who can't attend live sessions;
- new managers who rate a course highly but can't yet apply it;
- experts sent to beginner courses ("not worth my time, but recommend it for new hires");
- early-career enthusiasts who rate themselves experts and score lowest on the test;
- remote salespeople clicking through a compliance course;
- finance planners trained on a tool their team doesn't have yet.
Settle on the list of personas first, before asking GeAI to create the full persona document. Here's an example of a persona and its planted story:
ID: P1
Persona: Night-shift fab techs
Rule on learner attributes: business_unit = Manufacturing AND shift = Night
Story: Live sessions run at 9:00 Pacific or 16:00 Dublin, inside their sleep window. They like the content but can't attend, and many never get the survey at all.
GenAI can generate these for you but you will need to approve them before moving on.
Beyond persona-driven issues, I also added course or course delivery problems: one weak facilitator, lab failures in one region, outdated screenshots, a course that is fun but shallow, and a policy question about GenAI tools that appears in only one survey question.
Finally, I added 12 traps:
These are common issues I see in the real-world data:
▪ sarcasm ("Loved spending my Friday clicking next 87 times");
▪ comments that praise and complain at once;
▪ non-English comments;
▪ facilitator names and contact details;
▪ one long, vivid rant from a single person;
▪ duplicate submissions and learners who gave every question the same answer;
▪ the groups least likely to answer the survey being the ones with the biggest problems
Scoring: Keeping the test fair
An answer key records the story. It was sealed: no method saw it until its results were in.
Well, actually, I did a side-experiment with AI. I tested the same AI-only runs in three different ways: no answer key at all (no way to cheat), access to answer keys with instructions as a barrier under competitive pressure (told AI its results will be compared to other models), and access to answers keys under potential cheating (told AI I want it do their best even if other models might elect to cheat against it). In my experiment, Gemini did not cheat. That does not mean it wouldn't.
Every method was graded on the same 40-point scorecard:
▪ 22 points for finding the 11 patterns with the right group and the right cause;
▪ 12 for handling the traps;
▪ 6 for reliability: the same answer on repeat runs, accurate counts, and quotes that really exist;
▪ minus 1 for each claim with no support.
Each AI method ran several times, because AI can give a different answer to the same question.
A lesson from building it: the first version of the comments was too tidy, with whole sentences repeated word for word. The classic method grouped comments by sentence template instead of meaning, so the comments were rewritten to vary the way real people write.
What I learned
Contender 1: classic machine learning and text analytics
What it is, in plain terms: software that compares groups with statistics and sorts comments by which words appear together. It is fast, cheap and gives the same answer every time. For those who are interested in the geek part: TD-IF and NMF.
Score: 32 of 40.
What it did well:
▪ Found all 11 patterns, mostly by comparing ratings across groups such as shift, business unit, facilitator and region;
▪ Its numbers were exact;
▪ It caught duplicates, same-answer responses and contact details, and flagged the 12 accessibility comments through a keyword watchlist.
Where it struggled:
▪ Its word groups matched the real themes loosely, at best about 6 in 10;
▪ It scored every sarcastic comment as positive;
▪ It kept both halves of only 1 in 5 mixed comments;
Takeaway: excellent at "which group is different," weak at "what are people actually saying."
Contender 2: GenAI alone
What it is: a large language model (LLM) given the data and a detailed written brief (the same brief every time), asked to find patterns and write a report.
Two models: Claude Opus 5.5 (6 runs) and Gemini 4 Aragon (6 runs).
Scores: Claude 38 to 39; Gemini 33 to 37.
What it did well:
▪ Read tone, sarcasm and non-English comments;
▪ Found nearly every pattern;
▪ Wrote reports a leader could act on.
Where it struggled:
▪ In 10 of 12 runs it found that some C07 learners overrate themselves but tied it to the wrong group or no group. The data showed the early-career group; the AI didn't make the link.
▪ Gemini's course scores were built from 29% to 54% of the responses without saying so, and were off by up to 15 points.
▪ Mixed praise-and-complaint comments were still mostly lost.
▪ Cost: one Gemini run that wrote its own code read about 2 million units of text (tokens), and used up a $20 credit. Not workable at scale. Although, I did use the latest Aragon model.
▪ Privacy: comments and learner details all go to the model. Even with instructions of removing names, including the name of the facilitator, the models still slip.
Takeaway: a strong reader, an unreliable counter, and expensive when it does everything.
Conclusion
The most effective (and cost effective) method is to use the hybrid approach where classic machine learning with Python does all counting, averaging, sorting, deduplicating, PII handling, etc. and use GenAI for the open-text analysis while providing meaningful context. I ended up with a human-in-the-loop solution where GenAI gets a sample of 400 comments to generate themes first. Then I approve and tweak them. Then GenAI uses that approved pool of 20-30 themes to label every single comment in batches.