AI Coaching And Assessment: What To Automate And What To Keep Human

Proportional Controls For AI In Assessment

Advice on using LLM-based systems in learning keeps piling up, but what gets less attention is how AI is drifting further toward:

• Evaluating open-ended answers
• Identifying skill gaps
• Generating a readiness score
• Recommending whether the learner is ready to move forward

And since those jobs carry different amounts of responsibility, they feel like different breeds of AI. Yet it is usually the same model, just paired with a different prompt, rubric, or threshold.

But while the technical gap may be small, the stakes are not.

So how far up the decision chain are we willing to let LLM-based systems climb?

What Is AI Allowed To Decide?

Instead of sorting LLM-supported learning into the "AI-led" and "human-led" camps, a better lens is how much weight the AI's output can carry.

Level 1: AI As Coach

At this level, the output helps the learner think, practise and reflect. A custom AI coach for certification prep, for example, can build personalized study plans, surface skill gaps, recommend what to practise next, and adjust guidance as the learner progresses.

And though Level 1 still comes with hallucinations and hollow praise, here these failures are usually easy to correct and unlikely to determine what happens next.

Level 2: AI As Assessment Assistant

AI may detect apparent skill gaps, calculate a readiness indicator, flag unusual patterns, or surface work that deserves attention, and this may be seen by a trainer to decide who needs intervention.

So Level 2 still leaves the final call with a human, but AI moves upstream of that call: it can shape what the reviewer notices, questions, and acts on.

Level 3: AI As Part Of A Consequential Decision

The dividing line between Level 2 and Level 3 is where the AI output goes next.

At Level 2, it informs human judgment. But at Level 3, AI-generated evidence directly shapes a consequential outcome. It may feed into certification, pass-or-fail decisions, educational progression, admissions, or placement.

What makes the distinction easy to miss is that the model itself can stay unchanged across all three levels, even as its output gains more authority.

Where AI Already Works Well

AI is at its most useful in learning when it is used for:

• Instant formative feedback
• Role-play and scenario practice
• Personalized practice
• Learning reinforcement
• Skill-gap detection
• Rubric-based first-pass evaluation
• Triage at scale to surface ambiguity and patterns

These functions become even more valuable when they work together. An AI admission coach is a good example here: by combining personalized prep, progress tracking, and a readiness score that updates with new data, the coach contributed to 22% higher retention and 38% increase in SAT course completion.

However, the OECD's 2026 Digital Education Outlook flags an important caveat here: better performance with AI does not automatically mean better learning. That only holds when the system nudges, questions, scaffolds, and adapts without doing the thinking for the learner.

What Changes When AI Starts Producing Evidence, Not Just Advice

AI usefulness is only half the story. Once its output starts shaping someone else's judgment, these should come into play:

1. Validity

A readiness score means little unless the behaviors and data feeding it are proven to reflect the capability it claims to assess.

The International Coaching Federation makes a similar point for AI coaching systems in the Assurance and Testing domain of its 2024 AI Coaching Framework and Standards, which calls for claims about what a system can measure or improve to be formally validated.

2. Reliability

Evidence on AI scoring remains mixed: some studies find close alignment with human raters in narrow settings, while others show weak agreement, scoring bias, and poor separation between performance levels.

Reliability therefore has to be demonstrated through repeated scoring, test-retest consistency, agreement with expert raters, and agreement among the experts themselves.

A dependable assessment system should ignore irrelevant changes such as wording, format, or answer order, while any change to the model, prompt, or scoring logic should be documented and retested.

3. Fairness

A model's abstract reputation for fairness means little until it is tested on your learners, rubric, and workflow.

Equivalent competence should earn equivalent treatment, regardless of how polished the prose sounds.

Clear evidence-based rubrics, matched test sets across different writing styles, sufficiently diverse groups, and ongoing checks for accuracy and bias keep the score anchored to skill.

4. Accountability

Accountability means an instructor or learner can inspect the evidence behind a result, trace how it was produced, and challenge it when something does not add up.

What matters is a verifiable trail tying each score to specific rubric criteria and parts of the learner's response, with a clear route for human review.

5. Privacy

Using a third-party model raises questions about what leaves the organization, where it is processed, how long it is retained, and whether it can be reused.

The rule is to collect and expose no more learner data than the task requires.

Human-In-The-Loop Control Architecture

"Keep a human in the loop" sounds like one safeguard, while it is a bundle of controls that enter the workflow at different points:

1. Before AI Makes The Judgment

• Human-defined rubric: A person defines the criteria the AI is expected to apply, giving the system a clearer target and reducing arbitrary scoring. But a sound rubric does not guarantee sound application; the model can still interpret or weigh the criteria incorrectly.

2. During The Judgment

• Sample review: A human checks a portion of AI-scored work to catch drift, recurring errors, or unexpected patterns. It is useful for monitoring the system as a whole, but it cannot protect every learner from an individual scoring error.
• Low-confidence escalation: Ambiguous or uncertain cases are routed to a human instead of forcing the model to make the call. This can reduce review workload, but only if the confidence signal itself is well calibrated.
• Human validation of consequential scores: A reviewer checks AI-generated scores before they feed into a meaningful decision. This creates a stronger safeguard, provided the reviewer can inspect the evidence and has real authority to disagree with the system.

3. At The Decision

• Human final decision: AI may organize evidence, score performance, or recommend an outcome, but a person remains responsible for the decision itself.

4. After The Decision

• Appeal or review pathway: Once a result has been issued, the learner needs a way to challenge it, trigger another review, or correct an error that earlier controls failed to catch.

Building Controls Proportional To The Stakes

The more authority the output acquires, the more evidence and control the system needs around it.

If AI does this...:

1. Gives practice feedback
2. Produces rubric scores
3. Produces readiness or risk indicators
4. Influences a consequential decision

…you need more of this:

1. Grounding in approved content, content-quality checks
2. Validation dataset, scorer calibration, model and prompt version control
3. Ongoing monitoring, empirically tested thresholds, escalation and human review
4. Traceability, auditability, stronger fairness testing, independent review and appeal workflows

Moving toward the more consequential end of the table, a few engineering practices start doing heavier work. Each matters on its own, but their real strength comes from a well-designed AI system architecture that makes them operate as one control system and reinforce one another.

Model and version tracking matters because every score should be traceable to the exact configuration that produced it. That includes the model snapshot, prompt and rubric versions, sampling settings such as temperature where applicable, decision thresholds, post-processing logic, and any retrieval context used through RAG.

Evaluation datasets give teams a fixed set of representative cases against which they can test changes.

Regression testing answers an important operational question: did the latest improvement inadvertently make the system worse somewhere else?

Logging lets reviewers reconstruct what the system received and produced when a result is disputed. Since LLM outputs are not deterministic and a rerun may not reproduce the original result, the system should retain the inputs, retrieved context, intermediate processing where appropriate, and the final output.

Thresholds and escalation workflows mark where automation stops and human judgment begins, and they need their own validation, since "90% confidence" can sound reassuring without meaning much.

Five Questions Before Automating Another Learning Decision

Before automating another learning decision, five questions help show how far automation should be allowed to travel and where to locate the use case on the three-level ladder:

1. What decision will this output influence?
2. What happens to the learner if the AI is wrong?
3. What evidence shows that its judgment is good enough for this particular use case?
4. Where does human judgment enter the workflow, and what can it actually override?
5. Can the result be reconstructed, reviewed, and challenged?

For organizations operating in EU education markets, that choice also carries legal weight. Under the EU AI Act, systems that influence access, evaluate learning outcomes, or determine educational level can fall into high-risk categories, while preparatory or advisory uses may remain outside them.

So AI still has a place in assessment. But as the outputs start carrying more consequence, integrating AI into learning products demands greater engineering rigor, stronger validation, and tighter human control.