The Streetlight Effect: When Your Training Evaluation Is Guided By What’s Easy To Measure
Here's an old joke. Stop me if you've heard this one before.
A cop notices a drunk man searching for something under a streetlight. The cop asks the man what he's lost, and the man replies that he's lost his keys. They both look under the streetlight together.
After a few minutes of this, the cop asks, "Are you sure you lost them here?"
The drunk man replies, "No, I lost them in the park."
The cop asks, "Then why are you looking here?"
The drunk man replies, "Because this is where the light is!"
The joke is ridiculous, but it illustrates a real-life phenomenon called the streetlight effect. Under the influence of this observational bias, we look for something where it's easiest to look for it, rather than where we're most likely to find it. Our training evaluation forms ("smile sheets") are a good example of the streetlight effect.
In paper form or online, smile sheets are a quick, easy, and cheap form of training evaluation. We collect some data, average the scores, and bingo! We have a neat little number that tells us we did a good job. We can even hold up our little gold stars and show our bosses! But, if we're looking to find out how effective our training us, we're not likely to find it in those smile sheets.
It Doesn't Matter If They Liked It
The typical smile sheet measures reaction. In other words, it assesses whether or not the learner liked our training. But we don't get paid to make likable training. We get paid to make effective training. Training is supposed to help employees acquire new skills and knowledge that they can use to improve their performance on the job. Our smile sheets don't measure that, but we continue administering them anyway. Why? Maybe for the same reason the drunk man looked for his keys under the streetlight. It's easy.
Now, to be fair, we also use smile sheets because there's a long tradition of using them. There's even a theory that explains why. Kirkpatrick's Four Levels of Learning Evaluation are:
Each of these levels is supposed to build on the one below it. That sounds right, doesn't it? If people like the training, they're more likely to learn the material, right? Well, not so much, as it turns out. There's actually very little correlation between learner reaction and the higher levels of evaluation. In other words, whether or not the learners liked a course tells us very little about how much they learned, what they're applying on the job, or what the business outcomes are.
The Smile Sheet Is Dead, Long Live The Smile Sheet
Getting rid of smile sheets isn't going to happen any time soon. They are so much a part of how we do things that any attempt to eliminate them will result in massive resistance. And, to be fair, it's not as if they're totally useless. At the very least, they communicate to our learners that we care about their thoughts and opinions.
Will Thalheimer is a guy who has thought about this problem a lot. In fact, he wrote the book on it. Literally. It's called Performance-Focused Smile Sheets: A Radical Rethinking of a Dangerous Art Form. If you want to create better smile sheets, buy this book. (You won't believe how entertaining a book about smile sheets can be.)
In the meantime, here are 4 "thou shall nots" for improving your smile sheets:
- Thou Shall Not Ask Questions About Things You Can't Or Won't Change Anyway.
So, for example, if there's a legal or regulatory requirement for you to present certain content, don't bother asking learners if they liked it. You can't do anything about it if they don't.
- Thou Shall Not Ask Learners To Predict The Future.
Typically, the learner fills out a smile sheet immediately after they've finished the course. If they haven't had time to apply their new knowledge or skills, how could they possibly know whether or not the course was relevant or helpful at this point? They're just guessing based on how they feel. (And –see #1– you're not going to change a course based on learner hunches are you?)
- Thou Shall Not Ask Learners To Do Your Job.
Don't ask them how "effective" the visuals, audio, or course length is unless they happen to be learning, and development experts. Of course your learners have opinions, but they probably don't have the training or experience to know how effective your course is. (And –see #1– you're not going to change a course based on uninformed opinion, right?)
- Thou Shall Not Change Thy Course Based On A Few Negative Comments.
We are primed for negativity. If you've got a whole stack of positive evaluations and just one negative comment, guess which comment is going to grab your attention? But if you up and change your course every time you get a negative comment, you are, in fact, changing those elements of the course that the rest of your learners thought were good.
There are ways to make better smile sheets. If you want to learn them, consider reading Dr. Thalheimer's book. In the meantime, you can ease the pain and enhance your training evaluation by avoiding the four mistakes mentioned above.