Measuring eLearning ROI: No Learning = No ROI
It’s commonly heard that using a professional voice actor for eLearning is out of the question since there are no funds for it, or that using a professional voice actor for the backlog of eLearning projects waiting to be completed would put a major crimp in ROI. ROI is simply calculated as the gain or return / cost of instruction. This got me thinking: What exactly ROI might be when it comes to eLearning and through what lens is it measured? Here are a few scenarios for measuring eLearning ROI.
Scenario 1: Be Compliant
The Method
OSHA compliance training is needed before a new employee can be certified to use a potentially dangerous piece of industrial equipment. The usual way this training had been delivered was by a manager in a face-to-face classroom.
One day someone got hurt by the machine and sued the company, stating that the pertinent information that would have prevented the injury was not covered in the training. The employee won a large monetary judgment against the company.
The company then decided to standardize the training as web-based training and hired the cheapest Instructional Designer that could be found to take the training manual and turn it into eLearning, using TTS (text-to-speech), or no voice at all, static graphics of the machine, and some rudimentary questions at the end. When finishing the course, the employee signed off that all of the information has been understood and all questions answered.
The Question Of ROI
This fulfills OSHA requirements, since it contained all the pertinent information, and there was signed proof of completion. The ROI was easy to measure: Cost of writing training manual + Cost of the Instructional Design firm to produce a low-end eLearning module. The company was protected from litigation. Whether or not learning took place was irrelevant.
The Results And Challenges
Optimally, it would make sense for learning to be the goal. Even in creating training for nothing more than compliance, it would be quite advantageous for the machine operator to fully understand and practice safety precautions. This should prevent personal injury while benefiting even the most Machiavellian of companies since less injuries would prevent increases in workman’s comp insurance costs. However, the National Training Lab in Betel, Maine, demonstrated in their Learning Pyramid that only 10% retention comes from reading.
Scenario 2: Cookie Cutter
The Method
An Instructional Design firm has taken on ten projects of equal length and complexity. The cost is fairly low, due to the in-house template used to create such instruction, and the TTS software had been bought and expensed last year. The firm also owns a library of stock royalty-free graphics and clip-art that are mined for all projects.
The projects are developed using a formula where all available documentation is provided by the client and a Subject Matter Expert (SME) will be interviewed by an Instructional Designer for a set dollar amount per hour to determine required objectives and tasks that will lead to mastery of objectives.
Specified in the quote of each project are 2 hours of Subject Matter Expert interviews, no more than 3 objectives, and 3 modules. Additional time or modules are billed at pre-determined fees. Completed eLearning modules are delivered according to a set deadline. Often, in this form of design, as a deadline looms, corners are cut and the quickest and cheapest way out becomes quite attractive. When this happens, the potential for learning suffers.
The Question Of ROI
The ROI for the Instructional Design firm is easy to calculate. The ROI for the client is not as simple, since learning may or may not take place, and a definition of “learning” in this context was not defined. The exception would be a student bringing personal motivation to the table, since there was a clear and critical reason for learning the material.
The Results And Challenges
The “cookie cutter scenario” is all too common and, unfortunately, the resulting instruction does not take into account the learner, who could quite easily fake a successful outcome no matter how poorly engaged they might be with the instruction.
Without engagement, the default position for the student would be to get finished as quickly as possible and find the easiest way to do it. Often, it would take no more than reading the questions at the end and going back and cherry-picking the answers from the text.
If TTS is used and is designed to finish playing the audio before instruction can continue, it wouldn’t be hard to wait it out by surfing the internet or pulling out a smartphone to pass the time as the TTS voice drones slowly on, incorrectly emphasizing words and ending up not instructing but becoming “noise”. Any learning with this form of instruction would be coincidental.
The Learning Pyramid shows that only 30% retention is a result of instruction as demonstration.
Scenario 3: Learning Determines Instructional Design
The Method
Using an interactive approach, discussions are held between a client and a design group starting with an initial form being filled out by the client to determine the scope of the project and specific characteristics of learners who will undertake the instruction.
After submission to the design group meant to focus the project, possible further forms are required and further discussions are held with the Subject Matter Expert to determine a hierarchy of critical, important, and enhancement concepts. After the client signs off on this initial aspect of the project, the Instructional Designers codify objectives and, for each objective, chunk instruction into what tasks must be completed, the order of tasks, how successful completion of a task would be determined using practice and feedback before the next task started… as the skeleton of the instruction is determined or flowcharted.
Creating instruction that is engaging and relevant to the personal situation of each learner is critical to retention. Human voices of trained eLearning talent, along with animation, is as close as possible to being actualized and human and is far more engaging than stock and static images and poorly voiced TTS generated speech. The more realistic and lifelike the instruction, the more likely it is to engage, interest, and inspire thought.
The instruction must answer the question of why this is worth learning. In effect, it must sell its worth to the student before attention is paid and engagement is attained.
Instruction is built with a perspective of investigation, problem solving, and finding not only the right solutions, but why the answers are right. Practice, feedback, remediation, and more practice and feedback until a task is mastered are keys to progression toward an objective. Using formative evaluation, the instruction is revised and evaluated iteratively until both the Instructional Design firm and the client are satisfied with the result.
The Question Of ROI
The ROI for the Instructional Design firm is measured in billable hours with a predetermined budget governing the number of objectives tackled. If more instruction than the budget allows needs to be covered, the budget can be raised, or, if not feasible, instruction will be in smaller stand-alone modules each covering less objectives. The ROI for the client is based on the success of learning and retention; far more complex and nuanced concepts than a learner signing off on completion. It will take longer to calculate this sort of ROI than merely crunching test result numbers.
The Results And Challenges
It’s worth the effort and expense to plan for building elements meant to engage and not repel. Engaging and relevant instruction will have a much better chance of being remembered and transferred than an electronic version of a lecture with a few questions.
Success of a well-crafted post-test will measure short term learning, and, if a second differently constructed post-test can be administered a few weeks later, that would measure retention. This is not cheap nor easy, but the potential of retention through practice according to the Learning Pyramid averages 75% while teaching it to others can achieve 90% retention. Both can be built into simulation in the design.
Full blown professionally designed, voiced, structured, engaging, and creative eLearning production is costly and complex. But, if deep learning and transfer from short to long-term memory is critical, the creation of this type of eLearning is the best way a client can be assured of a real ROI.
So when you hear that it’s too expensive to hire a professional eLearning voice talent or clip-art is fine, just remember that you get what you pay for and run the risk of losing your training investment and any potential for ROI.
