Measuring Online Training Results
After you incorporate any online training program into your organizational structure, how do ensure its effectiveness? Is it teaching your employees skills/information that it is supposed to in order to make the employee proficient in their position? Is it relevant, standardized, useful, or even vital? How do you judge an online training program against all these measures? Fact is that it is extremely important to extensively review your organization's online training program according to some guidelines, such as those listed below, to have an evaluative measure that judges effectiveness. Its strengths and weaknesses should be analyzed on a strict performance parameter consisting of feedback, analytics, cost-to-performance ratio, and technological advancements.
According to the paper titled, “An Assessment of the Effectiveness of e-learning in Corporate Training Programs” by Strother (2002) , online corporate training can be evaluated on a five-level progressive scale based on Kirkpatrick’s model (1954). Strother states that, “Level I: Reaction is a measure of learners' reactions to the course. Level II: Learning is a measure of what they learned. Level III: Transfer is a measure of changes in their behavior when they return to the job after the training program. Level IV: Results is a measure of the business outcomes that occur because they are doing their jobs differently.”
Accordingly, you can measure your online training effectiveness on a similar scale:
Level I – Reaction
Reaction is measuring the learners’ response to the course. The best way to understand the learners’ response is to include a detailed feedback system in the form of employee satisfaction surveys, questionnaires, etc. This will provide an informative gauge of the learners’ perception of the course. They can state whether the content could be easily comprehended, what its strengths and weaknesses were, whether it was engaging enough, and whether all these factors led to it being memorable enough.
Gather and analyze the feedback to find constructive suggestions on improving your course. You will realize that the learners are the best critics and can pinpoint the areas of the course that need improvement. The Learning Management System (LMS) itself provides a feedback gathering system in some form or the other. It should be programmed to ask 10-15 short and succinct questions where the answers are on a graded scale (avoid yes/no answers) against the parameters discussed. Asking open ended questions may seem like a better way to gather feedback but measuring and evaluating that feedback on a fixed scale is complicated. Since the feedback provided is not on an objective scale, it makes it very difficult to analyze the performance of the online training program at different time intervals. Subjective feedback should be avoided, especially in the case of a mass automated system.
Level II – Measure
Measuring training can be done via a few methods, one is to measure the broader statistics of the course as a whole, and the other is to measure individual trainee performance regarding their learning. Before beginning, you should ascertain what you want to measure, this should correlate with the learning objectives you would have outlined for the course.
Learning analytics consist of vital pieces of information that, “are recorded throughout the duration of the eLearning course such as… how quickly they are progressing through a module, how many times have they have logged in, whether they have participated in a discussion board, etc.” (Moffett). These pieces of data offer managers a comprehensive understanding of the learners’ training. They can be collated to evaluate the broader training program, such as whether the trainees are taking longer to progress through the material than presumed which may mean that the material could be difficult to comprehend. This information can be further taken and converted into action by making the course content more user-friendly. Most LMS’s support the gathering of analytics data based on pre-determined keys that are ideally defined during the implementation of the LMS and can present reports on users, courses, chapters, and the overall system.
Assessments are a great way to gain insight into the actual level of employee information. It can be administered between modules or after the entire course, and can be intelligently designed to test against the learning objectives of that course. The scores on these assessments can be used to highlight whether the trainee is grasping the concepts/skills and whether they require any extra help. LMS’s support in-built assessment features that can report on employee performance and can further provide certification for the completed courses. However, it’s important to define the questionnaires based on the objectives and blooms level applicable for the training program. The questions should be designed in such a way that complexities are clearly defined and scoring a 100% and a 60% should aptly reflect the expertise of the trainee. An ideal certification questionnaire can prove to be a good instrument to gauge the content knowledge in the organization and can be intermittently implemented to minimize the knowledge gap.
The combined understanding gained from these two measures can assist learning managers or the training department to improve on their existing training course. They can use this information to personalize the learning experience by analyzing the performance of the trainees and noting areas of difficulty or a lack of interest; this in turn will boost learner retention rates and ensure effective learning.
Level III – Transfer of Information
There is no definitive way of measuring the actual transference of information. The best approach is for the managers to observe the trainees and make note of any difference in job performance prior-to and post training. Over time, it should be apparent whether the employee retained the information received during training or not. In online training this assessment can be achieved using scenario based tests where the user is made to react to a certain situation and their reactions can be gauged to determine their extent of learning internalized. Specific scenario-based learning and certification are designed not to check the knowledge directly, rather to check for the implementation of the knowledge. Good instructional design ensures a well-planned extensive scenario, which assesses for more than one objective and gives higher confidence to the learner on the concepts taught.
Level IV – Results and ROI
Measuring the monetary and business outcome is challenging and requires time; however, similar to the above points, this is also important. Without a true measure of the results and ROI, there will consistently be a doubt of the training program’s efficacy. A system should be put in place to measure pre-training and post-training data and watching for any changes in the measures that would indicate success in the program. In order to measure the results, you must first identify which results are closely related to the administered training. These can range from increased content recall rates, improved sales and/or production, faster execution reduction in quality issues, quicker SLAs increased customer satisfaction, etc., depending on the objectives of the training. With time you should be able to measure the cost-to-performance ratio, where you will be able to place cost of training on one side and evaluate it against the performance results. This will lead to a calculation of ROI and an indicator of the effectiveness of training.
Measuring training is important to ensure that the employees are receiving the best training possible by narrowing on the programs strengths and weaknesses and improving on the latter. No matter your method of training your employees, assessment of the program is vital. Systematic and regular research is required to ensure that the employees are learning what they are supposed to in order to be productive in their team and job. Similar to any other investment, check in once in a while to make sure that they are providing the desired ROI.