How To Use Web Analytics In Your LMS Performance Analysis To Improve eLearning Delivery
A Learning Management System (LMS) is the heart of every online training program, and links all of the business and educational processes together. Therefore, LMS performance is instrumental for achieving successful online delivery of training and assessments. In order to optimize the performance and tailor it best to the users’ needs, training providers need to be able to identify business and IT processes, as well as content elements that still have feasible scope for further improvement.
As an eLearning consultant, I usually get to work with LMSs that are already in use by the training providers (universities, colleges, corporate training departments etc.). Making improvements to “live’’ systems that are in full functional use is obviously far more taxing than working on a prototype or a development project. However, working on fully functional systems holds a distinct advantage of being able to assess the current system usage data, base improvements, and further developments on the “ user’s feedback’’. This can be done through the use of web analytics. While the analytics usage needs to be customized to the specific project objectives, this article aims to highlight some of the general concepts that could be considered throughout the system performance analysis, as well as examples of how these concepts could be applied when assessing an LMS.
Web Analytics Methods
The first step in commencing a web analytics project is to identify the Key Performance Indicators (KPIs) that the LMS should be assessed by. Each of the KPIs should be defined clearly and assigned a relative value. This task should be completed in partnership with the training provider so both the back-end and the front-end business process performance expectations are interpreted in the same way that the LMS stakeholders do.
Below are brief descriptions of 3 common web analytics methods that are used in LMS performance analysis projects, namely:
- Learner Web Log Analysis
- Learner Life-Cycle Analysis
- Learning Progression Analysis
1. Learner Web Log Analysis
Learner Web Log Analysis will help to determine the current state of learning activities. Web Log Analysis (WLA) involves establishing when, how, and by whom various sections of the system are visited. It is also possible to generate activity reports based on the data collected and, based on those reports, monitor nature of the activities and changes to the “demand”.
Data collected and analyzed via the WLA can vary depending on the KPIs investigated. Some of the standard WLA indicators to be investigated are:
- Page/site/section visit times, durations & number of visits
- Most/least viewed pages
- User authentication/access times per individual user
When applied to an LMS, such analysis can provide a lot of useful data sets that reflect upon various system performance indicators, such as identifying sections that are rarely accessed by learners or vice-versa—identify sections that are accessed more frequently than others. For example, an LMS may incorporate a forum for the learners to discuss course-related issues and to get together into study groups, but the WLA may reveal that few of the users access this section of the system/site and those who do visit it rather infrequently. This data can be useful for reviewing current settings and layout of the forum. The next step of the LMS forum redesign project would be to establish reasons for the forum’s failure to attract learners and to see whether it is underemployed due to the users’ trivial unawareness of its existence, poor layout, or lack of linkage to the courses offered.
Likewise, if particular sections of a learning site receive a lot of clicks, it means they are truly adding value to the system (at least as far as the learners are concerned), unless learners access the pages, then depart immediately without spending considerable amount of time to investigate the content or to download study materials.
2. Learner Life-Cycle Analysis
The ‘’Life-Cycle’’ Analysis (LCA) focuses on the so-called visitor-centered approach and investigates KPIs related to individual learners or separate learner groups, such as classes or study majors. Some of the parameters that could be applied are:
- Comparative use of the resources across different subjects/study areas
- Technology adoption level for the groups (e.g. how many of the study-enhancing apps provided are being utilized)
- Comparative participation (learning activities, discussions etc.)
During the early days of eLearning, training providers could (at times justifiably) rely on certain assumptions rather than on the data collected, as the online learner stereotypes were consistent and rarely deviated from the reality. Today, the situation is clearly different. Depending on the nature of the learning and assessment tasks, students of Psychology or Natural Therapy may surprisingly turn out to be using the online systems to a greater extent than students of IT or Engineering.
LCA can also be used to establish how the learning resources/assessments are accessed and used by these groups throughout the course. It is an important element of the analysis, as the key challenge behind developing an efficient LMS is not linked to technical requirements (balancing traffic load, download times etc.) but to business and learning processes. Also, it may provide valuable information on how user-friendly different facets of the LMS are from the perspective of various user groups.
3. Learning Progression Analysis
LPA is based on an effective combination of data available via web metrics-driven analysis of LMSs and data (including qualitative data) available on the learning outcomes achieved via the online training programs.
LPA is fairly complex when compared with traditional web metrics analysis methods, as it cannot be accomplished without looking beyond system usage and understanding not only nature and context of the activities that the learners undertake while logged in but also relative effectiveness of these activities from a knowledge building perspective. For example, if an LMS is employed to provide learning support to students of a certificate course in Chinese Medicine, LPA can establish the effectiveness of the online facilities available based on the students’ performance. If a link between going through online exercises and practice quizzes and achieving higher grades becomes evident, the LMS is obviously fulfilling its purpose well, but if not further investigations should be carried out to understand the reasons behind the poor positive impacts of these tools and facilities on the learners’ performance.
LMS performance analysis is always a large analytical task involving combinations of tools and approaches so it is impossible to provide a “ detailed step-by-step guide’’ on how it can be carried out in a brief article. Furthermore, all LMS platforms differ from one another, and so do training deliveries (scope of the training programs, intensity, testing requirements, etc.) and the objectives behind these deliveries. However, understanding the principles of the web analytics is always the very first step in ensuring that the resulting assessment is going to be valid!
