In The Presence Of Consequence

VR Assessment And Experience A New Perspective
Summary: In this article, I will suggest a shift from the instructional design philosophy that has driven our VR work to date to an assessment-focused design strategy that more closely resembles a game design.

Assessment-Focused VR Experiences

Hello VR friends, in this article, I will describe the predominant design philosophy that has driven our VR work to date. It’s a philosophy that has resulted in producing primarily instructional-based VR experiences for the purposes of teaching, learning, and research. I will describe what I see as flaws in this philosophy and I will propose a new approach that might influence our work going forward. This new design philosophy will favor producing assessment-focused experiences over instructional-based experiences.

I recently finished reading a book called Experience on Demand, by Jeremy Bailenson, who is a professor of communication at Stanford University and the founding director of the Virtual Human Interaction Lab. This book chronicles some of professor Bailenson’s extensive research in virtual reality. Bailenson has been working with VR technology since the early 2000s but he is also somewhat of a historian and references work that was done as far back as the ’80s. The point being VR research is not new, however, what is new is that for the first time virtual reality technology is now small and affordable enough for mass adoption to take place. While previously VR equipment could only be found in labs like Bailenson’s for a cost upwards of $30k, now anyone can own an HMD (head-mounted display) for under $300.

My experience with VR is small compared to Bailenson’s. While I have been developing interactive media for over 22 years, including educational game design, I have only been working with VR since 2014. I was excited to read his book to see if any of his theories, findings, and philosophies around VR development were similar to mine. It turned out they were! The book is largely stories about some of the VR experiences his lab has developed, with some brief reporting about research findings around those experiences. While the book does not offer any concrete advice on what makes for successful VR, it does provide some guidelines. It’s these guidelines where I found the most commonality between what Bailenson’s lab has done at Stanford and the work I try to do and encourage at Penn State.

Here are my 7 biggest takeaways from the book:

  1. While VR might be more engaging than traditional instruction, getting people interested and caring about knowledge, actual knowledge transfer shows only minor gains over traditional, far less expensive methods.
  2. When you directly experience something the way you can in VR, you see it in a different light.
  3. "VR is about exploration; storytelling is about control." Using a technique called "Discovery Narrative" is more effective than traditional linear storytelling.
  4. What makes a good VR is when the entire environment surrounding a viewer is important to the experience. The experience should require a user to explore by either turning their head or moving around.
  5. "Ask yourself: Does this need to be in VR?" By considering all other options first, you can design better VR experiences.
  6. At all costs, don’t make people sick. Framerate over graphics, always!
  7. VR experiences should be supplemental to other learning methods, not a replacement. Experiences should be short in duration; even twenty minutes is a long time in VR.

Working for teaching and learning with technology at Penn State University, it’s my job to discover ways of integrating VR in a classroom. Most of the work we have done so far focuses on the instructional side of teaching and learning. We primarily have been trying to find ways of using VR as a knowledge transfer tool. After reading Bailenson’s book and learning about some of the experiments and studies his lab has done, I thought a shift in our design philosophy might be beneficial.

The shift I am proposing is to change our current "design an instructional experience" mindset to a "design an assessment experience" mindset. This is not a new approach. It is a method I’ve used over and over when designing educational games. With VR experiences, we have drifted away from assessment and measurable metrics as the foremost design considerations. (Even though for research this is ultimately what we are looking for and in some ways have been tacking on at the end.) I believe there has been a rush to cram instructional content into a headset and then simply examine if people respond well to it. After being involved with the creation of several underwhelming instructional-based VR experiences and after reading Bailenson’s book where he concludes from multiple studies that knowledge transfer from VR experiences has only minor gains over traditional methods of teaching, it’s time for our design philosophy to move on and create richer, more meaningful user experiences.

While thinking about this shift in design philosophy, a few (possibly research-worthy) questions came to mind:

What Separates Instructional-Based Design And Assessment-Based Design?

Instructional-based design is filled with a temptation to create content not suitable for VR. If we always carefully consider takeaway #5 from Bailenson’s book: "Does this need to be in VR?" we should realize most instructional content can be more effective and more efficiently delivered by other means. Users can quickly identify aspects of VR experiences that will not require them to have a headset on. Wearing a headset is uncomfortable. To ask users to listen to a lecture, or read large amounts of text, or participate in any other poorly conceived VR content can result not only in distraction but in rejection of the purpose of an experience.

There is also a temptation to cram too much content into instructional-based VR experiences. Remember takeaway #7: VR experiences need to be short, definitely not exceeding twenty minutes. The visceral reaction to a VR experience feels more like a rollercoaster than a long slow train ride to most users. It’s a difficult proposition to expect a user to absorb a great deal of information in a short amount of time, especially if the content does not clearly justify itself as appropriate for VR.

The goal of designing assessment-based VR experiences over instructional-based designed experiences is to avoid using a screwdriver to hammer a nail. We want users to care about experience, discover on their own why they are participating in it, and realize it wouldn’t be the same if the experience happened any other way. VR is a chance to experience situations, not be fed with information. Properly designed assessment-based VR will allow instruction to happen more organically. It should allow users to explore while being observed and measured in real time, as opposed to being told information and tested later.

Bailenson points out that VR experiences should be supplemental to other learning methods. Assessment-based VR can fit nicely as a bookend to other learning. Provided as pre and post-tests, a VR experience can provide users with measurable tasks and expose where challenges lie. These challenges, which may be unique to every one, can be addressed with feedback and instruction outside of VR before letting them try an experience again. Not only will this validate a VR experience but also the instruction that accompanies it. A well-designed VR assessment experience is an instructional experience that disguises a lesson in an activity suitable for the technology.

Can VR Be The Most Effective Assessment Tool Ever Developed?

Maybe. I don’t like to prognosticate too much about the future with these technologies. I would rather focus on what we have today and what we can do with it. But it’s not hard to imagine that five, ten, or fifty years down the road, VR might be capable of creating experiences undisguisable from reality. If this is true, then combining a VR experience design with a game design can result in assessment methods that will give participants an unrivaled measurement of realistic experience and feedback. Imagine a police training that could assess when officers are ready to handle hostile situations because they had performed well in virtual ones.

From Bailenson’s experiments, we know VR can make users care more, even feel more empathy, and that direct experience leads to seeing situations in a different light. With hyper-realistic VR, it will be possible to give people experience in any situation, allow them to fail safely but make them feel as if though the consequences are real. The experience, then, can be evaluated, provide feedback, and repeated as many times as needed until an acceptable level of performance is achieved.

Can VR Be Effective Enough At Convincing Users They Are Present In A Moment That They React As Though They Are?

If VR experiences can make users feel present in a moment, then they might also make them feel the consequences of their actions at that moment are real. Obviously, there are real consequences to traditionally written exams that come in the form of a letter grade. But, in real-life scenarios, examinations of knowledge don’t always come in written form and consequences are rarely revealed as a letter grade. VR experiences can give users a heightened understanding of the impact and the importance of the knowledge they are being asked to display beyond an evaluation of themselves.

Displaying knowledge or skill is one thing; but can you do it in a moment that matters? Understanding, feeling, and caring about the consequences of your actions add pressure to any examination. I’m a golfer. I can tell you there is a big difference between playing golf with my friends and playing in a tournament. The psychological difference between casual performance and pressure-filled performance is real. It can affect one’s ability to both mentally and physically react. Any professional athlete will tell you that "game-time" experience is difficult to simulate. VR experiences might be uniquely equipped to provide pressure-filled performance assessments because of their ability to make users feel present in a moment.

Does The Assessment-Based Design Plan Include Good VR Mechanics Tied To Quantifiable Metrics That Define Success, Failure, And Progress?

When designing educational games, we always identify what the learning objectives are first. We then search for metrics that can be used to indicate those objectives are met. Finally, we design actions and events that allow players and responding systems to affect those metrics. These actions and events are referred to as the mechanics or gameplay aspects of an experience. It’s vital that the mechanics are designed to utilize the interactivity affordances of a delivery platform. If all goes well, this design methodology results in an effective assessment tool in "game" form.

For example, here’s an assessment-focused design plan we developed for a game targeted at managers of a walk-in health care clinic. The game is called "Time and Patients." The learning objective of the game was to successfully manage the clinic for an entire week. The metrics we came up with as measurements of success were: a) revenue, how much money is the clinic making, b) throughout time, how long does each patient spend in the clinic from arrival to check out, c) satisfaction, how satisfied are the patients with their experience in the clinic, and d) walkouts, how many people leave the clinic because the wait is too long. Next, we designed actions like reviewing resumes, hiring staff, and scheduling staff. We also gave players the ability to add amenities to their clinic, like a fish tank or WiFi, for example. Every action we made available to players was designed to somehow affect the metrics of the game.

We were not yet using VR when I developed Time and Patients. If we had been, we might have considered building this as a VR experience. However, it would not have been a very good one. Why not? We had a perfectly good design plan but we still had to consider #5 on the list of takeaways from Bailenson’s book: "Does this need to be in VR?" Our design plan’s main actions included reading resumes and filling out a schedule, neither of which required users to explore or interact with anything spatially. Accessorizing the clinic could have been done in a full-scale virtual clinic, seemingly satisfying #4 on the list of takeaways from the book. However, the spatial arrangement of the amenities was not particularly important; only the fact that they were there was a factor affecting the metrics. The mechanics in our design plan, while effective in assessing knowledge of learning objectives, were not good at VR interactions. Time and Patients ended up being a successful browser-based HTML5 game.

Time and Patients is a good example of why planning is important. A design process that uncovers learning objectives and methods to get to them is critical regardless of the delivery platform. It is the process that will identify how technology might be useful or why it might be ruled out. Like games, creating VR experiences is expensive and takes a long time to produce. However, designing assessment-based experiences using metrics and the actions and events that affect them can be prototyped fairly quickly using even a simple text. This is what is referred to in the game industry as paper prototyping. I actually developed a web-based tool for doing this years ago called "WIND." This cycle of planning and prototyping allows for a quick iteration before any time-consuming production of graphics and programming takes place.

Does VR Technology Have Any Unique Ability To Capture Assessment Data?

Absolutely. There is already built-in technology in some HMDs that allows for eye-tracking and heartbeat monitoring. This is the start of capturing data that can provide answers to what users pay attention to and how they are reacting to it. Decision-making, reaction time, movement through space, and identification of priorities are just some examples of data that could be collected during an assessment-based VR experience. This ability to generate assessment data, not usually found in traditional grading transcripts, can have a profound impact on evaluating the level of one’s knowledge and the effectiveness of education.

Is VR Assessment A Chance To Rewrite The Grade Book?

Maybe. There is an opportunity in VR experiences to evaluate and provide feedback from a wider spectrum than traditional letter grades. There are grey areas between success and failure in many aspects of life. The separation between an A and B letter grade might be feeling significantly different if conveyed as consequences in a VR experience. VR experiences could provide insights into a user’s knowledge beyond right and wrong answers.

This shift in design philosophy away from a purely instructional to an assessment-based design is not as radical as it may seem. I mentioned above that, essentially, assessment-based design is instructional design with instructions hidden in the mechanics of an experience. In many ways, the term "assessment-based design" is a way of rephrasing the term "game design" to make it seem less frivolous. It still seems to many educators that the word "game" is a four-letter word. We often disguise that word by saying "simulation," or we tack on other adjectives like "educational" game or "serious" game.

The truth is: Instructional design, game design, and even software development cycles have a lot of similarities. We’ve drifted away from game design in VR for a number of reasons. We need to circle back to some established design theories rather than try to learn everything about VR from scratch. I believe we’ve become too enamored of the new(ish) technology of VR. There has been a lot of experimentation with taking content and throwing it in a headset without considering users' experience in design. Ultimately this is what VR is: a technology vehicle for user experiences. It is not simply a content delivery mechanism; we already have plenty of those. The fact is not all content is a good fit for VR but with a proper and proven design philosophy in place, we can better position ourselves to find out what is.