Seeking Integrity In VR Educational Research 2: PwC VR for Soft Skills
Credit: Me and Midjourney
My first article in this series garnered so much attention! But many folks tried to pass me Mirjam Neelen & Paul A. Kirschner's Truth or Truthiness? Analysing a VR Study Using Gorard’s Sieve article on the PwC report entitled "The effectiveness of virtual reality soft skills training in the enterprise: a study" and all of its associated webpages like this one. I was like, I know! Mirjam & Paul wrote their article 2020 and I wrote about it in 2021. What's cool is that separately, we both came to the same conclusions. That's a good sign for our conclusions!
Short version: we both cast strong doubt on any conclusions.
Still, I realize the world does not revolve around me (sigh!). Some folks might have missed my long stream-of-consciousnesses article about the PwC report. I decided that the second article in this series should be an abbreviated and updated critique. Bear in mind that to reach the LinkedIn audience, I have to leave much nuance by the side of the road. If you have questions, just ask!
As Mario says "Here we go!"
What is Said About The Report
This infographic summarizes the dominant conclusions:
- 275% more confident to act on what they learned after training
- 4x faster than classroom training on average
- 4x more focused than e-learners
- 3.75x more emotionally connected to the content than classroom learners.
What do these Google results have in common?
They are all companies that sell some sort of VR product or service.
Because I was curious, I checked out that vrowl dot io link ("Virtual Reality training is not effective") just to see if it was presenting an alternate opinion. It's a strawman argument; it puts up "not really real" protests against VR for learning and then explains them away. I'm telling ya, Beware the VR Strawman.
What the Report Says
Eckert, D., & Mower, A. (2020). The effectiveness of virtual reality soft skills training in the enterprise: a study. https://www.pwc.com/us/en/services/consulting/technology/emerging-technology/assets/pwc-understanding-the-effectiveness-of-soft-skills-training-in-the-enterprise-a-study.pdf
Let's ask Google Scholar what it thinks. It's coming up with 11 cites. That's not much at all. But as I showed above, the money shot is on the Internet, not in academic articles.
Truly, the 4x faster learning quote is the runaway train of this report.
BTW, Google Scholar isn't loving this report (I'm snickering) and it seems to go by at least 2 citations:
Eckert, D., & Mower, A. (2020). The effectiveness of virtual reality soft skills training in the enterprise: a study.
Eckert, D., & Mower, A. The effectiveness of virtual reality soft skills training in the enterprise: a study. 2020. PricewaterhouseCoopers:[Place of publication not identified], 73.
I found one paper that referred to it and sadly Eckert didn't make the final proof as an author:
My Take On the Report
The other 3 major conclusions don't really mean much to learning results:
- Being more confident - the 275% number looks bogus (Dunning-Kruger effect?), there is no 100% and it is NOT clear how this was measured.
- Focused - was self-reported data backed up by no instrumentation.
- Emotionally connected - this was VR headset training in a 1st person perspective, again bogus comparable setup to classroom.
All of these learner-centric measures do not have a correlation to actual learning results.
Said another way, learners have about a 50/50 chance at self-estimating their own current and future abilities. They suck at it.
I know, I'm putting poor Thalheimer in the way of a lot of educators that think these things MUST lead to better learning. Sorry folks, this is where the parting of the educational research waters happens. It's either proven in research or it's not. Just hoping for something does not make it so.
=The 4x Faster
Let's look at the 4x faster:
Their data:
Classroom training: 2 hours or 120 minutes
E-Learning: 45 minutes
VR: 29 minutes (I'll round up to 30 minutes)
Time for math!
Q: What is 30 minutes compared to 120 minutes?
A: 30 is one-quarter of 120. 30 is 1/4 of 120. Therefore 120 is 4 x 30. From there you get the 'VR is 4x faster' line.
So the data metric they used is VR is 4x faster. That metric -- on the face of it -- is legit--they are not lying about their own disclosure of their own data. Taking 30 minutes do some learning is 4x faster than taking 120 minutes to do the same learning.
But...I have to flag the play twice here.
Non-comparable instructional methods
This time data is literally NOT COMPARABLE. The cognitive load on a learner to engage in 30 minutes in a VR headset is not the same work on the brain as learning in a classroom with other learners.
Let me show you. And this works without VR.
Let's pretend you have to complete mandatory HR training for work.
You can either:
A. Take the 2 hour class, where you are bunched together with all kinds of learners, some who learn faster than you, some that learn slower than you, etc. It doesn't matter because the teacher has been contracted for 2 hours and she's got to help everyone cross the finish line. That means that the 5 minutes she spent to re-explain something to one student made all of the students wait. It's just the way the [industrialized] ball bounces. I've probably described what your 4th grade class was like.
Or
B. You can take a 1:1 class with teacher. She's got a similar mission: to get YOU across the finish line with the training. Some of it is easy, you breeze through. You got stuck at one point, but because the teacher was observing you so closely (she's focused on you), she easily pinpoints the place where you got stuck and gives you help to get through.
Choice A took 120 minutes. No less, no more.
Choice B took 30 minutes, because it moved at YOUR pace.
See how the learning experiences are not comparable considering your brain burden? In Choice A, you sighed and checked your phone when the class dragged. In Choice B, you moved right along, skipping parts you already knew.
It isn't the VR that caused the learning to be accomplished faster, it was simply the 1:1 nature of the training.
The line "VR causes faster learning"
My second flag on the play is that the "line" that has entered our common conversation is that VR is faster or that VR causes faster learning.
But have we determined that here? In whole or in part? No. The learner's brain cells didn't fire off faster, nor form more neural connections, nor embed deeper in the emotional cortex, nor all kinds of other not-experimented-upon and therefore not known neuroscience measurements of learning.
There are side arguments from PwC that the VR headsets caused more focus because they isolate the learner's field of view; the less distraction causes more focus hypothesis. The experimental research that we have so far is mixed.
ICYMI: What about those blue and red brain scans when learners used VR?
If you'd like to read more on the research on presence, embodiment, and immersion, check out Mayer's Multimedia Learning text (2020, see Chapter 18) or my XR book chapter from 2021 or my summary video.
But wait there's more!
In this report, it has a little gem hidden away on page 44:
Section: "Higher information retention
We quickly discovered retention scores were inconclusive, as the delta between pre- and post-assessments in each modality was not significant. Indeed, the assessment team underestimated the previous knowledge experience our test population had on the diversity and inclusion topic. In hindsight, we should have selected a topic that was not already in our curriculum or selected a different test group that had not already been immersed in similar training.
We did test several associates and senior associates (1-2 levels below manager) who had not been exposed to diversity and inclusion behaviors in leadership functions,and the results demonstrated significant retention scores and learning in a v-learn course. However, these were not approved study participants, therefore their data was not part of our assessment or these results.
Despite our inability to demonstrate learning via assessment, 75% of those surveyed reported experiencing a wake-up call moment about the inclusivity of their behaviors. This indicates that learning did take place, even though our assessments were unable to demonstrate it."
Plain language: the learners didn't learn. Their scores didn't go up.
So after all of those 4 infographic statements, this one isn't touted.
Buckle up your seatbelts and prepare your passports for stamps because we are going to find quite a few #NoSignificantDifference results in my continuing article series.
What I find funny about this section is that it reads exactly like some perplexed person in HR with no educational psychology background. It's like she seems confused why after thousands of dollars of development and training (oh, excuse me, the training was sponsored by Oculus and Tailspin...so...freebie?), the learners didn't learn more.
In these paragraphs, the result is getting explained away. 'We should have picked an unfamiliar topic' or the learners 'reported experiencing a wake-up call'.
Sorry Charlie, none of this explaining away a not significant different set of results (and truly, given the overall doubt cast on this entire paper, I can't be sure that they know how to calculate a significant difference) clears up what happened. The learners didn't learn more.
Ironically, as much as I don't like this report, the two parts I like are this #NoSignificantDifference result and the cost estimation. If you can put the dots together, this report showed that, once the cost of developing VR for learning drops below e-learning (and the report says it's at 3,000 + 1 learners), VR for learning will increase in adoption. That's 2020 dollars. The cost of developing VR is dropping.
So it's a silver lining. Just not one that made the infographic.
As Hill Street Blues would say, "Let's be careful out there."
Post script
I took a stroll around the PwC website. Looks like this is going on.
#VirtualReality #VR #XR #VRForLearning #Technology #Future #edtech #learning #education #UserExperience #InstructionalDesign #research #ComparisonResearch #Media #MediaForLearning #ImmersiveExperience #Design #ResearchIntegrity #VRIsFaster #NoSignificantDifference
This post is simultaneously published on LinkedIn at
https://www.linkedin.com/pulse/seeking-integrity-part-2-vr-soft-skills-heather-dodds-ph-d-