Seeking Integrity in VR Educational Research
.png)
I'm starting a new article series today, calling out 'bad research' or research that is quoted badly in virtual reality for educational use. I thought I would start with a whopper - a really egregious example to start this series with a bang. Then I checked my notes and realized that this example is from LAST MONTH, June 2023. I'm not even going into the vault for this. I'm barely picking myself up off the ground from the shock wave.
So, like Mario says "Here we go!"
What Is Said About The Research Versus What The Research Says
June 2023, LinkedIn Post:
"According to a study from the University of Maryland in 2018, learners remember an astounding 90% of what they experience in VR compared to merely 10% of what they read and 20% of what they hear."
I believe this is the research referred to:
Krokos, E., Plaisant, C., & Varshney, A. (2019). Virtual memory palaces: immersion aids recall. Virtual reality, 23, 1-15. https://obj.umiacs.umd.edu/virtual_reality_study/10.1007-s10055-018-0346-3.pdf
Hey, I'll give you the abstract because I know you don't like to read long papers:
"Virtual reality displays, such as head-mounted displays (HMD), affords us a superior spatial awareness by leveraging our vestibular and proprioceptive senses, as compared to traditional desktop displays. Since classical times, people have used memory palaces as a spatial mnemonic to help remember information by organizing it spatially and associating it with salient features in that environment. In this paper, we explore whether using virtual memory palaces in a head-mounted display with head-tracking (HMD condition) would allow a user to better recall information than when using a traditional desktop display with a mouse-based interaction (desktop condition). [OK skip to here because this is the interesting part:] We found that virtual memory palaces in HMD condition provide a superior memory recall ability compared to the desktop condition. We believe this is a frst step in using virtual environments for creating more memorable experiences that enhance productivity through better recall of large amounts of information organized using the idea of virtual memory palaces."
Google Scholar tells me this study has been cited 461 times. That's a low-medium citation number. Not bad, and remember that's in ~3 years of time.
Believe it or not, I'm walking RIGHT PAST that 90%, 10%, and 20% because it has already be debunked here and here. Also, to be fair to the research paper, it never quotes those 10 and 20% numbers.
My Take on the Research
Research found 90.48% recall in the headset condition, with a 78.57% score from the desktop display control group. So that’s ~10% higher with the headset.
From Section 4.1 "Using a paired t test with Bonferroni–Holm correction, we calculated p = 0.0017 < 0.05 which shows that our result was statistically significant."
Interesting. I'm not familiar with Bonferroni-Holm correction. Just looking at it, it appears to be a method of discarding some data. I wonder if NOT using it showed a not statistically significant difference between the 90 and 78. Their n was 40. Smaller group sizes means it can be harder to justify the data as fitting a normal bell curve.
Figure 5 shows the data and just looking at it, you can see that the numbers landed in similar scores. The boxes overlap, so whatever the effect of VR is, it's not that substantial in this study. Students were learning, regardless.
But here comes the whopper. Check out this little detail in the Materials section:
"For this study, we used a traditional desktop with a 30 inch (76.2) cm—diagonal monitor and an Oculus DK2 HMD. The rendering for the desktop was configured to match that of the Oculus with a resolution of 1920 × 1080 pixels (across the two eyes) with a rendering field of view (FOV) of 100◦. In order to give the desktop display the same field of view as the HMD, the participants were positioned with their heads 10 inches (25.4 cm) away from the monitor."
10 inches away
The "control group" sat 10 inches from their desktop monitor to use the desktop condition.
WHO DOES THAT?
You know, I was curious. I grabbed my ruler.
I'm currently sitting 24 inches from my monitor. I leaned in to feel what 10 inches is like.
At that point, it became no wonder to me that the control group scored about 10 points lower. It was maddening. Remember, the learners had to look all around themselves so completing learning at 10" from the monitor would be...uh...weird?
This is a great example of not seeing the forest for the trees in VR in education design. In order to match the field of view, they forced learners to unusually use their desktop monitors.
There is too much. Let me sum up.
The quote is from a keynote speaker at a research conference. I can't believe anyone in the audience did not flag the play on the quote, the percentages, or the design setup of the U. of Maryland study. At the industry.
- The difference between 90 and 78 *might* be too close to call a difference caused by VR.
- Setting up learners to use a monitor from 10 inches away is unusual, to say the least.
- When research sets up unfair comparison conditions, the results should be questioned.
As Hill Street Blues would say, "Let's be careful out there."
What do you think?
#VirtualReality #VR #XR #VRForLearning #Technology #Future #edtech #learning #education #UserExperience #InstructionalDesign #research #ComparisonResearch #Media #MediaForLearning #BonferroniHolm #ImmersiveExperience #Desktop #Design #MemoryPalace #ResearchIntegrity
This article is co-published to my LinkedIn account here: https://www.linkedin.com/pulse/seeking-integrity-vr-educational-research-heather-dodds-ph-d-
Attribution-NonCommercial-ShareAlike
CC BY-NC-SA