Instructional Design in the Metaverse Part 3 Myths versus Reality
.png)
This conceptual series proposes instructional design principles for the metaverse. You've arrived at Part 3.
TL;DR
- XR causes more learning and faster - Myth
- XR is active learning - Myth
- More immersion is better than less - Myth
- XR causes greater retention - Jury is still out.
- XR increases empathy - Danger Will Robinson Danger
- Learners and instructors like it - True! But this means nothing to learning.
- XR can impact far transfer - Jury is still out.
- Positives get published - True, so negative or null results often do not.
Ready? Let's do some myth busting.
Myths versus Reality
The metaverse in education is an emerging topic and a potentially lucrative field, possibly set to supplant learning management systems as the next industry-wide educational platform (Spilka, 2023). Improvements in technology are fostering an “anytime at anywhere” implementation of the metaverse (Tlili, Huang, Shehata, et al., 2022, p. 4). As interest in the metaverse as educational media has increased, misleading claims or myths have already circulated. These myths are often shrouded under the title of research until the curious probe a little deeper. This part will examine claims such as the metaverse will cause learners to learn more and faster, it represents active learning and is therefore better, it is more immersive than ever, learners retain more, it increases empathy, and learners like it so therefore they learn better. This section will mention areas where the research is still unknown, publishing bias, and what to look for when IDs read educational research on the metaverse.
1 More and Faster
Learners in the metaverse will learn more and faster; this is the first claim examined. IDs should maintain a healthy skepticism of claims that a certain media causes dramatic learning improvements. Claims often do not communicate instructional methods as Beck, Morgado, and O’Shea (2023) pointed out. Instead, current publishing focuses on outcomes-based research or what Reigeluth and Honebein called research-to-prove results that are “typically operationalized by comparing a new medium to a traditional one” (2023, p. 2). For instance, a lesson in XR could be compared to a lesson in a textbook. Similarly, the National Academies of Science, Engineering, and Medicine advised that possible external validity of some studies is low in that “there is considerable evidence that a single instructional technology can lead to different outcomes when used by different learners in different contexts” (2018, p. 194). These claims usually represent the pitting of two very different instructional methods and thus cognitive workloads against each other.
Here is an example of this claim. In a study incorporating virtual reality (VR) headsets for soft skills training, Scott Likens of PricewaterhouseCoopers claimed, “We found the realism and performance feedback in virtual reality simulations helped people learn faster and retain more information around soft skills,” (Zielinski, 2021, para. 9). However, the accompanying published report contradicted these claims. When comparing information retention in VR versus an e-learning course, the authors “quickly discovered retention scores were inconclusive, as the delta between pre and post-assessments in each modality was not significant” (Eckert & Mower, 2020, p. 44, emphasis added). Thus, the two different media showed no different learning outcomes.
Another claim is that the isolation effect of a headset causes faster learning, perhaps arguing that less distraction equals more focus. In the same study, Likens stated, “A lot of courses that normally take an hour could be completed in 20 minutes through VR because people are so immersed in scenarios, there are fewer distractions and the learning is very concentrated” (Zielinski, 2021, para. 10). Referring to the same study, “VR was x4 faster than classroom and x1.5 faster than e-learning” (D. Clark, 2022, p. 190). Claims that learning is completed faster attempt to represent XR as a more efficient learning method, i.e., less time to learn equals learning faster. When compared to classroom learning, it is already known that 1:1 personalized learning is faster. In this case, the classroom learning was allotted to two hours and the VR experience took 29 minutes. Given that 29 minutes is approximately one-quarter of two hours, the touted line was that XR was four times (4x) faster. In fact, the XR media did not cause the learning to be completed faster, it was the 1:1 nature of the learning experience.
[Editor Heather here: this is the same study I wrote about extensively here and here and my colleagues wrote about here, in case you want to read more.]
Further, there is at least one study that refutes this focusing-causes-faster-learning claim. Makransky, Terkildsen, & Mayer have found that immersive metaverse environments can be sensory overload for learners and therefore decrease the learner’s focus (2019). On the whole, claims for increased speed can often be attributed to more efficient learning methods.
Lauding the media that manipulated instructional methods hides the fact that the learner could achieve the same results in a different media, given comparable time and resources.
I'm curious that the 4x was applied to "retention rate" and "attention span" and was compared to Teams or Zoom, which, to the best of my knowledge WAS NOT in the 4x PwC study.
I feel some, ahem, elaboration has occurred here. And I find it interesting that while propping UP Teams (because it is rolling out immersive team meeting environments) this paragraph highlighted actually disses Teams. I'm thinking this person is so excited and into rolling out stats that he's confused stuff.
2 Active Learning
Some claims state that learner-instigated avatar movement, in the form of moving hands, heads, or bodies, or the first-person point of view makes XR learning inherently active as opposed to passive. Intentional avatar movement is associated with manipulating content, which is the term embodiment or embodied learning (Johnson-Glenberg, 2018; Markowitz, Laha, Perone, et al., 2018). The claim begins with the given that active learning is known to be better than passive learning. Because XR is body-movement active, it must be active learning and thus cause more learning (Johnson-Glenberg, 2018). However, research has shown that while embodiment does have a connection to learning, it does not exclusively cause learning. Truly, “platform is not destiny” as Johnson‐Glenberg, Bartolomea, & Kalina stated in 2021 (p. 20). Just putting a learning experience with movement into XR does not make it active learning.
3 More Immersion
There are some claims that take issue with the second assumption stated earlier in this series: Learning outcomes are expected to be equal to other media (Mayer, 2020). These claims state that earlier comparative media studies did not show improved results because the technology then was older. Thus, technology now utilizes a better quality of immersion. This claim reflects a modernist philosophical approach: newer is better. Cummings and Bailenson (2016) reported that head tracking, stereoscopic visuals, and wider fields of view created more immersion than other visual or audio improvements. Yet Mayer wrote in 2020, “these comparisons between low-immersion and high-immersion media do not provide strong evidence for the instructional value of converting a 2D lesson rendered on a computer screen into a 3D lesson displayed with a head-mounted display in immersive virtual reality” (p. 365). Moreover, Abbas, Seoo, Ahn et al. (2023) found that high levels of presence did not impact user behaviors. Ochs and Sonderegger (2022) reported that learners that felt an increased sense of presence in VR scored worse on measures of memorization even when the learner simultaneously self-reported that they expected to do better in VR versus 2D. Finally, Makransky, Terkildsen, & Mayer found that VR causes more presence but less learning (2019).
Overall, these claims also fail to acknowledge that the main subject in media studies are humans (not the media), and we already know a great deal about how humans learn in 3D environments. These spaces exist outside of technology and are called classrooms. These claims that newer XR will cause more learning look more like calls to buy the latest technology.
That is not instructional design research, it is marketing.
4 Greater Retention
This claim states that XR enhances knowledge retention (Victor, 2023). Studies of retention are still ongoing and difficult to find. Indeed, broad reviews such as those conducted by Hamilton, McKechnie, Edgerton, and Wilson (2021) commented that finding “learning outcomes, intervention characteristics, and assessment measures associated with immersive virtual reality use has been sparse” (2021, p. 1). This aligns with Beck, Morgado, and O’Shea who contended that, “Very few literature reviews focus on the educational practices and strategies used in immersive learning environments. Thus, the problem is that we are evaluating outcomes without a comparable way to describe the educational approaches that led to those outcomes” (2023, p. 2). Therefore, retention could be achieved with XR implementation, but without more research detail, greater results might be attributable to the method, not the media.
The use of the metaverse in education is not yet common. It is difficult to find studies that measure retention within learners more than 10 to 21 days after instruction. Practical workplace implementation would require much longer retention times. Therefore, this claim has not yet been supported or refuted.
5 Increase Empathy
Research on empathy indicates that this is an area of risk. Because of a first-person point of view in many XR experiences, learners perceive a direct impact of the experience which is meant to foster empathy. However, empathy, like presence, is nuanced. Indeed, in some empathy research, learners did not react with a positive and caring response, but instead with disgust and rejection (Bailenson, 2018). Thus, the objective of the experience might be not only missed but soured.
Clark, when writing about accessible pedagogy in immersive learning, advised to avoid first-person depictions of marginalized groups because XR experiences cannot portray the depth and spectrum of a person’s life. “Instead of teaching students what it’s like to be blind, consider having them deconstruct the ways vision is assumed in how spaces are designed, as well as the ways their understandings of vision impact how they interact with others,” then “focus on bringing awareness to the assumptions built into the physical world around them” (J. Clark, 2021, Recommended Administrative Considerations section, para. 4). Therefore, XR to foster empathy should be approached with extreme caution.
6 Learners and Instructors Like It
This learning claim, that learners liking a learning experience will then learn more, is perhaps the most common. To the contrary, there is no research-based connection between liking an experience and learning success (Hughes, Gregory, Joseph, et al., 2016; Uttl, White, & Gonzalez, 2017) . Thalheimer further discounted the connection between learners liking their learning and achieving their learning, when he stated that “measuring interest is an inadequate way of measuring learning” (2018, p. 26). Therefore, while it is pleasant for learners to enjoy an experience, it has no firm connection to a learning outcome. Beyond the learner, education professionals should use caution when thinking that emotional motivation will work over the long term. Instructors often become enthralled with possibilities of XR and start to believe that the feeling of being there (a combination of presence, embodiment, and immersion) will make a positive difference. Research and theory are not forecasting this. However, the flame of excitement among professionals should not be extinguished. Clark and Mayer advocated for a tempered approach where the ID can keep to the best learning practices and not be distracted by the media:
“The challenge in e‐learning, as in any learning program, is to build lessons in ways that are compatible with human learning processes. To be effective, instructional strategies must support these processes. That is, they must foster the psychological events necessary for learning. While the computer technology for delivery of e‐learning is upgraded regularly, the human side of the equation—the neurological infrastructure underlying the learning process—is very old and designed for change only over evolutionary time spans. In fact, technology can easily deliver more sensory data than the human nervous system can process. To the extent that attention‐grabbing audio and visual elements in a lesson interfere with human cognition, learning will be depressed” (2016, p. 24, emphasis added).
7 Near Versus Far Transfer
Claims about near transfer outcomes will be addressed just ahead [look for Whitney]. Results showing positive far transfer from XR applications, however, are elusive in a similar way to the retention results. Research shows equivalent or mixed performance to traditional media (Kaplan, Cruit, Endsley, et al., 2021; Makransky, Borre‐Gude, & Mayer, 2019) or worse performance (Makransky, Terkildsen, & Mayer, 2019; Parong & Mayer, 2018). In particular, Mayer offered the Immersion Principle in 2020 which stated: “People do not necessarily learn better in 3D immersive virtual reality than with a corresponding 2D desktop presentation” (p. 357).
Intuitively, because XR can replicate real world environments where the learning would be applied, far transfer seems like a reachable goal. (Tlili, Huang, Shehata, et al., 2022) wrote that the technology can enhance and allow for transfer. Johnson-Glenberg noted that despite requests for more research into XR, “resources and affordable technologies were not readily available” (2018, p. 7) for educational research and that “longitudinal effects of VR exposure are unknown at this point” (2018, p. 11). It is possible that not enough time has passed for the research community to measure the ‘far’ in far transfer. In general, the ability to do worked practice exercises repeatedly in simulated real-world contexts suggests that XR should be at least equivalent when compared to other media for far transfer.
8 Positives Get Published
Research with positive results is published more often than research with negative or no results. This is not unique to metaverse applications. This is known as publication bias or the file drawer problem (Lederman & Lederman, 2016). It limits the results of a meta-analysis because if a particular form of learning is not effective, it usually is not published (Cofré, Núñez., Santibáñez et al., 2019). Thus, the published collection showing positive results with XR dominates over the no significant difference results.
As an added caveat, IDs should closely examine research funding sources and sponsors.
How Will I Know?
IDs are cautioned to examine metaverse research studies for these two major characteristics:
Novelty effect. The novelty effect is when learners are exposed to a new media and they engage in increased effort and attention. It tends to positively impact learning outcomes (Metcalf, Chen, Kamarainen, et al., 2019) but not always (Huang, Roscoe, Johnson‐Glenberg, et al., 2021). Further, "studies of virtual reality-based learning are based on only short-term implementations, and although they might show statistically significant learning outcomes, the novelty effect is an important caveat to the research because many of these studies do not account for the decay of outcomes over longer periods of time." (Metcalf, Chen, Kamarainen et al., 2019, p. 97)
If a research study implements a one-time 20-minute XR intervention and claims to show learning improvement, the learners are likely experiencing the novelty effect.
Non-cognitively comparable methods. Studies where the learner is not put into the same cognitive workload with two different media should be viewed with skepticism over claims of better results (Reigeluth & Honebein, 2023). For example, if a study stated that learners performed better in XR than paper-and-pencil-based learning, the results should be discounted due to the varying cognitive impact that the different media had on the learner (Parong & Mayer, 2021). In one example, the experimental learner group was exposed to VR training after the standard training and then scored higher than the control group (Seymour, Gallagher, Roman, et al., 2002). The total training time increased. This could have caused higher scores. The two conditions, therefore, were not comparable. Furthermore, the National Academies of Sciences, Engineering, and Medicine noted that the prevalence of WEIRD populations (Western, educated, industrialized, rich, and democratic) used in educational research inherently exclude diverse learner populations and this makes it difficult to draw solid conclusions for all humans in all learning situations (2018). Thus what works for one group of learners might not work, nor even be comparable, for another group.
Clark and Mayer summarized how to examine research claims for e-learning, but these questions equally apply to XR research.
“Are the methods, content, learners, and context like yours?
Does the experimental group outscore the control at a significance level of p < .05? [Editor Heather here: How many of y'all KNOW what the phrase "statistically significant" means? I thought so. I'll write a future article on it so that you stop banging the "XR will make a significant difference in education" phrase around. I hate that. Be warned.]
Does the effect size favor the experimental group at a 0.5 level or higher (2016, p. 63)?"
IDs will likely encounter innovators and early adopters who have anecdotal stories of how XR improved learning.
These stories should be accepted with grace, as every form of media has the possibility to hit the perfect instructional moment with the right learning at the right time for the right learners.
Longer term and wider implementation decisions, however, should be made more systematically, by thinking and rethinking the design decisions over time. IDs rarely have control over the large financial decisions that XR development requires, so their role can be one of consultant: offering all the options and pros and cons of each to the decision makers (Dodds, 2021). After reviewing research, IDs are ethically bound to point out if a learning objective can be met with a cheaper, more environmentally responsible, or more socially just media.
In summary, “As a consumer of experimental research, you need to be picky!’ (Clark & Mayer, 2016, p. 56)
Part 4 will answer "How do I know I'm on the right trail with this [assigned] XR project?" (Yeah, more Whitney!)
Part 2 covered theory and scope
Part 1 was the introduction.
Want to see my full references? Have at it.
#InstructionalDesign #XR #Myth #LearningMyths #XRMyths #Multimedia #Principles #Mayer #LXD #ID #InstructionalDesigner #WebXR #3D #2D #VRCausesFasterLearning #VRCausesMoreRetention #WEIRD #ActiveLearning #Immersion #Empathy #NearTransfer #FarTransfer #PositivesGetPublished #AcademicPublishing #NoveltyEffect #NonCognitivelyComparableMethods #HowToReadResearch #Anecdotes #Ethics #ExperimentalResearch
This blog post is simultaneously posted to a LinkedIn article here: https://www.linkedin.com/pulse/instructional-design-metaverse-part-3-heather-dodds-ph-d-