From Myths to Principles Part 8 Ethical Labyrinths, Interpreting Research

 

From Myths to Principles: Navigating Instructional Design in Immersive Environments
Part 8 Ethical Labyrinths, Interpreting Research

Photo by Mohammad Bagher Adib Behrooz on Unsplash

Ethics, as a set of rules of practice, is something that instructional designers deal with on a daily basis in the form of assuring learner privacy, coursework security, instructor authorship and institutional ownership (Moore, 2021). These topics are recognizable within instructional designers’ professional work lives. However, many instructional design models like ADDIE, Backwards Design, and ASSURE do not include any acknowledgment of possible ethical concerns (Warren et al., 2023). As such, instructional designers might not recognize some ethical decisions which are a critical part of their professional job (Moore, 2021). Within immersive environments, the stakes are higher as learners are primed to experience environments far beyond a classroom or home.

A scoping review of relevant research topics for immersive environments that covered access, content production, and deployment does not mention ethics (Gaspar et al., 2018). However, research on ethics in immersive educational environments is beginning to appear (Moore, 2021; Glaser & Moore, 2023; Zallio & Clarkson, 2022). Zallio, Huang, Osaki, Hong, Chang, Liu, and Ohashi (2024) completed a review of ethical issues in VR and AR technologies and found 15 different and broad ethical concerns including the dichotomy between the virtual and the real world (for example, abuse in immersive experiences), concerns related to user safety (for example, sensory overload) and the ethical concerns of people who surround immersive headset users (for example, caregivers). This series will look at some areas where instructional designers can exert influence even after the decision to incorporate immersive experiences has been made.

Interpreting research

Relying on what the research portrays on the surface does not fully illuminate what is happening within the immersive experiences. Research results were at the core of the myths illuminated earlier in this series. What might be a kernel of truth could be turned into a claim that immersive experiences will revolutionize education.

Instructional designers can conduct literature reviews and quickly review research paper abstracts for studies that are similar to the situation being considered. R. C. Clark and Mayer (2016) summarized how to examine research claims for e-learning, but these questions equally apply to sorting for immersive experience research.

1. “Are the methods, content, learners, and context like yours?

2. Does the experimental group outscore the control at a significance level of p < .05?

3. Does the effect size favor the experimental group at a 0.5 level or higher? (p. 63)

Despite experimental results that tout learning success in immersive experiences, those results might not apply to another situation due to different variables, effect size, and other appropriate measures. Readers of research need to become adept at identifying effect sizes, immersion times, and the presence of comparison groups. In summary, “as a consumer of experimental research, you need to be picky” (R. C. Clark & Mayer, 2016, p. 56).

Disgust embodies 'you need to be picky'

 

When reviewing research, the reader may sleuth for two primary problems that might appear in immersive experience studies: the presence of novelty effect and the bane of media comparisons.

Novelty effect

This series defines novelty effect as the phenomena when learners are exposed to something new during instruction and the new treatment causes increased motivation, excitement, and effort. There is usually a corresponding learning gain from the increased attention (Lodico et al., 2010). R. E. Clark and Craig (1992) succinctly refer to the novelty effect as the “attitude advantage” (p. 9). Novelty effect can be suspected within a research design when the learners are exposed to a media with which they are not familiar and the learners’ time within the experience is limited. The presence of the novelty effect is generally a negative threat to external validity of a study; the study results cannot necessarily be generalized to be true for other populations.

Certainly, an educator might be buoyed up by the illusory increase from incorporating immersive experiences. Just as motivation increases, however, it can also decrease. When the newness of the technology wears off, the learning gains tend to equilibrate to be comparable with other media choices (Clark & Craig, 1992).

It is valid to ponder how long the novelty effect can be expected to last with immersive experience. The answer is it depends. Novelty effect is unique to each learner. Some learners might personally use immersive headsets outside of learning environments and the novelty of the experience will end sooner for them. At the time of this series’s writing, headsets and immersive learning environments are not ubiquitous, so the novelty effect can be expected for some time into the future.

Media comparison studies

Photo by Dietmar Becker on Unsplash

Much research about immersive experiences for learning has focused on the hardware and the learners' reaction to it in the form of comparison studies (Glaser & Moore, 2023; Stefan et al., 2023). Studies often measure learning gains and do not give balanced consideration of the constraints of time, money, space, and connectivity that might have been present (McGivney, 2023). Indeed, media comparison studies are a debatable topic in instructional design. We must look at the root of the problem.

With the arrival of personal computers into education in the early 1980s, a debate arose of what causes the ideal conditions of learning: the media (which at this time was the personal computer) or the method (which is the approach taken to conduct the learning). R. E. Clark’s initial salvo in 1983, drawing on what was then already decades of empirical research, asserted that,

There are no learning benefits to be gained from employing any specific medium to deliver instruction. Research showing performance or time-saving gains from one or another medium are shown to be vulnerable to compelling rival hypotheses concerning the uncontrolled effects of instructional method and novelty. (p. 445)

With this, R. E. Clark called the media emperor naked. He pointed at two possible causes of learning gains seen in media comparison studies: the novelty effect (which was covered in the last section) and uncontrolled instructional methods. This latter item is when two different media experiences are pitted against each other to determine which is better. The problem is that use of different media often requires correspondingly different instructional methods. Thus, if something is taught differently, any differences cannot be the result of the media’s impact alone. The learning accomplished between the two media can be very different.

An example of a poor media comparison would be when learners in an immersive experience are compared to learners in paper and pencil-based learning. The results of a comparison like this should be discounted due to the varying cognitive impact that the different instructional methods have on the learner (Parong & Mayer, 2021). In another example, a control group was exposed to the standard training and an experimental group was exposed to VR training in addition to and after the standard training (Seymour, et al., 2002). The VR group scored higher. The extra training time with the content could have caused higher scores, not the media. The two media conditions of one with and one without immersive experiences were not comparable. 

Honebein and Reigeluth (2020) refer to media comparison studies as “a good guys versus bad guys competition” (p. 6). The comparison scenario has been repeated between many media. But R. E. Clark doubled down on this claim against media comparison studies in 1994 by making the “replaceability challenge” wherein he asked “whether there are other media or another set of media attributes that would yield similar learning gains” (p. 21). The research record since 1994 has supported R. E. Clark’s stance, now referred to at times as the no significant difference phenomena with media.

Honebein and Reigeluth (2020) contended that the entire research-to-prove approach, striving to prove which media is better, needs to be replaced with a research-to-improve approach acknowledging the complexity and systemic components for each individual situation. Instructional designers can draw from this research-to-improve idea by advocating for the specific affordances that immersive experiences media might bring that stand separate from learning gains. More discussion of those affordances will be mentioned within the future directions section of this series.

Missing design theories and models

You do plan to have 

some learning theory in

 your learning experience, 

right?

The design work for immersive experiences in education is complex. To design for the highest possible chance of learning, there should be instructional models or beacons for developers and designers to follow. Immersive experiences, as replications of real world experiences, could reasonably utilize any major learning theory. Radianti et al. (2020) reported that in their review of immersive virtual reality applications, 68% of studies did not mention a learning theory. Most papers focused on XR usability and did not connect theory with use. Checa and Bustillo (2023) asserted that constructivism, behaviorism, cognitivism, and connectivism can be foundations for a wide variety of immersive pedagogical approaches. Similarly, Marougkas et al. (2023) found that constructivism was the most commonly cited learning theory in VR studies. However, the specific affordances of presence and embodiment in the metaverse point to simulations and experiential learning as the most appropriate design theories (Reigeluth & Carr-Chellman, 2009; Johnson-Glenberg, 2018; Checa & Bustillo, 2023; Marougkas et al., 2023).

Similarly, Castelhano et al. (2023) conducted a systematic literature review for instructional design models and found that no current model combines the best of what we know about pedagogy from two-dimensional learning with the affordances of three-dimensional technologies. For example, traditional pedagogical research has shown the importance of having clear learning objectives, a consideration of the audience, planned and structured learning, and alignment of assessment choices. All of these are standard instructional design expectations. By contrast, immersive experience research identifies the importance of segmenting training to avoid overload in intensely stimulating and surrounding environments. Also, the research stresses the equal importance of both advance briefings (on-boarding) to prepare learners for what they will experience and post-briefings (off-boarding) to allow the learners to process and engage in generative activities (Dede, 2021). Thus, researchers seem to be not putting the best of what are separate knowledge pools together.

Similar gaps in theory-driven designs were found by Kim et al. (2023) and McGowin, Fiore, and Oden (2023). The emergent use of immersive experiences technology has precipitated haphazard designs lacking guidance:

In these early days, trial and error plays an outsized role in design. Education researchers borrow heavily from the entertainment designers, who focus on engagement, and not necessarily on retention of content. The dearth of studies highlights the urgency for a set of guidelines for designing content that allows users to make appropriate choices in a spherical space. (Johnson-Glenberg, 2018, p. 7)

Indeed, “theoretical frameworks devised to inform design, research, and practice in the field are rare” (Southgate, 2020).

Problematic data

Even after the learning event is done, assessing the results has been problematic. In a systematic review of computer-aided technologies in safety training, Gao et al. (2019) found that evidence supporting the effectiveness of the training is poor. Narciso et al. (2021) observed that the most common form of assessment used in published research of immersive experiences for learning was questionnaires. This contradicts the advice recommended by experts who point out that assessments should be tied closely to future performance (Ziker, et al., 2020). According to Stefan et al. (2023), only one-third of published studies contained some form of evaluation at all. Of those, Kirkpatrick’s Level 1, learner reaction, measurements were found 66% of the time. Some research studies do not seem to go further than asking the learners if they liked the immersive experience (Kavanagh at al., 2017; Stefan, et al., 2023). While liking an experience is pleasant, it is known that what learners like or prefer to engage in for their learning often has no positive correlation to their actual learning (Thalheimer, 2018; Ruiz-Martin et al., 2024).

Further problems appear once research is published. Lanier et al. (2019) noted that the median sample size in published studies was 25 participants. This number might not represent a large enough data pool to detect anything but large effects. If the impact effect of immersive experiences is supposed to be moderate, pools of 25 participants would only statistically detect the impact in about 50% of the experiments (Lanier et al., 2019, p. 14). This means that even if the inclusion of immersive experiences do positively impact learning, most published research studies cannot detect it because the sample sizes are too small. Despite researchers and educational influencers using the word significant to describe future anticipated impacts of immersive experiences, there is room for doubt that statistical thresholds are being met. 

 

In the next part of these series, I'll cover the ethical problems inside of the biased content creation process - both in terms of XR content and research publishing.

References

Castelhano, M., Morgado, L., & Pedrosa, D. (2023, November 1). Instructional design models for immersive virtual reality: a systematic literature review. http://hdl.handle.net/10400.2/15232

Checa, D., & Bustillo, A. (2023). Virtual reality for learning. Current Topics in Behavioral Neurosciences, 289–307. https://doi.org/10.1007/7854_2022_404

Clark, R. E. (1983). Reconsidering Research on Learning from Media. Review of Educational Research, 53(4), 445–459. https://doi.org/10.3102/00346543053004445

Clark, R. E. (1994). Media will never influence learning. Educational Technology Research and Development, 42(2), 21–29. https://doi.org/10.1007/bf02299088

Clark, R. E., & Craig, T. G. (1992). Research and Theory on Multi-Media Learning Effects. In Springer eBooks (pp. 19–30). https://doi.org/10.1007/978-3-642-77705-9_2

Clark, R. C., & Mayer, R. E. (2016). E-Learning and the science of instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. John Wiley & Sons.

Dede, C. (2021, May 17). Looking back: Insights from a century of cumulative research in immersive learning. [Video]. YouTube. https://www.youtube.com/live/l3tw6O8Hn-s?si=Ey6l-Na4t7YPYLu3

Gao, Y., Gonzalez, V. A., & Yiu, T. W. (2019.). The effectiveness of traditional tools and computer-aided technologies for health and safety training in the construction sector: a Systematic review. Computers & Education, 138, 101–115. https://doi.org/10.1016/j.compedu.2019.05.003

Gaspar, H., Morgado, L., Mamede, H. S., Manjón, B., & Gütl, C. (2018). Identifying immersive environments’ most relevant research topics: an instrument to query researchers and practitioners. iLRN 2018 Montana. Workshop, Long and Short Paper, and Poster Proceedings From the Fourth Immersive Learning Research Network Conference, 48–71. https://doi.org/10.3217/978-3-85125-609-3-10

Glaser, N., & Moore, S. (2023). Redefining immersive technology research: Beyond media comparisons to holistic learning approaches. Digital Psychology, 4(1S), 4–8. https://doi.org/10.24989/dp.v4i1s.2272

Honebein, P.C. & Reigeluth, C.M. (2020). The instructional theory framework appears lost. Isn’t it time we find it again? RED. Revista Educación a Distancia, 20(64). http://dx.doi.org/10.6018/red.405871

Johnson-Glenberg, M. C. (2018). Immersive VR and education: embodied design principles that include gesture and hand controls. Frontiers in Robotics and AI, 5. https://doi.org/10.3389/frobt.2018.00081

Kavanagh, S., Luxton-Reilly, A., Wuensche, B., & Plimmer, B. (2017). A systematic review of Virtual Reality in education. Themes in science and technology education, 10(2), 85-119. http://earthlab.uoi.gr/theste

Kim, T., Planey, J., & Lindgren, R. (2023). Theory-driven design in metaverse virtual reality learning environments: Two illustrative cases. IEEE Transactions on Learning Technologies, 16(6), 1141–1153. https://doi.org/10.1109/tlt.2023.3307211

Lanier, M., Waddell, T. F., Elson, M., Tamul, D. J., Ivory, J. D., & Przybylski, A. (2019). Virtual reality check: Statistical power, reported results, and the validity of research on the psychology of virtual reality and immersive environments. Computers in Human Behavior, 100, 70–78. https://doi.org/10.1016/j.chb.2019.06.015

Lodico, M. G., Spaulding, D. T., & Voegtle, K. H. (2010). Methods in educational research: From Theory to Practice. John Wiley & Sons.

Marougkas, A., Troussas, C., Krouska, A., & Sgouropoulou, C. (2023). Virtual reality in education: a review of learning theories, approaches and methodologies for the last decade. Electronics, 12(13), 2832. https://doi.org/10.3390/electronics12132832

McGivney, E. (2023). Improving Technology- Enhanced Immersive Learning With Design-Based Implementation Research. Proceedings of the 17th International Conference of the Learning Sciences-ICLS 2023. https://doi.org/10.22318/icls2023.213038

McGowin, G., Fiore, S. M., & Oden, K. (2023). Towards a theory of learning in immersive virtual reality: designing learning affordances with embodied, enactive, embedded, and extended cognition. In Cherner, T. & Fegely, A. (Eds.), Bridging the XR technology-to-practice gap: methods and strategies for blending extended realities into classroom instruction, Association for the Advancement of Computing in Education (AACE). https://www.learntechlib.org/primary/p/222242/

Moore, S. (2021). The design models we have are not the design models we need. Journal of Applied Instructional Design, 10(4). https://doi.org/10.51869/104/smo

Narciso, D., Melo, M., Rodrigues, S., Paulo Cunha, J., Vasconcelos-Raposo, J., & Bessa, M. (2021). A systematic review on the use of immersive virtual reality to train professionals. Multimedia Tools and Applications, 80, 13195-13214. https://doi.org/10.1007/s11042-020-10454-y

Parong, J., & Mayer, R. E. (2018). Learning science in immersive virtual reality. Journal of Educational Psychology, 110(6), 785–797. https://doi.org/10.1037/edu0000241

Radianti, J., Majchrzak, T. A., Fromm, J., & Wohlgenannt, I. (2020). A systematic review of immersive virtual reality applications for higher education: Design elements, lessons learned, and research agenda. Computers & education, 147, 103778.

Reigeluth, C. M., & Carr-Chellman, A. A. (Eds.). (2009). Instructional-design theories and models, volume III: Building a common knowledge base. (Vol. 3). Routledge.

Ruiz-Martín, H., Blanco, F., & Ferrero, M. (2024). Which learning techniques supported by cognitive research do students use at secondary school? Prevalence and associations with students’ beliefs and achievement. Cognitive Research: Principles and Implications, 9(1), 44.

Seymour, N. E., Gallagher, A. G., Roman, S. A., O’brien, M. K., Bansal, V. K., Andersen, D. K., & Satava, R. M. (2002). Virtual reality training improves operating room performance: results of a randomized, double-blinded study. Annals of surgery, 236(4), 458.

Southgate, E. (2020, June). Conceptualising embodiment through virtual reality for education. In 2020 6th international conference of the immersive learning research network (iLRN) (pp. 38-45). IEEE.

Stefan, H., Mortimer, M. & Horan, B. Evaluating the effectiveness of virtual reality for safety-relevant training: a systematic review. Virtual Reality 27, 2839–2869 (2023). https://doi.org/10.1007/s10055-023-00843-7

Thalheimer, W. (2018). The learning-transfer evaluation model: Sending messages to enable learning effectiveness. In Design Thinking Conference and the Learning Technologies Conference. London. https://www.worklearning.com/wp-content/uploads/2018/02/Thalheimer-The-Learning-Transfer-Evaluation-Model-Report-for-LTEM-v11.pdf

Warren, S., Beck, D., & McGuffin, K. (2023). In support of ethical instructional design. S. Moore y L. Dousay (Eds.). Applied ethics for instructional design and technology, 15-37.

Zallio, M., & Clarkson, P. J. (2022). Designing the metaverse: A study on inclusion, diversity, equity, accessibility and safety for digital immersive environments. Telematics and Informatics, 75, 101909.

Zallio, M., Huang, T., Osaki, Y., Hong, S., Chang, X., Liu, W., & Ohashi, T. (2024). The ethics of immersion: A scoping review of VR and AR technologies. Accessibility, Assistive Technology and Digital Environments, 121(121).

Ziker, C., Ydo, E., Zapata-Rivera, D., Hillier, M., & Casale, M. (2020, June). Special session—Challenges and opportunities for assessment in XR. In 2020 6th International Conference of the Immersive Learning Research Network (iLRN) (pp. 421-423). IEEE.