"What Happened When Student Brains -- on VR -- Were Scanned" is Analyzed.
I believe that VR has a place in education. We’ll get there and it will be awesome.
However, along the way, I will analyze and call out poor virtual reality and learning research EVERY CHANCE I GET. Poor research helps no one.
I’ll be writing about several poor VR studies soon. I’m writing just like I did for my “Survey does NOT show that instructional designers drive better student outcomes.” Even though poor research should be called out (yes!) I am NOT in the business of embarrassing or humiliating a person nor damaging or ending someone’s business. Therefore, I will block out names as much as possible. If the source blatantly has names within it though (for example, if someone names a company), I’ll keep that in the screen capture. Note however, the minute time traveling teenagers arrive in my home office asking me to truly name names related to this, I AM SO TELLING THEM.
Seat belt sign on.
On July 25, 2021, an author that I’ll call Author A posted this story to Medium: What Happened when student brains on VR were scanned Before you click on that link, remember that Medium is a site where authors are paid for article reads. Now I post to Medium too (as a matter of fact, I’m posting this article there as a sort of rock’em sock’em method of seeing my article go head to head against his) but just bear in mind that the more you click, the more the author is paid. Author A appears to have posted 6 articles to Medium and 4 of them are about VR. (Update: during publication, this has increased, I’ll pick this point up again at the end of this article).
(July 2021 version of this image, I’ve blocked the author’s account. Note the added #VEC2019. I think it’s very obvious that the #VEC2019 was overlaid.
VEC2019 is the VIVE Ecosystem Conference held in 2019.
Inside the article, I did hope to find sources.
Here is what I found (names blocked):
On July , 2021, Author A posted to their own LinkedIn account linking to the Medium article.
Partial capture of post here (blocked out the link and a name):
While you are looking at this though, does anything stand out yet?
Already for me:
- The writing style feels like it’s pulling me along (FIRST, SECOND, THIRD) but I tend to like my separate points to be actually separate and new points. If you look at what’s written for each item, the points are more chronological as if someone was talking than factual as if someone was listing.
- I’m zinged by that “ [read the rest of the story here: link ] That was actually written by the author, NOT hard coded in by LinkedIn. Bummer. I wish I had NOT clicked on the link but I sincerely thought that there was “the rest of the story” at the link. It was the Medium article link so by clicking on it, I “paid” this Author A some money. This means something. Stay tuned.
- I do a quick mental check of the numbers in the post versus the image. As your math teachers always said “Stand back and look at the numbers. Do they make sense?” The 4th bullet just further describes (aka says the same thing but with DIFFERENT numbers as the 3rd bullet so… is 15 roughly 6 times 2.5? Yes.
- OK, the number “6X” checks out within the diagram (meaning that the data I’m provided with so far does align with the head/images). However, your hackles should be going up because you should be asking yourself “Why was the image of the heads not enough? Why do I need to be told-- in red font with a drawn line-- “six times”? Answer: because the phrase “Six times” will stick in my head and slow down my reading and I will -- as I’m sure MANY have done-- quickly breeze over the “six times WHAT” part and read this to say:
Six times more brain activity on VR
Rather than what it says, which is
The difference between the brain states is 6 times traditional.
- See? Didn’t you read this as “Wow, the brain is 6 times more active on VR, that’s got to be good!” (VR and autism researchers right here are roiling; I can feel you. They are saying “An active brain isn’t necessarily a learning brain. Right on. You get your chance coming up later.)
- One further point I noticed later. The “Traditional Class” is pulling a 58.1 (unitless) over a Base State of 55.5. That’s a 2.6 difference. The image says 2.6, the writing says 2.5. I’m willing to overlook this; that’s minor. But think about it for a second. Traditional class work is pulling nearly the same brain activity as “base state”. So bad news teachers: your students in class are one tick over coma. (I HOPE NOT!)
I’m linked with Author A and that’s how I saw that this image was circulating again.
Author A linked to the Medium article within their post but they added this image to their LinkedIn post. Therefore, the image appears prominently like this:
Here is a version of the same image from 2020.
So the Chinese line under the title is present in the newer version but I don’t see any other changes. I did a Google image search and this appears often on LinkedIn, but it’s not coming up linked directly to another source (so far).
The images seem to have a “source” credited in the lower right corner.
Retyped here with kept capitalization but not formatting:
Study on the Effect of VR on Students Concentration, Saga University, Japan, N = 30, Age (12~13).
Capitalization matters because it can indicate how much the author is following a particular reference style (APA, MLA, IEEE, etc.) Also some statistical and mathematical symbols change meaning if they are capitalized or not.
So six times the improvement of concentration on VR! That is an attention grabbing number!
I’m not the first person to report that FINDING that article by that name or some derivative appears to be impossible. However, looking at that citation, a few thoughts pop up:
- There is a formatting change in the line that looks like text has been added or overlaid. In particular, the age information is odd. Researchers don’t usually add the sample groups’ age in a citation particularly when everyone in the research study (supposedly) was of the same age. Ages didn’t vary within the study. The N info can be properly included in a figure caption, which is different from a citation. So this info looks like a blend of an attempt to give a citation and more information...for perhaps “people who were asking”? (I BET.)
- It is entirely possible that this source was published in a language other than English. The title is academic-looking but academics are also sticklers for choosing the exact words to reflect what we mean and this title is actually a little wordy. That tips me off that it might be a translation. For example, tight academic English would have been “The effect of VR on students’ concentration.” All through some of the sources I go through next, I have the feeling that I’m dealing with a good-hearted translation. Good intentions, yes, but not the original author’s thoughts. Hmm.
In 2020, a few other images and a link circulated associated with this research.
Shared on Facebook as “slides” from a conference talk:
There was a suggestion that THIS is the actual study here: A Case Study - The Impact of VR on Academic Performance
For the next part of this article, I’m really getting in the weeds of the research. Get a cup of coffee and keep up.
As a summary of the research a teacher separated students into a control group and a VR group. Then each of those groups was measured for learning either in the same class period or two weeks later. The VR group was compared to the control group.
Note that the control group is the “Traditional” is a teacher talking with a PowerPoint for 30 minutes.
Remember that Intermediate refers to the ‘same day’ test and Retention refers to the test 2 weeks later.
Before I get into some problems, I’d like to say a couple of items in this teacher’s defense:
- First, it is clear that the teacher means well and is supportive about the use of VR in the classroom.
- Second, the teacher taught an astrophysics lesson. Physics content is an area of ideal use for VR because of the conceptual and sometimes “hard to see” type of content (hard to see atoms, hard to conceive of galaxies, etc.). There is always a direct relationship between “seeing” and understanding when trying to use VR.
- Technically, self-publishing isn’t awful. I do that myself. I think more publishing will be self-publishing in the future. But I’m also NOT SELLING SOMETHING and not dangling the data behind paid links (cough, Medium) or concealing references to support eye-popping results. Don’t forget, it said SIX TIMES.
Inside this paper, there are some concerning misfires.
The first would be the experiment design. It is poor ground to stand on to compare not cognitively equivalent experiences. PowerPoint versus VR is not a fair fight and just with that item, that research shouldn’t be published.
Second, as I read through to get my bearings, the author seems to put some of the wrong data in the wrong places through the paper (they talk about the Intermediate data last and the Retention data first when those were administered in the opposite order) and then they attribute the averages backwards. It’s like the paper had a cut & paste festival run through it.
Example page 9, in a section talking about the Intermediate test, subtitled “VR Improves Test Scores”
“The average score of the VRIT group is 93, CIT, 73. On average, VRIT group has registered a 27.4% growth in terms of score, indicating the great advantage of VR in the teaching of astrophysics.”
OK, that first sentence is true. Here’s my data check:
I also get averages 93 and 73. But the difference between 93 and 73 is 20, not 27.4.
Then on page 10, there is a graph that immediately followed this text:
Both the text here and the graph indicated that the gap between 93 and 73 is 27.4%. It is not. It is 20. But I’m trained to look for “accuracy” that suddenly arrives that wasn’t present before. Where did that 0.4 accuracy come from?. And how did the percent symbol sneak in? The percent symbol isn’t anywhere else in this image (red flag). Does that 27.4% look pasted on? Why? I’d cry “Significant Figures!” here but the 27. 4% does show up on Page 14:
"4.2.1. VR Improves knowledge Retention
In Retention Test, the average score of VR group is 90, while that of the traditional teaching group is 68. The gap between the two average scores is 32.4%, higher than that in the Immediate Test 27.4% (# 4.1.1-1), suggesting that knowledge taught in traditional mode is more inclined to be forgotten, while VR-based teaching could help students get a deeper impression and maintain long-term memory because it creates a quosi-real environment, interacts with students and make students more involved in the teaching." [spelling in context
OK, so now the text says that the difference between 90 and 68 , which is 22 points, is 32.4%! OK, so they are not trying to communicate the point difference (a number), they are communicating, on purpose, a percent difference. Ah! OK. Points and percentages are different, and they know that. Now I feel better.
But notice, why would you convert one set of numbers like this:
- Hey the difference was 20 points!
To another set of numbers like this:
- Hey the difference was 27.4%!
Answer: 27.4 is interpreted as “bigger” than 20 even though, in this case, they are the same (percentage and points, respectively). Sigh. It’ another instance of the SIX TIMES difference. If I make the number appear more impressive, I have more of your attention.
Folks, this is exactly what peeves me here. Remember that I WANT this research to be positive and true. But when you manipulate the numbers just to get me to go wow, I’m on to you. I get extra angry when you are making money off of this manipulation.
In 2020, I commented:
"Can we get more eyes on “Case Study - The Impact of VR on Academic Performance” Several red flags with that paper:
1) Self-published by a mobile training solutions company.
2) I calculate the standard deviation of the controlled group as 19.6 (by taking the first test results ONLY) and when one is claiming the difference between the groups is 27% (uh, I’ll go with 20 percentage points difference between 73 and 93) that means the standard deviation is enough to cast doubt on the results.
Also students were allowed to retest and there appear to be no randomization of assigning the students. This means that students better in the subject could have landed in the VR group.
Hmm….I am just not comfortable with recommending this source. Hey, I could be wrong."
I want to spend just a little more time on the stats. First, standard deviation. Refresher! Standard deviation is a descriptive number that describes how well the average describes the group. Quick example:
One class: 2 students. Students score 45 and 55 on an exam. The average is 50. The standard deviation is 5. That means that a random student, if I could mix all the students up and just pick one student out, differs from the average score by 5 points. Said another way: most students are scoring with 5 points of 50. And this is true. In one case, if I picked out a student, they’d be scoring 45, which is 5 points off the average. In another case, the student would be scoring 55, which is 5 points off the average. Five points off and five points off. That’s good, that’s ‘tight”. Therefore the average of “50” is a nice tight description of how the class is scoring.
One class: 2 students. Students score 0 and 100 on an exam. The average is 50. The standard deviation is 50. That means that a random student differs from the average by 50 points. Said another way: most students are nowhere near the average score. They are missing it hugely. Both students are off by 50 points. The average of “50” does NOT describe this class scores very well at all. It’s junk.
So you generally want small standard deviation numbers if you want to believe that your average number is a good descriptor.
Let’s go look at those standard deviations (I calculated) again.
So I calculated the standard deviation in 2020 of the Control Group Intermediate Test Average Score to be 19.6. Same result by recalculating that in 2021 (so I’m using myself as my own data checker by inserting time between the 2 calculations).
That is troublesome. A standard dev of 19.6 when the difference between the 2 groups was 20 means that the control group could, reasonably, waver from ~53 to ~93 on their score and still be considered “OK” and reasonably near the group average. But that means that the control **could have scored** near the VR group. 93 is quite near 93. (#fact)
So, a statistical flag on that play. When you have reason to think that your experimental group and control group **could have* scored the same, you do not have reason to think that cause and effect has happened in your experiment, you should doubt that your independent variable caused your dependent variable’s results.
In plain language, VR could not be causing higher scores. Higher scores could have happened by chance.
This is what I meant when I said that this data does hint that “students good at VR” could have simply been sorted into the VR group unintentionally and thus, voila, do great at the content. (The photos of the students admittedly look like they are ‘having a good time.’) Statistics is meant to help us know how much to believe in some numbers and doubt other numbers. (Remember the line: Lies, Damn Lies, and Statistics?) I know this is deep math for some, but it’s just saying that these numbers are not believable as they stand.
Also, I want to point out that I’m NOT strong in “power” talk in stats, but this is the field of being able to know how many individual data points you need before you can trust a whole set of data points. Data points are known as “N”s. An N of 10 is quite ridiculously low for a sample size. 10 students are nice but no one should be spending thousands of dollars buying headsets or software because of data from 10 students. And remember, when you are seeing these numbers, you are being convinced to buy. You are NOT being convinced to research deeper. More on this is coming after some more scrolling down.
Finally, one last big item that you might have forgotten down in these weeds:
Where’s the EEG data?
Well, not in this paper, that’s for sure. But the image you are looking at has only ONLY half-hearted reference on it (lower right). Where does the EEG data come from? I tried to find it both by topic and image search. Maybe that’s where “Saga University” comes in? I don’t know. Word on the street is that that EEG data is made up. I really don’t know. I can tell you that I’m suspicious because I’m not sure why one would research on VR with EEG (a reasonably expensive test), publish your results in color, find six times more activity and then bury your published paper about it? Seems fishy to me. Again, though, this could have happened all in another language and me and Google are failing to find it.
There’s possible other stats problems in the paper but I’ll let it rest.
Also circulating from 2020 was this one long infographic that I screen captured into slices. The images are grainy, I know. (sad face)
This infographic appears to be supporting this image. The “Every child can be a genius” phrase and the numbers appear to be the same. I have only one item to say about this infographic: cute UFO theme.
Note that the student clothes looks similar:
I tried to find “Study of VR Education and Effects upon Academic Learning” Google Scholar and Google can’t seem to find it.
Searching on iBokan Wisdom Tech Training instead, it seems to be used as a reference from here: https://www.degruyter.com/document/doi/10.1515/phys-2017-0114/html
To: here:
[5] Beijing Bluefocus E-Commerce Co., Ltd. and Beijing iBokan Wisdom Mobile Internet Technology Training Institutions, A Case Study - The Impact of VR on Academic Performance, 2016. Search in Google Scholar
That is a dead end, as in, dead link:
Back to the August 2021 comments. Notice how even if this data cannot be substantiated, folks are still eating it up! 99 reactions last I checked:
Who needs sources?
(Blocking all these names is making me crazy!)
Or at least, they are eating it up if they work at that same company as Author A.
But there are a few of us VR Research Jedi’s prowling that send up warnings.
My colleague is much more even in tone than I am but they do strike the right balance with saying that we ARE hoping to find positive learning research as it relates to VR.
But this isn’t it.
P.S. I don’t know why Author A posted twice. Zealous much?
I’m coming in here. This is the link I share: https://www.analyticsinsight.net/extended-reality-enhancing-healthcare-industry/
Uh. I didn’t respond after this. Author A apparently didn’t realize that I was questioning the research in HIS OWN reply immediately above about the “Miami Children’s Hospital CEO”. Perhaps he thought I was so excited about his first graphic.
If you cannot properly mansplain to me, I just don’t know what else to say.
Note: side fight broke out! Although, I’m not sure what they are fighting about… I suspect autocorrect did it because that’s one perky disagreement. Tee hee hee...
There is one honorable mention of a person that asked for “we need a concerted effort to also communicate the methodology and external validity for such research. For industry adoption, particularly healthcare, the core elements of research must be extracted and communicated alongside the highlights”
Hey, nice try buddy!
Note that all of the positive supportive comments came from people working in the VR industry. That’s not bad, it should just be noted.
If you are selling me something, don’t I have a right to doubt the research you hand me?
FYI, during the writing of this article, this same Author A continues with the “I’ll tell you something tantalizing about VR!” and then makes you click on a link where:
He makes money.
He doesn’t have to give you more information or sources.
That LinkedIn link right there? Goes to Medium, for a “2 minute” article. I didn't click on this.
Why does all of this matter?
Because media will never influence learning. Yup. I’m a Clarkist. Studies that say that VR makes eye-popping differences in learning is not supported by every media study we’ve ever done as humanity since the beginning of time.
If we go around telling everyone that all students will be geniuses and that VR will cause six fold increase in grades, those of us that DO support VR for education are going to be out of jobs pretty quickly as that bubble will pop.
~~
I recently watched Exodus: Gods and Kings and I love this exchange between Moses and Malak (who is the messenger of God). Moses is fed up with Egypt/slavery and is being told to cool his jets by God.
Moses: So what do I do, nothing?
Malak/God: For now, you can watch.
~~
You can watch. You can watch as I tear this research up.
Clark, R.E. Media will never influence learning. ETR&D 42, 21–29 (1994). https://doi.org/10.1007/BF02299088
#edtech #research #VR #VReducation #VRResearch #TooGoodToBeTrue #MediaWillNeverInfluenceLearning #InstructionalDesign
This is a copy of the article I published on Linked In on August 17, 2021
https://www.linkedin.com/pulse/what-happened-when-student-brains-vr-were-scanned-analyzed-dodds/