Misleading Headline or I snagged a live one!

 

Oh, I snagged a live one! I really didn't think this image would garner much attention as my posts rarely do on these topics. 2022 has been a bad year for keeping XR friends.

But I've been contacted directly by a nice LinkedIn link.  

Incoming message

(DM'ing instead of commenting on your recent post here.) 

 Why are you distorting the facts?

The article says: "The survey revealed that students showed an ​average of 68 % improvement​ in a test taken after learning with Futuclass VR lessons compared to the results before the 30-minute VR lesson." This is very different from "started from zero and learned 68 out of 100 percent". 

The source is here, wasn't hard to find. https://futuclass.com/blog/how-VR-lessons-increased-chemistry-test-scores-by-68-percentage/ 

And looking up the teacher involved, quoted in the article, she has multiple publications at hard science institutions at Tallinn University. I somehow doubt she'd agree in your assessment on her scientific rigor. https://www.etis.ee/Portal/Persons/Display/95da33ee-9b4c-4e72-9cab-3e72e3791c0c? 

I'm all for criticizing messaging and methodology of these "research" reports, but when you skew the actual facts you're undermining your own point.

So, my response:

Good sleuthing!  I like that you want to look at the actual numbers.   

I did so. I did read the article before I made my post. 

May I direct to a few more sentences lower down in the article?   Some of these will be confusing, so by the end of this, we'll have to look at the entire article and not just stick with exact sentences. 

Here is the one you quoted and I do see it: "As a result, students showed an ​average of 68 % improvement​ in a test taken after learning the metal oxides during a 30-minute lesson using VR compared to the results before."   

This one is not clear what "to the results before" is actually referring to. Could it be referring to: other teaching, to non-teaching, to "before" teaching?  I read further down in the article to find out.   

This appears to be describing more of the method:  After testing different approaches, the best results proved to be with group work and paper worksheets. Students with VR headsets were expected to communicate what was going on and solve the worksheets together with the students without the VR sets.   

I thought that was interesting because it implies other approaches that but that this one was settled on and it included a non-tech component (paper worksheets).   

The next sentence is problematic: "Several studies have revealed that first-hand experience is four times more effective than traditional theory learning. “I believe that we should mix the experience from VR equipment and practical work and this would be effective learning,” admits Katrin Soika."  

It quotes "four times more effective than traditional learning theory".  The "four times" is clued me in. The statement implies that VR is 4 times as better than traditional teaching. This is debunked research (by me).  If it referring to the PwC study (https://www.5discovery.com/wp-content/uploads/2020/09/pwc-understanding-the-effectiveness-of-soft-skills-training-in-the-enterprise-a-study.pdf ), that study had learning take place in 1/4 the number of minutes that a classroom equivalent would have taken. Those are not apples to apples comparison, but it was published by PwC that they thought VR was, then, 4x as effective. 

 

Here is where we get to what the 68% was actually measuring: "With VR education, learners are inspired to discover for themselves. Students have an opportunity to learn by doing rather than passively reading or listening. After the VR chemistry lessons students were surveyed about their experience and the results were significant. The survey revealed that students showed an ​average of 68 % improvement​ in a test taken after learning with Futuclass VR lessons compared to the results before the 30-minute VR lesson." 

In that paragraph, it becomes clear that the 68% was the "average" improvement **compared to the results before the 30-minute VR lesson**.     

That's why I wrote: Learners learn after learning.  

The 68% was the score that they received (on average) after engaging in a learning event - in this case - the 30 minute VR lesson.     

So that's why I wrote what I wrote.   Do you think I missed anything else quantitative in the article? 

_ _

The source isn't here

About "the source isn't here"

That's purposeful from me. I'm not out to point out the company or publisher's name or link my comments with them on the LinkedIn network - especially when I have something negative to say. I let my hashtags carry the weight of passing my opinion around my network:

#XR #VR #VREducation #VRForGood #edtech #Chemistry #ChemistryIsHardIGetIt #VRForLearning #ReadCarefully #WriteBetter

Note: this article came to me via my LinkedIn network - via 2 friends, actually. So I'm only recirculating my opinion back out and NOT pointing out the article, company, or researchers.

Additionally, I posted to LinkedIn one image and one line of text "Let's be careful out there" (a reference to the TV show Hill Street Blues). That's it. I didn't include a link.

_ _

The researcher's qualifications

About the comment about the researcher's qualifications: I don't actually think the researcher/educator whose experience is IN this article wrote this article. There is no author quote, so I'd have to go with someone at Futuclass wrote this. I say that because some of the quotes for the researcher are put up close to sentences on something different-- and thus, it doesn't make sense in terms of writing. Usually evidence supports a claim, not says something very different and off tangent.  By the way, when I see this, I always try to figure in possible changes in translation between languages.  For example, could this article have been originally written in Estonian and then translated to English and thus some things just don't sound like native English?  Yes, that's possible. However, the problems I see in the writing are more along the lines of illogical ideas or "pushed" writing (advertising) than straight research.  So that would not invade actual word-for-word translations, but it would remain at the sentence & paragraph level- which is what I think I see.

Here is an example:

Several studies have revealed that first-hand experience is four times more effective than traditional theory learning. “I believe that we should mix the experience from VR equipment and practical work and this would be effective learning,” admits Katrin Soika.

So the quote shows that Katrin wanted a mix of technology and practical work to create effective learning. That seems to be her point.  But the sentence just before makes the (wild) claim of the four times (but I'm not sure what of four times).  So the ideas don't connect.

BTW, the link to her qualifications is in another language. Regardless, I respect that she might have done some nice work here. Actually if you look at her quotes in isolation AND look at the "other properties" section - particularly with reference to chemistry education - I think that this educator is cooking with gas, as in, she's doing well.

Conclusion

Overall my opinion is that I wish the "other properties" section would have actually made the headline and NOT the quantitative data. There is some really good stuff there. It has been 5 days since this interaction and no follow-up response from my link. 😟

This is just another case of  if it looks too good to be true (68% improvement!!), it probably is.