Analysis of The Total Economic Impact™ Of Mixed Reality Using Microsoft HoloLens 2
This report, The Total Economic Impact™ Of Mixed RealityUsing Microsoft HoloLens 2, dated (November 2021) came through my LinkedIn feed in this post:
What most caught my eye was the third bullet:
Mixed reality increased training efficiency by 60%, saving $1,440 per trainee while improving knowledge acquisition and retention.
I posted my response:
"With
respect, a company that sells a product that then claims the product
increases training efficiency with these kinds of numbers is to be
doubted.
In the case of the 60% increase, my eyes land on the
table on page 21 and the source of the 60% increase in task efficiency
data is "Interview data", not quantified data. That means the
interviewees that Microsoft picked provided verbal estimates to the
interviewers. While the report takes great pains to describe how the
results were calculated-- and I respect that-- this is a "commissioned
report" (page 1)-- that was paid for by the very company selling the
product.
Long term, results like these do not hold up."
To which, an Associate Fellow at Lockheed Martin replied:
Heather Dodds while the numbers seem outlandish they are very real based on our findings.
We
use mixed reality to build spacecraft and see well over these numbers
(~93% reduction) in touch labor. This was analyzed across 10
manufacturing sites and numerous spacecraft manufacturing programs.
Technicians consistently finish 8 hours worth of traditional work in 45 minutes, and other companies are seeing similar results.
For
training we see an 85% reduction in training time. Mixed reality also
offers greater comprehension than traditional methods.
While the numbers seem incredible, they have been validated with more than four years of shop floor implementation.
If you would like to chat details, just let me know. I would be happy to provide insight to our findings.
I did not reply further at the time to this post because I was not actually arguing time-based measurements. I was arguing improving knowledge acquisition and retention via task efficiency-- admittedly their own squirrely wording. Said another way, claims that users learned more.
This blog post is, in part, my response.
A scan of the report reveals a great deal of "hand wave" language and it is, of course, a report so I'm not going to spend a great deal of time picking it apart. It's just not worth the time. I'll just point out a few red flags for posterity.
1. A report commission by Microsoft about Microsoft technology.
DISCLOSURES
Readers should be aware of the following:
This study is commissioned by Microsoft and delivered by Forrester Consulting. It is not meant to be used as a competitive analysis.
Forrester makes no assumptions as to the potential ROI that other organizations will receive. Forrester strongly advises that readers use their own estimates within the framework provided in the study to determine the appropriateness of an investment in mixed reality.
Microsoft reviewed and provided feedback to Forrester, but Forrester maintains editorial control over the study and its findings and does not accept changes to the study that contradict Forrester’s findings or obscure the meaning of the study.
Microsoft provided the customer names for the interviews but did not participate in the interviews.
I interpret this as I stated above:
"the interviewees that Microsoft picked provided verbal estimates to the interviewers."
Said another way, random Microsoft Hololens users did not provide the data. Users specifically picked by Microsoft did. That's a non-random sample. Page 5 tells me the 23 interviewees are decision-makers at 21 independent software vendors (ISV) and SI mixed reality partners. There is more data about them on page 44.
Don't forget that "commissioned" also tends to mean "paid for."
2. Their key findings are listed on Page 2:
- Enhanced training efficiency for up to 1,000 annual trainees, saving $2.1 million in labor.
- Increased task efficiency for 50 field workers, saving $1.3 million in labor.
- Increased task efficiency for 120 onsite workers, saving $1.4 million in labor.
- Improved productivity for 15 leaders, saving $428,000 in labor.
- Improved productivity and avoided travel for 15 specialized experts, saving $1.6 million in labor.
- Avoided travel for experts and field workers, saving $1.1 million in travel and incidentals costs.
- Reduced operational costs, saving $2.9 million in excess expenses.
- Protected and grew revenue, boosting operating income by $1.1 million.
From this list, most of the metrics are mentioning time savings. Only the first one had a comment of
"Mixed reality increased training efficiency by 60%, saving $1,440 per trainee while improving knowledge acquisition and retention"
which appears to be the same line from the marketing of this document (LinkedIn image above).
Therefore, I have no comment on all of these bullets except the first one. Also, I am still looking for the source of that 60%.
3. Page 7 starts with shockingly tone-deaf statement 😦:
The “roaring 2020s” have started with a force: a global COVID-19 pandemic that is propelling organizations into the future of work. Like its namesake decade from the 20th century, the 2020s will be full of challenges. Systemic risk now leaves every organization globally exposed, and all must react today. But this time, many of the forces that determine who will win or lose are already visible — and just got fast-tracked.
Wow, nothing like up-spinning a global market crash which happened at the END of the 1920s.
The roaring 1920s is a phrase conjuring good market of the decade before that crash and it was a good time all around for markets rebounding from a World War and the 1918 flu pandemic.
The 2020s is far from a happy time, being marked by death and global market downturns. I can't help but think that someone heard the phrase "roaring 2020s" on a TV show and thought it would make a nice start to a optimistic tech report that would then stir a bunch of expensive headset purchases!
See what I mean by tone-deaf? Someone just used death--- to sell headsets.
Just because I'm now looking--- what do those quote marks mean? And the little "2" at the end of the paragraph? Perhaps you are telling the reader that something here comes from another source? Oh! I'll check it out.
Your references- in Appendix D: Endnotes (ahem, pick one name please) show the source to be a...nother Forrester source. Actually scanning all eight sources used in your 46 page document, six of them are Forrester sources.
Wow.
Heard of circular logic?
If you look just above, the "Appendix C: Supplemental Material" is an inconsistent listing of sources that also comes mostly from Forrester. I might need an Escape Room partner soon.
So no one we know said "roaring 2020s". Ick. I need a shower now.
4. I'm still looking for that 60% and here it is on page 16. There is some "hand wave" wording here:
Self-guided mixed reality instructions leverage 3D models, simulations, and real-world overlays to enable employees or students to learn more quickly, better understand content, and practice skills. Learners can also observe and participate in remote demonstrations or be evaluated by viewers as they test their new skills. Mixed reality not only accelerates training to allow labor recapture but also improves knowledge acquisition and retention to improve outcomes and reduce future training needs. Fast and effective training continues to gain importance as organizations across industries struggle to attract, hire, and retain crucial talent.
The phrases get mixed here:
enable employees or students to learn more quickly, - possibly true, depending what it was compared to
better understand content, - extremely debatable and I never did find anything in the report to substantiate this.
and practice skills.- yes, MR or XR can help practice skills.
So the "hand wave" is where they start to mix things that they want the reader to take as "known" or give the hand wave past, we know this, with things we don't know. By combining, the truth gets muddled.This one sentence (above) contains things that are true and not true at the same time.
Most of the next few pages focus all training gains on time.
There are three tables included that show this line (highlighted):
Page 17:
Metric: Reduction in training time with MR
Source: Interview data
Year 1 - 30%, Year 2 - 30%, Year 3 - 60%
So it doesn't say anything here about better understanding of content, it just says time will be shorter.
Page 19:
Metric: Increase in task efficiency with MR
Source: Interview data
Year 1 - 40%, Year 2 - 40%, Year 3- 40%.
But by the time we get to page 20 and 21, it feels like the report is trying harder:
AEC firms improved efficiency by 60% for 15% of onsite tasks and reduced rework by 50%, saving $44 per hour. Complexity of work and expectations also drive high stakes for AEC firms, maximizing potential savings from MR.
So here it is trying to say that because of the MR factor the "onsite tasks" can actually be done faster and therefore more efficiently and even, with less rework. That's that savings of time feature, which I mentioned at the beginning, I didn't have a problem with. So that cannot be the 60% I'm looking for.
Here is page 21 in total:
The wording really feels like it is straining now but in writing, it's not clear where the 60% is coming from. So, looking at the table:
It says
Metric: Increase in task efficiency with MR
Source: Interview data
Year 1: 60%, Year 2: 60%, Year 3:60%
We went from "Reduction in training time" to "Increase in task efficiency" (note the hint to savings of time...and for the third time I'll state, I'm not debating TIME) and did not really receive any justification of "improving knowledge acquisition and retention" other than that 60% number which was reported in these 3 tables as "Interview data."
Glancing through the rest of the report- every other statement made rests on savings of time.
If you've followed my research and presentations in the past, you'll know that I'm for the Microsoft Hololens and I do write the XR does save time, money, and danger in education. But this report...as many business reports do, plays fast and loose with the facts.
Maybe for Microsoft it IS the roaring 2020s. Not for the rest of us.
