Just a decade ago, holographic projections and virtual reality seemed like the stuff of the distant future—a Star Trekkian fantasy that sounded as tenable as teleportation and light-speed space travel. Yet, on March 2, 2021, Microsoft unveiled Microsoft Mesh and demonstrated how the distant future might not be so distant after all.
Mesh is a multi-device communication platform that allows users from different locations across the world to collaborate in a shared 3D space. Mesh operates by pairing desktop or mobile devices with virtual reality (VR) headsets and holographic projections to create a unified experience. Whether it is designing, training, conducting project meetings, or attending a simple social gathering, users will be able to project avatars or photorealistic holograms of themselves that can share and interact with 3D/2D content, as well as communicate with their teammates as if they are in the same geographic location.
Mesh’s launch was rife with this blend of reality. Alex Kipman, technical fellow at Microsoft, led the keynote presentation—not standing on a stage as you would expect— but in an immersive, underwater 3D VR setting. The audience joined the event from all over the world as 3D avatars of themselves whose likes and other emojis bubbled into the presentation in real time. Guy Laliberté, the cofounder of Cirque du Soleil, joined the presentation as a guest speaker—not in person but as a holographic avatar of himself. He concluded the presentation with a Cirque du Soleil performance rendered entirely in a VR island setting with holographic performers and 3D avatars of the audience.
Not-so-Magical Realism
Combining the physical world with the digital world is known as mixed reality (MR). As Industry 4.0 and computer vision continue to advance rapidly, our ability to interact with the digital world through MR devices is not only possible but already happening.
Video games like Pokémon Go are an example of MR where digital characters exist in the players’ physical geographic locations. Online concerts and fitness apps frequently employ user avatars to allow interaction with other users online. Gucci has developed an app that empowers its customers to scan their feet and select the shoes they are interested in—all without needing to visit the store in person. In 2020, Mercedes-Benz introduced Virtual Remote Support, which allows technicians to connect with expert service representatives and address any issues they are experiencing with their cars. The technicians can show the service rep exactly what they are seeing in real time using virtual reality goggles, enabling the technician to scale the image, mark it or even use lines and other shapes to highlight specific areas. Using Virtual Remote Support has allowed Mercedes-Benz to reduce costs and time delays while eliminating errors that would otherwise have occurred. MR is also being implemented in the oil and gas sector, where it is used in design stages as well as for training end users.
With Mesh, the scope of the MR is both expansive and interactive. Mesh is powered by Azure, a public cloud platform that grants users the chance to create, design, test and manage applications. It offers more than 200 cloud products and services not limited to AI, machine learning, mixed reality and Internet of Things (IoT), all of which can be integrated into the platform.
“Microsoft Mesh can build immersive, multi-user, cross-platform MR applications leveraging AI-powered tools for spatial maps, immersive presence, holographic rendering, and multi-user sync,” explained Simon Skaria, partner director of product for Microsoft Mesh. “So now you can more easily interact with others as if you are face-to-face and collaborate with 2D and 3D content.”
Through Mesh, collaborations are rendered in 3D and 2D with high fidelity regardless of the format. Users have a choice between rendering their content on a stand-alone platform or through Mesh’s accelerated cloud-connected platform, where the content can be rendered in as many as 100 million polygons.
The Microsoft HoloLens 2 enables users to truly immerse themselves in Microsoft Mesh. Once coupled with the solutions and apps available on Mesh, the HoloLens 2 is a cordless, hands-free device that renders a high-fidelity hologram of your content on top of your real-world view. This hologram content not only pins itself on top of what you are seeing but also can be interacted with as if it were a physical object. Users can pan, zoom, translate, redact and annotate on the content, and adjust their point of view (POV) as well as that of the participants so that everyone is seeing exactly the same thing if needed. Users can even collaborate on otherwise dangerous projects like construction inspection using HoloLens 2, where images of the construction project can be taken with a drone and uploaded into Mesh and a full 3D model of the project can be created. The model can be scaled to life-size so that the team has a full 360-degree view from angles that would be dangerous or downright impossible to obtain in the real world.
“The fundamental parameters of digital presence are realism and reach. Microsoft Mesh covers both ends of this spectrum. For example, on the one end of the spectrum, Mesh delivers the most accessible 3D presence delivering representative avatars with inside-out sensors already on the devices that you carry. And on the other end of the spectrum, Mesh delivers the most photorealistic 360 holoportation with outside-in sensors,” elaborated Skaria. “[Mesh] also brings spatial audio so even as you move in the space, the audio will be directionally emitted in the 3D space consistent with your relative position and those of the participants in the session. It’s truly magical and you have to experience it to understand the connection and engagement it delivers to your apps.”
Despite being scattered across the world, Microsoft claims Mesh’s latency to be under 100 milliseconds because of its smart compression technology. As a result, Mesh combines all the advantages of in-person meetings (clarity, direct access to information, opportunities to ask questions, saving time) with all the advantages of working remotely (flexibility, cost-effectiveness, saving time during commuting, safety). The meetings can also be saved, enabling other users or teams to pick up exactly where they had last left off, and in the desired POV. For example, the design team can save an annotated 3D model in a shared space that can then be accessed by the engineering team later on. The engineering team would then have access to not just all the 3D models, but also any notes and instructions left behind by the design team.
A Safe Space
Microsoft Mesh is fully integrated with Azure Active Directory (AAD), a cloud-based directory and identity management service. AAD enables users to access multiple services using a single username and password. Mesh is also integrated with Microsoft Accounts (MSA) so that hosts can access multiple Microsoft devices and services using a single set of login credentials. By being integrated with both AAD and MSA, Mesh users can invite other users into a secure conference room where the shared information will be protected.
A New Real
That Mesh users can invite each other into a shared space, that they can touch the same objects and see them together, that objects can be rendered in 3D in such high definition that they are indistinguishable from the real objects—all of this presents us with a gateway into a reality that was not possible even a decade ago. Indeed, it feels more like a sci-fi dream than reality. Yet, it is this nature of reality that Kipman deliberated upon during the reveal of Mesh. He observed that while a dream one dreams alone is only a dream, a dream people dream together is reality. Given how rapidly technology is evolving, it may be only a matter of time before mixed reality and reality are the same.