With the rise of virtual reality (VR), augmented reality (AR) and mixed reality, and with increasingly affordable hardware associated with them, society has been experiencing new ways of seeing our favorite entertainment in larger-than-life spectacles. The next realm to hit this medium: live entertainment.
Sports have already become a big hit in VR. Professional basketball, hockey, racing and baseball leagues are already broadcasting in this medium. This is just the beginning. Many content creators and engineers are working on the next generation of live VR/AR entertainment.
It’s all about the experience when it comes to concerts—the crowd of people swarming the stage trying to get a glimpse of the artist, the stereos blasting with music, and the graphics that make the entire act come together. Clearly, it’ll be hard to capture that sense of togetherness without fellow fans jumping and dancing alongside the entertainers at a large arena, especially during the time of social distancing and stay-at-home orders.
Enter VR. Once a user puts on the headset, they are transported into another world. They get front seat tickets to see the artist up close and experience the fans cheering during the show. If the user is among multiple people watching the same event, they’ll be able to see them amid this virtual crowd. They’ll get the best seat in the house while being in their living room.
We Get VR and AR, but What’s Mixed Reality?
As the user puts on a VR headset, they enter an immersive 3D environment that eliminates any sign of the real world. The two headset displays, one for each eye, create the illusion of depth in the virtual world. The higher the display framerate and field of view, the better and more immersive the user experience. Sound effects that are consistent with the graphics can add to the immersive nature of this medium. Some of the most popular VR hardware today includes the Oculus Rift, PlayStation VR and HTC Vive.
In contrast, users can see virtual images on real-world objects when they use AR. The camera gathers data on real 3D geometry and superimposes a virtual image at the right spot. Some of the most popular AR hardware includes the Microsoft HoloLens and, for consumer use, regular smartphones.
Mixed reality was first defined in 1994 by Paul Milgram and Fumio Kishino as the mixture of real environments and virtual environments, which are at opposite extremums. However, the medium was first used in the Virtual Fixtures platform by Louis Rosenberg at the Armstrong Laboratories of the U.S. Air Force in 1992. There has been an improvement in environmental input, spatial sound and positioning since then. The experiences between augmented and virtual reality refer to mixed reality such as any layered digital objects, avatars, holographic objects, and the incorporation of any virtual environment in the real world.
Mixed reality is produced from the combination of human-computer interaction such as voice and skeletal tracking, computer input such as sensors and processing, and environmental inputs such as the positioning of people, surfaces and objects in the real world. The hardware consists of holographic devices, which place digital content in the real world, and immersive devices, which hide the physical world and replace it with a digital experience.
In the Real World
We have already seen many companies hop on the bandwagon of creating virtual live entertainment.
Sensorium teamed up with Mubert to create a new AI-based DJ that jams out at virtual concerts. Teaming up with many big players, including Universal Music Group, Sony Music Entertainment, Warner Music Group, and Beggars Group, the company developed photorealistic avatars to constantly play music for various moods and types of entertainment.
Blank XR: Setting a New Stage
Denise E. White, founder and CEO of BLANK XR.
“I know it’s hard to imagine, but think pre-phone, pre-tablet, pre-everything, when there was just pen and paper. We’re at that same moment again where technology is getting ready to unleash and move into a new paradigm.… We’re taking holograms, we’re making them intelligent through our artificial intelligence [AI] and machine learning [ML] algorithms and we’re placing them into your contextual real-world environment. That’s what’s meant by mixed reality, mixing your real environment with holographic 3D data that’s interactive and talking to you versus VR where you put the headset on and … you can’t see your real world but you’re in an immersive environment,” said White.
BLANK XR’s new channel Immortal XR is combining virtual, augmented and mixed reality entertainment to create interactive, immersive experiences. With the ongoing pandemic, there has been advancing technology that drives social connections and progress in cloud computing to help deliver new ways of interacting with content creators.
“We’re actually crafting a mixed reality experience, so what you’ll see is actually more of a live human being sitting there in front of you. They can be powered live, but there’s some proprietary technology that we’re developing to really animate them from an AI/ML standpoint. We have a deep [synthetic] voice engine that we’ve created … and it enables the artists to sort of extend themselves infinitely right around the world as a 3D hologram. The beautiful thing about that is that from a user perspective they can really get a sense of the person,” explained White.
Using artificial intelligence, music lovers will be able to converse with their favorite artists in a holographic form through a deep voice synthetic engine anytime of the day.
“It’s all based in that human element, which we feel is very important to create that emotional connection and the feeling of reality. We’re aiming so that you can’t really tell the difference between sitting there in front of a real person or a hologram,” added White. “We’re creating new ways to interact with the artist. One of the things that we’re working on right now is kind of a chat functionality so that you can literally speak to the holographic likeness of your favorite artist and they’ll talk back to you with their real voice, so they won’t sound like Alexa.”
Concerts will also be individualized by using AI to give hyper-personalized content such as shout-outs and localized language.
“We all have our favorite artists. Think of someone who you love listening to and imagine being able to speak to them,” said White. “You’re able to have a natural conversation. We’re using a lot of Microsoft Azure services for voice rendering and for our instant local translation.… It’s in real time [and] as 5G rolls out, that latency will get lower and lower.”
Thinking post-pandemic, the company will integrate its technology with artists to make them more interesting and even more immersive in person.
“We’re now thinking of the combination of doing things with live artists on the stage and putting volumetric, AR [VR and mixed reality elements] so they can tell a complete story instead of through a video projection. I can make dragons fly through a venue. I can do anything and everything with volumetric capture,” said Rene Ribbens, chief of Spatial Innovation at BLANK XR.
It will also allow people who missed a concert to see it from any city around the world with the bonus of having the best view in the arena.
Delivering the Future Through Screens
The VR/AR market is expected to grow to $692 billion in revenue by 2025, according to ABI Research. We can expect to see an increasing convergence of AR and VR gear, making them more affordable to everyday consumers.
While gaming has consistently stayed the number one driver of headsets, live entertainment is expected to gain positive momentum, especially during these unprecedented times. A recent poll by the Consumer Technology Association found that consumers are looking for more VR content for concerts, sports and exercises. A Greenlight Insights’ 2017 survey backs up that conclusion, citing that 65 percent of consumers are interested in more live events using VR.
VR concert experiences will remain an alternative to live entertainment as they continue to broadcast the show and feature behind-the-scenes exclusive content with artists. It’s a win-win for the fans and artists alike.