VROOM Combines Mixed Reality Avatars and Robots

Microsoft’s vested interest in their Mixed Reality program featuring HoloLens has taken a somewhat odd turn. VROOM, which stands for Virtual Robot Overlay for Online Meetings, is a way to synthesize AR, VR and telepresence robots in an amalgam of digital and physical technologies designed to create more immersive presences for distributed workers in the office. COVID-19 continues, as does social distancing, so the timing of this announcement is curious.

(Image courtesy of Microsoft).

At Microsoft Research, the creators of VROOM described the system in a paper posted by Microsoft. It starts with someone working remotely using virtual reality and another working remotely in a different location wearing a mixed reality headset such as Microsoft’s HoloLens.

The mixed reality headset tracks the worker’s position and tracks the movement of their head, while a telepresence robot performs a stereoscopic scan of the worker’s surrounding environment. A Unity app animates an avatar of the worker using a Mixed Reality headset, so the worker using a virtual reality headset can see their avatar in real-time through their headset.

A Unity app is used to animate the avatar for the person wearing the HoloLens based on the movement of the remote worker. The worker wearing the VR headset can see their hand movements and gestures as well.

Hand gestures and arm movements are recorded by controllers and seen by both participants. The system adds mouth movement when people are talking, as well as blinking and idle movements to make the avatar seem more lifelike. VROOM converts a 2D image into a 3D avatar to place the remote worker’s face on the avatar’s head.

The VROOM system also gives the VR user a first-person view so the remote worker can see their hand movements and gestures. The avatar then appears to walk when the remote worker instructs the robot to move.

Bottom Line

VROOM is only capable of demonstrating this AR-VR-telepresence robot connection in one-to-one interactions. It seems like a quite unwieldy process at this point in the evolution of AR and VR. Telepresence robots have great applications for people unable to attend various physical locations due to temporary infirmities, maladies and permanent disabilities.

VROOM researchers plan to expand the interaction to multiple people and with less cost-prohibitive mobile robots in the future.