The Greatest Engineering Challenge to Improve Mobile Augmented Reality Headsets

In 2016, tethered systems like the HTC Vive and Oculus Rift rose in popularity swiftly, and largely due to the fact that they used “outside-in tracking” technology, relying on powerful workstation computers to do most of the legwork. This was perceived as less cumbersome than devices like HTC Vive lighthouse base stations, which must be installed in the user’s immediate vicinity to create a tracking environment for measuring a user’s orientation and position.

Since then, Oculus Go, HTC Vive Focus and the Microsoft Mixed Reality headsets have evolved to inside-out tracking, meaning no base stations are required to track a user’s position, motion and orientation. This is due largely to the design of Microsoft HoloLens. The HoloLens popularized the concept of inside-out tracking, and DAQRI Smart Glasses followed suit after their evolution from the DAQRI Smart Helmet. DAQRI now uses inside-out tracking as well to make their Smart Glasses easier to set up and use—untethered and mobile.

Latency is the main challenge for mobile augmented reality headsets, more so than for virtual reality headsets, which up until recently, were mostly tethered to powerful workstation or laptop computers.

Too much latency and you receive the classic complaint about all augmented, virtual and mixed reality experiences: they make you nauseous or sick.

For mobile headsets that use inside-out tracking like the HoloLens, a technique known as visual-inertial fusion has to be incorporated into the user experience in order to offset latency to undetectable levels. Prediction, late correction and inertial measurement units (IMUs) are techniques and hardware that are combined by product design teams to bring end-to-end latencies under 5 milliseconds, greatly improving the user experience.

Correcting latency is much harder to do in the realm of augmented reality.

Latency

Specifically, it is known as motion-to-photon latency, which is a property of both augmented and virtual reality and is characterized in terms of the user experience by the amount of time it takes for a display to catch up with a user’s motion. If you move your head to the right, the displayed content has a lag and possible unwanted artefacts that appear relative to the time, orientation and position of the headset. This makes the experience unpleasant, making users nauseous and compelled to stop and take off the headset.

In virtual reality, the headset fully occludes physical reality from the user as they are visually and fully immersed in a virtual environment. The vestibular system within the ear combined with muscle motion causes the perception of misalignment between the person and the digital content, but a larger range of latency (up to 20 milliseconds) remains mostly inconspicuous. 

In augmented reality, digital content is overlaid by optical see-through (OST) systems onto the real world. Human perceptual systems do not have any latency. This is why any overlaid virtual content in AR that does have latency is way more pronounced than latency in the fully occluded virtual environment of VR. Couple this with the fact that mobile AR headsets do not have the computing power of a tethered workstation or laptop, and it’s easy to understand why latency is more of an engineering challenge for AR as opposed to VR.

Sensors

For mobile augmented reality devices like the HoloLens and DAQRI Smart Glasses, motion estimation—otherwise known as inside-out pose tracking—is accomplished by an array of IMUs and tracking cameras.

Think of every image as part of an assembly line. Tracking cameras (for both AR and VR headsets) typically run at a frame rate of 30 Hz, which translates to one image captured and read out for processing every 33 milliseconds. This frame rate also limits exposure time per frame to 33 milliseconds.

If the exposure rate is 20 milliseconds, the timestamp most convenient for processing is the middle point, which is 10 milliseconds. When the exposure is complete, the age of the image is 10 milliseconds old and will be read out and sent to the image processing system which adds 5 milliseconds to the age of the image, bringing it to 15 milliseconds.

But IMUs run at rates of one to several orders of magnitude greater than cameras. If an IMU operates at 1000 Hz, it will generate one sample per millisecond with virtually no latency and the average age of the sample will be 1 millisecond.

Tracking

Tracking extrapolates a 6 degrees-of-freedom (6DOF) motion estimation, showing which way the headset has changed position. When the camera image reaches the tracking system, the age of the image is 15 milliseconds old. The IMU has less latency and 1 millisecond-old IMU data, which is less old than the camera frame and generally not processed at this stage. If the camera is running at 30 Hz, then the tracking system should be running at the same rate, giving it a 33-millisecond processing budget.

Rendering

In rendering, a frame buffer (representing a 2D image) is sent to the display of the mobile AR unit. The renderer uses several different sources of data to create this 2D image. Since rendering processes at a higher rate than motion estimations, it can be done separately and without interference from tracking computations.

If rendering operates at 60 Hz, then a new frame is generated at a rate of approximately once every 17 milliseconds. If you consider the length of time that is 17 milliseconds, transmission of the frame to the display only begins at the tail-end of that length of time. 

Display

The mechanics of displays for mobile AR displays is crucial in terms of fending off unnecessary latency. When the data is sent one pixel at a time and one line at a time—which is also known as the scanout—and the display is running at 60 Hz, the frame’s transmission time to reach the display adds 17 milliseconds of additional latency.

There are two main types of displays that are used for AR and VR: Color Sequential Displays and Line Sequential Displays.

Color Sequential Displays

Known as “liquid crystalline on a silicon wafer”, or LCoS displays, these are found primarily in mobile AR units, versus Line Sequential Displays such as LCD and OLED displays, which are common in consumer electronics products and VR.

The LCoS data is organized in subframes of different colors instead of interleaved RGB frames, which is beneficial considering that GPUs create interleaved RGB frames by default. So the display process lines up with the GPU hardware process, not to mention that DisplayPort and HDMI also work with interleaved RGB frames.

If you sense a catch-22 here, you’re correct. If single color subframe isn’t ready until the whole RGB frame is received, a frame of latency occurs before a separated-color subframe is displayed.

To complicate matters further for mobile AR LCoS displays, each successive subframe in sequence has an offset delay of 4 milliseconds, adding to overall latency.

IMUs, Prediction and Late Correction to the Rescue

If a product design team for mobile AR could reduce camera exposure time, motion blur would be reduced. But small cheap cameras don’t do this very well without leaving artefacts that would counteract the improvement in the first place. 

Faster tracking sequences or processing units could reduce the amount of time it takes to process an image, but they will incur a cost in overall accuracy.

But IMUs can run at 1000 Hz, so every sample is 1 millisecond old at most. If you use IMU data to calculate a relative motion path from 1 timestamp to another in every sample within that length of time, this allows design teams to apply the relative motion over an absolute pose from the tracker, calculating a new pose that is only 1 millisecond old, which is way less latent than without this calculation, which would incur about 40-70 milliseconds of latency total.

This is a 2D operation, and as such, it only takes a few milliseconds compared to more complex 3D operations. On a modern GPU, this “late warping”—a type of late correction—can be executed just before the frame buffer is transmitted to the display.

The big difference between LCoS displays (used for mobile AR) and OLED displays (used for VR like HTC Vive and Oculus Rift) is that there is no standard protocol for LCoS, and as a result, LCoS manufacturers have different proprietary protocols. The reason protocols are different is due to the fact that GPUs produce color sequential frame buffers by design, so extra processing is needed to correct the color sequential format needed by the LCoS display. To correct this, LCoS display manufacturers include a chip that receives frame buffers in common display standards, process a color separation and finally passes the data on via proprietary protocol.

The HIMAX LCoS is used in both DAQRI Smart Glasses and Microsoft HoloLens. Since only a single-color is shown at a time, this can lead to the colors breaking up under motion, producing what’s known as the rainbow effect. 

Optimizing LCoS Display Mobile AR

If LCoS displays produce unwanted effects and artefacts because of the innate sequencing necessary to consistently refresh the display, late warping techniques can be used for correction. But mobile units are hindered by a lack of processing power versus systems tethered to powerful workstations and laptops. To balance catch-22s between requirements for high throughput, low latency and low power, the process of connecting a GPU to an LCoS display still requires proprietary display protocols.