Since we started covering AR, VR, MR (collectively known sometimes as XR), the goal has been to highlight practical applications used by engineers and product designers to speed up the product design lifecycle by improving quality, efficiency and lowering the overall time-to-market. The technology is split between two main sectors: media and entertainment (gaming, stereoscopic video) and engineering (including product design).
The major players in augmented reality headsets include Microsoft (HoloLens, HoloLens 2), well-funded startup DAQRI (Smart Helmet, Smart Glasses) and perhaps now Google Glass Edition 2.
The major players in virtual reality headsets are Oculus Rift and HTC Vive. Each have brought to market a succession of hardware and software that's improved incrementally since each released their first flagship headset within a week of each other in 2016.
Two Challenges to Professional Accuracy in AR
Motion-to-photon latency is a long-standing issue for users seeking professional levels of accuracy in their tools. Ensuring that the digital information generated by software within the headset is projected with 100 percent accuracy through the lens and projected image. When the image doesn't stay matched after a person fixes their gaze or head motion, there is latency, and this is an issue. Keeping a digital object fixed in on a spot in a room as the user moves their head any which way creates a lot of challenges for developers seeking to create tools for engineers, maintenance and manufacturing professionals. Our eyes have evolved to focus on actual objects relative to whatever three-dimensional environment we find ourselves in at any point in time. Foveated rendering is a term AR and VR developers use for systems with an eye tracker integrated in a headset. The goal of foveated rendering is to increase the efficiency of rendering by decreasing the image quality in peripheral vision. By knowing where the eyes are at all times, this focused rendering mimics how our own eyes and brain work.
Of all the complicated procedures and tasks professionals face daily, performing complex surgery is one of the most important. A recent study conducted at the University of Pisa suggests that the current swath of AR headsets is too imprecise to be considered reliable for tasks that are within two meters of the user. For developers hoping to create accurate and precise metrological tools, this isn't the greatest news.
The study specifically covers researchers who want to use AR headsets like Microsoft HoloLens to guide surgeons with digitally overlaid guides for incisions, or use a virtual axis over bone surfaces to keep realignment surgery on target. According to the study, the problem is that the human eye cannot focus on virtual objects and real objects at the same time, which is known as focal rivalry. The fundamental question arises: how can we use AR for precision-guided tasks if our natural eyes cannot focus on two objects at once?
AR Doesn't Help Individuals "Connect the Dots"
Vincenzo Ferrari, Sara Condino and a team of colleagues created a unique test and enlisted 20 participants. Participants wore the HoloLens, which the researchers consider to be the most sophisticated optical see-through (OST) device and were asked to connect a sequence of numbered dots displayed on the HoloLens onto real paper using a ruler and a pen. The focal rivalry occurs between the computer-generated dots which are projected onto the lens of the headset, and the user’s eyes observing the blank piece of paper simultaneously.
Four different situations were tested with each participant: completing the connect-the-dots task with one eye or both eyes open while wearing the HoloLens, and with one eye or both eyes open while not wearing the HoloLens. Timing and accuracy were measure by the research team at the University of Pisa. Interestingly, the participants felt as though there was no demonstrable difference between how they as individuals completed the task. Connecting the dots with the naked eye or eyes produced errors .9 mm in length on average, while those who completed the tasks with the HoloLens made errors up to 2.3 mm in length on average.
The basic engineering behind most augmented reality devices seems to hinge on a component that uses a beam combiner or half-silvered mirror to merge virtual and real content by sending display images right into the user's eyes. In order that the digital images appear at a viewing distance that is comfortable for the headset user, lenses are placed in between the display and beam combiner to change the focus of the virtual imagery. The virtual imagery than appears to the user at a comfortable distance on what's known as a semitransparent surface of projection (SSP).
As you can imagine, putting together a piece of technology that both accurately places digital information over physical information and actively keeps it positioned relative to the user’s eye motions and head motions isn't the easiest engineering task in the world. AR continues to promise, struggle and baffle its way into a potentially groundbreaking technology. But there are clear bottlenecks and longstanding challenges.
What are the perceptual limitations of augmented reality technology?
A few problems consistently dogging augmented reality are keeping the spatial registration accurate no matter the combination of eye and head movement, having too small of a Field-of-View (FoV), having micro displays with low luminance (too see through to discern the virtual imagery) and motion-to-photon latency among other issues at the SSP, a kind of crossroads where the 2D virtual image meets the real (3D) world.
Bottom Line
Augmented reality applications aren't going to replace the precision measuring of simple or advanced tools of metrology in use today. And why should they? Perhaps a readable stream of digital information fed to an engineer or field mechanic with precise measurements from a digital measuring tool would suffice, especially if both hands are needed.
The research team at the University of Pisa isn't dismissing augmented reality headsets or their potential, they are just sharing their findings from a very basic and straightforward research study. Diving further down the rabbit hole to experiment with AR-assisted surgery, researchers are developing an experimental hybrid AR system that switches between optical see-through (OST) and video see-through (VST)-AR. With video see-through, they will capture a real-view of the world by embedding external cameras on a headset and merging the digital and the video copy of the real world first, combing the images and then sending it to the user's eyeballs.
After comparing both augmented reality systems, the group hopes to come away with a better understanding of the differences in perceptual limitations experienced by each. Their end goal as one would imagine, is to improve the overall performance of AR-assisted surgery, so that it may one day rival our current practice of AR-unassisted surgery.