Researchers and engineers at Purdue University and the Indiana University School of Medicine are developing an “augmented reality telementoring” system to connect surgeons with specialists across the world.
Telementoring is already used to allow experts to guide surgeons remotely using telestrators, much like what you see when watching football commentary. Annotations are laid over video of a surgery, but this can potentially divert the doctor’s attention away from the patient.
The new System for Telementoring with Augmented Reality (STAR) uses modern technology like transparent displays and sensors to improve the quality of communication between surgeons and specialists.
The STAR system uses a tablet positioned between the surgeon and their patient, held in place with a robotic arm or surgical assistant. A video stream of the operation is sent to the expert who, with a pad of their own, can create annotations and communicate verbally through the device.
“The surgeon sees the operating field, the instruments and their hands as if the display were not there, yet the operating field is enhanced with the mentor’s graphical annotations,” states Juan Wachs, professor of industrial engineering at Purdue.
However, there is still plenty of room for error with this type of system. With all the motions involved in a surgery, doctors can easily lose track of annotation.
In response, the STAR system uses computer vision algorithms to keep annotations aligned. This is done by “anchoring” annotations so they remain in the same location relative to the patient no matter how the camera moves.
Voicu S. Popescu, professor of computer science at Purdue, points out that other limitations are still in the process of being solved.
“The video acquired by the tablet will be warped to the view of the surgeon, which will require acquiring the operating field with a depth camera similar to the Kinect camera, and will require tracking the surgeon’s head,” said Popescu.
Furthermore, because the surgeon’s hands are between the camera and the surgical field, they sometimes obstruct the specialist’s view. An algorithm may be able to detect the surgeon’s hands and render them semi-transparent for easier viewing.
Further research will also focus on improving the robustness of annotation anchoring with surgical field changes, the anchoring frame rate and the transparent display simulation fidelity.
So far, researchers have tested the system using commercially available tablets, robots and even Google Glass, while performing common operating room procedures on animals and synthetic patient simulators.
“The study provides preliminary indication that the system allows trainees to follow some mentor instructions more accurately than existing telementoring systems,” Wachs said. “Data suggests the system can provide meaningful improvements to the accuracy of surgical tasks.”
With this technology, experts could bring their skills to isolated rural and field hospitals or military front lines where specialists might not be available.
The researcher’s findings are detailed in an online paper appearing in The Visual Computer and will see print in the near future.