Epic Games has sponsored this post.

Aneurysm surgery application. (Image courtesy of Igarashi Lab, The University of Tokyo.)

Neurosurgery is up there with rocket science as one of the go-to examples of difficult endeavors. And for good reason: with roughly 86 billion neurons, the human brain may be the most complex object in the entire universe. Even a rocket engine pales in comparison.

Despite their own 86 billion neurons firing at full speed, for neurosurgeons, there's no such thing as an easy brain operation. But brain operations can be made easier, as researchers from The University of Tokyo have discovered, with a little help from artificial brains—computers, that is.

Computer Graphics in Medicine

What can computer graphics do for the medical field? That’s a driving question for some neurosurgeons, and researchers Hirofumi Seo and Takeo Igarashi from The University of Tokyo’s Department of Creative Informatics were asked to join the research by Taichi Kin and Nobuhito Saito from The University of Tokyo Hospital’s Department of Neurosurgery.¹

In 2018, Seo and Igarashi proposed an enhanced method for visualizing medical images from processes such as MRIs and CT scans. With a few simple graphical techniques—screen space ambient occlusion, depth of field, and depth cueing, among others—Seo and Igarashi found they could transform original medical scan renders into much richer and far more useful visuals for doctors.

Human anatomy visuals before and after Hirofumi Seo and Takeo Igarashi’s refined technique. (Image courtesy of Igarashi Lab, The University of Tokyo.)

“In the medical field, the visualization of 3DCG [3D computer graphics] is often really cheap,” said Seo. “The structure of the human body is very complicated, so beautiful lighting, shading, and rendering are really helpful for understanding the complex structure.”

After completing their work on enhanced anatomy rendering, Seo and Igarashi set their sights on a more specific problem: brain aneurysm surgery. A brain aneurysm is a bulge in a brain artery, and they’re found in 3.2 percent of the adult population worldwide. Ruptured brain aneurysms are the most common cause of strokes, which can be fatal or severely debilitating.

One treatment for brain aneurysms is to clip off the artery to prevent further blood flow to the aneurysm. But first, surgeons have to get to the artery. A common way to perform clipping is to take the so-called Transsylvian approach, where surgeons enter the brain through the Sylvian fissure, the region separating the frontal and temporal lobes of the brain.

The Sylvian fissure separates the temporal and frontal lobes of the brain. (Image courtesy of Igarashi Lab, The University of Tokyo.)

The Transsylvian approach is easier said than done (and it’s not that easily said). What the picture above doesn’t show is the mass of blood vessels and trabeculae (thin strands of connective tissue) that block the Transsylvian approach.

“During the surgery, neurosurgeons have to move each blood vessel branch located in the Sylvian fissure to either the frontal or the temporal lobe side, according to the blood-vessel-branch dominant region,” Seo explained. “However, the visible area during the surgery is very limited, so it is often very difficult to guess the correct direction to pull each blood vessel around the Sylvian fissure. Being pulled in the wrong direction might cause a dangerous situation such as instability of the blood vessel or hemorrhage.”

Simulating Aneurism Surgery

Fresh off their work on anatomy visualization, Seo and Igarashi saw immediately how they could address the difficulties of the Transsylvian approach. “We are trying to simulate the surgery before the actual surgery, using some medical images of each patient, which would make the surgery much safer,” Seo said.

The idea was to build an application that would allow neurosurgeons to see exactly what was in store for them during surgery. A dynamic 3D model of the brain, constructed from patient scans, would enable surgeons to simulate the surgery and practice the procedure before taking the patient in for surgery.

A simulated 3D model of a brain could help neurosurgeons practice the Transsylvian approach. (Image courtesy of Igarashi Lab, The University of Tokyo.)

To build their simulation application, the researchers turned to Unreal Engine, a real-time 3D development platform from Epic Games. Originally built as a game engine, Unreal Engine is finding increasing use in architecture, engineering, and now, medicine.

Starting with a patient’s MRI and 3D rotational angiography data, the researchers manually extracted the partial polygon mesh of the brain using Avizo, data visualization software from Thermo Fisher Scientific. The researchers also manually extracted the central blood vessel line with Autodesk Maya, and used their 2018 visualization techniques to improve the appearance of the vessels. Finally, they brought the 3D data into Unreal Engine to build the simulation application.

(Image courtesy of Igarashi Lab, University of Tokyo.)

“We wanted to use a game engine from the beginning of our research because of the wonderful rendering quality,” Seo said. “We selected Unreal Engine mainly because C++ is possible. We had to implement a super-fast physics simulation, so the speed was essential. Also, we wanted to integrate some CUDA-based code to the game engine, so a C++-based game engine was perfect.”

Unreal Engine offers developers the option of writing custom functionality with C++, in addition to providing a visual scripting language called Blueprint. Unreal Engine also includes many mathematical C++ APIs that Seo and the team needed for their application, including FVector for vector manipulation, FMath for various mathematical functions, and UKismetMathLibrary for additional math functions.

“Without Unreal Engine, we would have to write all mathematical structs or functions, all the mouse or keyboard interaction methods, and rendering processes,” Seo said, “which is almost impossible, especially if we want real-time, beautiful rendering or post-process effects such as blurring.”

Building the brain surgery simulation application in Unreal Engine. (Image courtesy of Igarashi Lab, The University of Tokyo.)

In Unreal Engine, the researchers developed a simulated brain that responds in real time to a user’s input (mouse or multi-touch). Users can manipulate the Sylvian fissure and trace blood vessel branches to their proper lobes. The vessels are even color-coded by an automatic algorithm that, while not 100 percent accurate, greatly simplifies the surgical process. The Unreal Engine application also automatically generates virtual trabeculae that users can cut during the simulation.

Blood vessels are automatically color-coded by an algorithm that estimates whether they belong to the frontal or temporal lobe. (Image courtesy of Igarashi Lab, The University of Tokyo.)

You can see the application in action in this video from the researchers:

Better Medicine Through Simulation

Though their work is preliminary, Seo, Igarashi, and the rest of their team see an incredible opportunity for the medical community.

“Neurosurgeons all over the world performing the brain aneurysm cerebral surgery want some kind of presurgical simulation, because the actual surgical view is very limited and the surgery itself is very difficult,” said Seo. “So many neurosurgeons have wanted to use 3DCG for a long time, but they don’t know how to implement it.”

This isn’t the first time computer simulations have been a source of innovation for neurology. Researchers at Stevens Institute of Technology have used computer simulations to better understand a type of neurological procedure called a decompressive craniectomy. Beyond the brain, Unreal Engine has also been used to develop virtual reality simulations to help train orthopedic surgeons.

To learn more about Unreal Engine and how it’s being used across a wide range of industries, visit the Unreal Engine website.

¹_{Grant number JP18he1602001 from the Japan Agency for Medical Research and Development.}