One touted goal of three-dimensional modeling—which encompasses virtual reality, augmented reality and 3D printing—is improved pre-surgical planning and intraoperative guidance.
Increasingly, organizations are experimenting with 3D models to help surgeons and interventional radiologists determine how to best perform intricate procedures. These advanced visualization approaches replace flat two-dimensional images with life-like replications of patient organs and other body parts that physicians can hold in their hands or manipulate in a video-game-type setting.
The need for 3D visualization methods is rising alongside the growth of minimally invasive procedures, which limit physicians’ ability to see inside the body because they are operating through tiny incisions or natural body openings. “It is … critical for physicians who are not doing traditional surgery with a scalpel, but rather a catheter-based approach, to understand what is going on in the 3D anatomy,” says Dmitry Levin, associate director of UW Medicine’s Center for CardioVascular Innovation.
The usefulness of traditional 2D images in surgical planning is questionable. In a small study at NYU Langone Health, in New York, Nicole Wake, a postdoctoral fellow at the Center for Advanced Imaging Innovation and Research, found that surgeons are not very good at identifying the location of kidney tumors based solely on 2D images. Wake asked three kidney surgeons to pinpoint tumors on 3D representations of 10 diseased kidneys based on their interpretations of CT and MRI scans. The surgeons’ estimates tended to be off-target compared with the actual tumor locations.
“This shows how bad we are at translating 2D into 3D,” Wake says. “Having a 3D model could really help … the surgeon [get] a better idea of where … they need to operate.”
As part of a study, Wake is developing 3D kidney and prostate models to help surgeons plan complicated surgeries. In one case, a patient only had one working kidney, and the surgeon had to determine how to remove the diseased part of the organ while preserving as much kidney function as possible. “The surgeon said, ‘I could not have done the case without the model,’” Wake says.
Early 3D modeling efforts described at the recent Radiological Society of North America annual meeting suggest that advanced visualization approaches can help surgeons conduct more precise surgeries. This, in turn, promises to improve patient outcomes and reduce turnaround times.
The basic process for creating 3D models is similar in virtual reality, augmented reality and 3D printing, Wake says. A patient’s radiology scans, such as CT or MRI images, are sliced into segments and reassembled as a pop-up computer image to reveal the breadth and depth of a body part. The computerized image can be printed using a 3D fabrication printer, or visually displayed on screens, objects or floating in a room using virtual and augmented reality systems and equipment, such as Oculus Rift and HTC Vive virtual reality headsets, Microsoft HoloLens augmented reality system, and Google Glass augmented reality eyeglasses.
Eliot Siegel, MD, associate vice chair at the University of Maryland School of Medicine and chief of imaging services at the VA Maryland Healthcare System explained the difference between virtual and augmented reality: “Virtual reality is immersive and completely replaces your environment; you are not seeing what’s going on outside. Augmented reality supplements your environment by superposing its images over your own space.”
Augmented reality can be space agnostic or cognizant. “Space-agnostic systems, like Google Glass, superimpose an image on the display/glasses and move with your head and eyes, Siegel says. “Space-cognizant systems, such as the HoloLens, display an object that appears to occupy a specified location in your environment so that you can walk around the object and look at it from above and below.”
UW Medicine, a Seattle-based academic medical center, is developing virtual reality models of patients’ hearts to help interventional cardiologists with a procedure called left atrium appendage (LAA) occlusion, which reduces the risk of stroke from blood clots in people with irregular heart rhythms. During the procedure, a catheter is inserted into a vein in the groin and threaded to the LAA where a parachute-shaped device is deployed to close the LAA.
The procedure can get tricky because of two anatomy-related challenges: One is that the LAA is shaped differently in different people. “[The cardiologists have] to adjust whatever they are doing to [a patient’s] particular anatomy,” says Dmitry Levin, associate director of UW Medicine’s Center for CardioVascular Innovation.
Just as challenging, the physician has to find and cross a dime-sized region between the right and left atriums to get to the LAA. If they don’t cross in the correct location, they may have to start over, increasing procedure time.
The 3D heart models that Levin creates are helping cardiologists individualize each LAA procedure to each patient’s anatomy. It is also helping cardiologists and other clinicians to collaboratively plan procedures. Multiple members at different sites can view the virtual model simultaneously, which enables collaboration.
At the University of Maryland School of Medicine, Siegel and his colleagues have been experimenting with various augmented reality models. “We are pursuing a ‘holy grail’ … of superimposing a CT, MRI or PET/CT image over a patient that stays registered to the patient as he or she moves on the table, in real-time. Whether used as a supplement to fluoroscopy-guided procedures (such as a stent graft, vertebroplasty or biopsy) or during surgery, we anticipate that this will reduce procedure time, radiation dose, and improve patient safety.” Siegel estimates that procedure time will decrease 15 to 20 percent.
Each of the three types of advanced imaging visualization has its pros and cons. Three-dimensional printing enables physicians to physically touch a model, providing tactile understanding of a patient’s anatomy; however, printing is expensive and time-consuming. It took 9 hours and 26 minutes to print one kidney model at NYU Langone, Wake said.
Virtual reality models can be developed fairly rapidly using a desktop computer with a lot of processing power. These models are also easy to manipulate. Physicians can grab a 3D image of a heart, for example, with a controller and rotate the organ or open a section to get a better understanding of the anatomy. Potential cons include ergonomic issues (such as wearing a headset for lengthy periods) and being cut off from the real world.
Augmented reality carries many of the same benefits as virtual reality, plus the added bonus of being able to view 3D models anywhere in the environment. Augmented reality systems currently have limited processing capabilities, but a separate streaming server can be used to speed up rendering, Siegel says.
Many current technology shortcomings will be resolved in the near future, Siegel believes. “Things are going to be changing really quickly … partly fueled by the entertainment business and military applications. Those huge technology investments outside of the healthcare space will end up being available to everybody very soon.”