What happens when physicians get 3-D vision inside a patient’s body while they’re performing a procedure?
By Julie Wood
Brian Park, MD, retrieves a CT scan of a torso on his computer monitor. He moves away from the screen and carefully puts on a headset with lightly tinted lenses aligning with his eyes. Turning to look into his office space through the lens of the headset, he lifts his hand in front of him to make a simple pinching motion. On the screen, the torso rotates. But from Park’s point of view through the headset, he is interacting with a floating 3-D CT scan in the room in front of him. This is not just a new way of looking at medical images; Park is demonstrating a method that is being used and refined in Penn Medicine procedures, one that could change the future of how surgery is practiced.
The headset, Microsoft HoloLens, is a wireless portable device designed to use “mixed reality” for learning and collaborating on projects in business settings. Park, an Interventional Radiology fellow at the Hospital of the University of Pennsylvania who has a background in engineering and contributed to the design of the Microsoft Xbox 360 processor, has combined his skills to program new uses for HoloLens in medicine.
Mixed reality projects virtual 3-D models that appear to coexist within a person’s own physical space in real time. Unlike virtual reality, which immerses the viewer in a completely artificial and simulated world, or augmented reality, which simply overlays 2-D images in one’s field of vision, mixed reality offers a fusion of both forms of visual technology. At Penn Medicine, the 3-D models have consisted of virtually constructed bones, organs, and tumors taken from patients’ scans for medical procedures.
Using HoloLens, physicians can manipulate and respond to these models within their virtual workspace as they perform a procedure. “You don’t have to be straining your neck or looking across the room for data,” Park says. Rather than disrupting the course of the procedure to check scans of a patient’s body or examine the progress of a needle being inserted on a screen located away from the patient, the physician can now view this imaging through their headset. “Using HoloLens, you could essentially watch your hands and the patient while you’re looking at the imaging at the same time,” Park explains. “The images are inside of the patient and in 3-D. We can physically see what's happening at the surface of the patient, but now we can virtually see inside of them, too, right in front of us.”
Over the last year at Penn Medicine, HoloLens has been primarily used in ablations performed by Stephen Hunt, MD, PhD, an assistant professor of Radiology—he has used it in several of these procedures to guide the placement of his instruments into the right location to destroy tumors. Penn neurosurgeons have also collaborated to use the device during spinal fixation surgery to direct screw placements into the spine, led by Isaac Chen, MD, an assistant professor of Neurosurgery, and Vivek Buch, MD, a Neurosurgery resident.
Park adds that HoloLens also makes CT-guided procedures more efficient. Rather than conducting multiple small needle adjustments and confirming each adjustment with scans—which also expose the patient to small doses of radiation each time—physicians using HoloLens can see in the 3-D anatomical projection where the needle needs to go and how much it needs to be adjusted to get to the right place. “We sometimes adjust the needle a little too much or not quite enough," Park explains. "HoloLens can help us adjust the needle just the right amount.”
HoloLens also helps physicians trace the exact line and angle of approach they have drawn on a scan in advance of a procedure for virtual guidance. “You can just use the virtual line as a guide and follow it with your needle,” Park says.
“Visually, it makes physicians more confident with doing the procedures,” Park says. Other hospitals have used HoloLens simply for practice or preparation prior to a surgery, Park explains, but Penn’s hospitals actually use it during the procedure itself.
HoloLens does have limitations, including limited battery life, making it insufficient for any prolonged operations; however, Penn physicians have typically only used the device for a short period of time during procedures.
The medical applications of HoloLens are also still early in development, but Park says his colleagues are excited about its potential. “It’s not quite there yet,” he says. “But we’re going in the right direction.”