"Organ on the Wall," Feature Article, Nov. 2002

organ on the wall

An English doctor consults a four-foot, sliceable, movable image of a patient's liver while operating on the real thing.

by Jean Thilmany, Associate Editor

These days, liver surgeon Rory McCloy's operations are guided by a four-foot, three-dimensional virtual profile of a patient's liver that rotates at his command on the operating room wall in front of him. Using a specially outfitted mouse and his laptop computer, McCloy can manipulate the virtual liver to find the exact location and extent of a tumor before he makes even one incision in the patient's actual liver on the table below.

This virtual medical technology isn't ahead of its time, McCloy said. Instead, it has been slow to be adopted by the medical establishment and he, for one, aims to speed the pace. McCloy is a surgeon at the Manchester Royal Infirmary at the University of Manchester in England.

In the past 20 years, virtual reality and virtual imaging have revolutionized the way physicians practice medicine, McCloy said. Magnetic resonance imaging, computer-aided tomography scans, and ultrasound images give doctors a vital peek into the inner workings of a patient's body in a way that X-rays cannot. Not only can these imaging tools diagnose disease, tumors, and the like, but they serve as a map to each person's unique body. Just as people are different on the outside—though they all have a nose, two eyes, and two ears—internal organs are also unique. Being able to see exactly what an organ looks like before operating allows doctors to perform closer and more precise work.

Liver surgeon Rory McCloy of the University of Manchester Royal Infirmary in England adapted virtual image technology to study three-dimensional images of the patient's liver as he operates.

McCloy is charged with cutting tumors from the liver and the pancreas. Nicking the tumor during surgery can release cancerous cells into the organ, so he needs to map his cuts.

But MRI, CT, and ultrasound techniques only touch on what technology is capable of now, McCloy said. The shame of it is that more virtual reality technology isn't exploited for the medical environment, he said.

As a doctor long interested in virtual reality, McCloy has gone to meetings and conferences on that subject for the past eight years and has seen firsthand the feasibility of using 3-D scans for more than just diagnostic purposes. The scans could be used in the operating arena to take the place of the X-rays hung around the room to help guide his knife.

"Here I am at a state-of-the-art teaching hospital in the U.K., one of the biggest hospitals in Europe, and I'm still given X-ray films or CT scans that are slices, like slices of salami," he said. "I have to reconstruct in my mind, in the operating room, the 3-D image."

In fact, many hospitals in the United States digitally reconstruct a patient's body or organ in 3-D by use of a CT scan across the body, he said. But those reconstructions are often available only at a computer workstation in the X-ray room.

"The surgeons go down there and look at it and then vanish to the operating theater, and still have to remember where the tumor is," he said. "And while they're very good at remembering, it's the difference between cutting here or cutting three millimeters away in hitting cancer or not hitting cancer."

McCloy likens his operations to extracting an entire plum from a pie with only a few knife slices allowed. It would be easier to do that if you were guided by images in three dimensions that exactly mirrored the pie, rather than looking at slices and trying to figure out how big the plum was and exactly where it was situated.

"I'm busy in my mind during operations trying to put the X-rays into 3-D and going into the patient's body and trying to figure out where this tumor is, and it's all a bit demanding," McCloy said. "Since we're in the digital age, wouldn't it be nice to use all the digits?"

Before using the virtual reality technology, which he and Nigel John, a computer scientist at the university, prepared for the operating room, McCloy studied patients' CT scans and X-rays before surgery to decide how to best operate. During surgery, he didn't have access to the CT scan, because it needed to reside on a computer and the images often didn't load fast enough to really help a doctor.

"On the X-ray, I can see the tumor on one slide and then on another slide, and I can work out that it's several millimeters long," he said. "The X-ray doctors are happy to look at the slide and see that there's a tumor there at that particular part of the liver. That's the diagnosis. But I'm faced with six pounds of liver or the pancreas, which is the size of the banana and buried deep in the body, and you can't see it because it's hidden behind a lot of other structures.

"I know the tumor is there because the X-ray doctors told me it was there," McCloy continued. "But it's a bit like a lucky dip. You put your hand into a barrel and pick out the sweeties. With the X-ray, you know there are six packages of sweeties in the barrel, but you don't know exactly where they are. Only it's not a barrel, it's a liver that bleeds when you cut it. And I have to remove the cancer whole because if I cut into it, the cells spread around the body."

Why Can't I See a Liver in 3-D?

Because of his affiliation with John and because he has attended conferences on virtual reality, McCloy knew that the technology he sought was available. However, he had to find his own way to make it available to him, to get it to meet his own needs. John helped with that.

"My frustration was sheer frustration that people kept giving me these X-ray films while the data has been there for years just sitting on the scanners," he said. "It's okay to look at a lovely 3-D image, but why can't I have that while I'm operating? I'm the guy that has to get the 3-D tumor out."

Because the amount of data contained in a series of CT scans is too large for the graphics card on a personal computer to handle, McCloy couldn't simply bring a computer into the operating room and refer to the patient's CT while operating. The scan would take forever to load. So he and John found a way to send the CT information to a supercomputer at the University of Manchester computing center more than a mile from the hospital. The supercomputer was then networked with a laptop McCloy used in his operating room.

The team used an Onyx 300 visualization computer from SGI, formerly Silicon Graphics Inc., of Mountain View, Calif. The machine is specially configured for high-powered graphics imaging by use of the company's InfiniteReality graphics capability. It simultaneously processes two-dimensional images, 3-D graphics—in this case, the CT scan—and data in real time. A team at the on-campus Manchester Visualisation Centre wrote the software that allows the raw 50 to 80 megabytes of data from the CT images to be graphically visualized.

The image of the patient's liver that McCloy follows while operating is composed of the patient's CT scan, which is sent to a university supercomputer and then to McCloy's laptop computer.

The high-powered graphics are available to McCloy by remote access on his decidedly non-supercomputer, thanks to software called OpenGL Vizserver, also from SGI, which allows the data from the supercomputer to be shifted to other computers, even much less powerful ones, via a network link. In this case, the laptop in McCloy's operating room serves as a remote station that lets him see and manipulate the CT in 3-D while operating. Because the images take no time to load, he can manipulate them in real time. For instance, he can rotate the image of the liver.

To keep the files small for negligible loading time, only the pixels of the graphics are transmitted to the laptop, not the graphics themselves. The computer generates about 20 images a second. McCloy uses a common projector, the same one he uses for PowerPoint presentations, to display the image on the wall.

Because the graphic is what's called a volume reconstruction of the entire data set—in this case, an image of the entire liver, both inside and out—McCloy can slice the image—the liver—for a 2-D look inside the organ to mirror the X-rays he formerly saw. He likened the technology to a 3-D anatomy book customized for each patient and projected at a height of six feet on the operating-room wall. The wall is, in McCloy's words, a tasteful shade of pale green and the images show up clearly even under the bright operating-room lights.

Practicing Before Operating

McCloy performed his first liver surgery using the technology in April, a second in May, and a third in July. The next surgery will be done on a pancreas rather than on a liver. And he's been using the technology to plan surgeries before going into the operating room.

"But the exciting bit is getting the scan on the wall while operating," he said.

Because images take no time to load on his laptop, McCloy can manipulate them in real time. They're reconstructions of the 3-D liver, so he can slice them to see inside the organ.

Still, McCloy runs through the surgery on another laptop computer before going into the operating theater.

"I can practice before I make the cut," he said. "I can say to myself, ÔIf I cut it at this angle, will it work?' And I can see on-screen that it won't. Then, I can cut through it another way and see that, yes, I can get it out this way."

Still, the scan won't accurately reflect the shape of the real liver McCloy will see because organs aren't rigidly conformed inside the body. They move around a bit and spread, or are contracted by other organs and their shape changes accordingly, he said.

"When I actually have the patient's organ in front of me, it may not be in the same shape or at the same angle it is on screen," he added. "But I can rotate the organ on screen so it shows up at the same angle."

Joystick Leads the Change

The trick in making the technology useful for the operating room lay in finding a way to manipulate the images with one hand while performing the surgery with the other, said John, of the school's computing center. A traditional mouse is slow to work with because of the sanitary requirements of the operating room. The mouse can't be sealed within a sterile plastic bag to meet operating room requirements because the roller ball doesn't work so well when encased in plastic, John found.

McCloy has to put on a sterile glove before using a standard mouse in the operating room. When he needs both hands to operate, he takes off that glove and puts on another. It's a time-consuming process that he expects a plastic-encased joystick to eliminate.

A joystick works fine encased in plastic. John's team is now linking an $18 joystick, the type used for computer gaming, to the software program.

Many university teaching hospitals have been working with SGI to find uses for visualization technology, according to Chodi McReynolds, director of marketing at the SGI sciences division. One area of continued study, for example, is pairing a CT scan with customized graphics of the patient's blood flow through veins and arteries provided courtesy of computational fluid dynamics software, McReynolds said.

The pairing of a patient's body scan with graphics that depict blood flow could be used to study how surgery would affect that particular person.

"If they wanted to do a bypass in one spot, they could do that on the image, and it would show how blood flow to the leg would be affected," she said.

For their part, McCloy and John hope to network their supercomputer to other laptops in operating rooms throughout the United Kingdom. First, McCloy will use the technology during 30 surgeries over the next year to document its usefulness. In the next few weeks, a gynecologist colleague will begin looking at ways a digitized and projected CT scan of the pelvis, ovaries, and uterus might be used during surgery.

The visual technology that McCloy and John implemented was funded by a $25 million grant from the Department of Education in the U.K.

"This is a collaboration between a computer scientist, a surgeon, and a company that has the power of computation with large amounts of number crunching," McCloy said. "Put the three of us together, shake us up, and we've come up with this as an answer."