Bringing 3D to the operating room

Jan 31, 2011 By Tiffany Fox
One of VPL's disparity (proportional to depth) estimation methods. Left image: original view (one from the stereo pair). Middle image: initial disparity estimates. Right image: Refined disparity using space-time minimization.

(PhysOrg.com) -- For all of their high-tech advantages, laparoscopic surgical systems are only capable of providing a two-dimensional visualization -- or in other words, no depth perception.

“This means that often surgeons can’t pinpoint the exact location of an organ until they brush up against it with their tools,” explains Ramsin Khoshabeh, a Ph.D. candidate in Electrical and Computer Engineering at the University of California, San Diego. “I hate to put it this way, but some surgical procedures are still done by brute force.”

Khoshabe and his colleagues are part of the Video Processing Laboratory (VPL), which is run by electrical engineering professor Truong Nguyen from the UC San Diego Jacobs School of Engineering. The researchers, who are also part of the UC San Diego division of the California Institute for Telecommunications and Information Technology (Calit2), are working to bring three-dimensional video feeds into the operating room by leveraging the use of autostereoscopic displays. Their hope is that one day surgeons will not only be able to perform minimally invasive surgeries in 3D, they will be able to do so without having to wear potentially cumbersome 3D glasses.

“Surgeons are often slow to adapt to new technologies,” says Khoshabeh. “They worry that a new technology will be dangerous, that it will interfere with the procedure. They say, ‘I’ve been doing this the same way for 40 years; why change things?’

“To accommodate for all these types of mentalities, we are pitching 3D laparoscopic surgery as a supplement. It’s a way for surgeons to enhance their abilities, and it’s also a way for interns and everyone else in the room to learn from what they are doing by watching the video feed.”

Professor Truong Nguyen, from the Department of Electrical and Computer Engineering (ECE), is an expert in digital signal processing. Other collaborators include world-renowned UCSD surgeons Dr. Mark Talamini (chairman of the UCSD Department of Surgery) and Dr. Santiago Horgan (director of Minimally Invasive Surgery), as well as Dr. Horgan’s researcher, Dr. Noam Belkind. The work being done at the VPL — with collaboration from visiting scholars from all around the world — is at the forefront of research in 3D processing.

Researchers on the fourth and fifth floor lab space at Calit2’s Atkinson Hall focus their efforts on all aspects of 2D and 3D image and video processing, including disparity estimation, 3D perceptual measures, image stabilization, frame-rate up-conversion, saliency detection, motion estimation and compensation, coding, and object detection, extraction and tracking.
Khoshabeh and a few of his colleagues — including Electrical and graduate students Can Bal, Lam Tran, Stanley Chan, Ankit Jain, Kyoung-Rok Lee and Jason Juang — have concentrated their efforts on developing algorithms that address three of the greatest challenges in 3D research: virtual view synthesis, disparity estimation and the development of proper 3D evaluation metrics.

Since 3D stereo is actually comprised of two images (one for the left-eye view and one for the right-eye view), both images must be properly aligned in order for the viewer to accurately perceive the 3D effect. For many 3D applications, virtual views need to be synthesized from the existing two images to allow viewers to see a 3D scene from another perspective. When these virtual views are generated, however, previously hidden regions in the scene (perhaps occluded by an object in the foreground) become uncovered, which means that ‘holes’ appear in the virtual images. Khoshabeh and his colleagues in the VPL are looking at ways to fill in holes by extrapolating neighboring information from, say, the left-eye view to fill a hole in the right-eye view, or to improve the method for matching up points in one view to points in another view (also known as disparity estimation).

Notes Khoshabeh: “For humans, this is easy, but for computers this is a non-trivial task. And as you can imagine, this is very important in terms of surgery. If you have something that looks strange in a 3D laparoscopic feed, a surgeon might think it’s a tumor, but really, it’s just a poorly synthesized hole.”

“As the technology evolves, not only will the holes go away, surgeons will be able to see a real live 3D effect, where they can look behind something. But right now, that’s the tricky part — capturing what’s behind and providing surgeons with accurate visualization.”
Although it can create challenges in terms of creating a true 3D effect, Bal notes that extrapolating from one view to capture another is an effective technique for maximizing transmission efficiency — and that’s important when viewing and potentially sharing such large file sizes.

“Having two views means you’re dealing with twice the amount of data,” he explains. “It’s really dumb to send all that information twice when there’s so much overlap, so there are some intelligent things we can do to reduce the file size and bit rate.”

Parallels exist with current 2D video technologies, such as H.264 encoding, which uses motion estimation to correlate sequential video frames. Something similar can be done with 3D videos, except that, by contrast, they have angular correlation (between left and right views) and temporal correlation (between, say, frame ‘t’ and frame ‘t+1’). Khoshabeh says that fusing these two cues will be crucial to achieving high 3D coding efficiency in the compression and transmission of high-resolution 3D data.

The researchers are hoping that one day their algorithms will be standardized by the Moving Picture Experts Group (MPEG), a working group of experts that was formed to set standards for audio and video compression and transmission. Tran’s algorithm for virtual view synthesis is already the top performer in the field.

“But,” Khoshabeh says, “there’s still a lot of room for innovation.”

“The reason I got involved with this research is because I wanted to take general engineering practices and apply them to medicine,” he adds, noting that he intends to apply to UCSD’s Medical Scientist Training program. “Even though we’ve had so many breakthroughs, surgeons are still sewing people up just like they have for hundreds of years.

“The problem right now is that the engineers don’t know what the doctors need, and the doctors don’t know what the engineers are capable of doing. I want to bridge that gap. This work in 3D laparoscopy is only the beginning.”

Explore further: For Google's self-driving cars, learning to deal with the bizarre is essential

Related Stories

Modern society made up of all types

Nov 04, 2010

Modern society has an intense interest in classifying people into ‘types’, according to a University of Melbourne Cultural Historian, leading to potentially catastrophic life-changing outcomes for those typed – ...

A 360 degree camera that sees in 3D (w/ Video)

Dec 01, 2010

Surround sight has come to the camera. Inspired by the eye of a fly, EPFL scientists have invented a camera that can take pictures and film in 360° and reconstruct the images in 3D.

Health-conscious future could stem from smartphones

Dec 14, 2010

The latest smartphones are equipped with a range of technologies that can pinpoint your location. It's only a matter of time before they'll also be able to detect your every movement, says Reetika Gupta, assistant professor ...

Recommended for you

Audi to develop Tesla Model S all-electric rival

12 hours ago

The Tesla Model S has a rival. Audi is to develop all-electric family car. This is to be a family car that will offer an all-electric range of 280 miles (450 kilometers), according to Auto Express, which ...

A green data center with an autonomous power supply

18 hours ago

A new data center in the United States is generating electricity for its servers entirely from renewable sources, converting biogas from a sewage treatment plant into electricity and water. Siemens implemented ...

After a data breach, it's consumers left holding the bag

19 hours ago

Shoppers have launched into the holiday buying season and retailers are looking forward to year-end sales that make up almost 20% of their annual receipts. But as you check out at a store or click "purchase" on your online shopping cart ...

Can we create an energy efficient Internet?

19 hours ago

With the number of Internet connected devices rapidly increasing, researchers from Melbourne are starting a new research program to reduce energy consumption of such devices.

Brain inspired data engineering

20 hours ago

What if next-generation ICT systems could be based on the brain's structure and its cognitive and adaptive processes? A groundbreaking paradigm of brain-inspired intelligent ICT architectures is being born.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.