Bringing 3D to the operating room

Jan 31, 2011 By Tiffany Fox
One of VPL's disparity (proportional to depth) estimation methods. Left image: original view (one from the stereo pair). Middle image: initial disparity estimates. Right image: Refined disparity using space-time minimization.

(PhysOrg.com) -- For all of their high-tech advantages, laparoscopic surgical systems are only capable of providing a two-dimensional visualization -- or in other words, no depth perception.

“This means that often surgeons can’t pinpoint the exact location of an organ until they brush up against it with their tools,” explains Ramsin Khoshabeh, a Ph.D. candidate in Electrical and Computer Engineering at the University of California, San Diego. “I hate to put it this way, but some surgical procedures are still done by brute force.”

Khoshabe and his colleagues are part of the Video Processing Laboratory (VPL), which is run by electrical engineering professor Truong Nguyen from the UC San Diego Jacobs School of Engineering. The researchers, who are also part of the UC San Diego division of the California Institute for Telecommunications and Information Technology (Calit2), are working to bring three-dimensional video feeds into the operating room by leveraging the use of autostereoscopic displays. Their hope is that one day surgeons will not only be able to perform minimally invasive surgeries in 3D, they will be able to do so without having to wear potentially cumbersome 3D glasses.

“Surgeons are often slow to adapt to new technologies,” says Khoshabeh. “They worry that a new technology will be dangerous, that it will interfere with the procedure. They say, ‘I’ve been doing this the same way for 40 years; why change things?’

“To accommodate for all these types of mentalities, we are pitching 3D laparoscopic surgery as a supplement. It’s a way for surgeons to enhance their abilities, and it’s also a way for interns and everyone else in the room to learn from what they are doing by watching the video feed.”

Professor Truong Nguyen, from the Department of Electrical and Computer Engineering (ECE), is an expert in digital signal processing. Other collaborators include world-renowned UCSD surgeons Dr. Mark Talamini (chairman of the UCSD Department of Surgery) and Dr. Santiago Horgan (director of Minimally Invasive Surgery), as well as Dr. Horgan’s researcher, Dr. Noam Belkind. The work being done at the VPL — with collaboration from visiting scholars from all around the world — is at the forefront of research in 3D processing.

Researchers on the fourth and fifth floor lab space at Calit2’s Atkinson Hall focus their efforts on all aspects of 2D and 3D image and video processing, including disparity estimation, 3D perceptual measures, image stabilization, frame-rate up-conversion, saliency detection, motion estimation and compensation, coding, and object detection, extraction and tracking.
Khoshabeh and a few of his colleagues — including Electrical and graduate students Can Bal, Lam Tran, Stanley Chan, Ankit Jain, Kyoung-Rok Lee and Jason Juang — have concentrated their efforts on developing algorithms that address three of the greatest challenges in 3D research: virtual view synthesis, disparity estimation and the development of proper 3D evaluation metrics.

Since 3D stereo is actually comprised of two images (one for the left-eye view and one for the right-eye view), both images must be properly aligned in order for the viewer to accurately perceive the 3D effect. For many 3D applications, virtual views need to be synthesized from the existing two images to allow viewers to see a 3D scene from another perspective. When these virtual views are generated, however, previously hidden regions in the scene (perhaps occluded by an object in the foreground) become uncovered, which means that ‘holes’ appear in the virtual images. Khoshabeh and his colleagues in the VPL are looking at ways to fill in holes by extrapolating neighboring information from, say, the left-eye view to fill a hole in the right-eye view, or to improve the method for matching up points in one view to points in another view (also known as disparity estimation).

Notes Khoshabeh: “For humans, this is easy, but for computers this is a non-trivial task. And as you can imagine, this is very important in terms of surgery. If you have something that looks strange in a 3D laparoscopic feed, a surgeon might think it’s a tumor, but really, it’s just a poorly synthesized hole.”

“As the technology evolves, not only will the holes go away, surgeons will be able to see a real live 3D effect, where they can look behind something. But right now, that’s the tricky part — capturing what’s behind and providing surgeons with accurate visualization.”
Although it can create challenges in terms of creating a true 3D effect, Bal notes that extrapolating from one view to capture another is an effective technique for maximizing transmission efficiency — and that’s important when viewing and potentially sharing such large file sizes.

“Having two views means you’re dealing with twice the amount of data,” he explains. “It’s really dumb to send all that information twice when there’s so much overlap, so there are some intelligent things we can do to reduce the file size and bit rate.”

Parallels exist with current 2D video technologies, such as H.264 encoding, which uses motion estimation to correlate sequential video frames. Something similar can be done with 3D videos, except that, by contrast, they have angular correlation (between left and right views) and temporal correlation (between, say, frame ‘t’ and frame ‘t+1’). Khoshabeh says that fusing these two cues will be crucial to achieving high 3D coding efficiency in the compression and transmission of high-resolution 3D data.

The researchers are hoping that one day their algorithms will be standardized by the Moving Picture Experts Group (MPEG), a working group of experts that was formed to set standards for audio and video compression and transmission. Tran’s algorithm for virtual view synthesis is already the top performer in the field.

“But,” Khoshabeh says, “there’s still a lot of room for innovation.”

“The reason I got involved with this research is because I wanted to take general engineering practices and apply them to medicine,” he adds, noting that he intends to apply to UCSD’s Medical Scientist Training program. “Even though we’ve had so many breakthroughs, surgeons are still sewing people up just like they have for hundreds of years.

“The problem right now is that the engineers don’t know what the doctors need, and the doctors don’t know what the engineers are capable of doing. I want to bridge that gap. This work in 3D laparoscopy is only the beginning.”

Explore further: Cadillac CT6 will get streaming video mirror

Related Stories

Modern society made up of all types

Nov 04, 2010

Modern society has an intense interest in classifying people into ‘types’, according to a University of Melbourne Cultural Historian, leading to potentially catastrophic life-changing outcomes for those typed – ...

A 360 degree camera that sees in 3D (w/ Video)

Dec 01, 2010

Surround sight has come to the camera. Inspired by the eye of a fly, EPFL scientists have invented a camera that can take pictures and film in 360° and reconstruct the images in 3D.

Health-conscious future could stem from smartphones

Dec 14, 2010

The latest smartphones are equipped with a range of technologies that can pinpoint your location. It's only a matter of time before they'll also be able to detect your every movement, says Reetika Gupta, assistant professor ...

Recommended for you

Cadillac CT6 will get streaming video mirror

5 hours ago

Cadillac said Thursday it will add high resolution streaming video to the function of a rearview mirror, so that the driver's vision and safety can be enhanced. The technology will debut on the 2016 Cadillac ...

Poll: Americans skeptical of commercial drones (Update)

Dec 19, 2014

Americans broadly back tight regulations on commercial drone operators, according to a new Associated Press-GfK poll, as concerns about privacy and safety override the potential benefits of the heralded drone ...

Cheaper, more powerful VR system for engineers

Dec 17, 2014

It's like a scene from a gamer's wildest dreams: 12 high-definition, 55-inch 3D televisions all connected to a computer capable of supporting high-end, graphics-intensive gaming.

Nokia HERE prepares maps for autonomous cars

Dec 17, 2014

Autonomous cars will need a new kind of map, a crucial element that until now has been given a back seat to the more popularly discussed issues of sensors and legal questions. Senior Writer Greg Miller in ...

Dutch launch 'intelligent bicycle' that warns of danger

Dec 15, 2014

The Netherlands on Monday launched its first-ever "intelligent bicycle", fitted with an array of electronic devices to help bring down the high accident rate among elderly cyclists in the bicycle-mad country.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.