Kinect launches a surgical revolution

Jun 08, 2012 By Douglas Gantenbein
Kinect for Windows enables surgeons to use gestures to manipulate medical imagery while in the surgical theater.

Medical imaging today gives surgeons an ability to obtain a virtual peek inside the human body in a way that rivals the campy 1966 movie Fantastic Voyage, in which a team of physicians is miniaturized and injected into a wounded diplomat to save his life.

In today’s operating theater, though, those virtual images can be difficult to retrieve and manipulate. A surgeon can’t use a mouse or keyboard, because they are unsterile and pose a risk of infection. A surgeon can give instructions to someone else, who then can operate a mouse or keyboard, but the surgeon must break concentration to say, “Right … right … up a little … OK.”

Now, a team at Microsoft Research Cambridge consisting of social scientists, computer scientists, and designers has used the for Windows hardware and SDK to simplify this process. It enables doctors to use simple hand gestures to change, move, or zoom in on CT scans, MRIs, and other medical images. In initial development, the Kinect for Windows-based system has thrilled surgeons who have seen it and who believe it could help make surgery faster and more accurate. The hope is that these systems will deliver better outcomes to patients when fully field-tested and approved.

The project, called Touchless Interaction in , to bring Kinect for Windows into surgical suites has been guided by Helena Mentis, a Philadelphia native and a postdoctoral researcher with the Socio-Digital Systems group at the Cambridge lab. She has been intrigued for a long time by how technology can improve medical practices. She earned a Ph.D. in Information Sciences and Technology from The Pennsylvania State University, writing a thesis on how expressions of emotion between health-care workers helped foster collaboration in emergency rooms.

Microsoft Research Cambridge’s extensive work in exploring new technology attracted Mentis to that facility. The lab’s researchers also were instrumental in delivering many of Kinect for Windows’ capabilities, so when Mentis heard about Kinect, she soon began to investigate how it might be deployed in medical settings.

“Even before Kinect came on the market, my colleagues and I started to do some fieldwork with an eye on whether something could be developed for surgical theaters,” Mentis says. “The initial idea was to learn where we would want to use the system and what type of systems could interact with it in the surgical area.”

When Kinect for Xbox 360 hit the market in November 2010, Mentis was ready to go. Since then, her team has worked closely with surgeons to test it at several hospitals, particularly vascular surgeons at St. Thomas’ Hospital, located prominently across the Thames River from the British Houses of Parliament.

Vascular surgeons often can avoid performing open-heart surgery by placing stents to open clogged arteries or to support aortas with an aneurysm. But proper placement of the stent requires a great deal of precision, and medical imaging plays a big role in enabling that precision.

By using the Kinect for Windows sensor in conjunction with a large monitor, a surgeon easily can change the image view of a patient’s chest area, look at a new image, or zoom in on details—all without touching a screen, a keyboard, or a mouse. That can help the surgeon determine more quickly the proper placement of a stent.

“They love it,” Mentis says of surgeons’ reactions to seeing Kinect in action. “The idea of being able to control almost anything with just a wave of the hand really speaks to how surgeons want to feel like they have control over things. They want to be able to save as much time as possible and to be able to get a patient in and out of anesthesia as quickly as possible. Those all are things that they’re motivated by, so they think this is the greatest thing ever.”

In the operating room, the setup consists of a Windows PC connected to a fluoroscopy scanner through an external video-capturing device, a Kinect for Windows sensor attached to the PC with a USB cable, and an LCD monitor. The Kinect for Windows sensor is placed below or above one of the computer monitors that the surgeons currently use for visualising medical images.

Key 3-D medical images are acquired before the surgery using a CT (computed tomography) scanner. During surgery, the fluoroscopy scanner captures real-time X-ray images, which are overlaid on one another, registered, and displayed in an image viewer. A surgeon then uses gestures to rotate images, zoom into details, and more.

Although still being tested, the Kinect for Windows sensor, Mentis said, works well in the surgical suite. It’s able to track and follow multiple people, something other camera-based gestural systems cannot do. It also could be “trained” to know which individual in a surgical suite wanted to use the system, which can be a challenge, given that everyone essentially is masked and dressed the same.

To do that, the Kinect sensor is programmed to respond to the command, “Kinect, control,” in conjunction with a raised hand. The sensor then “watches” that individual for instruction, ignoring other people who might themselves be gesturing and talking.

Surgeons also must avoid swinging their arms in too wide of an arc while working with the Kinect for Windows hardware to avoid touching a contaminated object. The system compensates by supporting gesturing in a limited area in front of the surgeon’s body, using the Near Mode feature in Kinect for Windows.

“This is very different than designing for a game, where one can flail about wildly,” Mentis says. “The surgeon’s sterile area is from chest to waist and from shoulder to shoulder. We worked hard to ensure the surgeon could fully use the system while staying within a safe and sterile zone.”

But otherwise, sterility isn’t a particular challenge. If the Kinect for Windows sensor and its accompanying monitor are kept clean and left untouched, the risk of infection is substantially reduced.

In March, Mentis and her team began to test the first iterations of the system with several vascular surgeons in a surgical theater without patients. By the beginning of May, the first field-trial test with patients was under way.  

Surgeons who have worked with the research team are eager to give the system a regular place in surgery, once finalized.

“As a surgeon, being able to manipulate information in a touch-free way while operating will have enormous benefits,” says Dr. Neville Dastur, a vascular surgeon at the Brighton and Sussex University Hospitals National Health Service (NHS) Trust. “Breaking the flow of the operation to get someone to do it for you not only distracts from performing a delicate surgery, but it’s also time-consuming.”

Dr. Tom Carrell, vascular surgeon at Guy’s and St. Thomas’ NHS Foundation Trust, says adding Kinect for Windows to the surgical theater could be a big step forward.

“With Kinect, we could revolutionize the way we do complex operations,” Carrell says. “Patients and clinical teams will spend less time in theater, and surgeons will be more in control of the information they need.”

More Possibilities

The Kinect for Windows-based imaging system, Mentis says, already is being evaluated for other new uses. One example is the manipulation of 3-D models of the brain for neurosurgery at Addenbrooke’s Hospital in Cambridge. 

“Neurosurgeons are interested in being able to go deep into the brain and extract a tumor or to even do a biopsy for a tumor,” she says. “Unfortunately, tumors never are right on the outside of the brain—they typically are within the brain tissue. The idea is to get in there with the minimal amount of brain damage possible, and better ability to view imaging can help with that.”

Another aim is to develop a touchless interaction-in-surgery tool kit that can be used by any hospital or system interested in applying touchless interaction to an imaging system.

Mentis is working with an extensive team to make the Kinect for Windows-based surgical-suite setup a reality. Partners at Microsoft Research Cambridge included principal researcher Abigail Sellen, senior researcher Antonio Criminisi, designer Robert Corish, and vendor Kenton O’Hara. The team also worked with Mark Rouncefield and Gerardo Gonzalez from Lancaster University’s Computing Department. The team, which also included participation from King’s College London, hopes that innovations in technology using Kinect for Windows that make surgery easier and faster will obtain appropriate governmental approval.

Explore further: Bringing history and the future to life with augmented reality

Provided by Microsoft Corporation

5 /5 (4 votes)
add to favorites email to friend print save as pdf

Related Stories

Pioneering touchless surgical technology

Jun 01, 2012

Researchers and surgeons from King’s College London and Guy’s and St Thomas’ NHS Foundation Trust are piloting ‘touchless’ technology in the operating theatre.

ShakeID tracks touch action in multi-user display

Jun 03, 2012

(Phys.org) -- How do you determine who is doing the touching with a multi-user touch display? Microsoft Research has published a paper that presents a technique for doing so. The researchers make their attempt ...

Austin lab team rolls out Kinect-controlled skateboard

Jan 10, 2012

(PhysOrg.com) -- An Austin, Texas, team has developed an electric skateboard that makes use of Microsoft’s Kinect and a Samsung tablet running Windows 8 to go places on a piece of transport dubbed The ...

Microsoft offers $20,000 payouts for Kinect apps

Nov 23, 2011

(PhysOrg.com) -- Microsoft has launched a project where it will give 10 companies $20,000 each for making the best use of Kinect on Windows or the Xbox. The project is Microsoft’s strategic push for extending ...

WorldWide Telescope lights up with Kinect

Apr 19, 2011

A few weeks ago Microsoft Research held an event on Microsoft Campus called TechFest. We show a lot of new projects and prototypes from our labs but we keep a lot of stuff behind closed doors. There was one demo that blew ...

Recommended for you

Patent talk: Google sharpens contact lens vision

Apr 16, 2014

(Phys.org) —A report from Patent Bolt brings us one step closer to what Google may have in mind in developing smart contact lenses. According to the discussion Google is interested in the concept of contact ...

Neuroscientist's idea wins new-toy award

Apr 15, 2014

When he was a child, Robijanto Soetedjo used to play with his electrically powered toys for a while and then, when he got bored, take them apart - much to the consternation of his parents.

Land Rover demos invisible bonnet / car hood (w/ video)

Apr 14, 2014

(Phys.org) —Land Rover has released a video demonstrating a part of its Discover Vision Concept—the invisible "bonnet" or as it's known in the U.S. the "hood" of the car. It's a concept the automaker ...

User comments : 0

More news stories

TCS, Mitsubishi to create new Japan IT services firm

India's biggest outsourcing firm Tata Consultancy Services (TCS) and Japan's Mitsubishi Corp said Monday they are teaming up to create a Japanese software services provider with annual revenues of $600 million.

Finnish inventor rethinks design of the axe

(Phys.org) —Finnish inventor Heikki Kärnä is the man behind the Vipukirves Leveraxe, which is a precision tool for splitting firewood. He designed the tool to make the job easier and more efficient, with ...

Atom probe assisted dating of oldest piece of earth

(Phys.org) —It's a scientific axiom: big claims require extra-solid evidence. So there were skeptics in 2001 when University of Wisconsin-Madison geoscience professor John Valley dated an ancient crystal ...