Wearable depth-sensing projection system makes any surface capable of multitouch interaction (w/ video)

Oct 17, 2011

(PhysOrg.com) -- OmniTouch, a wearable projection system developed by researchers at Microsoft Research and Carnegie Mellon University, enables users to turn pads of paper, walls or even their own hands, arms and legs into graphical, interactive surfaces.

OmniTouch employs a depth-sensing camera, similar to the Microsoft Kinect, to track the user's fingers on everyday surfaces. This allows users to control by tapping or dragging their fingers, much as they would with touchscreens found on smartphones or . The projector can superimpose keyboards, keypads and other controls onto any surface, automatically adjusting for the surface's shape and orientation to minimize distortion of the projected images.

This video is not supported by your browser at this time.

"It's conceivable that anything you can do on today's mobile devices, you will be able to do on your hand using OmniTouch," said Chris Harrison, a Ph.D. student in Carnegie Mellon's Institute. The palm of the hand could be used as a phone keypad, or as a tablet for jotting down brief notes. Maps projected onto a wall could be panned and zoomed with the same finger motions that work with a conventional multitouch screen.

This video is not supported by your browser at this time.

Harrison was an intern at Microsoft Research when he developed OmniTouch in collaboration with Microsoft Research's Hrvoje Benko and Andrew D. Wilson. Harrison will describe the technology on Wednesday (Oct. 19) at the Association for Computing Machinery's Symposium on and Technology (UIST) in Santa Barbara, Calif.

The OmniTouch device includes a short-range depth camera and laser pico-projector and is mounted on a user's shoulder. But Harrison said the device ultimately could be the size of a deck of cards, or even a matchbox, so that it could fit in a pocket, be easily wearable, or be integrated into future handheld devices.

"With OmniTouch, we wanted to capitalize on the tremendous surface area the real world provides," said Benko, a researcher in Microsoft Research's Adaptive Systems and Interaction group. "We see this work as an evolutionary step in a larger effort at Microsoft Research to investigate the unconventional use of touch and gesture in devices to extend our vision of ubiquitous computing even further. Being able to collaborate openly with academics and researchers like Chris on such work is critical to our organization's ability to do great research — and to advancing the state of the art of computer user interfaces in general."

Harrison previously worked with Microsoft Research to develop Skinput, a technology that used bioacoustic sensors to detect finger taps on a person's hands or forearm. Skinput thus enabled users to control smartphones or other compact computing devices.

The optical sensing used in OmniTouch, by contrast, allows a wide range of interactions, similar to the capabilities of a computer mouse or touchscreen. It can track three-dimensional motion on the hand or other commonplace surfaces, and can sense whether fingers are "clicked" or hovering. What's more, OmniTouch does not require calibration — users can simply wear the device and immediately use its features. No instrumentation of the environment is needed; only the wearable device is needed.

Explore further: For Google's self-driving cars, learning to deal with the bizarre is essential

More information: www.chrisharrison.net/index.php/Research/OmniTouch
(Videos and images: Chris Harison)

Related Stories

Skinput turns your arm into a touchscreen (w/ Video)

Mar 01, 2010

(PhysOrg.com) -- If you find yourself getting annoyed at the tiny touchscreens on today's mobile devices, you might be interested in a "new" yet overlooked input surface: yourself. A new skin-based interface ...

Microsoft Kinect makes moves on computers

Jun 17, 2011

Microsoft on Thursday began letting software developers imbue computers with voice and motion-sensing technology from its Kinect controller for the Xbox 360 videogame console.

Touch Mouse ready for Windows 7 after two long years

Jan 12, 2011

More than two years of hard work, countless prototypes, and intense collaboration with team members around the globe are just a few of the things that went into the creation of the Microsoft Touch Mouse, a ...

Recommended for you

Hackers force message on websites via US firm

9 hours ago

A U.S. firm that helps connect more than 700 companies with customers through social media says a Syrian group hacked the company's web address to upload a message to other websites.

Shedding light on solar power

15 hours ago

Everyone wants to save energy, but not everyone knows where to start. Grid Resources, a startup based out of the Centre for Urban Energy's iCUE incubator, is developing a new website that seeks to help homeowners ...

Energy transition project moves into its second phase

15 hours ago

Siemens is studying new concepts for optimizing the cost-effectiveness and technical performance of energy systems with distributed and fluctuating electricity production. The associated IRENE research project ...

User comments : 14

Adjust slider to filter visible comments by rank

Display comments: newest first

iPan
3 / 5 (2) Oct 17, 2011
It's neat, but no one is going to wear it.
Isaacsname
not rated yet Oct 17, 2011
I would wear it if it were integrated into a decent set of glasses/goggles, maybe some eye-tracking for an optical mouse.

It seems to be very similar to the Sixth Sense system

http://www.youtub...=related

http://www.youtub...=related

Dr_Doe
3.3 / 5 (3) Oct 17, 2011
^^I highly doubt that's how big it will remain and even if it did, I bet people would wear it.... There's always someone either rich enough or "trendy" enough. Fact is, it will become almost invisible it will be so small and I can't wait. People really need to think outside the box more often.
Skultch
not rated yet Oct 17, 2011
The question is, will the engineering of the miniaturization happen before we figure out thought recognition and/or heads-up displays that are either on contact lenses or connected directly to the optic nerve?
TrinityComplex
not rated yet Oct 17, 2011
I agree that it's unlikely both that it would remain that size when put on the consumer market, or that many people would wear it, but the power draw and size could be reduced by removing the need for such a powerful projector. The platform might even be able to fit into a set of glasses, which people wouldn't have as many qualms about wearing. The camera lenses would be on the outside of either of the glasses' lenses, and a third could be on the bridge. The projector could be reduced to projecting on the lenses, or a retinal projector could be used. Taken a step further it could even track eye movement for additional control. A slim cable could run from one or both arms of the glasses like earphone cables for power and data, connecting to a mobile device carried elsewhere. I'd imagine it they would be about the same size as those MP3 glasses Oakley manufactures.
blazingspark
not rated yet Oct 17, 2011
The question is, will the engineering of the miniaturization happen before we figure out thought recognition and/or heads-up displays that are either on contact lenses or connected directly to the optic nerve?
I can tell you straight up that we are not close to wiring anything directly to the bodies nervous system. Mostly because it is surgically invasive and the interfaces dont last long in the body.
Yes it has been done in trials but only short term. There still no convenient way to connect to the bodies nerves long term.
MediocreSmoke
not rated yet Oct 17, 2011
It's neat, but no one is going to wear it.


I'm sure people said similar things to/about the early generation of room sized computers, or giant Zack Morris cell phones.
Isaacsname
not rated yet Oct 17, 2011
The question is, will the engineering of the miniaturization happen before we figure out thought recognition and/or heads-up displays that are either on contact lenses or connected directly to the optic nerve?
I can tell you straight up that we are not close to wiring anything directly to the bodies nervous system. Mostly because it is surgically invasive and the interfaces dont last long in the body.
Yes it has been done in trials but only short term. There still no convenient way to connect to the bodies nerves long term.


I'm sure we could find some body modification freaks to volunteer for experiments..

MorganW
5 / 5 (1) Oct 17, 2011
I saw this on TED a few years back. They were able to put the whole thing together (and link it to a 2-D scanner and Google) for a couple hundred bucks.
Nice to see Microsoft & Co is still "innovating"... :/
amit_darji
not rated yet Oct 18, 2011
I concur with MorganW, the idea has been around for sometime, Pranav Mistry developed the idea at MIT calling it Sixth Sense and presented it on TED a while back and I believe made the concept open source (well at least that what he mentioned on TED). Great idea, if Microsoft can develop the Sixth Sense idea into a consumer product than great. Not sure how valuable it might be for the average consumer, but perhaps there may be greater use for it industry. In any case for any gadget lover it will be an item on the wish list.
Jimbaloid
not rated yet Oct 18, 2011
It's neat, but no one is going to wear it.


I agree, it is one of those solutions that is looking for the problem to solve. It might well find it, but the need to wear it about the upper body makes it an unlikely candidate for mainstream consumer devices.
visual
3 / 5 (1) Oct 18, 2011
I can tell you straight up that we are not close to wiring anything directly to the bodies nervous system. Mostly because it is surgically invasive and the interfaces dont last long in the body.
Yes it has been done in trials but only short term. There still no convenient way to connect to the bodies nerves long term.

I can tell you we've been doing exactly that for decades now, with cochlear implants.
OK, I do know what you meant... the cochlear implants case isn't exactly qualifying as central nervous system example.
But I still think you are wrong. Of course, your "not close" is up to interpretation, but to me the various work that's been done with carbon nanotube electrodes, optical nerve stimulation, inductive (contact-less) reception of single-neuron signals and so on means "close". As in, quite likely to happen within my lifetime.
Skultch
not rated yet Oct 18, 2011
The question is, will the engineering of the miniaturization happen before we figure out thought recognition and/or heads-up displays that are either on contact lenses or connected directly to the optic nerve?
I can tell you straight up that we are not close to wiring anything directly to the bodies nervous system. Mostly because it is surgically invasive and the interfaces dont last long in the body.
Yes it has been done in trials but only short term. There still no convenient way to connect to the bodies nerves long term.


And this tech here might reach a critical roadblock that can't be overcome to make viable. We don't know, was really my only point.
powerup1
1 / 5 (1) Oct 19, 2011
It's neat, but no one is going to wear it.


@iPan. You do understand that this is a prototype and not a finished product, don't you? :-D

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.