In-place: Interacting with large displays by combining pen and touch technology

Oct 05, 2012 by Rob Knies
In-Place: Interacting with Large Displays

Computing in the 21st century increasingly is embracing touch interaction. Whether it be on mobile handhelds, large electronic displays, or something in between, such user interfaces are becoming commonplace.

Interestingly, researchers from Microsoft Research Redmond are expanding that modern-day model of interaction using research breakthroughs from the 1990s combined with the latest in touch functionality.

In-Place, a project by Michel Pahud, Ken Hinckley, and Bill Buxton, is the result. The work uses "bi-manual commands," which means two hands working seamlessly together, such as touch with an to move a toward the so it can select a tool.

It doesn't hurt, though, that Buxton worked on one of the technique's predecessors, T3, a prototype two-handed drawing program, while with Alias|Wavefront Research in 1997. In addition, he also helped develop Toolglass and Magic Lens, also used for the In-Place project, while working for Xerox PARC in the early '90s.

What goes around comes around …

"This work was done using indirect manipulation," explains Pahud of the 1990 technologies, "because, at that time, the technology of direct touch on displays was not mature enough and the ability to have simultaneous pen and touch on a digitized screen was not possible. Now that pen and touch is possible on many devices, ranging from slates to large displays, it's a great time to revisit this work."

This video is not supported by your browser at this time.

It also doesn't hurt that Hinckley, through his work on the InkSeine project, is a thought leader in pen computing. Through the prior contributions of Buxton and Hinckley, and the new techniques built atop that 20th-century foundation, In-Place holds promise to enhance significantly human-computer  interaction in the second decade of the 21st century.

The larger displays get, the more out of reach fixed location icons and commands become, making systems more cumbersome to use and preventing multiple people from working at the same time on the same surface. In-Place commands are what one could equally well call right-place commands, where the right place is where you need them—just as a painter's palette it is right by the canvas when the painter needs it, but out of the way when it's not needed. The whole display is freed for content and not for interface. 

In-Place enables you to invoke contextual menus where you want them. You touch the screen in a convenient place, and the menu appears. You move your finger, the menu follows. Remove your finger, and the menu remains where you left it. For normal drag and swipe actions, the In-Place menu stays completely out of the way.

There are other enhancements, as well. You can expand a menu to see a collection of submenus by placing your index finger and thumb together on the menu and spreading them apart. A simple tap away from the menu makes it disappear. A shortcut menu enables you to change modalities. All of these functionalities—and others—are intuitive, natural, and easy to grasp.

Then there is In-Place's Toolglass menu, which provides the ability to create shapes on the touchscreen-enabled device— an example of what the researchers like to call a "direct-input device." Using a pen in one hand and touching the screen with the other, users can create and manipulate a line, a circle, or a rectangle and combine them in various ways. An offset enables accurate placement without one's finger blocking the target. Colors and line thicknesses are also at hand.

"Personally," Pahud says, "I find the Toolglass menu very compelling, because you can use both hands to size and place shapes. For instance, try to draw a circle inside a square in PowerPoint on a slate with no simultaneous pen and touch—or at least multitouch. It's difficult, because you cannot position and size the shapes at the same time. With the Toolglass and two hands, manipulation is very intuitive and straightforward."

The researchers are experimenting with new types of menus that would benefit from the pen-and-touch interactivity. One is a radial menu in which menu options are displayed in circular fashion rather than rectangular. And then there is the Finger Clipboard: With a finger touching the display, you can drag items such as documents, photos, or music files into the finger's shadow. Remove the finger, the items disappear; touch the screen in a different location, and the items reappear in that new position. To unload the "clipboard," you need only swipe the collection of objects across the screen.

For Pahud, the naturalness of the gestures used to operate the In-Place UI is the main selling point for the research.

"I love the extension of the menu and the ability to switch the default tool palette by moving the index finger while keeping the thumb down," he says. "It might appear difficult to perform when you see it in the demonstration, but it's actually very simple, it feels good, and it's intuitive."

Explore further: Bringing history and the future to life with augmented reality

add to favorites email to friend print save as pdf

Related Stories

Sharp Releases Notebook PC with Optical Sensor LCD Pad

Apr 21, 2009

Sharp Corporation will release into the Japanese market a new notebook PC employing an optical sensor LCD for the touchpad. Its new touch-sensing recognition method allows handwritten input and intuitive direct-touch ...

TVs, Cell Phones to Learn 'Sign Language'

Jan 13, 2008

The days of pressing tangible buttons to operate TVs and cell phones may one day be replaced by waving, clapping, and pointing. With two upcoming products, Sony Ericsson´s Z555 gesture phone and JVC´s "Snap ...

Recommended for you

Patent talk: Google sharpens contact lens vision

Apr 16, 2014

( —A report from Patent Bolt brings us one step closer to what Google may have in mind in developing smart contact lenses. According to the discussion Google is interested in the concept of contact ...

Neuroscientist's idea wins new-toy award

Apr 15, 2014

When he was a child, Robijanto Soetedjo used to play with his electrically powered toys for a while and then, when he got bored, take them apart - much to the consternation of his parents.

Land Rover demos invisible bonnet / car hood (w/ video)

Apr 14, 2014

( —Land Rover has released a video demonstrating a part of its Discover Vision Concept—the invisible "bonnet" or as it's known in the U.S. the "hood" of the car. It's a concept the automaker ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Oct 05, 2012
Technologies like these are going to be very interesting when devices like the LeapMotion sensor come out. Essentially, it will make any display, no matter what size, into a 'touch screen' of sorts.

More news stories

Ex-Apple chief plans mobile phone for India

Former Apple chief executive John Sculley, whose marketing skills helped bring the personal computer to desktops worldwide, says he plans to launch a mobile phone in India to exploit its still largely untapped ...

Airbnb rental site raises $450 mn

Online lodging listings website Airbnb inked a $450 million funding deal with investors led by TPG, a source close to the matter said Friday.

Health care site flagged in Heartbleed review

People with accounts on the enrollment website for President Barack Obama's signature health care law are being told to change their passwords following an administration-wide review of the government's vulnerability to the ...

A homemade solar lamp for developing countries

( —The solar lamp developed by the start-up LEDsafari is a more effective, safer, and less expensive form of illumination than the traditional oil lamp currently used by more than one billion people ...

NASA's space station Robonaut finally getting legs

Robonaut, the first out-of-this-world humanoid, is finally getting its space legs. For three years, Robonaut has had to manage from the waist up. This new pair of legs means the experimental robot—now stuck ...

Filipino tests negative for Middle East virus

A Filipino nurse who tested positive for the Middle East virus has been found free of infection in a subsequent examination after he returned home, Philippine health officials said Saturday.

Egypt archaeologists find ancient writer's tomb

Egypt's minister of antiquities says a team of Spanish archaeologists has discovered two tombs in the southern part of the country, one of them belonging to a writer and containing a trove of artifacts including reed pens ...