Bringing gesture-control technology from Hollywood to corporate conference rooms

September 8, 2014 by Rob Matheson
Oblong's collaborative-conferencing system, Mezzanine, being used in a conference room. Credit: Oblong Industries

When you imagine the future of gesture-control interfaces, you might think of the popular science-fiction films "Minority Report" (2002) or "Iron Man" (2008). In those films, the protagonists use their hands or wireless gloves to seamlessly scroll through and manipulate visual data on a wall-sized, panoramic screen.

We're not quite there yet. But the brain behind those Hollywood interfaces, MIT alumnus John Underkoffler '88, SM '91, PhD '99—who served as scientific advisor for both films—has been bringing a more practical version of that to conference rooms of Fortune 500 and other companies for the past year.

Underkoffler's company, Oblong Industries, has developed a platform called g-speak, based on MIT research, and a collaborative-conferencing system called Mezzanine that allows multiple users to simultaneously share and control digital content across multiple screens, from any device, using gesture control.

Overall, the major benefit in such a system lies in boosting productivity during meetings, says Underkoffler, Oblong's CEO. This is especially true for clients who tend to pool resources into brainstorming and whose meeting rooms may remain open all day, every day.

"If you can make those meetings synthetically productive—not just times for people to check in, produce status reports, or check email surreptitiously under the table—that can be electrifying force for the ," he says.

Mezzanine surrounds a conference room with multiple screens, as well as the "brains" of the system (a small server) that controls and syncs everything. Several Wii-like wands, with six degrees of freedom, allow users to manipulate content—such as text, photos, videos, maps, charts, spreadsheets, and PDFs—depending on certain gestures they make with the wand.

That system is built on g-speak, a type of operating system—or a so-called "spatial operating environment"—used by developers to create their own programs that run like Mezzanine.

"G-speak programs run in a distributed way across multiple machines and allow concurrent interactions for multiple people," Underkoffler says. "This shift in thinking—as if from single sequential notes to chords and harmonies—is powerful."

Oblong's clients include Boeing, Saudi Aramco, SAP, General Electric, and IBM, as well as government agencies and academic institutions, such as Harvard University's Graduate School of Design. Architects and real estate firms are also using the system for structural designing.

A Wii-like wand used to manipulate data with gestural control in the Mezzanine system. Credit: Oblong Industries

Putting pixels in the room

G-speak has its roots in a 1999 MIT Media Lab project—co-invented by Underkoffler in Professor Hiroshi Ishii's Tangible Media Group—called "Luminous Room," which enabled all surfaces to hold data that could be manipulated with gestures. "It literally put pixels in the room with you," Underkoffler says.

The group designed light bulbs, called "1/0 Bulbs," that not only projected information, but also collected the information from a surface it projected onto. That meant the team could make any projected surface a veritable computer screen, and the data could interact with, and be controlled by, physical objects.

They also assigned pixels three-dimensional coordinates. Imagine, for example, if you sat down in a chair at a table, and tried to describe where the front, left corner of that table was located in physical space. "You'd say that corner is this far off the floor, this far to the right of my chair, and this much in front of me, among other things," Underkoffler explains. "We started doing that with pixels."

One application for urban planners involved placing small building models onto a 1/0 Bulb projected table, "and the pixels surrounded the model," Underkoffler says. This provided three-dimensional spatial information, from which the program casted accurate, digital shadows from the models onto the table. (Changing the time on a digital clock changed the direction of the shadows.)

In another application, the researchers used a glass vase to manipulate digital text and image boxes that were projected onto a whiteboard. The digital boxes were linked to the vase in a circle via digital "springs." When the vase moved, all the graphics followed. When the vase rotated, the graphics bunched together and "self-stored" into the vase; when the vase rotated again, the graphics reappeared in their first form.

These initial concepts—using the whole room as a digital workplace—became the foundation for g-speak. "I really wanted to get the ideas out into the world in a form that everyone could use," Underkoffler says. "Generally, that means commercial form, but the world of movies came calling first."

"The world's largest focus group"

Underkoffler was recruited as scientific advisor for Steven Spielberg's "Minority Report" after meeting the film's crew, who were searching for novel technology ideas at the Media Lab. Later, in 2003, Underkoffler reprised his behind-the-scenes gig for Ang Lee's "Hulk," and, in 2008, for Jon Favreau's "Iron Man," which both depicted similar technologies.

Seeing this technology on the big screen inspired Underkoffler to refine his MIT technology, launch Oblong in 2006, and build early g-speak prototypes—glove-based systems that eventually ended up with the company's first customer, Boeing.

Having tens of millions of viewers seeing the technology on the big screen, however, offered a couple of surprising perks for Oblong, which today is headquartered in Los Angeles, with nine other offices and demo rooms in cities including Boston, New York, and London. "It might have been the world's largest focus group," Underkoffler says.

Those enthused by the technology, for instance, started getting in touch with Underkoffler to see if the technology was real. Additionally, being part of a big-screen production helped Underkoffler and Oblong better explain their own technology to clients, Underkoffler says. In such spectacular science-fiction films, technology competes for viewer attention and, yet, it needs to be simplified so viewers can understand it clearly.

"When you take technology from a lab like at MIT, and you need to show it in a film, the process of refining and simplifying those ideas so they're instantly legible on screen is really close to the refinement you need to undertake if you're turning that lab work into a product," he says. "It was enormously valuable to us to strip away everything in the system that wasn't necessary and leave a really compact core of user-interface ideas we have today."

After years of writing custom projects for clients on g-speak, Oblong turned the most-requested features of these jobs—such as having cross-platform and multiple-user capabilities—into Mezzanine. "It was the first killer application we could write on top of g-speak," he says. "Building a universal, shared-pixel workspace has enormous value no matter what your business is."

Today, Oblong is shooting for greater ubiquity of its technology. But how far away are we from a consumer model of Mezzanine? It could take years, Underkoffler admits: "But we really hope to radically tilt the whole landscape of how we think about computers and user interface."

Explore further: 'Minority Report' software hits the real world

Related Stories

'Minority Report' software hits the real world

July 22, 2012

The software behind the film "Minority Report" -- where Tom Cruise speeds through video on a large screen using only hand gestures -- is making its way into the real world.

A 'hands-on' approach to computers

April 6, 2009

(PhysOrg.com) -- At a time when ever more aspects of our lives are moving toward the virtual, online world -- stores, newspapers, games and even social interactions -- Hiroshi Ishii seems to be swimming against that current: ...

Team shows calibrated multiple-projector spherical display

August 26, 2014

Researchers from Canada and Brazil offer designers a hard-to-resist invitation—a spherical display that lets you use gestures to interact with three-dimensional display objects. Their formal description is "A 3D perspective-corrected ...

Recommended for you

Samsung to disable Note 7 phones in recall effort

December 9, 2016

Samsung announced Friday it would disable its Galaxy Note 7 smartphones in the US market to force remaining owners to stop using the devices, which were recalled for safety reasons.

Swiss unveil stratospheric solar plane

December 7, 2016

Just months after two Swiss pilots completed a historic round-the-world trip in a Sun-powered plane, another Swiss adventurer on Wednesday unveiled a solar plane aimed at reaching the stratosphere.

Solar panels repay their energy 'debt': study

December 6, 2016

The climate-friendly electricity generated by solar panels in the past 40 years has all but cancelled out the polluting energy used to produce them, a study said Tuesday.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.