Researchers make augmented reality a group experience

July 26, 2017, Disney Research
Credit: Disney Research

Sit on Disney Research's Magic Bench and you may have an elephant hand you a glowing orb. Or you might get rained on. Or a tiny donkey might saunter by and kick the bench.

It's a combined augmented and mixed reality experience, but not the type that involves wearing a head-mounted display or using a handheld device. Instead, the surroundings are instrumented rather than the individual, allowing people to share the magical experience as a group.

People seated on the Magic Bench can see themselves in a mirrored image on a large screen in front of them, creating a third person point of view. The scene is reconstructed using a depth sensor, allowing the participants to actually occupy the same 3D space as a computer-generated character or object, rather than superimposing one video feed onto another.

"This platform creates a multi-sensory immersive experience in which a group can interact directly with an animated character," said Moshe Mahler, principal digital artist at Disney Research. "Our mantra for this project was: hear a character coming, see them enter the space, and feel them sit next to you."

The research team will present and demonstrate the Magic Bench at SIGGRAPH 2017, the Computer Graphics and Interactive Techniques Conference, beginning July 30 in Los Angeles.

The researchers used a color camera and depth sensor to create a real-time, HD-video-textured 3D reconstruction of the bench, surroundings, and participants. The algorithm reconstructs the scene, aligning the RGB camera information with the depth sensor information.

To eliminate depth shadows that occur in areas where the depth sensor has no corresponding line of sight with the color camera, a modified algorithm creates a 2D backdrop. The 3D and 2D reconstructions are positioned in virtual space and populated with 3D characters and effects in such a way that the resulting real-time rendering is a seamless composite, fully capable of interacting with virtual physics, light, and shadows.

"The bench itself plays a critical role," Mahler said. "Not only does it contain haptic actuators, but it constrains several issues for us in an elegant way. We know the location and the number of participants, and can infer their gaze. It creates a stage with a foreground and a background, with the seated participants in the middle ground. It even serves as a controller; the mixed reality experience doesn't begin until someone sits down and different formations of people seated create different types of experiences."

The Magic Bench was the work of a large team of Disney Research scientists. In addition to Mahler, it includes Kyna McIntosh, John Mars, James Krahe, Jim McCann, Alexander Rivera, Jake Marsico, Ali Israr and Shawn Lawson.

Combining creativity and innovation, this research continues Disney's rich legacy of leveraging technology to enhance the tools and systems of tomorrow.

Explore further: Disney Research transforms movie-quality animations for interactive viewing

More information: "Magic Bench – A Multi-User & Multi-Sensory AR/MR Platform-Paper" [PDF, 6.28 MB]

Related Stories

Computers using linguistic clues to deduce photo content

July 21, 2017

Scientists at Disney Research and the University of California, Davis have found that the way a person describes the content of a photo can provide important clues for computer vision programs to determine where various things ...

Virtual reality app makes haptics as immersive as visuals

November 4, 2016

Disney Research has developed a 360-degree virtual reality application that enables users to enhance their experience by adding customized haptic effects that can be triggered by user movements, biofeedback or timelines.

Using virtual reality to catch a real ball

March 20, 2017

Disney Research scientists have found innovative ways to enhance virtual experiences involving interactions with physical objects by showing how a person using a virtual reality system can use it to reliably catch a real ...

Recommended for you

Apple closing iPhone security gap used by law enforcement

June 14, 2018

Apple is closing a security gap that allowed outsiders to pry personal information from locked iPhones without a password, a change that will thwart law enforcement agencies that have been exploiting the vulnerability to ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.