Researchers combinehundreds of videos to reconstruct 3D motion without markers (w/ Video)

Jul 17, 2014
Thousands of video trajectories of a man swinging a baseball bat, captured in Carnegie Mellon University's Panoptic Studio, are reconstructed to create this image. Credit: Carnegie Mellon University

Carnegie Mellon University researchers have developed techniques for combining the views of 480 video cameras mounted in a two-story geodesic dome to perform large-scale 3D motion reconstruction, including volleyball games, the swirl of air currents and even a cascade of confetti.

Though the research was performed in a specialized, heavily instrumented video laboratory, Yaser Sheikh, an assistant research professor of robotics who led the research team, said the techniques might eventually be applied to large-scale reconstructions of sporting events or performances captured by hundreds of cameras wielded by spectators.

The video lab, called the Panoptic Studio, also can be used to capture the fine details of people interacting, whether it be college students casually conversing or a child being evaluated by a psychologist for signs of autism.

In contrast to most previous work, which typically has involved just 10 to 20 video feeds, the Carnegie Mellon researchers didn't have to worry about filling in gaps in data; their camera system can track 100,000 points at a time. Rather, they have to figure out how to choose which of the hundreds of video trajectories can see each of those points and select only those camera views for the reconstruction.

"At some point, extra camera views just become 'noise,'" said Hanbyul Joo, a Ph.D. student in the Robotics Institute. "To fully leverage hundreds of cameras, we need to figure out which cameras can see each target point at any given time."

Carnegie Mellon's Panoptic Studio used 480 cameras to capture the motion of a man swinging a baseball bat. Credit: Carnegie Mellon University

The research team developed a technique for estimating visibility that uses as a cue. In contrast to that use balls or other markers, the researchers used established techniques for automatically identifying and tracking points based on appearance features—in this case, distinctive patterns. For each point, the system then seeks to determine which cameras see motion that is consistent with that point.

For instance, if a point on a person's chest is being tracked and most cameras show that point is moving to the right, a camera that picks up motion in the opposite direction is probably seeing a person or object that is in between the target and the camera. Or it may indicate the person has turned and the chest is no longer visible to the camera. In either case, the system knows that cannot see the target point and that its video feed is not useful for 3D reconstruction involving that point.

Other researchers have been able to use images from a large number of cameras, such as smartphones, to create 3D reconstructions of still images, Joo noted. But without methods such as the visibility estimation technique, 3D motion reconstruction at such a large scale has not been possible.

In the Panoptic Studio, the researchers have 480 , plus an additional 30 high-definition video cameras, arrayed all around and halfway up the walls of a geodesic dome that can easily accommodate 10 people.

This video is not supported by your browser at this time.
Credit: Carnegie Mellon University

Such a dense array of cameras enables the researchers to perform 3D motion reconstructions not previously possible. These include 3D reconstructions of a person tossing confetti into the air, with each piece of paper tracked until it reaches the floor. In another case, confetti is fed into a fan, enabling a motion capture of the air flow. "You couldn't put markers on the paper without changing the flow," Joo explained.

This video is not supported by your browser at this time.
Full Length Video. Credit: Carnegie Mellon University

Likewise, such techniques might be used for of the motion of animals, which typically can't be instrumented.

Explore further: Filmmakers to get thrill-ride scenes with IMAX 3D digital camera

add to favorites email to friend print save as pdf

Related Stories

Body-mounted cameras turn motion capture inside out

Aug 08, 2011

Traditional motion capture techniques use cameras to meticulously record the movements of actors inside studios, enabling those movements to be translated into digital models. But by turning the cameras around — mounting ...

Researchers demonstrate markerless motion capture

Aug 06, 2012

Conventional motion capture for film and game production involves multiple cameras and actors festooned with markers. A new technique developed by Disney Research, Pittsburgh, has demonstrated how three-dimensional motion ...

A 360 degree camera that sees in 3D (w/ Video)

Dec 01, 2010

Surround sight has come to the camera. Inspired by the eye of a fly, EPFL scientists have invented a camera that can take pictures and film in 360° and reconstruct the images in 3D.

Recommended for you

Key decisions on drones likely from Congress

9 hours ago

The Obama administration is on the verge of proposing long-awaited rules for commercial drone operations in U.S. skies, but key decisions on how much access to grant drones are likely to come from Congress ...

Building a machine that sorts candy colors with iPhone

Dec 23, 2014

The very idea of a machine being able to color-sort M&Ms teases an inventor's imagination and interest in machines, electronics and programming. A person with a website called "reviewmylife" had heard about ...

Laser technology aids CO2 storage capabilities

Dec 23, 2014

DOE's National Energy Technology Laboratory is attracting private industry attention and winning innovation awards for harnessing the power of lasers to monitor the safe and permanent underground storage ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

dayblakely_donaldson
not rated yet Jul 17, 2014
Is this for military weapons?

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.