Researchers develop fast, economical method for high-definition video compositing

May 20, 2013

Video compositing to create special effects, replace backgrounds or combine multiple takes of an actor's performance is an integral, but highly labor-intensive, part of modern film making. Researchers at Disney Research, Zürich, however, have found an innovative way to create these composite videos that is simple, fast, and easy to use.

Rather than perform a painstaking of that are to be added or subtracted from a video, the Disney system, called DuctTake, uses to find a spatiotemporal "seam" through the video frame that enables two or more videos to be joined together.

These seams can be highly irregular, following the contours of people, furniture and other objects that are common in each take of the scene. Because it can only combine scenes that have overlapping content, DuctTake isn't useful for combining arbitrary . But it works like a charm when combining multiple takes of the same shot.

The Disney Research, Zürich researchers showed the technique can be used across a wide range of video composites. For example, it can combine the best performances by actors from several different takes into a single seamless output, reducing the number of on-set takes that are required to be filmed. Furthermore, the same technique can be used to make a cut seamless, allowing the first half of one take to be combined into a second take. In another case, the researchers eliminated unwanted cars from a street scene by offsetting the vehicle with empty street from the same video take. Hard-to-shoot scenes involving animals, they demonstrated, become a bit easier when an animal trainer can be easily removed from the shot, or other actors can be added at the moment of time when the animal does the right thing.

"The most delicate component is alignment," said Oliver Wang, post doctoral researcher at Disney Research, Zürich. "But given properly aligned views, we can almost always generate good composites with minimal work."

The findings by Wang and his Disney Research, Zürich team of Jan Rüegg, Aljoscha Smolic and Markus Gross were presented at Eurographics 2013, the European Association for Computer Graphics conference.

Most video compositing is accomplished now by the digital equivalent of "cut-and-paste." Rotoscoping is the process by which elements can be added by drawing segmentation outlines. Chroma-keying, familiar to viewers of TV weathercasts in which news announcers appear to stand in front of large, animated maps, separates actors from backgrounds based on color hues; it's cheap and robust, but restricts filming to studio environments, and can require challenging color balancing in post-production.

"Our approach solves a simpler problem," Gross acknowledged, "but as a result it is robust, fast to compute and easy for artists to use, enabling compositing techniques to be used on lower budget shots and productions."

A DuctTake user can combine two videos by making a few quick brush strokes to indicate which parts of the video to keep in each take. An algorithm developed by the Disney Research, Zürich team then computes an optimal seam and merges the two videos together.

DuctTake also includes a number of tools necessary to create a composite that looks realistic, such as adjusting the seam between frames to compensate for camera movement or content movement. Other tools adjust for differences in brightness, contrast and hue between takes, blend images along seams that are visible in a common background, and increase the blurriness in some video to match blurring that occurs in the with which it is being combined.

Explore further: Computerized emotion detector

More information: Paper: www.disneyresearch.com/wp-cont… uploads/DuctTake.pdf
Project: www.disneyresearch.com/project/ducttake/

Related Stories

Disney comes to YouTube

Nov 23, 2011

Disney films were available for rent on YouTube on Wednesday in the latest bid by the Google-owned website to transform into an online stage for the gamut of digital video content.

Researchers demonstrate markerless motion capture

Aug 06, 2012

Conventional motion capture for film and game production involves multiple cameras and actors festooned with markers. A new technique developed by Disney Research, Pittsburgh, has demonstrated how three-dimensional motion ...

YouTube, Disney teaming up

Nov 08, 2011

YouTube and The Walt Disney Co. announced on Monday they are teaming up to produce an original video series and feature "family-friendly" Disney programming on the popular video-sharing site.

Recommended for you

Computerized emotion detector

19 hours ago

Face recognition software measures various parameters in a mug shot, such as the distance between the person's eyes, the height from lip to top of their nose and various other metrics and then compares it with photos of people ...

Cutting the cloud computing carbon cost

Sep 12, 2014

Cloud computing involves displacing data storage and processing from the user's computer on to remote servers. It can provide users with more storage space and computing power that they can then access from anywhere in the ...

Teaching computers the nuances of human conversation

Sep 12, 2014

Computer scientists have successfully developed programs to recognize spoken language, as in automated phone systems that respond to voice prompts and voice-activated assistants like Apple's Siri.

Mapping the connections between diverse sets of data

Sep 12, 2014

What is a map? Most often, it's a visual tool used to demonstrate the relationship between multiple places in geographic space. They're useful because you can look at one and very quickly pick up on the general ...

User comments : 4

Adjust slider to filter visible comments by rank

Display comments: newest first

EyeNStein
1 / 5 (2) May 20, 2013
I give then 5/5 just for calling it "DuctTake" (for sticking two takes together).
This was an evolutionary development once algorithms existed to convert 2D to 3D footage.

In your smart 3D TV these algorithms effectively split a 2D scene into layers, 3D process them and "DuctTake" them back together .(Normally not very well really.)
alfie_null
not rated yet May 21, 2013
Extrapolating on compositing technology: One day, perhaps, a face, a body, movement, and voice, all from different sources. A conundrum, should an award ever be given to the amalgam.
EyeNStein
1 / 5 (2) May 21, 2013
Best director and best special Fx still get awards.
But does it spoil your enjoyment of a Bond movie knowing the action scene with the fantastic exploding Kremlin backdrop was filmed explosions and all on a disused street in Pittsburgh?
EyeNStein
1 / 5 (2) May 21, 2013
In your smart 3D TV these algorithms effectively split a 2D scene into layers, 3D process them and "DuctTake" them back together .(Normally not very well really.)


I guess I earned 1/5 for that. I should have said that once the algorithms to intelligently componentise images are written and proven and 'get out into the wild'. They pop up everywhere from smart camera multi-frame compositing, to 3D smart TV's, to studio special FX. I only meant that after seeing basic 2D to 3D conversion appear in domestic kit, professional high-end versions of this facility, and other complex scene/component manipulations like "DuctTake" were inevitable.