Researchers develop fast, economical method for high-definition video compositing

May 20, 2013

Video compositing to create special effects, replace backgrounds or combine multiple takes of an actor's performance is an integral, but highly labor-intensive, part of modern film making. Researchers at Disney Research, Zürich, however, have found an innovative way to create these composite videos that is simple, fast, and easy to use.

Rather than perform a painstaking of that are to be added or subtracted from a video, the Disney system, called DuctTake, uses to find a spatiotemporal "seam" through the video frame that enables two or more videos to be joined together.

These seams can be highly irregular, following the contours of people, furniture and other objects that are common in each take of the scene. Because it can only combine scenes that have overlapping content, DuctTake isn't useful for combining arbitrary . But it works like a charm when combining multiple takes of the same shot.

The Disney Research, Zürich researchers showed the technique can be used across a wide range of video composites. For example, it can combine the best performances by actors from several different takes into a single seamless output, reducing the number of on-set takes that are required to be filmed. Furthermore, the same technique can be used to make a cut seamless, allowing the first half of one take to be combined into a second take. In another case, the researchers eliminated unwanted cars from a street scene by offsetting the vehicle with empty street from the same video take. Hard-to-shoot scenes involving animals, they demonstrated, become a bit easier when an animal trainer can be easily removed from the shot, or other actors can be added at the moment of time when the animal does the right thing.

"The most delicate component is alignment," said Oliver Wang, post doctoral researcher at Disney Research, Zürich. "But given properly aligned views, we can almost always generate good composites with minimal work."

The findings by Wang and his Disney Research, Zürich team of Jan Rüegg, Aljoscha Smolic and Markus Gross were presented at Eurographics 2013, the European Association for Computer Graphics conference.

Most video compositing is accomplished now by the digital equivalent of "cut-and-paste." Rotoscoping is the process by which elements can be added by drawing segmentation outlines. Chroma-keying, familiar to viewers of TV weathercasts in which news announcers appear to stand in front of large, animated maps, separates actors from backgrounds based on color hues; it's cheap and robust, but restricts filming to studio environments, and can require challenging color balancing in post-production.

"Our approach solves a simpler problem," Gross acknowledged, "but as a result it is robust, fast to compute and easy for artists to use, enabling compositing techniques to be used on lower budget shots and productions."

A DuctTake user can combine two videos by making a few quick brush strokes to indicate which parts of the video to keep in each take. An algorithm developed by the Disney Research, Zürich team then computes an optimal seam and merges the two videos together.

DuctTake also includes a number of tools necessary to create a composite that looks realistic, such as adjusting the seam between frames to compensate for camera movement or content movement. Other tools adjust for differences in brightness, contrast and hue between takes, blend images along seams that are visible in a common background, and increase the blurriness in some video to match blurring that occurs in the with which it is being combined.

Explore further: Disney teams with EA on 'Star Wars' video games

More information: Paper: … uploads/DuctTake.pdf

Related Stories

Disney comes to YouTube

November 23, 2011

Disney films were available for rent on YouTube on Wednesday in the latest bid by the Google-owned website to transform into an online stage for the gamut of digital video content.

Researchers demonstrate markerless motion capture

August 6, 2012

Conventional motion capture for film and game production involves multiple cameras and actors festooned with markers. A new technique developed by Disney Research, Pittsburgh, has demonstrated how three-dimensional motion ...

YouTube, Disney teaming up

November 8, 2011

YouTube and The Walt Disney Co. announced on Monday they are teaming up to produce an original video series and feature "family-friendly" Disney programming on the popular video-sharing site.

Recommended for you

Google to serve next version of Android as 'Oreo"

August 22, 2017

An upcoming update to Google's Android software finally has a delectable name. The next version will be known as Oreo, extending Google's tradition of naming each version after a sweet treat.

Forget oil, Russia goes crazy for cryptocurrency

August 16, 2017

Standing in a warehouse in a Moscow suburb, Dmitry Marinichev tries to speak over the deafening hum of hundreds of computers stacked on shelves hard at work mining for crypto money.

Researchers clarify mystery about proposed battery material

August 15, 2017

Battery researchers agree that one of the most promising possibilities for future battery technology is the lithium-air (or lithium-oxygen) battery, which could provide three times as much power for a given weight as today's ...

Signs of distracted driving—pounding heart, sweaty nose

August 15, 2017

Distracted driving—texting or absent-mindedness—claims thousands of lives a year. Researchers from the University of Houston and the Texas A&M Transportation Institute have produced an extensive dataset examining how ...


Adjust slider to filter visible comments by rank

Display comments: newest first

1 / 5 (2) May 20, 2013
I give then 5/5 just for calling it "DuctTake" (for sticking two takes together).
This was an evolutionary development once algorithms existed to convert 2D to 3D footage.

In your smart 3D TV these algorithms effectively split a 2D scene into layers, 3D process them and "DuctTake" them back together .(Normally not very well really.)
not rated yet May 21, 2013
Extrapolating on compositing technology: One day, perhaps, a face, a body, movement, and voice, all from different sources. A conundrum, should an award ever be given to the amalgam.
1 / 5 (2) May 21, 2013
Best director and best special Fx still get awards.
But does it spoil your enjoyment of a Bond movie knowing the action scene with the fantastic exploding Kremlin backdrop was filmed explosions and all on a disused street in Pittsburgh?
1 / 5 (2) May 21, 2013
In your smart 3D TV these algorithms effectively split a 2D scene into layers, 3D process them and "DuctTake" them back together .(Normally not very well really.)

I guess I earned 1/5 for that. I should have said that once the algorithms to intelligently componentise images are written and proven and 'get out into the wild'. They pop up everywhere from smart camera multi-frame compositing, to 3D smart TV's, to studio special FX. I only meant that after seeing basic 2D to 3D conversion appear in domestic kit, professional high-end versions of this facility, and other complex scene/component manipulations like "DuctTake" were inevitable.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.