Harvard group takes complexity out of video face replacement (w/ video)

December 5, 2011 by Nancy Owano, Phys.org report

The method for face replacement requires only single-camera video of the source (a) and target (b) subject, which allows for simple acquisition and reuse of existing footage. It tracks both performances with a multilinear morphable model then spatially and temporally align the source face to the target footage (c). Researchrs then compute an optimal seam for gradient domain compositing that minimizes bleeding and flickering in the final result (d). Image credit: Kevin Dale / Harvard University
(PhysOrg.com) -- From Facebook to YouTube to on the fly film projects, the presentation of content that entertains or instructs or both draws on visual tools, ranging from simple to complex. Novice as well as expert creatives are being increasingly equipped with technologies to help them make something creative. Out to prove that point even further, a computer scientist from Harvard University’s School of Engineering and Applied Sciences and colleagues have come up with face-transplant software.

The tool claims to be able to replace faces using only single-camera video with minimal user input. The software can render a ten-second video in about twenty minutes. The software is promoted as a system that can be used by amateur and budget-wary film makers.

Kevin Dale, part of the school’s Graphics, Vision and Interaction (GVI) Group, is co-author of a paper, “Video Face Replacement,” which discusses the approach. The facial replacement method used, according to the authors, requires no substantial manual operations or complex hardware, only single-camera video.

For tracking facial performance, a 3-D multilinear model is used in both videos. With the corresponding 3-D geometry, the source is warped to the target face and the source is retimed to march the target performance. “We then compute an optimal seam through the video volume that maintains temporal consistency in the final composite,” according to the team. They note that the results are difficult to distinguish from real video footage.

The software has been met by favorable comments that it can be useful, though not a high end tool to compete with tools and techniques at major studios. Quoted in New Scientist, computer graphics researcher Paul Debevec sees Dale’s work as a potential YouTube plug in or just generally an easy to use tool.

While easy to use, the difficulty may arise in questions about technology privileges of fair use and abuse, as with many controversial software tools that make use of people’s faces with and without their explicit permission. Gizmodo Australia opines,"it could open a whole new world of piracy issues when even an actor’s face and performance are used without their permission.”

The face-swapping tool, meanwhile, is just one extension of goals for image and compositing at the Harvard GVI group.

We are likely to hear more from them. “Merging images and videos to create high-quality composites is a very difficult problem, and even professional artists using sophisticated can take many hours of work to create results that are photo-realistic,” says the group statement. They want to deliver tools that make compositing easier by automating most of the process.

The group says its work involves developing algorithms that analyze and match the visual appearance of objects in images--color, contrast, noise, texture, and blur. They fundamentally want to make the creation of composites from diverse images easy.

Similar interest was sparked in September, when Arturo Castro and Kyle McDonald demonstrated a technique called Real-Time Face Substitution. The software made use of the open source platform openFrameworks, which Castro helped to drive along with Zachary Lieberman and Theodore Watson.

Explore further: Software developer shows face-swapping in realtime (w/ video)

More information: gvi.seas.harvard.edu/node/318

Related Stories

Software developer shows face-swapping in realtime (w/ video)

September 23, 2011

(PhysOrg.com) -- Software developer Arturo Castro and media artist Kyle MacDonald have put out a video demo of their software that replaces their faces with other faces in realtime. Their face-swapping, face-morphing technology ...

A new window to the face

August 9, 2011

The human face is a complicated thing—powered by 52 muscles; contoured by the nose, eyebrows, and other features; and capable of an almost infinite range of expressions, from joy to anger to sorrow to puzzlement.

YouTube adds video editing tool

September 15, 2011

YouTube on Wednesday added an editing tool that lets creators of videos make changes to snippets after they have been uploaded to the popular website.

Faster computer graphics

June 13, 2011

Photographs of moving objects are almost always a little blurry — or a lot blurry, if the objects are moving rapidly enough. To make their work look as much like conventional film as possible, game and movie animators ...

A 360 degree camera that sees in 3D (w/ Video)

December 1, 2010

Surround sight has come to the camera. Inspired by the eye of a fly, EPFL scientists have invented a camera that can take pictures and film in 360° and reconstruct the images in 3D.

Researcher sees new angles in visual search

October 26, 2011

Engineering professor Shih-Fu Chang is trying to make visual search technology as effortless as typing a keyword like “Morningside restaurants” into Google.

Recommended for you

Technology near for real-time TV political fact checks

January 18, 2019

A Duke University team expects to have a product available for election year that will allow television networks to offer real-time fact checks onscreen when a politician makes a questionable claim during a speech or debate.

Privacy becomes a selling point at tech show

January 7, 2019

Apple is not among the exhibitors at the 2019 Consumer Electronics Show, but that didn't prevent the iPhone maker from sending a message to attendees on a large billboard.

China's Huawei unveils chip for global big data market

January 7, 2019

Huawei Technologies Ltd. showed off a new processor chip for data centers and cloud computing Monday, expanding into new and growing markets despite Western warnings the company might be a security risk.


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Dec 05, 2011
This is the beginning of a very very special technology - but the aligned faces doesn't look anything like either one. I guess it could be used by a one man film studio to generate diversity, for one.
not rated yet Dec 05, 2011
Good enough for a beginning. Think of the deceptions possible if ever it is perfected!
not rated yet Dec 06, 2011
When they perfect the ability to manipulate the ears then I will be impressed.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.