Harvard group takes complexity out of video face replacement (w/ video)

The tool claims to be able to replace faces using only single-camera video with minimal user input. The software can render a ten-second video in about twenty minutes. The software is promoted as a system that can be used by amateur and budget-wary film makers.

Kevin Dale, part of the school’s Graphics, Vision and Interaction (GVI) Group, is co-author of a paper, “Video Face Replacement,” which discusses the approach. The facial replacement method used, according to the authors, requires no substantial manual operations or complex hardware, only single-camera video.

This video is not supported by your browser at this time.

For tracking facial performance, a 3-D multilinear model is used in both videos. With the corresponding 3-D geometry, the source is warped to the target face and the source is retimed to march the target performance. “We then compute an optimal seam through the video volume that maintains temporal consistency in the final composite,” according to the team. They note that the results are difficult to distinguish from real video footage.

The software has been met by favorable comments that it can be useful, though not a high end tool to compete with tools and techniques at major studios. Quoted in New Scientist, computer graphics researcher Paul Debevec sees Dale’s work as a potential YouTube plug in or just generally an easy to use tool.

While easy to use, the difficulty may arise in questions about technology privileges of fair use and abuse, as with many controversial software tools that make use of people’s faces with and without their explicit permission. Gizmodo Australia opines,"it could open a whole new world of piracy issues when even an actor’s face and performance are used without their permission.”

The face-swapping tool, meanwhile, is just one extension of goals for image and video compositing at the Harvard GVI group.

We are likely to hear more from them. “Merging images and videos to create high-quality composites is a very difficult problem, and even professional artists using sophisticated can take many hours of work to create results that are photo-realistic,” says the group statement. They want to deliver tools that make compositing easier by automating most of the process.

The group says its work involves developing algorithms that analyze and match the visual appearance of objects in images--color, contrast, noise, texture, and blur. They fundamentally want to make the creation of composites from diverse images easy.

Similar interest was sparked in September, when Arturo Castro and Kyle McDonald demonstrated a technique called Real-Time Face Substitution. The software made use of the open source platform openFrameworks, which Castro helped to drive along with Zachary Lieberman and Theodore Watson.

More information: http://gvi.seas.ha … edu/node/318

© 2011 PhysOrg.com