New method captures facial details at high fidelity and real time

August 5, 2015, Disney Research

Forehead wrinkles that rapidly deepen and crow's feet that appear suddenly around the eyes might distress the average person, but the ability to quickly and realistically incorporate such details in a facial reconstruction is the key feature of a new performance capture method developed at Disney Research.

The method, which requires only a single such as a webcam, is the first to both operate in real-time and to capture facial features in , including such details as wrinkles.

"This could open up a variety of new applications, from casual home users who wish to give their own likeness to a game character to commercial applications where, for example, a customer can apply and view virtual makeup," said Derek Bradley, research scientist at Disney Research. "In film production, our method would allow a director to assess an actor's performance transferred to a digital creature even as he is filming."

In addition to Bradley, the research team included Chen Cao and Kun Zhou of Zhejiang University as well as Thabo Beeler of Disney Research. The researchers will present their facial performance capture method at ACM SIGGRAPH 2015, the International Conference on Computer Graphics and Interactive Techniques, in Los Angeles Aug. 9-13.

Capturing facial details has proven critical for conveying emotions or the state of mind of a character and audiences are sensitive to subtle characteristics of the face. Animators thus devote great care to tracking facial performances of actors so that they can create realistic virtual characters. Until now, these facial performance capture methods have been designed to either capture these details in high resolution offline, or to record less detail but at high speed.

The new method represents a substantial step towards the goal of attaining both high quality and high speed.

As with other real-time facial capture methods, the researchers begin by using generic, low-resolution face models as the basis for reconstruction. A face model is virtually fitted to the user by the system as the user changes expressions and contorts his or her face in front of a video camera.

The system then automatically calculates a personalized wrinkle probability map, based on the set of extreme expressions the user assumed to train the program. The wrinkles may not match the user's actual wrinkles exactly, but they exhibit a very similar appearance because of the way they are formed.

"By learning the relationship between local image appearance and local wrinkle formation, we can reconstruct plausible face wrinkles in real-time from a single camera," Beeler noted. Once trained, the online capture approach can be applied to any new user without additional training. The system is generic and can be used with most common types of real-time face trackers.

Explore further: Wrinkles and all: Hi-res eyelid reconstruction makes digital doubles look more realistic

More information: … eperformancecapture/

Related Stories

Creating an avatar from a 3-D selfie

August 5, 2015

Generating a 3D duplicate of someone without the aid of a Hollywood studio: this is the challenge taken up by EPFL researchers, who have successfully condensed an expensive and complex process to use only a smartphone camera.

A new window to the face

August 9, 2011

The human face is a complicated thing—powered by 52 muscles; contoured by the nose, eyebrows, and other features; and capable of an almost infinite range of expressions, from joy to anger to sorrow to puzzlement.

Recommended for you

Coffee-based colloids for direct solar absorption

March 22, 2019

Solar energy is one of the most promising resources to help reduce fossil fuel consumption and mitigate greenhouse gas emissions to power a sustainable future. Devices presently in use to convert solar energy into thermal ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.