Computer graphics research team to present new tool for sketching faces

July 2, 2018, Association for Computing Machinery
Faceshop consists of a web-based user interface (left) that enables a user to specify a region to be edited. Moreover, the user can draw strokes and color scribbles to guide the editing process. The core of the back end (right) is a deep image completion network that takes the user input and the original image to synthesize the edited result. Credit: Tiziano Protenier

Across popular social media platforms, users are posting countless images every day. On Instagram alone, there are more than 40 billion uploaded photos—a figure that's skyrocketing by 95 million daily. This presents a clear need for intuitive yet robust photo-editing tools that allow the average user to perform advanced editing functions.

And while there is a definite need for interactive image editing with respect to , improved editing tools and systems also remain an important aspect of computer graphics and computer vision. However, there is a lack of tools that feature more complex editing functions for inexperienced users, such as changing the facial expression in a photo.

A research team, led by computer scientists from the University of Bern-Switzerland and University of Maryland-College Park, have devised a sketch-based editing framework that enables a user to edit their photos by "sketching" a few strokes on top of them. Their system, called FaceShop, also offers a copy-paste function, which allows users to edit any part of a photo by copying-and-pasting the portion to be edited from another (better) photo, eliminating the need to hand-draw or sketch anything at all.

The team's approach is built on machine-learning techniques, which, in the end, give users more control over their desired edits in real-time and produce more realistic results.

"Most other approaches rely on more traditional, handcrafted techniques, which impose some limitations. For instance, these systems are either [by design] restricted to limited sets of predefined editing operations, or they are very flexible but hard to use and require experienced users to spend a considerable amount of time to perform rather basic edits," says Tiziano Protenier, lead author of the work and Ph.D. candidate at the University of Bern. "In contrast, our system is very flexible and allows untrained users to perform complex edits within minutes using an intuitive interface."

Faceshop consists of a web-based user interface (left) that enables a user to specify a region to be edited. Moreover, the user can draw strokes and color scribbles to guide the editing process. The core of the back end (right) is a deep image completion network that takes the user input and the original image to synthesize the edited result. Credit: Tiziano Protenier

Protenier developed the new system with collaborators Qiyang Hu, Attila Szabó, Siavash Arjomand Bigdeli and Paolo Favaro from University of Bern, and Matthias Zwicker from University of Maryland. The team will present their work at SIGGRAPH 2018, held 12-16 August in Vancouver, British Columbia. This annual gathering showcases the world's leading professionals, academics, and creative minds at the forefront of computer graphics and interactive techniques.

In their paper, "FaceShop: Deep Sketch-based Face Image Editing," the researchers demonstrate their new system via several examples. In one, a woman's nose is slightly manipulated, and a strand of hair is edited so that it is pushed away from her face, removing a dark shadow that had appeared on just one side of her face in the original photo. Another photo shows how a user is able to enhance a woman's eye make-up and bring out the color of her eyes. The examples highlighted in the paper showcase how the novel system works, intuitively, and produces high-quality, realistic results.

The team's method is based on generative adversarial neural networks (GANs), a form of artificial intelligence (AI) that, in recent years, has attracted a lot of research interest for its ability to generate realistic looking images. These GANs consist of two AIs that fight against each other. The first component tries to distinguish the generated images from genuine images, whereas the second component tries to produce images that fool the other AI. During training, the two components learn from each other, eventually resulting in a system that autonomously learned to produce realistic looking images, without any human judgement in the loop.

Unlike other AI-driven image editing methods, once the system is trained, users are more in control of the edits to their photos. "Our system features an optimum degree of control, which makes it very interesting from an application point of view," notes Portenier. "Other techniques that try to incorporate user control are either limited in the scope of possible edits or have been shown to work well on the data that was used to train the systems, but fail to produce convincing results with real user input. We are presenting a technique that mitigates this issue, resulting in a system that works surprisingly well in practice."

In future work, the researchers intend to explore additional user interaction tools to add to their framework, and consider how to leverage AI for sketch-based editing of videos.

Explore further: Instagram vamps up photo editing tools

More information:

Related Stories

New Facebook AI application can unblink your eyes in a photo

June 26, 2018

Two researchers at Facebook, Brian Dolhansky and Cristian Canton Ferrer, have posted a paper on the social network giant's site detailing a new AI application they are working on. The goal of the app, they report, is to open ...

Colorizing images with deep neural networks

July 25, 2017

For decades, image colorization has enjoyed an enduring interest from the public. Though not without its share of detractors, there is something powerful about this simple act of adding color to black and white imagery, whether ...

NVIDIA researchers raise the bar on image inpainting

April 26, 2018

For those not yet familiar with photo reconstruction tools, the title of this paper on arXiv might be totally puzzling: "Image Inpainting for Irregular Holes Using Partial Convolutions." The research, though, from a NVIDIA ...

Turning photos into an interactive experience

November 27, 2017

Human facial expressions, big and small, have the ability to convey what a person is feeling. Imagine being able to bring out a wide range of human emotions in any still photo, and to do so, automatically.

Recommended for you

Coffee-based colloids for direct solar absorption

March 22, 2019

Solar energy is one of the most promising resources to help reduce fossil fuel consumption and mitigate greenhouse gas emissions to power a sustainable future. Devices presently in use to convert solar energy into thermal ...

EPA adviser is promoting harmful ideas, scientists say

March 22, 2019

The Trump administration's reliance on industry-funded environmental specialists is again coming under fire, this time by researchers who say that Louis Anthony "Tony" Cox Jr., who leads a key Environmental Protection Agency ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.