Colorizing images with deep neural networks

July 25, 2017, Association for Computing Machinery
The proposed system uses AI to colorize a grayscale image (left), guided by user color 'hints' (second), providing the capability for quickly generating multiple plausible colorizations (middle to right). Credit: Photograph of Migrant Mother by Dorothea Lange, 1936 (Library of Congress, Prints & Photographs Division, FSA/OWI Collection, reproduction number: LC-USF34-9058-C.).

For decades, image colorization has enjoyed an enduring interest from the public. Though not without its share of detractors, there is something powerful about this simple act of adding color to black and white imagery, whether it be a way of bridging memories between the generations, or expressing artistic creativity. However, the process of manually adding color can be very time consuming and require expertise, with typical professional processes taking hours or days per image to perfect. A team of researchers has proposed a new technique to leverage deep networks and AI, which allows novices, even those with limited artistic ability, to quickly produce reasonable results.

The research, entitled "Real-Time User Guided Colorization with Learned Deep Priors," is authored by a team at UC Berkeley led by Alexei A. Efros, Professor of Electrical Engineering and Computer Sciences. They will present their work at SIGGRAPH 2017, which spotlights the most innovative in computer graphics research and interactive techniques worldwide. The annual conference will be held in Los Angeles, 30 July to 3 August.

In prior work, the team trained a deep on big visual data (a million images) to automatically colorize grayscale images, with no user intervention. While the results were sometimes very good, it was prone to certain artifacts. One major limitation was that the color of many objects—for example, shirts—may be inherently ambiguous. The could only ultimately decide on one possibility.

"The goal of our previous project was to just get a single, plausible colorization," says Richard Zhang, a coauthor and PhD candidate, advised by Professor Efros. "If the user didn't like the result, or wanted to change something, they were out of luck. We realized that empowering the user and adding them in the loop was actually a necessary component for obtaining desirable results."

The new network is trained on a grayscale image, along with simulated user inputs. The system improves upon previous automatic colorization systems by enabling the user, in real-time, to correct and customize the colorization. The user provides guidance by adding colored points, or "hints", which the system then propagates to the rest of the image. The network also learns common colors for different objects and makes appropriate recommendations to the user. Though the new system is only trained on natural images—for instance, elephants are typically brown or gray—the system is also happy to follow the user's whims, enabling out-of-the-box coloring. For example, a pink elephant—though unnatural—is not off limits.

To evaluate the system, the researchers tested their interface on novice users, challenging them to produce a realistic colorization of a randomly selected grayscale image. Even with minimal training and limited time—just one minute per image—these users quickly learned how to produce colorizations that often fooled real human judges in a real vs. fake test scenario.

Deep networks are being more heavily used in graphics. Perhaps after conquering remaining challenges, such as streamlining memory usage and hardware requirements, along with integrating with existing image editing tools, a system like this one could find its way into commercial tools for image manipulation.

Explore further: Information processing: Adding a touch of color

More information: For the full paper, visit . Download the software at .

Related Stories

Information processing: Adding a touch of color

June 8, 2012

Creating a high-quality realistic color image from a grayscale picture can be challenging. Conventional methods typically require the user’s input, either by using a scribbling tool to color the image manually or by ...

Novel, computer-assisted method for colorization

March 8, 2005

A novel, computer-assisted method for colorizing black and white images and movies has been developed by researchers at the Hebrew University of Jerusalem's Benin School of Computer Science and Engineering. The method is ...

Team accelerates rendering with AI

June 30, 2017

Modern films and TV shows are filled with spectacular, computer-generated sequences which are computed by rendering systems that simulate the flow of light in a 3D scene. However, computing many light rays is an immensely ...

Recommended for you

Galactic center visualization delivers star power

March 21, 2019

Want to take a trip to the center of the Milky Way? Check out a new immersive, ultra-high-definition visualization. This 360-movie offers an unparalleled opportunity to look around the center of the galaxy, from the vantage ...

Ultra-sharp images make old stars look absolutely marvelous

March 21, 2019

Using high-resolution adaptive optics imaging from the Gemini Observatory, astronomers have uncovered one of the oldest star clusters in the Milky Way Galaxy. The remarkably sharp image looks back into the early history of ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.