Defusing photobombs—researchers find ways to remove distractions from photos

September 11, 2015 by John Sullivan, Princeton University
Researchers including members of the Princeton computer science department have developed a program that automatically removes distracting elements from photographs. Although they said the technique needs further work before it could be ready for general use, the researchers demonstrated its use by removing distracting elements from the photos shown here. Credit: Ohad Fried and Adam Finkelstein, Department of Computer Science, Princeton University; and Eli Shechtman and Dan Goldman, Adobe Research

Researchers have found an automated way to identify and eliminate those stray soda cans, roaming cars and photobombing strangers that can send favorite photos to the recycle bin.

Called distractors, they are elements that take away from the central focus of the photo.

"Clutter," said Ohad Fried, a graduate student in at Princeton University and the lead author of the paper on this research. "These are things in the photo that, if they were removed, would make the photo better."

In an article delivered at the 2015 Computer Vision and Pattern Recognition Conference in Boston this summer, a team from Princeton and the software company Adobe presented an early version of a one-click system to remove distracting elements from photos. Currently, Fried said, many programs such as Photoshop allow users the ability to manually define and remove distracting elements. But that typically takes time and requires a degree of skill that casual users often do not have.

"For most casual photographers, this effort is neither possible nor warranted," the researchers wrote. "Last year Facebook reported that people were uploading photos at an average rate of 4,000 images per second. The overwhelming majority of these pictures are casual—they effectively chronicle a moment without much work on the part of the photographer."

The researchers distinguished distractors from more general flaws in photos such as tone or lighting. They noted that photo editing programs already handle global problems quite well, but identifying distractors is more difficult because it is a question of content selection rather than changing a universal aspect of a photo such as color saturation.

"I find this work intriguing because detecting distracting elements in an image is in fact a semantic analysis, a high-level analysis of a scene," said Daniel Cohen-Or, a professor of computer science at Tel Aviv University who was not involved in the research.

To identify elements that people find distracting in photos, the researchers used two data sets of images. They then turned to Amazon's Mechanical Turk system (which enlists large numbers of people to perform set tasks such as reviewing photographs) to go through the first data set of 1,073 images, and identify distracting elements.

"The nice part about this set is that we have multiple entries per image," said Fried, who works in the lab of Adam Finkelstein, a professor of computer science at Princeton and a co-author of the paper. "The problem is that these are people who don't necessarily care about the images."

For the second set—over 5,000 images—the researchers used an iPhone app called Fixel to note changes to images that photographers had retouched and exported, shared or saved. (Fixel, an experimental app developed at Adobe, asked users to check a box to allow photos to be used for research; the researchers said about 25 percent of users did so.) The set was the mirror of the first—images had only one entry, but the users cared enough about the shots to take the time to edit them.

The researchers used the data to create detectors for a wide range of distracting elements.

"We have a specific car detector in the code because people often want to eliminate cars that wander into the frame," Fried said. "We have a face detector. If the face is large and in the center of the photo, we probably don't want to remove it. But if it is coming in from the side, it might be a photobomb."

Besides creating task-specific detectors, the researchers also created a weighting system that assigned values for different arrangements of colors and shapes in . They then created programs to train the software to identify and eliminate elements with the characteristics of distractors.

The researchers said the results were promising but definitely left room for improvement. Many elements in the testing dataset were successfully removed. But in some cases critical elements of the photo were identified as distractions. In others, particularly with more complex photo elements, the distractors were correctly identified but the program was unable to properly fill the removed area to match the photo. The researchers said each of the failures suggested new methods for improvement.

"In the future, it might become accurate enough to be used in a photo editing product," Fried said. "Right now, it is a very interesting research question."

Explore further: Software can automatically critique composition of digital photographs

More information: "Finding Distractors In Images." … d2015distractors.pdf

Related Stories

New website can identify birds using photos

June 4, 2015

In a breakthrough for computer vision and for bird watching, researchers and bird enthusiasts have enabled computers to achieve a task that stumps most humans—identifying hundreds of bird species pictured in photos.

Recommended for you

Technology near for real-time TV political fact checks

January 18, 2019

A Duke University team expects to have a product available for election year that will allow television networks to offer real-time fact checks onscreen when a politician makes a questionable claim during a speech or debate.

Privacy becomes a selling point at tech show

January 7, 2019

Apple is not among the exhibitors at the 2019 Consumer Electronics Show, but that didn't prevent the iPhone maker from sending a message to attendees on a large billboard.

China's Huawei unveils chip for global big data market

January 7, 2019

Huawei Technologies Ltd. showed off a new processor chip for data centers and cloud computing Monday, expanding into new and growing markets despite Western warnings the company might be a security risk.


Adjust slider to filter visible comments by rank

Display comments: newest first

Sep 11, 2015
This comment has been removed by a moderator.
5 / 5 (2) Sep 11, 2015
In case you're wondering, the second pair of photos, the landscape, has been cut off above the distraction. IOW, there is no difference between them.

Actually if you click on the image you'll find they removed a sign in front of the tree at the bottom of the photo (which can't be seen until you enlarge the image)

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.