Researchers Teach Computers to Search for Photos Based on Their Contents

Oct 08, 2008
ALIPR assigned the following keywords to this photo of Biscayne Bay in Miami, Florida: landscape, lake, mountain, ocean, building, grass, water, ice, glacier, historical, house, rock, man-made, train, and tree. Credit: Penn State

A pair of Penn State researchers has developed a statistical approach, called Automatic Linguistic Indexing of Pictures in Real-Time (ALIPR), that one day could make it easier to search the Internet for photographs. The public can participate in improving ALIPR's accuracy by visiting a designated Web site (http://www.alipr.com), uploading photographs, and evaluating whether the keywords that ALIPR uses to describe the photographs are appropriate.

ALIPR works by teaching computers to recognize the contents of photographs, such as buildings, people, or landscapes, rather than by searching for keywords in the surrounding text, as is done with most current image-retrieval systems. The team recently received a patent for an earlier version of the approach, called ALIP, and is in the process of obtaining another patent for the more sophisticated ALIPR. They hope that eventually ALIPR can be used in industry for automatic tagging or as part of Internet search engines.

"Our basic approach is to take a large number of photos -- we started with 60,000 photos -- and to manually tag them with a variety of keywords that describe their contents. For example, we might select 100 photos of national parks and tag them with the following keywords: national park, landscape, and tree," said Jia Li, an associate professor of statistics at Penn State. "We then would build a statistical model to teach the computer to recognize patterns in color and texture among these 100 photos and to assign our keywords to new photos that seem to contain national parks, landscapes, and/or trees. Eventually, we hope to reverse the process so that a person can use the keywords to search the Web for relevant images."

Li said that most current image-retrieval systems search for keywords in the text associated with the photo or in the name that was given to the photo. This technique, however, often misses appropriate photos and retrieves inappropriate photos. Li's new technique allows her to train computers to recognize the semantics of images based on pixel information alone.

Li, who developed ALIPR with her colleague James Wang, a Penn State associate professor of information sciences and technology, said that their approach appropriately assigns to photos at least one keyword among seven possible keywords about 90 percent of the time. But, she added, the accuracy rate really depends on the evaluator. "It depends on how specific the evaluator expects the approach to be," she said. "For example, ALIPR often distinguishes people from animals, but rarely distinguishes children from adults."

Although the team's goal is to improve ALIPR's accuracy, Li said she does not believe the approach ever will be 100-percent accurate. "There are so many images out there and so many variations on the images' contents that I don't think it will be possible for ALIPR to be 100-percent accurate," she said. "ALIPR works by recognizing patterns in color and texture. For example, if a cat in a photo is wearing a red coat, the red coat may lead ALIPR to tag the photo with words that are irrelevant to the cat. There is just too much variability out there." Li currently is pursuing some new ideas that may help her to achieve better recognition of image semantics.

Provided by Penn State University

Explore further: A social-network illusion that makes things appear more popular than they are

Related Stories

Research shows 'mulch fungus' causes turfgrass disease

4 hours ago

Inadvertently continuing a line of study they conducted about 15 years ago, a team of Penn State researchers recently discovered the causal agent for an emerging turfgrass disease affecting golf courses around ...

Study on pesticides in lab rat feed causes a stir

5 hours ago

French scientists published evidence Thursday of pesticide contamination of lab rat feed which they said discredited historic toxicity studies, though commentators questioned the analysis.

Experiments open window on landscape formation

6 hours ago

University of Oregon geologists have seen ridges and valleys form in real time and—even though the work was a fast-forwarded operation done in a laboratory setting—they now have an idea of how climate ...

To conduct, or to insulate? That is the question

6 hours ago

A new study has discovered mysterious behaviour of a material that acts like an insulator in certain measurements, but simultaneously acts like a conductor in others. In an insulator, electrons are largely stuck in one place, ...

Recommended for you

EU open source software project receives green light

Jul 01, 2015

An open source software project involving the University of Southampton to extend the capacity of computational mathematics and interactive computing environments has received over seven million euros in EU funding.

Can computers be creative?

Jul 01, 2015

The EU-funded 'What-if Machine' (WHIM) project not only generates fictional storylines but also judges their potential usefulness and appeal. It represents a major advance in the field of computational creativity.

Algorithm detects nudity in images, offers demo page

Jul 01, 2015

An algorithm has been designed to tell if somebody in a color photo is naked. Isitnude.com launched earlier this month; its demo page invites you to try it out to test its power in nudity detection. You ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.