Researchers teach computers how to name images by 'thinking'

November 1, 2006

Penn State researchers have "taught" computers how to interpret images using a vocabulary of up to 330 English words, so that a computer can describe a photograph of two polo players, for instance, as "sport," "people," "horse," "polo."

The new system, which can automatically annotate entire online collections of photographs as they are uploaded, means significant time-savings for the millions of Internet users who now manually tag or identify their images. It also facilitates retrieval of images through the use of search terms, said James Wang, associate professor in the Penn State College of Information Sciences and Technology, and one of the technology's two inventors.

The system is described in a paper, "Real-Time Computerized Annotation of Pictures," given at the recent ACM Multimedia 2006 conference in Santa Barbara, Calif., and authored by Jia Li, associate professor, Department of Statistics, and Wang. Penn State has filed a provisional patent application on the invention. Major search engines currently rely upon uploaded tags of text to describe images. While many collections are annotated, many are not. The result: Images without text tags are not accessible to Web searchers. Because it provides text tags, the ALIPR system-Automatic Linguistic Indexing of Pictures-Real Time-makes those images visible to Web users.

ALIPR does this by analyzing the pixel content of images and comparing that against a stored knowledge base of the pixel content of tens of thousands of image examples. The computer then suggests a list of 15 possible annotations or words for the image.

"By inputting tens of thousands of images, we have trained computers to recognize certain objects and concepts and automatically annotate those new or unseen images," Wang said. "More than half the time, the computer's first tag out of the top 15 tags is correct."

In addition, for 98 percent of images tested, the system has provided at least one correct annotation in the top 15 selected words. The system, which completes the annotation in about 1.4 seconds, also can be applied to other domains such as art collections, satellite imaging and pathology slides, Wang said. The new system builds on the authors' previous invention, ALIP, which also analyzes image content. But unlike ALIP which characterized images by incorporating computational-intensive spatial modeling, ALIPR characterizes images by modeling distributions of color and texture.

The researchers acknowledge computers trained with their algorithms have difficulties when photos are fuzzy or have low contrast or resolution; when objects are shown only partially; and when the angle used by the photographer presents an image in a way that is different than how the computer was trained on the object. Adding more training images as well as improving the training process may reduce these limitations-future areas of research.

Source: Penn State

Explore further: Researchers trained neural networks to be fashion designers (sort of)

Related Stories

Wireless handheld spectrometer transmits data to smartphone

November 8, 2017

Spectral images, which contain more color information than is obtainable with a typical camera, reveal characteristics of tissue and other biological samples that can't be seen by the naked eye. A new smartphone-compatible ...

Why head and face pain causes more suffering

November 13, 2017

Hate headaches? The distress you feel is not all in your—well, head. People consistently rate pain of the head, face, eyeballs, ears and teeth as more disruptive, and more emotionally draining, than pain elsewhere in the ...

World's smallest tape recorder is built from microbes

November 23, 2017

Through a few clever molecular hacks, researchers at Columbia University Medical Center have converted a natural bacterial immune system into a microscopic data recorder, laying the groundwork for a new class of technologies ...

Researchers transform tomatoes into fluorescent carbon dots

November 28, 2017

(Phys.org)—Researchers have shown that tomato pulp dissolved in water can eventually be turned into a powder of nanoparticles containing carbon dots with diameters of less than 5 nm. Like all carbon dots, one of the main ...

Recommended for you

Tech titans ramp up tools to win over children

December 10, 2017

From smartphone messaging tailored for tikes to computers for classrooms, technology titans are weaving their way into childhoods to form lifelong bonds, raising hackles of advocacy groups.

Mapping out a biorobotic future  

December 8, 2017

You might not think a research area as detailed, technically advanced and futuristic as building robots with living materials would need help getting organized, but that's precisely what Vickie Webster-Wood and a team from ...

Lyft puts driverless cars to work in Boston

December 6, 2017

Lyft on Wednesday began rolling out self-driving cars with users of the smartphone-summoned ride service in Boston in a project with technology partner nuTonomy.

Researchers 3-D print lifelike artificial organ models

December 6, 2017

A team of researchers led by the University of Minnesota has 3D printed lifelike artificial organ models that mimic the exact anatomical structure, mechanical properties, and look and feel of real organs. These patient-specific ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.