Researcher sees new angles in visual search

Oct 26, 2011 by Beth Kwon
Engineering professor Shih-Fu Chang is developing technology to make visual search as effortless as typing keywords into a search engine. Image credit: Eileen Barroso/Columbia University

Engineering professor Shih-Fu Chang is trying to make visual search technology as effortless as typing a keyword like “Morningside restaurants” into Google.

In Chang’s ideal world, you could sift through a season’s worth of Major League Baseball games to find every double play or sort through your digital archives to find all the pictures you’ve taken of your kid blowing out her birthday candles.

Chang, the Richard Dicker Professor of Telecommunications, who has a joint appointment in electrical engineering and computer science, develops algorithms to identify and index data, as well as new techniques and software systems to help users manage large amounts of multimedia information. The director of Columbia Engineering’s Digital Video and Multimedia Lab, Chang was recently honored with a lifetime achievement award from the Association for Computing Machinery’s Special Interest Group on Multimedia.

Chang credits his associates for the prestigious award. “It really should be attributed to all the wonderful students and collaborators I have had the good fortune to work with during my career,” he said.

A native of Taiwan, he was fascinated by technology from a young age, he remembers getting his first computer, an Apple II, as a high school student in 1981 and envying classmates who were fortunate enough to have an Atari computer system.

“I grew up in a time when the information technology industry was rapidly expanding, and it caught the imagination of young people,” Chang recalls. “My parents didn’t know anything about computers, but we as students caught the wave and jumped on it.” He received his Ph.D. from the University of California, Berkeley, and joined Columbia in 1993.

In 1998, he developed one of the first video systems, VideoQ, and in the 1990s he pioneered search by sketch technology, in which a user can draw what he or she is looking for.

His work has been broadly funded by government and industry, including Eastman Kodak, and many video indexing technologies developed by his group have been licensed to companies. With the support of the National Science Foundation, he is working on technology to determine whether images and videos have been tampered with.

Recently, he’s been working on technology that lets users adjust the importance of multiple characteristics in a search. If you’re looking for an image of a sun setting over a mountain near a body of water, for instance, you would enter the search words “sunset,” “mountain” and “water” into a grid next to a database of images. Then you could home in on a specific feature—the mountain, for instance—by using your mouse to nudge the cursor closer to that word. All the images in the database would be indexed automatically without time-consuming human tagging.

Other research projects let users upload a specific image they want to match, where a photo can be matched with images all over the Internet. Chang is working on refining technology that allows people to specify the exact portion of an image to search for—a certain sculpture or building, for example—so that searching is more accurate and efficient, even on mobile devices like iPhones. His research could also be applied in the medical field, allowing technicians to find a moment in an ultrasound film that could help make a diagnosis.

Another aspect of his research involves using the human brain as a search tool. In collaboration with biomedical engineering professor Paul Sajda, Chang designed a device that monitors brain activity as a subject looks at pictures. First, an EEG machine records the “aha” moments of recognition, then a computer analyzes the recognition patterns to identify similarities in other photographs in large databases that elicited the same strong reaction.

“The machine does what it’s best at, and the human does what he or she is best at in the most natural way,” Chang says.

Explore further: System to automatically find a common type of programming bug significantly outperforms its predecessors

add to favorites email to friend print save as pdf

Related Stories

Professor develops mobile app to identify plant species

Jun 08, 2011

( -- Not every child can dream up a smartphone application and see it come to life. But that’s what happened when 8-year-old William Belhumeur suggested his father make an app that identifies ...

For software developers, more speed and mobility

Dec 14, 2010

Across the globe, technology and innovation are becoming increasingly more reliant on mobility and accessibility. For software developers working on highly complex projects, that means being able to save their ...

Understanding the social side of cyber-security issues

May 04, 2011

When Engin Kirda started focusing on cyber-security research 10 years ago, those primarily responsible for launching Internet attacks were teenagers out for kicks, he said. But the scope of threats existing ...

Recommended for you

Blue Freedom uses power of flowing water to charge

3 hours ago

Good friends may decide to tell you something that is not true but nonetheless sustaining: Nothing is impossible. That was the case of Blue Freedom co-founder who asked his friend if it would be possible ...

Special ops troops using flawed intel software

3 hours ago

Special operations troops heading to war zones are asking for commercial intelligence analysis software they say will help their missions. But their requests are languishing, and they are being ordered to use a flawed, in-house ...

Applications of optical fibre for sensors

8 hours ago

Mikel Bravo-Acha's PhD thesis has focused on the applications of optical fibre as a sensor. In the course of his research, conducted at the NUP/UPNA-Public University of Navarre, he monitored a sensor fitted to optical fibre ...

Twitter chief vows to help Indonesia fight disasters

8 hours ago

Twitter chief Dick Costolo said Thursday the microblogging site planned to work with Indonesian authorities to warn people about natural disasters that regularly hit the archipelago, from earthquakes to volcanic ...

Virtual robotization for human limbs

8 hours ago

Recent advances in computer gaming technology allow for an increasingly immersive gaming experience. Gesture input devices, for example, synchronise a player's actions with the character on the screen. Entertainment ...

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.