Sketch-based query for searching for relationships among objects in images

August 11, 2016, King Abdullah University of Science and Technology
Technology search for relationships
Credit: Wikimedia

Searching for specific images may become easier thanks to a new tool that generates image queries based on a sketch or description of objects in spatial relationships. The tool, which has been proposed by researchers from King Abdullah University of Science and Technology (KAUST), Saudi Arabia, and University College London, makes it easier to search the world's ever-expanding databases for pictures matching a wider and more powerful range of image queries.

The enormous collections of photographs and pictures now available in online databases represent a remarkable resource for research and creative arts. As unfathomably rich as these databases might be, they are only as useful as a user's capacity to use a query to search effectively.

"When searching for in a database like Flickr, the images need to include a short but informative description," explained Peter Wonka, the KAUST researcher who led the study. "The description needs to be short to allow the search algorithm to match against millions of possibilities, but also needs to be informative because the correct images need to be found based solely on this description."

Wonka and his colleagues Paul Guerrero and Niloy Mitra from University College London wanted to add something more powerful to the currently limited repertoire of image search tools without adding extra metadata to existing images.

"Instead of describing just the individual objects occurring in an image, we wanted to describe the relationships between objects—such as 'riding,' 'carrying,' 'holding' or 'standing on'—in a way that can be computed and searched for efficiently," noted Wonka.

The team came up with a query tool they call a relation-augmented image descriptor (RAID) that takes either a written description or sketch of objects in a specific spatial relationship and searches for matches in the image database based on relatively simple geometric processing.

"RAID allows us to using a sentence such as 'person standing on snowboard' or to use a simple sketch of the desired composition of objects or an example image with the desired object composition," said Wonka. "Our scheme uses a novel description based on the spatial distribution of simple relationships—like 'above' or 'left of'—over the entire object, which allows us to successfully discriminate between different complex relationships."

RAID provides a new way to describe images and has potential applications in computer graphics, computer vision and automated object classification. The team is currently working on a three-dimensional version of the descriptor that could help with computer interpretation of entire scenes.

Explore further: In a new method for searching image databases, a hand-drawn sketch is all it takes

More information: RAID: A Relation-Augmented Image Descriptor.

Related Stories

Computers can perceive image curves like artists

November 23, 2015

Imagine computers being able to understand paintings or paint abstract images much like humans. Bo Li at Umeå University in Sweden demonstrates a breakthrough concept in the field of computer vision using curves and lines ...

Recommended for you

Archaeologists discover Incan tomb in Peru

February 16, 2019

Peruvian archaeologists discovered an Incan tomb in the north of the country where an elite member of the pre-Columbian empire was buried, one of the investigators announced Friday.

Where is the universe hiding its missing mass?

February 15, 2019

Astronomers have spent decades looking for something that sounds like it would be hard to miss: about a third of the "normal" matter in the Universe. New results from NASA's Chandra X-ray Observatory may have helped them ...

What rising seas mean for local economies

February 15, 2019

Impacts from climate change are not always easy to see. But for many local businesses in coastal communities across the United States, the evidence is right outside their doors—or in their parking lots.

The friendly extortioner takes it all

February 15, 2019

Cooperating with other people makes many things easier. However, competition is also a characteristic aspect of our society. In their struggle for contracts and positions, people have to be more successful than their competitors ...


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.