Sketch-based query for searching for relationships among objects in images

August 11, 2016, King Abdullah University of Science and Technology
Technology search for relationships
Credit: Wikimedia

Searching for specific images may become easier thanks to a new tool that generates image queries based on a sketch or description of objects in spatial relationships. The tool, which has been proposed by researchers from King Abdullah University of Science and Technology (KAUST), Saudi Arabia, and University College London, makes it easier to search the world's ever-expanding databases for pictures matching a wider and more powerful range of image queries.

The enormous collections of photographs and pictures now available in online databases represent a remarkable resource for research and creative arts. As unfathomably rich as these databases might be, they are only as useful as a user's capacity to use a query to search effectively.

"When searching for in a database like Flickr, the images need to include a short but informative description," explained Peter Wonka, the KAUST researcher who led the study. "The description needs to be short to allow the search algorithm to match against millions of possibilities, but also needs to be informative because the correct images need to be found based solely on this description."

Wonka and his colleagues Paul Guerrero and Niloy Mitra from University College London wanted to add something more powerful to the currently limited repertoire of image search tools without adding extra metadata to existing images.

"Instead of describing just the individual objects occurring in an image, we wanted to describe the relationships between objects—such as 'riding,' 'carrying,' 'holding' or 'standing on'—in a way that can be computed and searched for efficiently," noted Wonka.

The team came up with a query tool they call a relation-augmented image descriptor (RAID) that takes either a written description or sketch of objects in a specific spatial relationship and searches for matches in the image database based on relatively simple geometric processing.

"RAID allows us to using a sentence such as 'person standing on snowboard' or to use a simple sketch of the desired composition of objects or an example image with the desired object composition," said Wonka. "Our scheme uses a novel description based on the spatial distribution of simple relationships—like 'above' or 'left of'—over the entire object, which allows us to successfully discriminate between different complex relationships."

RAID provides a new way to describe images and has potential applications in computer graphics, computer vision and automated object classification. The team is currently working on a three-dimensional version of the descriptor that could help with computer interpretation of entire scenes.

Explore further: In a new method for searching image databases, a hand-drawn sketch is all it takes

More information: RAID: A Relation-Augmented Image Descriptor. arxiv.org/abs/1510.01113

Related Stories

Computers can perceive image curves like artists

November 23, 2015

Imagine computers being able to understand paintings or paint abstract images much like humans. Bo Li at Umeå University in Sweden demonstrates a breakthrough concept in the field of computer vision using curves and lines ...

Recommended for you

China auto show highlights industry's electric ambitions

April 22, 2018

The biggest global auto show of the year showcases China's ambitions to become a leader in electric cars and the industry's multibillion-dollar scramble to roll out models that appeal to price-conscious but demanding Chinese ...

Robot designed for faster, safer uranium plant pipe cleanup

April 21, 2018

Ohio crews cleaning up a massive former Cold War-era uranium enrichment plant in Ohio plan this summer to deploy a high-tech helper: an autonomous, radiation-measuring robot that will roll through miles of large overhead ...

Virtually modelling the human brain in a computer

April 19, 2018

Neurons that remain active even after the triggering stimulus has been silenced form the basis of short-term memory. The brain uses rhythmically active neurons to combine larger groups of neurons into functional units. Until ...

'Poker face' stripped away by new-age tech

April 14, 2018

Dolby Laboratories chief scientist Poppy Crum tells of a fast-coming time when technology will see right through people no matter how hard they try to hide their feelings.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.