Search technology that can gauge opinion and predict the future

August 16, 2012
Search technology that can gauge opinion and predict the future
© Shutterstock

Inspired by a system for categorising books proposed by an Indian librarian more than 50 years ago, a team of European researchers have developed a new kind of internet search that takes into account factors such as opinion, bias, context, time and location. The new technology, which could soon be in use commercially, can display trends in public opinion about a topic, company or person over time - and it can even be used to predict the future.

'Do a search for the word "climate" on or another search engine and what you will get back is basically a list of results featuring that word: there's no categorisation, no specific order, no context. Current search engines do not take into account the dimensions of diversity: factors such as when the information was published, if there is a bias toward one opinion or another inherent in the content and structure, who published it and when,' explains Fausto Giunchiglia, a professor of computer science at the University of Trento in Italy.

But can search technology be made to identify and embrace diversity? Can a search engine tell you, for example, how public opinion about climate change has changed over the last decade? Or how hot the weather will be a century from now, by aggregating current and past estimates from different sources?

It seems that it can, thanks to a pioneering combination of and a decades-old classification method, brought together by in the LivingKnowledge project. Supported by EUR 4.8 million in funding from the , the LivingKnowledge team, coordinated by Prof. Giunchiglia, adopted a to developing new search technology, drawing on fields as diverse as , , semiotics and library science.

Indeed, the so-called father of library science, Sirkali Ramamrita Ranganathan, an Indian librarian, served as a source of inspiration for the researchers. In the 1920s and 1930s, Ranganathan developed the first major analytico-synthetic, or faceted, classification system. Using this approach, objects - books, in the case of Ranganathan; web and database content, in the case of the LivingKnowlege team - are assigned multiple characteristics and attributes (facets), enabling the classification to be ordered in multiple ways, rather than in a single, predetermined, taxonomic order. Using the system, an article about the effects on agriculture of climate change written in Norway in 1990 might be classified as 'Geography; Climate; ; Agriculture; Research; Norway; 1990.'

In order to understand the classification system better and implement it in search engine technology, the LivingKnowledge researchers turned to the Indian Statistical Institute, a project partner, which uses faceted classification on a daily basis.

'Using their knowledge we were able to turn Ranganathan's pseudo-algorithm into a computer algorithm and the computer scientists were able to use it to mine data from the web, extract its meaning and context, assign facets to it, and use these to structure the information based on the dimensions of diversity,' Prof. Giunchiglia says.

Researchers at the University of Pavia in Italy, another partner, drew on their expertise in extracting meaning from web content - not just from text and multimedia content, but also from the way the information is structured and laid out - in order to infer bias and opinions, adding another facet to the data.

'We are able to identify the bias of authors on a certain subject and whether their opinions are positive or negative,' the LivingKnowledge coordinator says. 'Facts are facts, but any information about an event, or on any subject, is often surrounded by opinions and bias.'

From libraries of the 1930s to space travel in 2034...

The technology was implemented in a testbed, now available as open source software, and used for trials based around two intriguing application scenarios.

Working with Austrian social research institute SORA, the team used the LivingKnowledge system to identify social trends and monitor public opinion in both quantitative and qualitative terms. Used for media content analysis, the system could help a company understand the impact of a new advertising campaign, showing how it has affected brand recognition over time and which social groups have been most receptive. Alternatively, a government might use the system to gauge about a new policy, or a politician could use it to respond in the most publicly acceptable way to a rival candidate's claims.

With Barcelona Media, a non-profit research foundation supported by Yahoo!, and with the Netherlands-based Internet Memory Foundation, the LivingKnowledge team looked not only at current and past trends, but extrapolated them and drew on forecasts extracted from existing data to try to predict the future. Their Future Predictor application is able to make searches based on questions such as 'What will oil prices be in 2050?' or 'How much will global temperatures rise over the next 100 years?' and find relevant information and forecasts from today’s web. For example, a search for the year 2034 turns up 'space travel' as the most relevant topic indexed in today's news.

'More immediately, this application scenario provides functionality for detecting trends even before these trends become apparent in daily events - based on integrated search and navigation capabilities for finding diverse, multi-dimensional information depending on content, bias and time,' Prof. Giunchiglia explains.

Several of the project partners have plans to implement the technology commercially, and the project coordinator intends to set up a non-profit foundation to build on the LivingKnowledge results at a time when demand for this sort of technology is only likely to increase.

As Prof. Giunchiglia points out, Google fundamentally changed the world by providing everyone with access to much of the world's information, but it did it for people: currently only humans can understand the meaning of all that data, so much so that information overload is a common problem. As we move into a 'big data' age in which information about everything and anything is available at the touch of a button, the meaning of that information needs to be understandable not just by humans but also by machines, so quantity must come combined with quality. The LivingKnowledge approach addresses that problem.

'When we started the project, no one was talking about big data. Now everyone is and there is increasing interest in this sort of technology,' Prof. Giunchiglia says. 'The future will be all about big data - we can't say whether it will be good or bad, but it will certainly be different.'

Armed with the project's Future Predictor, Prof. Giunchiglia is well equipped to make that prediction.

Explore further: Search engine mashup

More information:

Related Stories

Search engine mashup

July 6, 2007

A mashup of two different types of web search tools could make find the useful nuggets of information among all the grit on the Internet much easier.

Vertical search across the educational horizon

December 22, 2010

Searching the web usually involves typing keywords or a phrase into a search engine and clicking the "search now" button. It's very effective and several large companies have become prominent in the field by providing users ...

Social networking shortcut to finding medical experts

March 15, 2012

It can be difficult for someone outside of a specialist field to identify subject experts and the ever increasing amount of available data can be bewildering. New research, published in BioMed Central's open access journal, ...

Personal discrimination on the Web

May 21, 2009

How do you tell if a website you are browsing is a showing you a personal web page expressing the opinions of an individual or the marketing speak of a commercial site in disguise? Information engineers in India and Japan ...

The engines of change

November 5, 2010

In today's wired world, search engines have changed the way people find data, and social searches are making it even easier to find exactly what you're looking for, with a little help from your friends. For example, a recent ...

Recommended for you

World gears up for electric cars despite bumps in road

July 26, 2017

Technological advances mean fossil fuel in cars could be phased out within decades but switching to electric carries its own environmental and economic concerns as more and more countries announce radical plans.

Musk, Zuckerberg duel over artificial intelligence

July 25, 2017

Visionary entrepreneur Elon Musk and Facebook chief Mark Zuckerberg were trading jabs on social media over artificial intelligence this week in a debate that has turned personal between the two technology luminaries.

Adobe bidding Flash farewell in 2020

July 25, 2017

Adobe on Tuesday said its Flash software that served up video and online games for decades will be killed off over the next three years.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.