Scientists create computational algorithm for fact-checking

June 17, 2015
These charts map the 'truth scores; of statements related to geography, history and entertainment. Correct statements appear along the diagonal, with color intensity indicating the strength of the score. Credit: Giovanni Ciampaglia

Network scientists at Indiana University have developed a new computational method that can leverage any body of knowledge to aid in the complex human task of fact-checking.

In the first use of this method, IU scientists created a simple computational fact-checker that assigns "truth scores" to statements concerning history, geography and entertainment, as well as random statements drawn from the text of Wikipedia, the well-known online encyclopedia.

In multiple experiments, the automated system consistently matched the assessment of human fact-checkers in terms of their certitude about the accuracy of these statements.

The results of the study, "Computational Fact Checking From Knowledge Networks," are reported in today's issue of PLOS ONE.

"These results are encouraging and exciting," said Giovanni Luca Ciampaglia, a postdoctoral fellow at the Center for Complex Networks and Systems Research in the IU Bloomington School of Informatics and Computing, who led the study. "We live in an age of information overload, including abundant misinformation, unsubstantiated rumors and conspiracy theories whose volume threatens to overwhelm journalists and the public.

"Our experiments point to methods to abstract the vital and complex human task of fact-checking into a network analysis problem, which is easy to solve computationally."

The team selected Wikipedia as the information source for their experiment due to its breadth and open nature. Although Wikipedia is not 100 percent accurate, previous studies estimate the is nearly as reliable as traditional encyclopedias, but also covers many more subjects.

Using factual information from infoboxes on the site, IU scientists built a "knowledge graph" with 3 million concepts and 23 million links between them. A link between two concepts in the graph can be read as a simple factual statement, such as "Socrates is a person" or "Paris is the capital of France."

In what the IU scientists describe as an "automatic game of trivia," the team applied their algorithm to answer simple questions related to geography, history and entertainment, including statements that matched states or nations with their capitals, presidents with their spouses and Oscar-winning film directors with the movie for which they won the Best Picture awards, with the majority of tests returning highly accurate truth scores.

Lastly, the scientists used the algorithm to fact-check excerpts from the main text of Wikipedia, which were previously labeled by human fact-checkers as true or false, and found a positive correlation between the truth scores produced by the algorithm and the answers provided by the fact-checkers.

Significantly, the IU team found their could even assess the truthfulness of statements about information not directly contained in the infoboxes. For example, the fact that Steve Tesich —- the Serbian-American screenwriter of the classic Hoosier film "Breaking Away"—graduated from IU, despite the information not being specifically addressed in the infobox about him.

"The measurement of the truthfulness of statements appears to rely strongly on indirect connections, or 'paths,' between concepts," Ciampaglia said. "If we prevented our fact-checker from traversing multiple nodes on the graph, it performed poorly since it could not discover relevant indirect connections. But because it's free to explore beyond the information provided in one infobox, our method leverages the power of the full knowledge graph."

Although the experiments were conducted using Wikipedia, the IU team's method does not assume any particular source of knowledge. The scientists aim to conduct additional experiments using knowledge graphs built from other sources of human knowledge, such as Freebase, the open-knowledge base built by Google, and note that multiple information sources could be used together to account for different belief systems.

"Misinformation endangers the public debate on a broad range of global societal issues," said Filippo Menczer, director of the Center for Complex Networks and Systems Research and a professor in the IU School of Informatics and Computing, who is a co-author on the study. "With increasing reliance on the Internet as a source of information, we need tools to deal with the misinformation that reaches us every day. Computational fact-checkers could become part of the solution to this problem."

The team added a significant amount of natural language processing research, and other work remains before these methods could be made available to the public as a software tool.

Explore further: Network model for tracking Twitter memes sheds light on information spreading in the brain

More information: PLOS ONE, dx.plos.org/10.1371/journal.pone.0128193

Related Stories

Fact checking politicians gets results

October 18, 2013

New research indicates that American politicians are affected by the practice of fact-checking, thereby reducing the risk of misinformation and strengthening democratic accountability.

Wiki ranking: Bayesian statistics can score Wikipedia entries

August 6, 2014

Wikipedia the free, online collaborative encyclopedia is an important source of information. However, while the team of volunteer editors endeavors to maintain high standards, there are occasionally problems with the veracity ...

Recommended for you

MIT's flea market specializes in rare, obscure electronics

September 25, 2016

Once a month in the summer, a small parking lot on the Massachusetts Institute of Technology's campus transforms into a high-tech flea market known for its outlandish offerings. Tables overflow with antique radio equipment, ...

Tech titans join to study artificial intelligence

September 29, 2016

Major technology firms have joined forces in a partnership on artificial intelligence, aiming to cooperate on "best practices" on using the technology "to benefit people and society."

First test of driverless minibus in Paris Saturday

September 24, 2016

The French capital's transport authority will on Saturday carry out its first test of a driverless minibus, in the hope that regular routes for the hi-tech vehicles will be up and running within two years.

3 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

KBK
1 / 5 (2) Jun 17, 2015
So now, we are supposed to trust logic analysis to a computer algorithm?

If the program had any logic to it whatsoever, it would scream:

Do Not Trust Me! Do Not Invest in Me! At any Price!

This is a form of retardation, and subversive logic that I hoped to never be alive to see presented.

It is the end human logic formation, the end of human growth and the beginning of a chain, a fence of limits, within the idea of the pursuit of the furthering of the human race.

This kind of horseshit needs to die with the most extreme prejudice and permanent prejudice that is possible to muster. Forever. Until the end of fucking time.

This shit is death. Stay away from it.
Job001
5 / 5 (3) Jun 18, 2015
Considering human cognitive bias forms a powerful anti-truth input it is no wonder politics, marketing, legal myth, theology, and entertainment contain so little truth. Computer algorithms promise the possibility of low bias evaluation of misinformation.
antialias_physorg
5 / 5 (1) Jun 18, 2015
Computer algorithms promise the possibility of low bias evaluation of misinformation.

Given good input. With a broad knowledge base it will be hard to 'fact-hack' an item that is false into one as true. So this should be a first step when fact checking.

However the algorithm can be used the other way around (i.e. fact-hacking until you achieve the desired result or using the algorithm to identifiy the minimum number of places you need to fact-hack in order to achieve the desired level of 'truth').

Probably effective against a lone crank, but not so much against national distortions of history/current events.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.