Mapping the 'memory loss' of disinformation in fact checks

Fact-checking is an important tool in the fight against online disinformation that can have serious implications for individuals and society by influencing elections, conflict and health. However, according to a survey conducted as part of the vera.ai project, the crucial task of archiving appearances of disinformation is made extremely difficult by anti-scraping measures taken by social media platforms such as Facebook and Instagram.

What happens in this case? There are fewer documented fact checks, and less evidence of disinformation traces. As reported in a paper posted on the "Digital Methods Initiative" website, "many links to content disappear (erased by platforms, by end-users, or kept within private groups after debunks)." This harms fact-checking memory and hampers social scientists' efforts to assess the scope of disinformation on different platforms.

The focus on Ukraine

But what is the actual extent of fact-checking "memory loss?" To answer this question, researchers at vera.ai project partner University of Amsterdam, the Netherlands, used the "War in Ukraine" fact-check dataset published by the European Digital Media Observatory (EDMO). They analyzed 1,991 fact-checking articles, extracting more than 41,000 links, of which 6,002 were archived web pages.

The team identified the multiple fact-checking organizations contributing to archiving links, their contribution count and their country. They also investigated the evolution of the publications over time as the Ukraine war progressed.

The study revealed that at least 15% of the archived content links on EDMO's "War in Ukraine" dataset are poorly archived. "The 'memory loss' is even more important as many errors due to unplayable videos have been identified during a manual analysis of a sample of 100 links," the authors report in the paper. Additionally, 23% of the non-archived Facebook links in the same dataset have missing content.

The archive services used are also discussed: "Three main services (archive.today and its satellite sites with 44.1% in the dataset, 29.2% and 26.6%) are currently dominating the field. While archive.today relies on advertising, perma.cc is a freemium and commercial service built at Harvard University while the US Wayback Machine / Internet Archive remains the free-access web archive. Their use by fact checking organizations rely often on their ability to archive content from platforms and especially from Facebook, due to anti-bots and anti-scraping measures."

The researchers also found that although platform pages seem to remain accessible, many of them display archived warnings of different types of archiving issues ranging from login barriers to unplayable video content. Next, the vera.ai: VERification Assisted by Artificial Intelligence team intend to extend their study to more datasets.