Recognizing "spin" in the scientific literature

Facts are facts and data are data—both reveal objective truths. But, subjectivity is introduced the moment those facts and data are interpreted. In other words, they are susceptible to spin, or the biased interpretation of facts and data to persuade opinion in favor or against those facts and data. Therefore, individuals who are not entrenched in the science community are at a severe disadvantage when interpretations of data and fact are spun. Such individuals will take interpretations at face value without the training to analyze the hidden meanings or view topics from different angles. This disadvantage is further underscored when such individuals have little to no background in science and don't speak the language of jargon. Instead, the public relies on scientists to relay truth instead of being skeptical interpreters. And this has led to a large anti-intellectual movement among Americans who do not trust scientists or their interpretations of data. For example, see how the same climate change data lead the vast majority of climate scientists to implicate humans as a direct cause of climate change, whereas political pundits on the Right are skeptical and use the same data to spin their interpretation that humans are not responsible. How can American citizens trust scientists in their community, and how will they be able to tell who is a trusted source of information? This is a critical question in the current political environment, and one recently examined by Kellia Chiu and colleagues in a 2017 paper in PLOS Biology.

Should there be a specific criterion that members of the community should be able to draw their conclusions from? Are there some scientific topics that are more susceptible to spin than others? Furthermore, how is spin defined by the non-scientific community and by those who are in the science community?

How is Science Spun?

Spin is a common tool of propaganda, often employed by media outlets to push an agenda. Within science, it is commonly used to generate 'science hype.' In these cases, the importance of scientific findings is overstated and relayed in an inappropriate manner. Spin has gained renewed interest by researchers as they have noticed that findings—sometimes preliminary ones—are being put forth in positive light so that they can be viewed favorably by the public. This begs the question: what motivates scientists to spin their results? What are their goals? Are they mainly grant-oriented, or career-advancing, or are there other goals they are pursuing from overstating findings? Perhaps, another question that should be asked is whether the prevalence of spin renders entire groups of findings as false, and whether this makes scientific findings with spin questionable material for public consumption. If members of the public take what they read at face value, how are they to know any better?

The Effect of Spin

Chiu and colleagues investigated the effect of spin in science and how it impacts public opinion. Additionally, they investigated the following question: what is spin's impact on evidence-based policy decisions? For example, politicians of all ideologies often distort scientific information in order to achieve specific goals. If citizens are not aware of what scientists are discovering, they will not be able to make informed decisions, and as a result, citizens cannot pressure their representatives to do the same degree of thorough investigation of scientific findings to make informed policy decisions. The paper by Chiu et al. analyzed 35 reports that had investigated the concept of spin and its effects on clinical trials, observational studies, diagnostic accuracy studies, systematic reviews, and meta-analyses. Chiu et al. concluded that spin varied in different studies, and that the level or intensity of spin may be different in relation to the amount of funding that each study received, although there was inconclusive evidence to support this claim.

Types of Spin

So… how do we identify spin? As Chiu and colleagues point out, spin can take a variety of forms, including: (1) "inappropriate study given study data;" (2) "inappropriate extrapolations or recommendations for clinical practice;" (3) "selective reporting;" 4) and "more robust or favorable data presentation." Let's unpack these a little bit.

The first of these types, "inappropriate study given study data," occurs when findings simply are interpreted incorrectly. Specifically, Chiu et al. found that this type of spin is commonly used in conjunction with casual (or colloquial) language, which in many cases has the potential to alter interpretation of the data. So how do researchers strike a balance between using scientific jargon and communicating an idea in language understandable by those not in the science community? Should there be a glossary of scientific definitions provided at the end of an article which readers can refer to?

The second type of spin, refers to using statistical evidence—unintentionally—to support a statement when the evidence does not point to the conclusion. For example, how often do we lament the weather forecast being incorrect? From a statistical standpoint, we shouldn't ever be surprised when the forecast doesn't match the weather because the forecast is based on extrapolation of data, and extrapolated data are really just estimates. So, the relationship between final conclusions and the extrapolated data upon which they are based is weaker than if the data had been measured directly.

The final two examples of spin are related in the sense that "selective reporting" of some (but not all) data can lead to "more robust or favorable data presentation." In other words, leaving out some key pieces of information can enhance the interpretation of data and shed then in a positive light when they may not be favorable when viewed in total.

Where is the evidence of Spin?

Chiu et al. found evidence of spin in nine reports examined (9/35), across eight scientific findings that included spin. These researchers examined the conclusions of these scientific trials and found that conclusions were heavily spun the data were inconclusive to answer the question being asked. By saying inconclusive, we mean to say that the results neither confirmed nor disputed particular findings. Other studies suggested that high spin can be characterized by lack of (statistical) uncertainty when framing conclusions, no recommendation for further trials, or no acknowledgement of the statistically nonsignificant outcomes. Another study by Clement Lazarus and colleges reported that 128 abstracts of nonrandomized interventional studies included "tones" that suggested the outcomes came as a result of the intervention or used strong language to relate as much, without any actual evidence of a causal relationship. These "tones" included phrases such as: 'the study shows that' or 'the results explain,' even when there was no real evidence to support such claims. Therefore, spin such as that described here, can mislead the public and other researchers who may not read critically a paper that has been peer-reviewed in a major journal. The types of reports that were deemed most severe were concluding recommendations for clinical practice when not supported by results, titles that claim that the treatment is beneficial when not supported by the results, selective reporting, or overemphasizing results that favor a specific beneficial intervention.

What does this tell us?

What do these reports tell us the prevalence of spin in the scientific literature? Should readers place all their faith in one article they read? Or is it better for lay people to gain knowledge pertaining to specific findings from different reputable sources? Just as we colloquially advise to never place all eggs in a single basket, so it with scientific literature–reading more than one source is always better than relying on a single source. This is true for researchers and the public alike.

This story is republished courtesy of PLOS Blogs: blogs.plos.org.