Our survey found 'questionable research practices' by ecologists and biologists – here's what that means

April 10, 2018 by Fiona Fidler And Hannah Fraser, The Conversation
Negative results are still useful, and should not be hidden. Credit: www.shutterstock.com

Cherry picking or hiding results, excluding data to meet statistical thresholds and presenting unexpected findings as though they were predicted all along – these are just some of the "questionable research practices" implicated in the replication crisis psychology and medicine have faced over the last half a decade or so.

We recently surveyed more than 800 ecologists and evolutionary biologists and found high rates of many of these practices. We believe this to be first documentation of these behaviours in these fields of science.

Our pre-print results have certain shock value, and their release attracted a lot of attention on social media.

  • 64% of surveyed researchers reported they had at least once failed to report results because they were not statistically significant (cherry picking)
  • 42% had collected more data after inspecting whether results were statistically significant (a form of "p hacking")
  • 51% reported an unexpected finding as though it had been hypothesised from the start (known as "HARKing," or Hypothesising After Results are Known).

Although these results are very similar to those that have been found in psychology, reactions suggest that they are surprising – at least to some ecology and evolution researchers.


There are many possible interpretations of our results. We expect there will also be many misconceptions about them and unjustified extrapolations. We talk though some of these below.

It's fraud!

It's not fraud. Scientific fraud involves fabricating data and carries heavy criminal penalties. The questionable research practices we focus on are by definition questionable: they sit in a grey area between acceptable practices and scientific misconduct.

We did ask one question about fabricating data and the answer to that offered further evidence that it is very rare, consistent with findings from other fields.

Scientists lack integrity and we shouldn't trust them

There are a few reasons why this should not be the take home message of our paper.

First, reactions to our results so far suggest an engaged, mature scientific community, ready to acknowledge and address these problems.


If anything, this sort of engagement should increase our trust in these scientists and their commitment to research integrity.

Second, the results tell us much more about structured incentives and institutions than they tell us about individuals and their personal integrity.

For example, these results tell us about the institution of scientific publishing, where negative (non statistically significant results) are all but banished from most journals in most fields of science, and where replication studies are virtually never published because of relentless focus on novel, "ground breaking" results.

The survey results tells us about scientific funding, again where "novel" (meaning positive, significant) findings are valued more than careful, cautious procedures and replication. They also tell us about universities, about the hiring and promotion practices within academic science that focus on publication metrics and overvalue quantity at the expense of quality.

So what do they mean, these questionable research practices admitted by the scientists in our survey? We think they're best understood as the inevitable outcome of publication bias, funding protocols and an ever increasing pressure to publish.

We can't base important decisions on current scientific evidence

There's a risk our results will feed into a view that our science is not policy ready. In many areas, such as health and the environment, this could be very damaging, even disastrous.

One reason it's unwarranted is that climate science is a model based science, and there have been many independent replications of these models. Similarly with immunisation trials.

We know that any criticism of scientific practice runs a risk in the context of anti-science sentiment, but such criticism is fundamental to the success of science.

Remaining open to criticism is science's most powerful self-correction mechanism, and ultimately what makes the scientific evidence base trustworthy.

Scientists are human and we need safeguards

This is an interpretation we wholeheartedly endorse. Scientists are human and subject to the same suite of cognitive biases – like confirmation bias – as the rest of us.

As we learn more about cognitive biases and how best to mitigate them in different circumstances, we need to feed this back into the norms of scientific practice.

The same is true of our knowledge about how people function under different incentive structures and conditions. This is the basis of many of the initiatives designed to make more open and transparent.

The open science movement is about developing initiatives to protect against the influence of cognitive bias, and alter the incentive structures so that research using these questionable research practices stops being rewarded.

Some of these initiatives have been enthusiastically adopted by many scientists and journal editors. For example, many journals now publish analysis code and data along with their articles, and many have signed up to Transparency and Openness Promotion (TOP) guidelines.

Other initiatives offer great promise too. For example, registered report formats are now offered by some journals, mostly in psychology and medical fields. In a registered report, articles are reviewed on the strength of their underlying premise and approach, before data is collected. This removes the temptation to select only positive results or to apply different standards of rigour to negative results. In short, it thwarts .

We hope that by drawing attention to the prevalence of , our research will encourage support of these initiatives, and importantly, encourage institutions to support researchers in their own efforts to align their with their scientific values.

Explore further: Study examines state of social, personality psychology research

Related Stories

Questionable research practices surprisingly common

May 25, 2012

(Medical Xpress) -- Not all scientific misconduct is flat-out fraud. Much falls into the murkier realm of “questionable research practices.” A new study finds that in one field, psychology, these practices are surprisingly ...

Recommended for you

Matter waves and quantum splinters

March 25, 2019

Physicists in the United States, Austria and Brazil have shown that shaking ultracold Bose-Einstein condensates (BECs) can cause them to either divide into uniform segments or shatter into unpredictable splinters, depending ...

How tree diversity regulates invading forest pests

March 25, 2019

A national-scale study of U.S. forests found strong relationships between the diversity of native tree species and the number of nonnative pests that pose economic and ecological threats to the nation's forests.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.