Online games and remote experiments could reduce scientific fraud, cherry-picking

October 6, 2014

One way to combat the rising level of errors and fraud in life sciences research is through massive online laboratories, which use videogames to engage large numbers of non-professional investigators and prevent scientists from manually testing their own hypotheses, researchers from Carnegie Mellon University and Stanford University say.

Though unconventional, CMU's Adrien Treuille and Stanford's Rhiju Das argue that this online, game-like approach actually is more scientifically rigorous than the standard practice of scientists proposing an explanation for some phenomenon and then testing that hypothesis through experimentation. In a commentary published online today by the journal Trends in Biochemical Sciences, they maintain that massive online labs could be a model for the entirety of science.

Earlier this year, Treuille and Das reported in the Proceedings of the National Academy of Sciences on the success of their own online lab, an RNA-design project called EteRNA, which has produced unprecedented design insights that have advanced knowledge of RNA (ribonucleic acid).

"If you strip away the game part, projects such as EteRNA present a fundamentally new model of remote science that can prevent many common forms of scientific fraud," said Treuille, an assistant professor of computer science and robotics at Carnegie Mellon. "We registered more than 150,000 participants who contributed in excess of 2 million human-hours to EteRNA. That means there were a lot of eyes, a lot of people looking over each other's shoulders as hypotheses were developed and experimental results evaluated. Everything is out in the open."

Online participants use computer design tools to propose RNA designs that meet certain criteria. The designs are then synthesized in the Stanford lab of Das, an assistant professor of biochemistry. The results are made available to the entire EteRNA community for analysis and use in future design challenges.

Treuille said that transparency makes it difficult for any individual to retrospectively adjust scientific hypotheses to match , or to cherry-pick data to reflect a scientist's biases. "It's not typically acknowledged, but having the same team both develop a hypothesis and test it in the lab creates a conflict of interest—something that may be contributing to a plague of irreproducible results in many research studies," Treuille said. "EteRNA separates these two important processes without inhibiting the science."

The sheer number of people involved in the research also addresses a problem associated with a hallmark of modern biology—the use of high-throughput experimental techniques.

"A single DNA or RNA sequencing run can generate billions of data points," Das noted. "Following up those experiments with cycles of hypothesis generation and testing is critical to establishing scientific truth. But it can be expensive and time-consuming. It's tempting to skip the extra work and cherry-pick the data to produce a publishable manuscript.

"If you have thousands of people helping you, however, the data deluge isn't so bad," Das added.

Though people attracted to EteRNA by its game-like elements were not necessarily trained scientists, many developed deep understandings of RNA design as they used computerized tools to design molecules and then saw which designs were successful when synthesized in Das' lab.

Treuille and Das said setting up a massive online lab isn't cheap, but should be within the budget of a conventional life sciences grant.

"The major current barrier may instead be the career risks that these projects pose for their creators," they write in the new position paper. "In particular, videogames, which appear critical for recruiting scientifically engaged citizens, are generally viewed as incompatible with 'serious,' or rigorous research."

Explore further: Crowdsourced RNA designs outperform computer algorithms

More information:

Related Stories

Crowdsourced RNA designs outperform computer algorithms

January 27, 2014

An enthusiastic group of non-experts, working through an online interface and receiving feedback from lab experiments, has produced designs for RNA molecules that are consistently more successful than those generated by the ...

Recommended for you

No microbes? No problem for caterpillars

August 22, 2017

The microbiome seems ubiquitous: humans and many other species rely on billions of tiny organisms in their guts to aid in digestion, metabolism and other functions. Now, scientists at the University of Colorado Boulder are ...

How a bacterium can live on methanol

August 22, 2017

ETH Zurich researchers have identified all the genes required by a bacterium to use methanol as a food source. The results will help scientists advance the use of this resource in the field of biotechnology.


Adjust slider to filter visible comments by rank

Display comments: newest first

1 / 5 (3) Oct 06, 2014
Treuille said that transparency makes it difficult for any individual to retrospectively adjust scientific hypotheses to match experimental results, or to cherry-pick data to reflect a scientist's biases.
Nowhere are these problems more evident, than in the climate sciences.
3.7 / 5 (3) Oct 06, 2014
Nowhere are these problems more evident, than in the climate sciences.

Yes; global warming deniers sure do love cherry -picking the data and ignore inconvenient scientific facts for their hypothesis.
5 / 5 (1) Oct 06, 2014
There's two sides to this: Massive data is good. It gives you better statistical power.

Some downsides I can think of are:

Since you don't interact with the people who participate directly there's no guarantee that there will not be
- bots
- one user acting multiple times (to skew the data as he likes to see fit. An example of which can be seen e.g. here in the comment section when people create sockpuppets to vote)

There's also an inclination to skew data towards technophile people (i.e. those with an interent connection). This is less of an issue nowadays, but something that should not be entirely discounted. Especially when the games start to become complex will you see a shift towards people with longer attention spans (and probably a more 'geeky' attitude).

So it's probably a fine line to walk to make this work to the benefit of result quality.
Just throwing the 'game' out there may not achieve this.
Captain Stumpy
1 / 5 (1) Oct 06, 2014
Nowhere are these problems more evident, than in the climate sciences
and i suppose that you have empirical evidence of this, correct?
Please link the studies and their data so that we can see the "retrospectively adjust scientific hypotheses to match experimental results", or the "cherry-pick data to reflect a scientist's biases"
it is not like there isn't plenty of scientific studies to choose from
I just want to highlight this illuminating infographic by James Powell in which, based on more than 2000 peer-reviewed publications, he counts the number of authors from November, 2012 to December, 2013 who explicitly deny global warming (that is, who propose a fundamentally different reason for temperature rise than anthropogenic CO2). The number is exactly one.


Start with the above >2000 peer-reviewed publications

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.