Young and Karr propose ways to improve how observational studies are conducted

Aug 25, 2011

S. Stanley Young, assistant director for bioinformatics at the National Institute of Statistical Sciences (NISS), and Alan Karr, director at NISS, have published a non-technical article in the September issue of Significance magazine pointing out that medical and other observational studies often produce results that are later shown to be incorrect, and—invoking a quality control perspective—suggest ways to fix the system.

Their central point is that the current system of publication in peer-reviewed journals relies on post-production inspection to ensure quality, a practice that has disappeared from modern industry in favor of controlling the process instead: quality control is now process control, not product control. They cite W. Edwards Deming, considered by many the most innovative thinker ever about quality, arguing not only for process control, but also that the problem lies with the managers—funders and journals—rather with than the workers—individual researchers who respond rationally to the current set of incentives.

Young and Karr describe both their and others' studies of the extent to which observational studies do not replicate. Published claims such as "coffee causes pancreatic cancer," or "women eating breakfast cereal are more likely to have boy babies," have been refuted by subsequent studies and analyses. When these studies reach the popular media and influence individual consumers, the burden falls not just on science but also on society. And even if there were no impact on the public, scarce research resources, both money and personnel, have been squandered.

The paper describes several technical difficulties with observational studies, among them multiple testing (if enough questions are asked, some will yield false positive answers), bias (systematic error) and multiple modeling (searching among mathematical models until one is found that "fits the data"). Publication bias is another issue: papers reporting positive scientific results (for example, an association between Type A personalities and heart attacks) are more likely to be published than those reporting negative results, even though the latter may be as important scientifically.

Young and Karr recommend that when a study is submitted for publication, the data be split into two sets, a modeling data set and a holdout data set. Journals would then accept or reject papers based on the analysis of the modeling data set without knowing the results of applying the methods to the holdout set. But then the journal would also publish an addendum to the paper giving the results of the analysis of the holdout set.

Explore further: Researchers help Boston Marathon organizers plan for 2014 race

More information: Significance magazine is published by the Royal Statistical Society of the UK and the American Statistical Society. A copy of the article will be made available at:

add to favorites email to friend print save as pdf

Related Stories

Will eating certain cereal result in male babies?

Jan 14, 2009

Could eating cereal really make it more likely for someone to have a boy baby than a girl baby? Researchers wrote a paper, "Cereal-Induced Gender Selection? Most Likely a Multiple Testing False Positive," that will be published ...

For a less biased study, try randomization

Apr 13, 2011

A new review of existing research confirms that the so-called “gold standard” of medical research — the randomized controlled study — provides a safeguard against bias.

Detecting bias in the reporting of clinical trials

Aug 19, 2009

A study by researchers at the University of Leicester has revealed new ways to spot whether medical research has hidden biases. Writing in the prestigious British Medical Journal, Santiago Moreno and his colleagues demons ...

Recommended for you

Egypt archaeologists find ancient writer's tomb

Apr 19, 2014

Egypt's minister of antiquities says a team of Spanish archaeologists has discovered two tombs in the southern part of the country, one of them belonging to a writer and containing a trove of artifacts including reed pens ...

Study finds law dramatically curbing need for speed

Apr 18, 2014

Almost seven years have passed since Ontario's street-racing legislation hit the books and, according to one Western researcher, it has succeeded in putting the brakes on the number of convictions and, more importantly, injuries ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Sep 10, 2011
Significance costs $150 a year subscription. It is free if you belong to the American Statistical Society or the Royal Statistical Society. Some libraries may have free access.
For all the rest of us unemployed have-nots, it costs too much.

More news stories

Clippers and coiners in 16th-century England

In 2017 a new £1 coin will appear in our pockets with a design extremely difficult to forge. In the mid-16th century, Elizabeth I's government came up with a series of measures to deter "divers evil persons" ...

Growing app industry has developers racing to keep up

Smartphone application developers say they are challenged by the glut of apps as well as the need to update their software to keep up with evolving phone technology, making creative pricing strategies essential to finding ...

Making graphene in your kitchen

Graphene has been touted as a wonder material—the world's thinnest substance, but super-strong. Now scientists say it is so easy to make you could produce some in your kitchen.