Peer review system for awarding NIH grants is flawed, researchers say

February 16, 2016

The mechanism used by the National Institutes of Health (NIH) to allocate government research funds to scientists whose grants receive its top scores works essentially no better than distributing those dollars at random, new research suggests.

The findings suggest that the expensive and time-consuming peer-review process is not necessarily funding the best science, and that awarding grants by lottery could actually result in equally good, if not better, results. A report on the research, published online Feb. 16 in the journal eLife, was written by Ferric Fang, MD, at the University of Washington, Anthony Bowen, MS, at the Albert Einstein College of Medicine and Arturo Casadevall, MD, PhD, at the Johns Hopkins Bloomberg School of Public Health,

"The NIH claims that they are funding the best grants by the best scientists. While these data would argue that the NIH is funding a lot of very good science, they are also leaving a lot of very good science on the table," says Casadevall, Professor and Chair of the W. Harry Feinstone Department of Molecular Microbiology and Immunology at the Bloomberg School. "The government can't afford to fund every good grant proposal, but the problems with the current system make it worse than awarding grants through a lottery."

Notes Fang, a professor of laboratory medicine and microbiology at the University of Washington: "We are not criticizing the peer reviewers. We are simply showing that there are limits to the ability of peer review to predict future productivity based on grant applications. This suggests that some of the resources and effort spent on ranking applications might be better spent elsewhere. While the average productivity of grants with better scores was somewhat higher, the differences were extremely small, raising questions as to whether the effort is worthwhile."

NIH rejects the majority of research grant proposals it receives. To decide which proposals to fund, NIH relies on expert panels whose members score each application. Funding decisions are made on the basis of these scores and the amount of available funds. In recent years, the NIH has only funded those proposals ranked around the top 10 percent. The annual research budget for the NIH was $30.1 billion in 2015.

For their study, the researchers reanalyzed data on the 102,740 research project grants funded by the NIH from 1980 through 2008. Researchers who published a paper in the journal Science in 2015 had collected the data set. Their research suggested that peer review did in fact work—that the highest ranked research projects funded by the NIH earned the most citations. The researchers in this case chose to measure the success of a research grant by determining how many papers that resulted from the work were published in scientific journals and then tracked how many times those papers are cited in future research papers.

The original researchers looked at all of the grants funded by NIH in those years and a significantly larger number of grants were funded in many of those years. The percentage of grants funded in recent years has been at historic lows because of cutbacks resulting from sequestration budget cuts stemming from the October 2013 government shutdown

For the new study, Casadevall and his colleagues decided to only look at the top 20 percent of grants awarded and found very little difference between the top-ranked projects and those projects ranked in the 20th percentile when it came to which would go on to be the most-cited research. What the peer review process can do, they determined, is discriminate between very good science and very bad science —that is, those in the top 20 percent versus those below the 50th percentile.

Peer review isn't cheap. The annual budget of the NIH Center for Scientific Review is $110 million. Individual NIH institutes and centers also spend a lot on peer review. That money could go toward more grants, the researchers say. The costs are not only financial. Writing and reviewing grants is extremely time consuming and diverts the efforts of scientists away from doing science itself.

The process also allows for substantial subjectivity. The objection of a single member of the committee can effectively kill a grant proposal, whether that objection is legitimate or not.

"When people's opinions count a lot, we may be doing worse than choosing at random," Casadevall says. "A negative word at the table can often swing the debate. And this is how we allocate research funding in this country."

To solve this, the authors suggest that the top proposals are first chosen by and that those proposals then be put into a lottery, with grants awarded at random. Lotteries were used as part of the military draft during the Vietnam War and are used today to fill magnet schools with many qualified applicants and to award permanent residency applications. College student and low-income housing is often awarded by lottery. He says New Zealand has started using a lottery to make its scientific grants.

Adds Casadevall: "We're hoping people will look at this data and say, 'Can we do better? Can we create a fairer system that gives society the best science it can afford?'"

"NIH Peer Review Percentile Scores are Poorly Predictive of Grant Productivity" was written by Ferric C. Fang, Anthony Bowen and Arturo Casadevall.

Explore further: New study identifies gaps in NIH funding success rates for black researchers

Related Stories

Researchers claim NIH grant process is 'totally broken'

December 6, 2012

(Medical Xpress)—John Ioannidis, a researcher at Stanford University has, along with graduate student Joshua Nicholson, published a commentary piece in the journal Nature, taking the National Institutes of Health (NIH) ...

Should PCORI fund more primary care research?

February 11, 2016

The Patient-Centered Outcomes Research Institute (PCORI), established under the Affordable Care Act, is charged with funding research that ultimately helps patients make better-informed health care decisions. But some at ...

Recommended for you

An inflexible diet led to the disappearance of the cave bear

August 23, 2016

Senckenberg scientists have studied the feeding habits of the extinct cave bear. Based on the isotope composition in the collagen of the bears' bones, they were able to show that the large mammals subsisted on a purely vegan ...

Paleontologists discover major T. rex fossil (Update)

August 18, 2016

Paleontologists with the Burke Museum of Natural History and Culture and the University of Washington have discovered a Tyrannosaurus rex, including a very complete skull. The find, which paleontologists estimate to be about ...

Was 'Iceman Otzi' a Copper Age fashionista?

August 18, 2016

The 5,300-year-old Alpine mummy known as the Tyrolean Iceman died wearing leather clothes and accessories harvested from no less than five wild or domesticated species, a DNA analysis published Thursday revealed.

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.