Is the p-value pointless?

March 17, 2016 by Lauren Richardson, Plos Blogs
Is the p-value pointless?
This Figure, from the Head et al., "Perspective on p-hacking," shows that p-hacking alters the distribution of p-values in the range considered “statistically significant”

For the first time in its 177-year history, the American Statistical Association (ASA) has voiced its opinion and made specific recommendations for a statistical practice. The subject of their ire? The (arguably) most common statistical output, the p-value. The p-value has long been the primary metric for demonstrating that study results are "statistically significant," usually by achieving the semi-arbitrary value of p
ce of the p-value has been greatly overstated and the scientific community has become over-reliant on this one – flawed – measure.

In the associated article, published in The American Statistician, Ronald Wasserstein and Nicole Lazar explain how the dependence on the p-value threatens the reproducibility and replicability of research. Importantly, the p-value does not prove that scientific conclusions are true and does not signify the importance of a result. As Wasserstein says in the ASA press release, "The p-value was never intended to be a substitute for scientific reasoning."

As documented in a 2015 PLOS Biology Perspective by Megan Head, Michael Jennions and colleagues, the p-value is subject to a common type of manipulation known as "p-hacking," where researchers selectively report datasets or analyses that achieve a "significant" result. The authors of this Perspective used a text-mining protocol to reveal this to be a widespread issue across multiple scientific disciplines. The authors also provide helpful recommendations for researchers and journals.

The problem with the p-value cuts both ways. Over-interpretation of the p-value can lead to both false positives and false negatives. Dependence on a specific p-value can lead to bias as researchers may discontinue or shelve work that doesn't meet this arbitrary standard.

The hope is that the ASA statement will increase awareness of the problems of inappropriate p-value use persisting in the scientific practice. Their guidelines can aid researchers in determining the best practices for the use of the p-value, and help identify when other statistical tests are more appropriate.

Explore further: American Statistical Association releases statement on statistical significance and p-values

Related Stories

Scientists unknowingly tweak experiments

March 18, 2015

A new study has found some scientists are unknowingly tweaking experiments and analysis methods to increase their chances of getting results that are easily published.

Experimental economics: Results you can trust

March 3, 2016

Reproducibility is an important measure of validity in all fields of experimental science. If researcher A publishes a particular scientific result from his laboratory, researcher B should be able to follow the same protocol ...

Recommended for you

Experts uncover hidden layers of Jesus' tomb site

October 27, 2016

In the innermost chamber of the site said to be the tomb of Jesus, a restoration team has peeled away a marble layer for the first time in centuries in an effort to reach what it believes is the original rock surface where ...

Important ancient papyrus seized from looters in Israel

October 27, 2016

(—Eitan Klein, a representative of the Israel Antiquities Authority, has announced that an important papyrus document dated to 2,700 years ago has been seized from a group of Palestinian looters who reportedly ...

Ancient parrot fossil found in Siberia

October 26, 2016

(—A Russian paleontologist has discovered a parrot fossil uncovered in Siberia several years ago—the first evidence of parrots living in Asia. In his paper published in Biology Letters, Nikita Zelenkov describes ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Apr 20, 2016
Why is it that I could recognize this problem immediately, back in the day as a first-semester student in Statistics, but for almost 200 years the "best and brightest" in Mathematics and the sciences could or did not? (I'm not claiming to be smarter than others; but likely I am less impressed with "authority" or "bull-sh*tting" than most.)

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.