Using statistics ethically to combat 'a scientific credibility crisis'

Using statistics ethically to combat 'a scientific credibility crisis'
A recent survey suggests that many researchers have tried and failed to reproduce other scientists' experiments as well as their own. Georgetown University's Rochelle Tractenberg, Ph.D., is on a mission to help change that by promoting ethical statistical practices. She'll lead a panel discussion on the topic Sunday at the AAAS meeting in Boston. Credit: Georgetown University

Can statistics increase the value of science to society? Georgetown University's Rochelle Tractenberg, PhD, MPH, PhD, chair of the Committee on Professional Ethics of the American Statistical Association, will discuss "Promoting Ethical Science and Policy With Ethical Statistical Practice" on a panel presenting three disciplinary perspectives on Sunday, February 19, as part of a day long symposium titled Scientific Reproducibility and Social Responsibility at the AAAS 2017 Annual Meeting in Boston.

A survey of more than 1,500 investigators, published in a 2016 issue of Nature, showed that more than 70 percent of researchers have tried and failed to reproduce other scientists' experiments, and more than half have failed to reproduce their own experiments.

Additional studies have come to similar conclusions, says Tractenberg, associate professor of neurology at Georgetown University Medical Center with secondary appointments in biostatistics, bioinformatics and biomathematics, and rehabilitation medicine. "Irreproducible results do harm that can be difficult to discover and even more difficult to undo," she said.

A consulting statistician and practicing scientist for the past 20 years, Tractenberg started pursuing her interest in promoting ethical research skills in 2009 after being invited to join a GUMC task force to explore these challenges. Tractenberg has worked with colleagues at GUMC and other institutions around the country to promote this brand of responsible research, and she'll have a larger stage later this month.

Tractenberg will discuss responsible research, and its relevance for all statisticians, data analysts and data scientists, in a symposium she organized for the AAAS 2017 Annual Meeting.

Credibility Crisis

When most investigators have taken just a single course in statistics, and are therefore laymen when it comes to statistics - for typical experiments or if they wish to participate in big data analyses - it is perhaps not surprising that so many studies cannot be replicated, nor results reproduced, Tractenberg says.

"My focus on promoting ethical statistical practice arose because a scientific credibility crisis is emerging due partly to scientists who do not conduct - or insist upon - appropriate statistical analysis or interpretation, or both," she says. "If ethical statistical practice becomes the norm across statistics and data science, it may then be taken up into other domains where data analysis makes important contributions."

Several elements of a study can lead to irreproducible results, including incorrect analysis, improper interpretation of data, cherry picking results, or failing to transparently report the number of analyses that were done, Tractenberg says. Avoiding these are principles of ethical statistical practice as well as responsible conduct in research.

"Although it can often seem that data analysis is secondary to the 'main' science or study purpose, the analytic method and its interpretation are essential attributes of both rigor and reproducibility, and this is true for their own work and for their peer review of others' work," says Tractenberg.

A large number of these irreproducible studies may have never been published if peer reviewers that were unable to evaluate the statistics "just told the editor they don't feel qualified to evaluate the study's statistical argument, and that a formal statistical review is needed," she says. Having a formal statistical review does not guarantee reproducibility or rigor, but not having or insisting on one virtually guarantees the continuation of the reproducibility crisis.

"Treat Data Ethically"

A faculty member at Georgetown since 2002, Tractenberg was appointed to the national Committee on Professional Ethics of the American Statistical Association (ASA) in 2013, a committee that she now chairs. In her 90-minute panel at the AAAS meeting, "How Ethical Science Supports Ethical Policy: Disciplinary Perspectives," she will discuss the ASA Ethical Guidelines for Statistical Practice, which all those who analyze data can utilize, whether dealing with "small" or "big" data. She says that ethical statistical practice - by every data analyst - is integral to maintaining the value of science in society.

Tractenberg's panel will also bring together specialists in engineering and economics to describe their efforts to establish and promote ethical practices and policies within their disciplines. These three perspectives will then be discussed with respect to their potential to influence and support ethical policy and decision making.

"All scientific fields have different relationships to data and how the data should be interpreted," Tractenberg says. "But the core of all in this work is the data and its analysis, and I firmly believe these must be dealt with ethically.

Otherwise, decisions that are based on these results may be incorrect or indefensible, or both."

"The data analyst, whether a professional statistician or just the group member who is most skilled with the analysis software, has an obligation to treat and interpret the ethically," Tractenberg says. "In a post-truth world, this may be the best way to promote scientific integrity."

Explore further

American Statistical Association releases statement on statistical significance and p-values

Citation: Using statistics ethically to combat 'a scientific credibility crisis' (2017, February 19) retrieved 19 July 2019 from
This document is subject to copyright. Apart from any fair dealing for the purpose of private study or research, no part may be reproduced without the written permission. The content is provided for information purposes only.

Feedback to editors

User comments

Feb 20, 2017
Publish, Publish, Publish, write grant application, and repeat.

We've created an academic industry where success is measured not in scientific truth or the search of it, but in the number of publications and grants. The peer review process is no longer to weed out bad papers, but to advance careers and enforce a dogma.

Data, what is that ? Surely just provide a data file, no-one will check it anyways.

Feb 27, 2017
Hi FactsReallyMatter. :)

As I mentioned in that other thread, I too have been challenging astronomical/cosmological data acquisition/analysis based on methodologies depending on naive/simplistic 'data/maths. processing/techniques which are highly vulnerable to 'artifacts' and other serious GIGO potential.

So I am in agreement with the above PO article thrust/caution, and also your own above observations/cautions re big data handling etc etc without proper checks and balances NOT dependent on in-built naive/simplistic assumptions/treatments not fit for purpose enough to conclude any tenable level of 'confidence' when making claims.

If you choose your further 'battles' and challenges with the same care that you (and I) require from all true scientists, then your username will be more apt and less open to serious criticism. Good luck, good thinking and good science to you, mate! :)

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more