Certainty in complex scientific research an unachievable goal

February 1, 2017, University of Toronto
Credit: Petr Kratochvil/Public Domain

A University of Toronto study on uncertainty in scientific research could shed light on anomalies that arose in early attempts to discover the Higgs boson and even how polls failed to predict the outcome of Donald Trump winning the U.S. presidential election.

Published recently in the journal Royal Society Open Science, the study suggests that research in some of the more complex scientific disciplines, such as medicine or particle physics, often doesn't eliminate uncertainties to the extent we might expect.

"This is due to a tendency to under-estimate the chance of significant abnormalities in results." said study author David Bailey, a professor in U of T's Department of Physics.

Looking at 41,000 measurements of 3,200 quantities - from the mass of an electron to the carbon dating of a sample - Bailey found that anomalous observations happened up to 100,000 times more often than expected.

"The chance of large differences does not fall off exponentially as you'd expect in a normal bell curve," said Bailey.

A long tail of uncertainty

"The study shows that researchers in many fields do a good job of estimating the size of typical errors in their measurements, but usually underestimate the chance of large errors," said Bailey, noting that the larger-than-expected frequency of large differences may be an almost inevitable consequence of the complex nature of .

"As measurements become more and more accurate, the smallest things matter more and more," Bailey said.

"If two measurements agree, you're happy. If not, you see there's something you need to investigate," he said. "You track down the cause of the variation and report the cause or you say that you don't know the cause and this reduces the trust in your result."

But with finite time and financial resources, researchers often have to make a choice between having a large sample of data, such as tens of thousands of people in a survey, and having a large number of variables you want to understand.

"You start with a very large sample that just lumps everyone together. You then might have to ask if your result is the same for both men and women. Is it the same for different backgrounds, Canadians versus Americans, for example," says Bailey. "At that point, you have to ask if your results hold for the smaller data set. Your sample is getting smaller and more can go wrong."

Impossible not to be a little wrong?

Physics studies did not fare significantly better than the medical and other research observed. However, the highly quantifiable way in which values and uncertainties are reported, may make physics more useful in terms of the degree of reproduce-ability of results that researchers should reasonably expect.

"Scientists will still aim for the most accurate results, but their expectations of how well those aims are met may be tempered in light of this research," said Bailey.

He believes his study can help researchers better analyze their data, motivate more care with novel results, and encourage more realistic expectations by both scientists and the public about the accuracy of scientific research.

"These insights can be beneficial given the inherently complex nature of scientific research," says Bailey. "But the chance of avoiding being wrong in some way on some level is almost impossible."

Explore further: Christmas Eve asteroid to cruise past harmlessly: astronomers

More information: David C. Bailey, Not Normal: the uncertainties of scientific measurements, Royal Society Open Science (2017). DOI: 10.1098/rsos.160600

Related Stories

There's more than one way to silence a cricket

May 29, 2014

For most of us, crickets are probably most recognizable by the distinctive chirping sounds males make with their wings to lure females. But some crickets living on the islands of Hawaii have effectively lost their instruments ...

Recommended for you

Muons spin tales of undiscovered particles

April 20, 2018

Scientists at U.S. Department of Energy (DOE) national laboratories are collaborating to test a magnetic property of the muon. Their experiment could point to the existence of physics beyond our current understanding, including ...

Integrating optical components into existing chip designs

April 19, 2018

Two and a half years ago, a team of researchers led by groups at MIT, the University of California at Berkeley, and Boston University announced a milestone: the fabrication of a working microprocessor, built using only existing ...

10 comments

Adjust slider to filter visible comments by rank

Display comments: newest first

Chris_Reeve
Feb 01, 2017
This comment has been removed by a moderator.
Hyperfuzzy
1 / 5 (2) Feb 01, 2017
No, certainty is achievable with correct physics; however, the more complex a system is the more computational power is required. So our uncertainty exist only in theory. Absolute certainty exist with an absolute truth in physics. A truth: Diametrical Spherical Fields are all that exists. Then ...
Chris_Reeve
Feb 01, 2017
This comment has been removed by a moderator.
Spaced out Engineer
3 / 5 (4) Feb 01, 2017
Keep rocking in the dialect Hyperfuzzy. Physics is model based realism. Reality may be surreal in a manner we cannot understand. Bosonic sampling and the pilot wave seem to indicate so, but again that is still a choice feature regardless if it is extracted from reality or selected from inner categories. Just because a model using shapeness is unfalsifiable and works as a useful approximation, it does not make it certain. Non-local correlates only get you so close to synchronous non-locality, and because of uncertainty the name of the game is loosing in ever refined contexts, hopefully gaining near optimal marginal utility. You don't win, you just don't absolutely lose as you can always toss up the game. It seems as though completeness consistency, decidability, objectivity, independence, and determinism are reserved for Nature. There is a healthy balance between psychosis and pragmatic model based realism from an existential stance. Logical positivism is dead.
Whydening Gyre
not rated yet Feb 01, 2017
Keep rocking in the dialect Hyperfuzzy. Physics is model based realism. Reality may be surreal in a manner we cannot understand. ... . It seems as though completeness consistency, decidability, objectivity, independence, and determinism are reserved for Nature. There is a healthy balance between psychosis and pragmatic model based realism from an existential stance. Logical positivism is dead.

Does that mean - Long live logical negativism...?
Hyperfuzzy
5 / 5 (1) Feb 01, 2017
Re: "No, certainty is achievable with correct physics; however, the more complex a system is the more computational power is required."

What effort have you actually put into actually checking that this is correct? How are you ruling out competing ideas? If you have no process at work to do this, then your answer is meaningless.

I've served as a EE for over 40 years. Accuracy and precision is based upon what is known. Typically proper calibration is quite adequate. However, with Maxwell, the answers are accurate. The fields are accurate with absolute precision when everything is properly simulated. The imprecision is based upon our inadequacies and has nothing to do with proper physics.
ddaye
not rated yet Feb 01, 2017
There are no exotic phenomena in the Trump win. Polling on his national vote totals was quite close, and expertise much more mundane than complexity science can explain any residual questions.
BubbaNicholson
1 / 5 (1) Feb 02, 2017
I've found that any man can use his face grease to cure heroin addiction without any withdrawal symptoms. Just rub it off onto something the addict can put in his/her mouth. 250mg given by mouth is a flat out, unequivocal cure. Medicine need not be vague and an understanding of the facts does yield deliverable remedy. Subtleties are the province of incompetents.
Hyperfuzzy
not rated yet Feb 02, 2017
There are no exotic phenomena in the Trump win. Polling on his national vote totals was quite close, and expertise much more mundane than complexity science can explain any residual questions.

Any IT could hack this, really? You believe this was a fair election?
Spaced out Engineer
3 / 5 (2) Feb 04, 2017
Whydening-"Does that mean - Long live logical negativism...?"
That depends on your interpretation of ZFC, the empty set, or perhaps the holographic principle. Formal systems seem to imply mind moves. Objectivity gets vague. There are no laws. There are effective theories. We can only be so certain of the ontology of the waveform and it is contextually derived. Long live transactional complexity. Reality is far simpler than we can imagine.

Hyperfuzzy- Reality is not Maxwell's equations. It would be cool if we could fold the entire first order to encode it on those symmetry groups. This equations are a useful approximation of electromagnetism. Electrons tunnel and light has nonlinear effects we are just discovering.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.