Scientific evidence: What is it and how can we trust it?

Jul 04, 2013 by Manu Saunders
When presented with evidence, can you read the science behind it? Credit: funkandjazz

The phrase "scientific evidence" has become part of the vernacular – thrown about like a hot potato during discussions of major environmental, health or social issues. Climate change is one example. The EU's ban on neonicotinoid pesticides is another.

We've heard numerous mentions of the associated "evidence", indicating the importance of the issue and the need for action. This evidence is presented by proponents in much the same way that evidence is given in a court case, usually to back up policies or decisions that will impact people's lifestyles. But, unlike in a court case, we are rarely told exactly where the evidence comes from and why it's evidence.

Scientific evidence is information gathered from scientific research, which takes a lot of time (and patience!) to conduct. But there are a few things that all this research needs to have in common to make it possible for decision-makers, and ultimately all of us, to accept it as "evidence".

Objective and unbiased

Research needs money to pay for laboratory equipment, , and materials – not to mention the wages of all the people involved in the project. And money certainly doesn't appear out of thin air, even around the department!

The majority of researchers have to constantly apply for funds to carry out their research. These funds can come from different places, usually government bodies such as the Australian Research Council (ARC), academic or , non-profit organisations or even industry bodies. Applications are judged on scientific merit and their relevance to society or the funding body's interests.

Mostly, funds are distributed fairly. But if an organisation funds a research project that will benefit them financially, then we cannot accept the findings as "evidence" unless different researchers (from unrelated organisations) come to the same conclusions through their own independent research.

Ensuring results will be valid and accurate

Scientific evidence relies on data, and it is crucial for researchers to ensure that the data they collect is representative of the "true" situation. This means using proved or appropriate ways of collecting and analysing the data and ensuring the research is conducted ethically and safely.

Control scenarios may also be necessary when testing for effects or impacts – such as when developing new products (such as medicines), or evaluating management actions (such as farmland pesticide use). The control scenario represents the opposite of the scenario being tested. This is so the results that are seen in the test scenario are guaranteed to be from the tested product or impact, and nothing else.

If the scenario involves environmental processes of some kind, the test and control should ideally be carried out under natural conditions (or in an environment where these processes normally occur).

Sometimes this can be virtually impossible to do, and lab-based or combined lab/ field studies will need to be done instead so the "nuisance factors" can be controlled.

Take the recent neonicotinoid issue. If a researcher wants to prove that use of a pesticide does not affect bees flying about in the environment where the chemical is normally used, they will need to test two different scenarios.

One hive of bees will have to go about their business out in the field while being exposed to the pesticide. A second hive of bees will have to be in the same general environmental location as the first hive (to ensure both hives experience the same overall living conditions), but remain completely uncontaminated by the pesticide throughout the test.

It's obvious how impossible this would be to manage under natural conditions, where no one can control the drift of chemical droplets or the movement of tiny insects across the landscape! In this case, completely field-based studies may not exist, but it would be misleading to say that a "lack of field studies" means that the pesticide does not affect bees.

Peer-review and professional consensus

This step is the most crucial, and it turns research into the "evidence" that we all talk about. The researcher has to present their data, results and conclusions in the form of a scientific report or paper. This must be reviewed by their scientific peers – only they are qualified to assess the validity of the methods and the accuracy of the conclusions the researcher has drawn from the results.

Oscars are decided by international film industry professionals. Similarly, having research findings published in an international peer-reviewed journal essentially means that other professional scientists who specialise in that kind of research have verified the quality and validity of the research.

This process takes a long time – from submission of the manuscript to a journal, to the final publication date can take six months to a year, often longer.

For really important decisions, especially ones that will affect lots of people (how we should manage our national parks, for example), multiple studies may need to be sourced to show that a majority of scientists experienced with the issue agree on the evidence (just like a jury in a court case).

This is to show there is a "scientific consensus" on the evidence, and it provides even more reason for taking action on the issue at hand.

Of course, not everyone agrees on everything – think of any topic from the Earth being round, to what you and your family will eat for dinner tonight! So if a few scientists disagree with the majority group of scientists over a particular issue, that is not immediate proof that the evidence is wrong, and neither is it shocking or newsworthy.

After all, we don't talk about the Best Actor Academy Award nominee who got the fewest votes.

Interpreting the evidence presented to us

Most of us hear of "scientific evidence" from journalists, newsreaders, politicians or media commentators, and often we don't have the opportunity to check the facts ourselves. But understanding where true scientific evidence comes from, and what it means, is imperative to helping us tackle the most important issues affecting our own lives and the world we live in.

So the next time someone says they have "scientific evidence" to back up their case, ask a few questions. Who funded the research and why? How much evidence is there and how was it gathered? Was the sample size or location representative of the "real" situation?

Has the research been published in an internationally-accepted, peer-reviewed journal, or is it only available online on a personal or organisation's website? Do a majority of other scientists agree on these results? If a few disagree, are they qualified to evaluate the issue? (For example, a medical doctor and an astronomer are both scientists – but that doesn't mean the astronomer is qualified to perform heart surgery!)

And if someone claims there is a "lack" of on a contested issue, ask them to clarify. Do they mean that peer-reviewed research has been carried out, and found no proof of an effect? Or, do they mean that no one has yet funded research to examine the issue? These do not mean the same thing – as the saying goes:

Absence of evidence is not evidence of absence!

Explore further: What I learned from debating science with trolls

add to favorites email to friend print save as pdf

Related Stories

Climate change already affecting UK wildlife

May 09, 2013

UK wildlife is already feeling the effects of climate change, scientists say. According to a report out on Thursday, with input from many of the UK's top environmental scientists, many species are now found ...

Bees survival: Ban more pesticides?

May 03, 2013

Neonicotinoids are under intense scrutiny. But a ban of a broad variety of pesticides may be required to protect bees, humans and the environment.

Recommended for you

Freedom and responsibility of science

5 hours ago

Yesterday, the German Research Foundation (DFG) and the Leopoldina National Academy of Sciences presented their recommendations for "The Freedom and Responsibility of Science" in Berlin. Both research organizations appeal ...

What I learned from debating science with trolls

Aug 20, 2014

I often like to discuss science online and I'm also rather partial to topics that promote lively discussion, such as climate change, crime statistics and (perhaps surprisingly) the big bang. This inevitably ...

Activists urge EU to scrap science advisor job

Aug 19, 2014

Nine major charities urged the European Commission on Tuesday to scrap a science advisor position it says puts too much power over sensitive policy into the hands of one person.

More to a skilled ear in music

Aug 15, 2014

The first pilot study in Australia to give musicians the skills and training to critically assess music by what they hear rather than what they see begins this month at the Sydney Conservatorium of Music.The study aims to ...

User comments : 16

Adjust slider to filter visible comments by rank

Display comments: newest first

julianpenrod
1 / 5 (10) Jul 04, 2013
The fact is, there is a crucial step missing here, which demonstrates the attitude of "science" toward what they claim to be doing.
That step is proving the claims to the "rank and file"!
That is left out. "Science" simply says, "Believe what we tell you to believe! You don't need proof! The fact we tell you so is all the 'proof' your kind need!"
There is no evidence peer review takes place, there is no evidence even the "results" "scientists" claim actually come from "experiments" they performed! Shills say doubters can perform the "experiments" themselves, but even the article says those "experiments" are incredibly expensive. "Science" feels secure the public can't see their claims are untrue! And nothing really "proves" "science". There is no evidence electricity isn't tiny aliens carrying energy. Shills scoff, but never once do they provide proof!
julianpenrod
1.4 / 5 (10) Jul 04, 2013
In fact, frauds in "science" are legion.
All those who try to get some notoriety claiming to have derived the "secret" that everything from earthquakes to "terrorist" events follow "power laws".
To follow a "power law", one value varies as a constant power of another. To "prove" this, the logarithms of both variables are taken and, if their graph is a straight line, they follow a power law.
But logarithms "squash"data. The strangest shape, squeezed into a tight space, can look like a straight line.
And the "proof" the data are a straight line is that a linear regression is performed. A linear regression only gives the straight line that comes closest to all data points, it does not verify that those points form a straight line! But the majority don't know this! The result is an elaborate swindle, misrepresenting facts and depending on lack of understanding of the world and gullibility on the part of a specific target audience.
ValeriaT
1.5 / 5 (8) Jul 04, 2013
You should face ugly truth: our universe isn't nicely flat and homogeneous, being curved into black holes occasionally and the scientific evidence, which works well under common circumstances is often terribly biased. Actually the more, the more such an evidence could affect the jobs and salaries of scientists involved, because the contemporary science is occupation driven. It work in both directions: at the case of cold fusion all the evidence is ignored, at the case of another theories the absence of evidence is overlooked obstinately. The delaying of acceptation of reality doesn't help there, as it just makes situation even more and stance of scientists more stubborn.
ValeriaT
1 / 5 (6) Jul 04, 2013
The similar situation we can observe at Wikipedia: whereas the politically or culturally neutral articles are of high degree of relevance, those affected with social connections are often heavily manipulated instead. The public access and democracy aren't self-correctable under all circumstances - they can be source of bias as well.
julianpenrod
1 / 5 (7) Jul 04, 2013
"Psychologist" Gary Wells jumped on the bandwagon of "faulty witness identification". He concocted a "theory" about "photo lineups", individuals looking through books of known offenders, being illegitimate because, purportedly, witnesses tend to take pieces from different individuals and combine them unconsciously in their mind. Wells' "solution" was to require witnesses to stare for up to ten minutes at one picture after another and see if they recognize the individual. No mention of the fact that impressionable witnesses not aware this is a scam being led to believe their memory is faulty! As a result, they begin to see blemishes that they don't remember and question themselves. Also, it's tedious and many give up and don't try to recognize anyone. The "news" proclaimed that Wells' method helped solve crimes, but it only "decreased the number of false id's". But, if a witness stops and doesn't even try to id someone, there will be fewer id's and so fewer false id's!
julianpenrod
1 / 5 (8) Jul 04, 2013
Psychologist Margaret Beale constructed a characteristically fraudulent "experiment".
It was geared toward a politically approved finding, namely, proving that whites have "brainwashed" blacks from birth into valuing white skin more and thinking anything black is bad.
She constructed an "experiment" that, to the superficial and shallow, appears to say that, but, in fact, is based on other things entirely. She had the same cartoon child printed in different colors, so the only difference was the color. She then asked children which child was happy, smart, attractive, and so on. She didn't ask them their inherent thoughts. She told them what to think and left them to decide which figure they meant. It was already established that kids are attracted by bright colors, and fearful of dark ones. As a result, when told one of the figures was "bad" they "concluded" it had to be the dark one! A lie, a fraud, a swindle!
julianpenrod
1 / 5 (7) Jul 04, 2013
In much the same manner as Margaret Beale Spencer, "researchers" ordered by crooked New Jersey politicos to "prove" that cell phone use causes car accidents concocted an "experiment" that did just that. Crucial to this was their "definition" that "cell phone use caused an accident if it was used within ten minutes of the accident occurring". As a result, someone could take a call at a restaurant, get in their car, drive out, then be plowed into by the spaced out kid of a politician, and the driver with the cell phone will be blamed! This has the added crooked benefit of "exonerating" hard drug use from being blamed for car accidents!
julianpenrod
1.4 / 5 (9) Jul 04, 2013
Michael Crichton, in "The Andromeda Strain", proposed the "Rule of 48" that said, "All scientists are blind". The "explanation" being that when human chromosomes were imaged "scientists" thought there were 48 then, supposedly, recounted and found 46. This takes advantage of the lack of understanding of many as to what really is involved. They never imaged all the chromosomes at once, they imaged differently stained ones from different samples! So it wasn't a matter of counting the figures in the photograph! It was more a matter of drawing "conclusions" based on unwarranted things, such as chimps and gorillas have 48, therefore "concluding" humans must have 48.
julianpenrod
1 / 5 (9) Jul 04, 2013
And, in the area of DNA, so many without an understanding of the world are swindled into the confidence in DNA matching of suspects in crimes. A strip with colored bands is shown next to another strip with colored bands, the public are told "The DNA match". In fact, that is far from reliable! The basis for "DNA matching" is not correlating every gene from a sample to a person! There are trillions of genes, but "DNA analysis" only looks at eleven! One would think the craven reason is only that it's because those are simpler or less expensive rather than that they are the easiest to con! And "identifying" a suspect comes only from the calculated "odds" that two or more people would have that combination of markers. But they use unproved independent probabilities to calculate the "odds". They never assembled a full database of DNA to see if there might be hidden forces that cause two markers to combine more often than independent probability says!
Ojorf
1 / 5 (1) Jul 05, 2013
OMG
antialias_physorg
5 / 5 (3) Jul 05, 2013
Most important part of the article.
This must be reviewed by their scientific peers – only they are qualified to assess the validity of the methods and the accuracy of the conclusions the researcher has drawn from the results.

Let me stress the ONLY in this sentence.

And by this simple detail are (almost) all comments on this site regarding the research presented in the articles invalidated.
ryggesogn2
1.5 / 5 (8) Jul 05, 2013
only they are qualified to assess the validity of the methods

But those 'peers' do NOT actually duplicate the work and validate results.
And when a hybrid field like climate science emerges with only a handful of climate scientists, are they the ONLY peers qualified?
Modeling and Simulation is relatively new discipline with a few PhDs.
One of the first is John A. Sokolowski. His adviser was a computer scientist. Who would be his peers if only M&S PhDs can review?
This is a paper by Sokolowski and Banks. Banks PhD is in political science. How is she qualified to comment on M&S?
http://www.scs.or...anks.pdf
antialias_physorg
5 / 5 (4) Jul 05, 2013
Modeling and Simulation is relatively new discipline with a few PhDs.

Hello? Rip van Winkle? I think you missed quite a few decades worth of scientific work if you think that.

with only a handful of climate scientists

Erm - try a few tens of thousands.
PC76
1.8 / 5 (5) Jul 05, 2013
Question,
Back ground research,
construct hypothesis,
experiment to test hypothesis,
analyze experiment data,
draw conclulsion,
report out to community

If you have not done all of these, then you are not doing science. You are likely just writing a paper to justify spending grant money. Which, sadly, is majority of what passes for science today.
ryggesogn2
1.5 / 5 (8) Jul 05, 2013
Modeling and Simulation is relatively new discipline with a few PhDs.

Hello? Rip van Winkle? I think you missed quite a few decades worth of scientific work if you think that.

with only a handful of climate scientists

Erm - try a few tens of thousands.

A significant critique of Mann's hockey stick was the incestuousness of the peers.
antialias_physorg
4 / 5 (1) Jul 06, 2013
Question,
Back ground research,
construct hypothesis,
experiment to test hypothesis,
analyze experiment data,
draw conclulsion,
report out to community

If you have not done all of these, then you are not doing science. You are likely just writing a paper to justify spending grant money. Which, sadly, is majority of what passes for science today.

Not really, since the method you describe is known to the peers. And if the paper doesn't contain all these then it won't pass peer review.

(There's one expection: The first paper on a project is usually a "Here's the plan what we're going to do in the next 3 years"-paper...which does often pass peer review even though it does not contain the 'data collection' and 'experiment' parts but is usually rather heavy on the background research, motivation, analysis and hypothesis parts. It is important in its own right since it lets people know what you do and often leads to collaborations.)