Real benefits of scientific research measured

Jan 25, 2012 by Adele Rackley
Real benefits of scientific research measured

Scientists have developed a new way to measure the benefits of scientific research, by looking at the difference it has made to real-life issues.

In research circles these benefits are called 'impact': things like improvements in our health and life-expectancy, reduced losses from natural disasters, or simply more dollars in the bank from commercial exploitation of .

While impact is important, it can be very difficult to measure for a single piece of research. Traditional methods – like counting the number of times a published paper is 'cited' or referred to by other researchers – don't link the paper to its effect on the issue it's intended to address.

This new study shows it is possible to measure the impact of research publications on a defined objective – in this case wild-bee conservation.

The approach can also be used to direct future research, by identifying high-priority issues that remain unresolved, for example the benefit of increasing the proportion of natural habitat in farmed areas.

'This isn't the definitive answer to measuring impact,' says Professor William Sutherland of the University of Cambridge, who led the research, 'but it's an effective approach under certain circumstances.'

The authors of the paper, published in PLos One, prioritised various conservation measures that can be taken to benefit wild bees, then scored research papers according to their contribution to those measures.

'Working with an expert advisory board, the researchers identified 54 interventions that could benefit wild bee populations in the UK – such as increasing rough grassland for nesting, and providing conservation training to land managers – with 159 publications (papers, book chapters and theses) that tested the effectiveness of any intervention on that list.

A diverse group of people who apply to bee conservation, including policy-makers, ecologists, conservationists, academics and farm workers, then prioritised the interventions in terms of their likely effectiveness for bee conservation.

Three bee experts then assessed the evidence for each intervention and the contribution and relevance of each publication, including additions to knowledge about an intervention added by each subsequent paper.

Using models to adjust the scores for bias, the researchers arrived at impact scores for each publication based on their contribution to understanding of practical bee conservation issues.

This method meant the results could be plotted according to different factors, including total impact score, percentage contribution to knowledge, and relevance.

There are many problems with traditional means of measuring impact. It's rare that one piece of research will lead to impact by itself; more often it's the sum of many pieces of work over many years that produces applicable results. Impact might also be unexpected or unpredictable.

Another problem with traditional methods, as the authors of this paper point out, is that working out whether things actually change as a result of research is dependent on it being taken up by policy-makers or other users, which is based on factors quite separate from the quality or benefit of the research itself.

So this new qualitative method has many advantages.

'Rather than taking a research programme... as a starting point and asking what its contributions to society have been,' write the researchers, 'our approach takes the issues society wants answered as a starting point and asks how much each piece of research has contributed to answering them.'

The scoring identified a number of publications with high impact. The highest scoring research had evidence for one or more high-priority interventions from replicated, controlled trials.

Significantly, the scores for papers in this study had only a weak correlation with the impact factor of the journal they were published in, highlighting the problems with attributing real-life impact by traditional methods.

Because this new approach doesn't rely on take-up or publication statistics, and so allows for 'potential' or unrealised impact, it provides a more realistic assessment of the actual value of the research.

As Sutherland points out, this method isn't a silver bullet. It is more effective for applied rather than 'blue skies' research, because it depends on having an agreed set of possible solutions; and the impact score depends to an extent on who prioritises those solutions.

The literature review and scoring process is also time-consuming, but it could work well for subjects like climate change or medicine, for which syntheses of publications already exist.

Explore further: Rosetta comet-landing is Science's 2014 breakthrough

More information: Sutherland, WJ, Goulson, D, Potts, SG, Dicks LV (2011). Quantifying the impact and relevance of scientific research. PLoS One, 6(11): 327537. doi:10.1371/journal.pone.0027537

Journal reference: PLoS ONE search and more info website

Provided by PlanetEarth Online search and more info website

4.9 /5 (8 votes)

Related Stories

15 new conservation concerns

Dec 12, 2011

A review carried out by a group of international specialists has identified several emerging issues that are likely to damage biodiversity in the coming years.

Scientists call for cost-effective conservation

Jul 14, 2011

Britain could get more benefit from its conservation budget if it paid more attention to how effective and cost-efficient previous efforts to protect biodiversity have proved.

Fungus Foot Baths Could Save Bees

Jul 28, 2008

One of the biggest world wide threats to honey bees, the varroa mite, could soon be about to meet its nemesis. Researchers at the University of Warwick are examining naturally occurring fungi that kill the varroa mite. They ...

Recommended for you

Study: Alcatraz inmates could have survived escape

Dec 17, 2014

The three prisoners who escaped from Alcatraz in one of the most famous and elaborate prison breaks in U.S. history could have survived and made it to land, scientists concluded in a recent study.

User comments : 0

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.