Real benefits of scientific research measured

January 25, 2012 by Adele Rackley
Real benefits of scientific research measured

Scientists have developed a new way to measure the benefits of scientific research, by looking at the difference it has made to real-life issues.

In research circles these benefits are called 'impact': things like improvements in our health and life-expectancy, reduced losses from natural disasters, or simply more dollars in the bank from commercial exploitation of .

While impact is important, it can be very difficult to measure for a single piece of research. Traditional methods – like counting the number of times a published paper is 'cited' or referred to by other researchers – don't link the paper to its effect on the issue it's intended to address.

This new study shows it is possible to measure the impact of research publications on a defined objective – in this case wild-bee conservation.

The approach can also be used to direct future research, by identifying high-priority issues that remain unresolved, for example the benefit of increasing the proportion of natural habitat in farmed areas.

'This isn't the definitive answer to measuring impact,' says Professor William Sutherland of the University of Cambridge, who led the research, 'but it's an effective approach under certain circumstances.'

The authors of the paper, published in PLos One, prioritised various conservation measures that can be taken to benefit wild bees, then scored research papers according to their contribution to those measures.

'Working with an expert advisory board, the researchers identified 54 interventions that could benefit wild bee populations in the UK – such as increasing rough grassland for nesting, and providing conservation training to land managers – with 159 publications (papers, book chapters and theses) that tested the effectiveness of any intervention on that list.

A diverse group of people who apply to bee conservation, including policy-makers, ecologists, conservationists, academics and farm workers, then prioritised the interventions in terms of their likely effectiveness for bee conservation.

Three bee experts then assessed the evidence for each intervention and the contribution and relevance of each publication, including additions to knowledge about an intervention added by each subsequent paper.

Using models to adjust the scores for bias, the researchers arrived at impact scores for each publication based on their contribution to understanding of practical bee conservation issues.

This method meant the results could be plotted according to different factors, including total impact score, percentage contribution to knowledge, and relevance.

There are many problems with traditional means of measuring impact. It's rare that one piece of research will lead to impact by itself; more often it's the sum of many pieces of work over many years that produces applicable results. Impact might also be unexpected or unpredictable.

Another problem with traditional methods, as the authors of this paper point out, is that working out whether things actually change as a result of research is dependent on it being taken up by policy-makers or other users, which is based on factors quite separate from the quality or benefit of the research itself.

So this new qualitative method has many advantages.

'Rather than taking a research programme... as a starting point and asking what its contributions to society have been,' write the researchers, 'our approach takes the issues society wants answered as a starting point and asks how much each piece of research has contributed to answering them.'

The scoring identified a number of publications with high impact. The highest scoring research had evidence for one or more high-priority interventions from replicated, controlled trials.

Significantly, the scores for papers in this study had only a weak correlation with the impact factor of the journal they were published in, highlighting the problems with attributing real-life impact by traditional methods.

Because this new approach doesn't rely on take-up or publication statistics, and so allows for 'potential' or unrealised impact, it provides a more realistic assessment of the actual value of the research.

As Sutherland points out, this method isn't a silver bullet. It is more effective for applied rather than 'blue skies' research, because it depends on having an agreed set of possible solutions; and the impact score depends to an extent on who prioritises those solutions.

The literature review and scoring process is also time-consuming, but it could work well for subjects like climate change or medicine, for which syntheses of publications already exist.

Explore further: Fungus Foot Baths Could Save Bees

More information: Sutherland, WJ, Goulson, D, Potts, SG, Dicks LV (2011). Quantifying the impact and relevance of scientific research. PLoS One, 6(11): 327537. doi:10.1371/journal.pone.0027537

Related Stories

Fungus Foot Baths Could Save Bees

July 28, 2008

One of the biggest world wide threats to honey bees, the varroa mite, could soon be about to meet its nemesis. Researchers at the University of Warwick are examining naturally occurring fungi that kill the varroa mite. They ...

15 new conservation concerns

December 12, 2011

A review carried out by a group of international specialists has identified several emerging issues that are likely to damage biodiversity in the coming years.

Recommended for you

Just how good (or bad) is the fossil record of dinosaurs?

August 28, 2015

Everyone is excited by discoveries of new dinosaurs – or indeed any new fossil species. But a key question for palaeontologists is 'just how good is the fossil record?' Do we know fifty per cent of the species of dinosaurs ...

Fractals patterns in a drummer's music

August 28, 2015

Fractal patterns are profoundly human – at least in music. This is one of the findings of a team headed by researchers from the Max Planck Institute for Dynamics and Self-Organization in Göttingen and Harvard University ...

0 comments

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.