Real benefits of scientific research measured

In research circles these benefits are called 'impact': things like improvements in our health and life-expectancy, reduced losses from natural disasters, or simply more dollars in the bank from commercial exploitation of scientific discoveries.

While impact is important, it can be very difficult to measure for a single piece of research. Traditional methods – like counting the number of times a published paper is 'cited' or referred to by other researchers – don't link the paper to its effect on the issue it's intended to address.

This new study shows it is possible to measure the impact of research publications on a defined objective – in this case wild-bee conservation.

The approach can also be used to direct future research, by identifying high-priority issues that remain unresolved, for example the benefit of increasing the proportion of natural habitat in farmed areas.

'This isn't the definitive answer to measuring impact,' says Professor William Sutherland of the University of Cambridge, who led the research, 'but it's an effective approach under certain circumstances.'

The authors of the paper, published in PLos One, prioritised various conservation measures that can be taken to benefit wild bees, then scored research papers according to their contribution to those measures.

'Working with an expert advisory board, the researchers identified 54 interventions that could benefit wild bee populations in the UK – such as increasing rough grassland for nesting, and providing conservation training to land managers – with 159 publications (papers, book chapters and theses) that tested the effectiveness of any intervention on that list.

A diverse group of people who apply scientific research to bee conservation, including policy-makers, ecologists, conservationists, academics and farm workers, then prioritised the interventions in terms of their likely effectiveness for bee conservation.

Three bee experts then assessed the evidence for each intervention and the contribution and relevance of each publication, including additions to knowledge about an intervention added by each subsequent paper.

Using models to adjust the scores for bias, the researchers arrived at impact scores for each publication based on their contribution to understanding of practical bee conservation issues.

This method meant the results could be plotted according to different factors, including total impact score, percentage contribution to knowledge, and relevance.

There are many problems with traditional means of measuring impact. It's rare that one piece of research will lead to impact by itself; more often it's the sum of many pieces of work over many years that produces applicable results. Impact might also be unexpected or unpredictable.

Another problem with traditional methods, as the authors of this paper point out, is that working out whether things actually change as a result of research is dependent on it being taken up by policy-makers or other users, which is based on factors quite separate from the quality or benefit of the research itself.

So this new qualitative method has many advantages.

'Rather than taking a research programme... as a starting point and asking what its contributions to society have been,' write the researchers, 'our approach takes the issues society wants answered as a starting point and asks how much each piece of research has contributed to answering them.'

The scoring identified a number of publications with high impact. The highest scoring research had evidence for one or more high-priority interventions from replicated, controlled trials.

Significantly, the scores for papers in this study had only a weak correlation with the impact factor of the journal they were published in, highlighting the problems with attributing real-life impact by traditional methods.

Because this new approach doesn't rely on take-up or publication statistics, and so allows for 'potential' or unrealised impact, it provides a more realistic assessment of the actual value of the research.

As Sutherland points out, this method isn't a silver bullet. It is more effective for applied rather than 'blue skies' research, because it depends on having an agreed set of possible solutions; and the impact score depends to an extent on who prioritises those solutions.

The literature review and scoring process is also time-consuming, but it could work well for subjects like climate change or medicine, for which syntheses of publications already exist.

More information: Sutherland, WJ, Goulson, D, Potts, SG, Dicks LV (2011). Quantifying the impact and relevance of scientific research. PLoS One, 6(11): 327537. doi:10.1371/journal.pone.0027537

Journal reference: PLoS ONE

Provided by PlanetEarth Online

This story is republished courtesy of Planet Earth online, a free, companion website to the award-winning magazine Planet Earth published and funded by the Natural Environment Research Council (NERC).