Research impact can be measured through case studies

The large majority of Australian university research projects have considerable impact on society, according to new findings from the sector.

The Australian Technology Network of Universities (ATN) and the Group of 8 (Go8) have launched the results of the Excellence in Innovation for Australia Trial.

The trial was designed to measure research impact, asking researchers from nine ATN and Go8 universities to present case studies to a panel of industry and academic experts. Charles Darwin, Newcastle and the University of Tasmania also participated.

Seven panels assessed 162 case studies in terms of their economic, social and environmental benefit. Of these, 87% were deemed to have had considerable, very considerable or outstanding impact.

Professor Attila Brungs, Deputy Vice-Chancellor for Research at UTS, which participated in the trial, provides his thoughts on how it went below.

What is the Excellence in Innovation for Australia Trial (EIA)?

EIA is a trial to better understand and evaluate the broad impact of research that comes out of universities. Governments and universities use a number of tools to assess research excellence. These are often based on traditional academic metrics, such as publications and less traditional metrics such as creative works, design objects, exhibitions and patents.

While determining and supporting research excellence is critical, it is also vital that we develop tools to help us recognise and articulate the truly significant ways that university research impacts on, and benefits, our society, our environment and our economy.

This was tried in 2004-2005, then abandoned by the Rudd government, then picked by the UK and we are back doing it again. What makes you think it will make a difference this time?

Assessing broad ranging impact is quite a challenge. However, since 2004 four things have changed.

The first is that the methodology has matured and improved since it was first trialled in 2004. We have learned from how a similar assessment exercise in the UK was developed, and with their support subsequently addressed some of the problematic parts of the process.

This sort of exercise is quite complex and in parts becomes unavoidably subjective; for example, how could the benefit of a vaccine to cure a chronic disease be compared to the impact a new mineral process which doubles the amount of mine-able ore in Australia? Being able to generate reproducible results, which was part of this trial, is an important step forward.

Second, Australia is more sophisticated now in its approach to the evaluation of research outcomes and its appreciation of the broader impact that research has on society, including areas such as productivity and innovation, as a core part of those outcomes.

Given the great work by the Australian Research Council (ARC) developing a national research excellence assessment process, the implementation of CSIRO's impact driven Flagship Program plus the more strategic mindset evidenced in reports such as Strategic Framework for Research Infrastructure Investment, I think that the university sector and the government is now in a better position to tackle the impact questions. There is now more clarity on the importance of measuring both research excellence and research impact to deliberately foster both.

Third, I think the value in an exercise like this is that it produces easily understood cases and examples that demonstrate the benefit of research and the very real return the public gets from the investment of their taxpayer dollars into universities and research. Perhaps in 2004, there was a mistaken belief by government that everybody was familiar with the direct impact research and universities have on their everyday lives.

Finally, this trial is an initiative instigated by the bulk of the university sector rather than a government initiative (though supported by the government). Twelve universities – the 5 ATN (Curtin, RMIT, UniSA, UTS and QUT), 4 of the Go8 (UoM, UQ, UWA, UNSW) and Newcastle, University of Tasmania and Charles Darwin University all collaborated together on the design and implementation of the trial. By advancing the discussion on understanding impact, it brings national attention to appropriate incentive mechanisms to encourage the continued engagement of universities in delivering impact from research for society. As this delivery almost always includes partners outside universities, this will also foster collaboration between universities, other research providers, industry and government.

And how did it the trial go?

Overall, better than we could have hoped. It was a year-long effort, and required a considerable amount of goodwill and cooperation by all the participating universities and the members of the assessment panels. I think it brought together an unprecedented number of people it from Industry, Government, Community groups and Universities. The majority of the assessment panel constituents were individuals from outside the university sector, senior and busy people who generously gave their time.

I think this fact alone demonstrates how important the trial and its outcomes were to a wide range of people in sectors right across Australia. There were, of course, many challenges and important lessons learned during the trial. These lessons will be useful in guiding the next part of the conversation around impact assessment.

What do the results of the trial show?

They show three major findings. The first is that more than 87% of the case studies were assessed as having considerable, very considerable or outstanding impact an international, not just a national, level. That's pretty good. It demonstrates that high quality research carried out in Australian universities has had enormous benefits for health, security, prosperity cultural and environmental wellbeing of Australia.

The second finding is that EIA has shown us that it is possible to do this sort of impact assessment exercise. The case study methodology used, while it has problems, was determined to be sufficiently robust (across a spread of disciplines) by the expert panels involved in EIA. It was the finding of the panels that this approach could be a rigorous basis for an impact assessment approach.

Third, it was very useful having industry, government and university people involved, so having a mixed panel and combining the expertise and perspectives of people from a range of professional backgrounds is a good way of going about it, rather than having a panel made up of only academics or industry participants.

Of course, the problem with EIA, which is a problem we also see mirrored in other assessment activities like Excellence in Research Australia (ERA), is that the measurement tools are, by necessity, backward-looking. ERA informs us how good individual research groups were about four years ago.

EIA similarly looks at impact over the past few years, from research over the past decade. Both activities provide substantial insight into the research performance and impact outcomes of universities but are not a panacea. For example, their utility to determine your current environment to support research students, or assess performance in emerging or cutting edge research areas is of course limited.

Fortunately there are many other more specific approaches that universities use to answer these questions.

What are the dangers of using a case study approach, is there a chance it could not be particularly representative?

Yes, with a case study approach, it is harder to ensure that a sample is representative of all the output of a university. A case study approach is also quite resource intensive.

Each case study requires a huge amount of work to prepare, so there is a significant impost on the universities preparing the documentation. Similarly, panel chairs commented on the extensive time and effort required by the panels to assess the case studies.

This was a limited study so any national approach would have to solve these issues.

Can you talk about some of the case studies?

One of our key case studies looked at AustLII, a national free-access legal database. AustLII was a joint initiative of UTS and UNSW. Until the creation of AustLII in 1995 there was no free, public access to legal information in Australia.

Now they are an essential collection of more than 500 databases that hold an absolute wealth of historical and contemporary information about Australasian law; legislation, treaties and decisions from courts and tribunals.

The AustLII model, and its flow-on effect on public policy, has fundamentally changed the system to the point where it has been instrumental in the development of similar free access legal database systems around the world.

Another of our case studies looked at a project conducted at the Centre for Health Economics Research and Evaluation (CHERE). CHERE researchers undertook an economic evaluation of changes to the Extended Medicare Safety Net (EMSN) and, later a review of the measures to cap EMSN benefits.

The work raised important questions about the efficacy of the EMSN as a health care funding tool, and provided critical recommendations to help guide policies settings to shape decisions and affect outcomes for patients and providers, and ultimately the taxpayer.

This story is published courtesy of the The Conversation (under Creative Commons-Attribution/No derivatives).