Consulting firms provide low-quality research on crucial water policies: It shows we have a deeper problem

Management consulting revenue in Australia has grown from less than A$33 billion in 2010 to more than $47 billion in 2023. The increasing use of consultants, as well as the PwC scandal, highlights serious issues with vested interests, integrity and transparency.

Consequently, a Senate inquiry is investigating the management and integrity of consulting services. The deadline for the Senate committee's final report has been extended twice, partly due to the various revelations, to March 28. So far, all the big consulting groups in Australia have appeared before the committee.

Our recent review of research in the Murray-Darling Basin points to other serious concerns about the use of consulting studies, which are increasingly relied upon for policy-making, especially in water.

Of the studies we examined, 65 were on the economic consequences of water recovery. Almost half of these were low-quality studies, mainly from consultancies but also by think tanks and government departments. The low-quality studies were more likely to overestimate negative impacts on the economy and community from buying water back for the environment.

Unfortunately, these poor-quality studies were used to justify changes to water policy. Buying back water rights from "willing sellers" is a cost-effective way to redistribute water entitlements. But buybacks were halted under the former Coalition government. The policy will now be restored under Labor in the form of "voluntary water purchases."

Contested research into water buybacks

The $13 billion basin plan seeks to improve the health of our nation's largest river system by returning water from irrigation to the environment.

But such water reallocation has been blamed for huge job losses, reductions in irrigated production and consequently, economic decline in rural towns.

There are many groups with different interests in the basin. Research results are often contested.

To provide an objective assessment and comparison of the quality of basin water economic study results, we developed and applied a new economic quality assessment framework. This was inspired by health research, which has long applied grading systems to ensure robustness in research findings (such as the Grading of Recommendations Assessment, Development and Evaluation.

Our framework enables studies to be classified as low, medium or high quality, to suggest how robust each study's results may be.

Nearly half (45%) of the 65 water recovery studies in our review were classified as low quality. These low quality studies were much more likely to suggest large negative impacts on economic values from water recovery than higher quality studies. They were also more likely to be consulting studies.

The high quality studies (26%) were peer-reviewed, employed sophisticated modeling and extensive analysis. The estimated impact of water recovery ranged from none to small or modest. None of these studies were funded by industry.

Why is there such a difference in results?

The method used in each study is a major factor determining research quality. Consultants often rely on simple methods such as "input-output modeling" or "multipliers" to assess economic impact. These are models that often rely upon simplistic assumptions and links within sectors in the economy to predict changes in job numbers or production. These models are not able to consider all possible influences of change.

Input-output modeling is heavily criticized as inappropriate by the Australian Bureau of Statistics and many treasury departments. Given this modeling is used across many areas and subjects within Australia to illustrate "economic impact," its use and application needs greater scrutiny.

Higher quality studies use methods that allow for dynamic feedback and adaptation. They also account for other factors that influence outcomes such as climate or prices. As a result, higher quality studies in our review do not find anywhere near the same large decrease in jobs or economic impact from reduced water extraction.

For example, some feedbacks that can occur when farmers sell water include that the money is reinvested on the farm, increasing profits, or that the farm switches from irrigated to dryland agriculture, so production continues. Alternatively water recovery may increase community welfare through an improved environment, or better downstream water conditions for other farmers. Simplistic modeling approaches often ignore these other benefits.

Our review also indicated a relative lack of study in the basin on other downstream and Indigenous benefits and costs, as well as a need to pay closer attention to transition and adjustment issues within some small irrigation-intensive communities.

We need quality standards for water research

Basin communities will increasingly need to adapt and adjust as the climate changes. We need better ways to cope with such transitions, especially in the face of future upheavals from drought and extreme weather events.

Hopefully the recently released funding and other support for communities announced in the amended water law will help communities adjust to the reallocation of water. To date, such funds have not been allocated to areas most in need.

The negative socio-economic impacts predicted by low-quality studies are often used to justify changed water policies. We, along with other water economic professors, are calling for greater quality standards when it comes to government-funded research into the affects of water reallocation. The government is now required to update the impact analysis for the basin plan. It is essential that any assessment of impact is robust and defensible, following strict quality standards.

These quality standards could also be applied widely, across a variety of policies and areas. Although high quality research is difficult and takes time, relying on inadequate research can have serious consequences.

This article is republished from The Conversation under a Creative Commons license. Read the original article.