All academic metrics are flawed, but some are useful

June 2, 2015 by Merlin Crossley, The Conversation
A good scoring system can help the best rise to the top. Michael Coghlan/Flickr, CC BY-SA

Why are bean counters so fixated on counting? Why are universities overrun by metrics? Are we heading for a world where we know the cost of everything and the value of nothing?

Is an obsession with really corrupting science?

To me metrics are like models. To paraphrase George Box: all metrics are flawed but some are useful. What they are useful for is providing opportunities, helping install meritocracies and breaking through entrenched social elites.

A number of opportunities

My views are in part informed by my own family history. My parents were both the first in their families to go to grammar school and then to university. They went via Cyril Burt's much maligned "11-plus" examination. The flaws of this simple "intelligence test" are many fold. There is even evidence that the research on which his approach was based was fraudulent.

Nevertheless this process was an attempt to provide opportunities to those who would benefit. Despite all the problems the test was well-intentioned and did help many people. It was one of the first steps in breaking down class structures and providing social mobility. The exams were not perfect, but they were better than the class structures that had remained in place for hundreds of years.

Ultimately, we need to use and understand metrics better. We need to discuss them and improve them. But most of all, we need to interpret them with sophistication and remember they are a good beginning but a poor end of the conversation.

Better than the alternative

Metrics don't just affect students and student selection at schools and universities. Staff are also now subject to scrutiny.

There are many measures of productivity and quality in use. Student feedback is used to measure the quality of teaching and citations are counted to assess how well research papers are received.

Neither measure is perfect and both can certainly drive perverse outcomes if used poorly. But, again, both can be useful.

If students think a lecturer is good, it is worth knowing so that others can learn from that. If a lecturer is not appreciated, shouldn't university management know? If papers are highly cited, is that not one indication that the research is having an impact in the world?

Metrics are a good starting point for subsequent discussions informed by expert opinion and experience on how we can improve what we do. Metrics also provide independent evidence to tax payers and other supporters that their funding is making a difference.

Some people lament the fact that metrics may influence decisions about academic hiring and promotion, but surely this is better than the decisions being confined to darkened rooms where status quo prevails.

Although all numbers have limitations, they are often superior to political intrigues and the "who you know" and "who you trust" methodology, which tends to take hold when data is absent.

The scoreboard

Some idealists seem to have a very benign view of society and believe that fairness and quality can prosper on their own without the help of systems and numbers. But others consider that work is required to drive reforms. Numbers can do the work in a non-confrontational and impartial, although imperfect, way.

One area where meritocracy already holds sway is on the sporting field. Here the scoreboards are paramount and there are some interesting effects. It is no coincidence that it is often in sport that our indigenous Australians shine most brightly. In other fields they may face prejudice, but in sport their achievements and talents cannot be denied.

While racial slurs may still occur, no-one can deny the match statistics and no coach is likely to drop a productive player from the team. In other walks of life it is much harder to agree on talent. So prejudgements, often based on the shared background and experiences of the selection panel, may come into play.

Ultimately, metrics make a good starting point for social harmony at universities or elsewhere, provided they are used by academic managers who have been through the system and understand the pitfalls. It is also important that managers actually communicate the fact that they don't use metrics blindly, bluntly or as final arbiters of decision making.

We need to try harder to communicate the fact that metrics are not used in isolation and remember that excluding people via what appear to be dumb numbers can be extremely hurtful.

During my own highly privileged student days I played squash and rowing. In squash there were clear metrics. Any disagreement about selection could be settled quickly via a challenge.

In rowing it was the opposite. No one ever really knew who the best oarsman was. We talked at length behind the coach's back, and those who didn't make the team sometimes suspected the worst. The rowing machine –- "The Erg" – that measured strength helped settle a lot of questions.

Rowing machine metrics were never the final word in selecting talent for inclusion in the small crew, but they were usually a helpful place to start.

Explore further: Our obsession with metrics is corrupting science

Related Stories

Guidance offered for ethical referrals within ACOs

January 15, 2015

(HealthDay)—For accountable care organizations (ACOs), referrals within the ACO can be ethical as long as their selection is influenced in a manner that is transparent, with appropriate metrics and the right incentives, ...

Recommended for you

Research helps students excel on science exams

January 15, 2019

A new study released today reveals that helping lower-income high school freshman to regulate their test-taking anxiety can cut their biology course failure rates in half. The study, published in Proceedings of the National ...


Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Jun 02, 2015
The big problem with numeric representations of activities is that they lead to gaming the system. For example, the "number" in the stock market is used by investors in bidding stock prices up or down. So corporations routinely rig that number so that it comes within a penny of their prediction. Similar problem for all the tests given to students these days. Teachers are encouraged to get the test scores up, even if it means the students don't actually learn anything.
Jun 02, 2015
This comment has been removed by a moderator.
Captain Stumpy
5 / 5 (1) Jun 02, 2015
The people are cheaters by their very nature and the scientists aren't an exception
your conspiracy theory is showing
re-read above for content again
Better than the alternative
Neither measure is perfect and both can certainly drive perverse outcomes if used poorly. But, again, both can be useful
get it yet?
nearly nobody is doing really inquisitive original research today (like the research cold fusion or antigravity drives)
maybe because the fundamental research shows them to be non-viable
again, given the historical posts and claims- go build one if it is so easy
because the repetitive research is safe and appreciated well enough
fundamental research, just like ALL research, requires validation

this is the part that always gets zephir
just because a claim is made doesn't mead said claim is true
stop reading pseudoscience

the power of the scientific method insures that pseudoscience will always be obvious
Jun 02, 2015
This comment has been removed by a moderator.
Captain Stumpy
5 / 5 (1) Jun 02, 2015
The answer is, the scientist got corrupted with money, which they got from tax payers without public feedback
so if the scientists don't validate pseudoscience because you want it validated, then it is a conspiracy against the public and greed for money?

or maybe it is simply something like:
science doesn't like investing in pseudoscience
not everything is conspiracy against reality


by the way, i told you before-
i ain't clicking your links unless they are to a reputable science site and/or journal with actual reputable scientific publications and studies that can be or are validated through the scientific method

Jun 02, 2015
This comment has been removed by a moderator.
Captain Stumpy
5 / 5 (1) Jun 03, 2015
Such an embarrassing situation would just never happen, if the scientists wouldn't be a cheaters by their profession (i.e. as a whole community)
so you are calling all scientists cheaters by default ... but have absolutely NO evidence to support your conclusions? (BTW- that is called PSEUDOSCIENCE)
by your use of syntax and description above, that means anyone trying to define anything using logical scientific protocols is a liar and cheat as it is a profession and they belong in a community
Therefore, your precious pseudoscience is also full of cheaters and liars

you cannot judge all scientists based upon the interpretations of your perceptions of pseudoscience

just because you WANT to believe in something that has no empirical evidence doesn't mean it is true NOR that scientists are cheaters etc
it only means YOU SPECIFICALLY have a faith in something that is not proven or that doesn't have evidence supporting its existence
Captain Stumpy
5 / 5 (1) Jun 03, 2015
It's not a conspiracy - but a result of omnipresent pluralistic ignorance of literally every peer of this group
no, it isn't
it also doesn't prove pluralistic ignorance
perhaps before you throw out terms you should research them a little more?
the above posts by you (and your historical assertions and belief in pseudoscience and extreme unproven/unvalidated science) demonstrate your Dunning-Kruger
as well as your conspiracy theory ideation and unrealistic beliefs
The usage of the "conspiracy" word in this connection is ignorant, pejorative - and also nonsensical
no it isn't
you are attempting to justify your pseudoscience with denigrations of actual proven validated science by promoting a known fallacy that has been shown to be false on numerous occasions

THAT is essentially how conspiracy theories operate - regardless of the evidence, it is evidence of conspiracy
1 / 5 (1) Jun 07, 2015
Sports is not "meritocratic"

If it were, you wouldn't see so few, very highly payed players....its quite doubt able that the "league" represents all possible "elite" players. Its a losers game anyways..for people with little else but athletic talent

Plus team dynamics can make players look better or wors

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.