The complex role of citations as measure of scientific quality

Oct 14, 2013

Allocation of resources in the scientific community is increasingly based on various quantitative indicators. One of the most important indicators of scientific quality is how often research is cited. However, a new doctoral thesis in theory of science from the University of Gothenburg shows that the number of citations is a poor measurement of the quality of research.

'Citations occur when a researcher provides a reference to previous research results to for example back up a claim. However, references can be made for many different reasons,' says the author of the thesis Gustaf Nelhans, PhD in theory of science at the University of Gothenburg and lecturer in library and information science at the University of Borås.

Researchers sometimes refer to previous research to indicate the source of certain influences or to identify past work that they want to develop further. But they may also cite previous work in order to argue against it or maybe even refute it entirely. And sometimes sources are referred to out of tradition or routin-like because everybody else in a field seems to do it.

'The conclusion of this is that the number of times research is cited is a rather poor indicator of its scientific quality nor that more automatically means high quality, says Nelhans.

As a result of the so-called citation culture that has emerged in the , an increasing number of researchers have started to present their studies not only with the obvious goal of promoting the content, but also with an aim to attract as many citations as possible. The purpose of this is to gain acknowledgement in the scientific community and secure .

On the other hand Nelhans argues: You can on the other hand say that a cited article has been 'used' by the later literature and that it therefore is 'visible'. But this has to be done carefully since citations only show up for certain publications, namely articles published in certain peer-reviewed and internationally distributed scientific journals.

Nelhans' thesis points to how the awareness of the effects of citations on research has led to them being perceived as hard currency in the scientific community, from the national level down to the individual researcher at his or her department.

'The problem is that citation statistics offer a complex measurement that hides at least as much information as it reveals. It is therefore important to see the whole extent of this phenomenon and not treat citations as an automatic measure,' says Nelhans, who urges decision-makers to be more careful when basing allocation of funding on citation statistics.

Explore further: Cloning whistle-blower: little change in S. Korea

More information: hdl.handle.net/2077/33516

add to favorites email to friend print save as pdf

Related Stories

Scientists who share data publicly receive more citations

Oct 01, 2013

A new study finds that papers with data shared in public gene expression archives received increased numbers of citations for at least five years. The large size of the study allowed the researchers to exclude ...

How to Spot an Influential Paper Based on its Citations

Jul 04, 2009

(PhysOrg.com) -- At first it may seem that the number of citations received by a published scientific paper is directly related to that paper's quality of content. The higher the quality, the more people read ...

Recommended for you

Cloning whistle-blower: little change in S. Korea

21 hours ago

The whistle-blower who exposed breakthrough cloning research as a devastating fake says South Korea is still dominated by the values that allowed science fraudster Hwang Woo-suk to become an almost untouchable ...

Color and texture matter most when it comes to tomatoes

Oct 21, 2014

A new study in the Journal of Food Science, published by the Institute of Food Technologists (IFT), evaluated consumers' choice in fresh tomato selection and revealed which characteristics make the red fruit most appealing.

How the lotus got its own administration

Oct 21, 2014

Actually the lotus is a very ordinary plant. Nevertheless, during the Qing dynasty (1644-1911) a complex bureaucratic structure was built up around this plant. The lotus was part of the Imperial Household, ...

User comments : 5

Adjust slider to filter visible comments by rank

Display comments: newest first

HannesAlfven
1.6 / 5 (7) Oct 14, 2013
Re: "'The conclusion of this is that the number of times research is cited is a rather poor indicator of its scientific quality nor that more citations automatically means high quality, says Nelhans."

We know this for a fact, actually, since many now accepted theories were initially rejected by reviewers. What is very clear, as well, is that there are in fact a great number of good ideas which are simply neglected because they originate from worldviews that basically undermine existing worldviews.

The problem with mainstream science, at this moment, is that there is a widespread perception (confusion for some) that there is only one worldview in science worth talking about -- the one which aligns with consensus. Scientometrics reinforces this false premise in science.

Scientometrics is simply one small component to the far larger knowledge map. If science's knowledge map is to include innovative ideas as well, then it must be extended outwards beyond "mainstream" science.
alfie_null
5 / 5 (1) Oct 15, 2013
We know this for a fact, actually, since many now accepted theories were initially rejected by reviewers. What is very clear, as well, is that there are in fact a great number of good ideas which are simply neglected because they originate from worldviews that basically undermine existing worldviews.

No. Not clear. Would you deny there are cranks? Or do you accept that every single crank idea is valid? All theories start out in an "unaccepted" state. You might consider citation count to be one of those "worst way except for all the others" instances.

You speak of consensus as if it were a popularity contest. You are utterly missing the point. Is the process completely fair? No. But it's better than chaos that would ensue if everyone who thinks they have a good idea had equal footing. And over the long term, it works.
HannesAlfven
1 / 5 (3) Oct 15, 2013
Re: "No. Not clear. Would you deny there are cranks?"

You seem to not be aware of the incredibly destructive wrecking ball which has been deployed to eliminate the "crank problem". The mistake which has been made is a failure of imagination, and critical thinking is the unintended casualty.

Creating a label for a group of critics is not some sort of a solution to a complex communication problem. Your label for "cranks" fails to distinguish actual critical thinkers. We've yet to see technology deployed at differentiating cranks from critical thinkers. You guys just lump together anybody who disagrees with ideology in science as an enemy of science, and you point to the empty comments text box's failure to elicit wisdom like you actually expected an empty text box to generate wisdom.

This is totally juvenile and naive, and it will have obvious long-term consequences for science itself which apparently only some of us can see. Outsiders have always played a part in science.
HannesAlfven
1 / 5 (3) Oct 15, 2013
This is now "normal" ... You reap what you sow ...

From http://web.archiv...ogist%29

"SA's edits of my page have systematically aimed to create the false impression that no one in the community takes my work seriously, that I should be grouped with someone like, say, Velikovsky. For that reason he has eliminated references to my peer-reviewed publications, to leading academic institutions where I have been invited to present my work and to my stay at ESO as a Visiting Astronomer. [..]

"SA and his colleague BKramer have pursued this campaign against me and everyone else they see as minority thinkers. This has led to ludicrous stunts such as questioning my (BA!) degree from Columbia as "unsourced". I would like to see anyone's degree which is publicly sourced to anyone except themselves. University records are not generally on-line!"

- Eric Lerner discussing ScienceApologist
HannesAlfven
1 / 5 (3) Oct 15, 2013
Re: "Is the process completely fair? No. But it's better than chaos that would ensue if everyone who thinks they have a good idea had equal footing."

There appears to be a metric which you and others who ceaselessly complain of cranks rarely, if ever, make any mention of ... That of philosophy of science. The way out of the dilemma is to use philosophy of science as a fundamental metric for judging one another's comments (in combination with other types of metrics). The solution never apparently occurs to advocates of mainstream science, however, because modern day physicists have made a habit of routinely violating philosophical principles. A system which uses an actually useful metric would naturally invite people to critique conventional theory -- and so, we never see the suggestion broached by those in the mainstream.