Confirmation bias in studies of gamma ray bursts

Our understanding of gamma ray bursts (GRBs) – flashes of gamma rays from explosions in distant galaxies – since they were discovered more than 50 years ago may not be as solid as first thought.

Research by myself and colleagues, published in Monthly Notices of the Royal Society Letters and a preprint, found that journals tended to publish papers which supported an already widely accepted model of how GRBs form, and reject those that don't – a form of confirmation bias.

But our work shows that results that don't conform to the standard model still need to be considered. So can the standard model be salvaged in light of us uncovering the apparent confirmation bias?

NASA satellites detect an intense flash of gamma rays every few days, which are believed to originate from catastrophic explosions of massive stars (hypernovae) in galaxies at the edge of the known universe. From Earth, telescopes triggered by NASA usually detect a rapidly fading afterglow following the prompt short flash.

The most popular model that describes the physics of GRB afterglow creation is the standard model (not to be confused with the Standard Model of Particle Physics). One core prediction of the GRB model is that the total energy of the high-energy flash determines the brightness of the fading afterglow.

The standard model has been successful in describing many individual GRBs and has remained firmly entrenched as the favoured and only viable model to explain observations. It predicts a significant correlation between the total GRB energy emitted and afterglow luminosity.

It is what has been previously observed and reported in numerous published studies, adding further support to the standard model.

But is it the only model?

Other models

An alternative – the Cannonball Model – can explain several anomalous GRBs that don't fit the standard model. One important concern that applies to both models is that fine-tuning has to be done to make either model work for individual GRBs.

Since 2007 I had my doubts about the standard model as it seemed to fall short when I considered GRB properties of the whole population of several hundred.

My main issue was that many peer-reviewed GRB studies used the growing number of GRB distance estimates to determine how they are distributed throughout the universe. This is important (remembering GRBs result from massive stars exploding) because it provides a way to indirectly see how massive stars are distributed across the universe.

There is one problem with this idea: the distances are obtained from a rapidly fading optical afterglow to obtain spectroscopic redshift so many distances cannot be measured because the afterglow is too dim.

This causes a bias (known as the Malmquist Bias) to measure only the brightest and nearest GRBs. This implies that one preferentially selects some data while inadvertently removing other data.

Importantly, the Malmquist Bias then produces a "false" correlation between intrinsic properties, such as total energy and luminosity. I soon realised that the intrinsic properties of GRBs that relied on distance measurements were biased, but now had to prove this.

Rise to the challenge

In 2013, armed with my growing scepticism and new insight, I was determined to test the reported GRB correlations.

I led a small team that began analysing the published studies that showed significant correlations of intrinsic properties of GRBs (using the standard model). Although some of these works did attempt to remove biases, I soon realised an obvious one remained.

The GRB afterglow luminosity data was distance dependent, introducing a Malmquist bias, which was not corrected for!

After re-analysing the data, but removing the afterglow Malmquist Bias, I found that previous reported correlations could not be reproduced and are in fact non-existent. Furthermore, I found that by including only the biased data, the correlation returned.

What this really means is that many scientists ignored or did not see fundamental biases in the data. Our work showed that previously reported correlations are an artifact of observation – nothing to do with the intrinsic properties of GRBs.

How could this happen in a rigorous peer reviewed culture, and why did I see this and not anyone else?

The answer is rooted in the psychology of scientists participating in a relatively volatile field (for astronomy anyway) dominated by several strong collaborative groups. The GRB standard model has become a powerful dogma that cannot easily be challenged, especially with no viable alternative.

So the previous studies naturally interpreted the observed GRB correlations as a result of the standard model. In general terms this behaviour is termed confirmation bias (also called confirmatory bias or myside bias). It is a tendency for people to favour information that confirms their preconceptions or hypotheses regardless of whether the information is true.

Not perfect, but the best we have

From my personal account, it is understandable that the public may be sceptical of the scientific process. But ultimately the work we did to uncover the truth was vindicated by a rigorous peer review (not without setbacks, as we had to find scientists not connected with the GRB standard model) and now published results.

Despite the fact the scientific process is not perfect, it is the best system we have to converge to an informed understanding of the world around us. Sometimes this convergence is convoluted, as in my case, but ultimately science and society is a winner.

So what about the GRB standard model – is it in tatters? Actually, no. Its proponents are very good at fine-tuning the model to suit observations.

How much fine-tuning can you perform on a model before it should be abandoned? That is a question for another day.

This story is published courtesy of The Conversation (under Creative Commons-Attribution/No derivatives).