Assessing consistency in meta-analysis: A new measure considers statistical power

Researchers have improved the assessment of consistency in meta-analysis. The improved consistency measure considers statistical power, and it has potential to alter the interpretation of meta-analyses. The new measure was published in the European Journal for Philosophy of Science.

Meta-analysis refers to combining results of several studies mathematically. Especially, medicine considers meta-analyses as the highest level of evidence. If a meta-analysis recommends a treatment, the treatment most probably becomes established.

Meta-analysis must be consistent per se. Conventional measures of consistency straightforwardly favor large studies and do not consider statistical power as a source of inconsistency. Statistical power denotes the ability of a study to detect statistically significant results, such as effects of a treatment.

"Repeating a study with high statistical power is more difficult than repeating a study with low statistical power. Our proposed measure of inconsistency emphasizes this aspect," says Data Manager Ari Voutilainen from the University of Eastern Finland, one of the four developers of the measure.

"If a meta-analysis combines studies that differ from each other with respect to statistical power, the meta-analysis is inconsistent, even if within-study variances are low. Moreover, it is important that studies with the highest statistical power have the strongest effect on the results of a meta-analysis. Our proposed measure accentuates the meaning of statistical power also from this viewpoint."

Scientific journals publish approximately half a million meta-analyses annually. Consequently, the improved consistency measure can have a substantial effect on research in general.