In September 2007, a British polling company named ORB released an alarming estimate of "more than 1,000,000 Iraqis murdered" in the Iraq war. ORB’s poll-based estimate has been cited approvingly in much of the blogosphere, a number of mainstream media outlets and academic publications, and in official statements from the Bloomberg School of Public Health of Johns Hopkins University in support of their own work in this area.
A new study, recently published in ‘Survey Research Methods’, by Professor Michael Spagat, from Royal Holloway, University of London, and Josh Dougherty, of Iraq Body Count, describes in detail how the ORB poll is riddled with critical inconsistencies and methodological shortcomings. This first and only peer-reviewed analysis of the ORB estimate concludes that it is too flawed, exaggerated and ill-founded to contribute to discussion of the human costs of the Iraq war.
An internal validity check of ORB data across three separate polls reveals internal contradictions indicative of compromised data collection practices which greatly exaggerate the resulting estimate (section 4 of the paper). In particular, four governorates in central Iraq account for more than 80% of ORB’s estimated one million deaths. Yet in these governorates a higher percentage of respondents report deaths of household members than report deaths of extended family members in another ORB poll conducted only six months earlier. This pattern can not be seen as credible since extended family networks are far larger than households.
The percentage reporting deaths in the entire southern region of Iraq, on the other hand, does show the expected sharp drop between the two polls (from 35 to 7 percent) when ORB switches from its extended-family question to its household question. This more reasonable pattern casts strong doubt on precisely the data (in the four key central governorates) which provided the vast bulk (more than 80%) of deaths in the “million” figure.
The ORB poll is also marred by a number of serious quality problems, such as a claimed margin of error of plus or minus 8% that is unrealistically narrow, and critical problems with ORB’s published mortality question that invite respondents to report both non-violent deaths and deaths of extended family members in what is purportedly a household survey measuring “murders”. Other problems include a failure to disclose key methodological information such as the exact wordings of its questions as asked in the field, i.e., in Arabic and Kurdish, and an inadequate treatment of non-response in which ORB assumes that death rates of non-respondents were identical to those of responding households.
Professor Spagat said, “The ORB poll is not a serious piece of research and should never have seen the light of day.” Josh Dougherty added that: “There has been terrible carnage in the Iraq war but no useful purpose is served by promoting inflated death toll estimates.”
Explore further:
The strangely familiar browsing habits of 14th-century readers
