Researchers measure accuracy of forecasts in strategic intelligence

Jul 15, 2014 by Bob Yirka report
Model-based calibration curves. Dotted lines show 95% confidence intervals. Credit: PNAS, 2014. doi: 10.1073/pnas.1406138111

(Phys.org) —Two researchers have measured the accuracy of forecasts given by two teams within Canada's Intelligence Assessment Secretariat (IAS), a group that provides government leaders with intelligence information regarding the Middle East and Africa. David Mandel with Defence Research and Development and Alan Barnes, formerly with IAS report in a paper they've had published in Proceedings of the National Academy of Sciences on their findings and offer opinions on the overall quality of such intelligence gathering based on what they learned.

Intelligence gathering has been conducted by people for most of human history—knowing what friends and enemies alike are about to do plays a very large role in strategic planning. But, as the two researchers note, in modern times, very little information is available regarding the accuracy of strategic intelligence gathered by governments. To learn more, they focused their efforts on IAS, looking at 1,514 strategic intelligence forecasts (with particulars redacted for security purposes) that were made over the period March 2005 to December 2011. Comparing the forecasts with actual outcomes revealed the unit was right 76 percent of the time, which the researchers describe as "very good."

Not surprisingly, the researchers also found that more experienced analysts were better at their jobs (when making predictions based on judgment rather than informational value) than those with just a few years of experience—their work was more accurate though there were miscalibrations that the researchers attributed to assigning more uncertainty to some events than might have been necessary due to under-confidence that came about as a result of an air of caution. Such an environment has come about, the research pair note, from stated goals that strive for "informativeness" and an aversion to overstating what has been found. They noted also that most of the analysts had both on-the-ground experience in their area of study and post-graduate degrees, which suggests they were highly qualified for their positions.

The researchers conclude their analysis by offering their opinion on the quality of intelligence forecasting offered by the agency, suggesting they found "cause for tempered optimism" regarding the work done by the analysts in the IAS, but also suggest that forecasting could be made even more accurate by using postforecast transformations.

Explore further: River flow model assists in planning and extreme weather prediction

More information: Accuracy of forecasts in strategic intelligence, PNAS, 2014. www.pnas.org/content/early/201… /1406138111.abstract

Abstract
The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination. Underconfidence was more pronounced for harder (versus easier) forecasts and for forecasts deemed more (versus less) important for policy decision making. Despite the observed underconfidence, there was a paucity of forecasts in the least informative 0.4–0.6 probability range. Recalibrating the forecasts substantially reduced underconfidence. The findings offer cause for tempered optimism about the accuracy of strategic intelligence forecasts and indicate that intelligence producers aim to promote informativeness while avoiding overstatement.

add to favorites email to friend print save as pdf

Related Stories

Chimpanzee intelligence determined by genes

Jul 10, 2014

A chimpanzee's intelligence is largely determined by its genes, while environmental factors may be less important than scientists previously thought, according to a Georgia State University research study.

Study finds genetic link between height and IQ

Mar 05, 2014

A team of researchers at Edinburgh University in Scotland has found a correlation between genes associated with height and those associated with intelligence. In their paper published in the journal Behavior Genetics, the gr ...

Recommended for you

The changing landscape of religion

4 hours ago

Religion is a key factor in demography, important for projections of future population growth as well as for other social indicators. A new journal, Yearbook of International Religious Demography, is the first to bring a quan ...

Abusive leadership infects entire team

4 hours ago

Supervisors who are abusive to individual employees can actually throw the entire work team into conflict, hurting productivity, finds new research led by a Michigan State University business scholar.

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

COCO
not rated yet Jul 16, 2014
I think the litmus test of the value of the Canadian intel group was there assessment and projections of the ISIS rise to power - these chaps are like prophets.