Researchers measure accuracy of forecasts in strategic intelligence

July 15, 2014 by Bob Yirka report
Model-based calibration curves. Dotted lines show 95% confidence intervals. Credit: PNAS, 2014. doi: 10.1073/pnas.1406138111

( —Two researchers have measured the accuracy of forecasts given by two teams within Canada's Intelligence Assessment Secretariat (IAS), a group that provides government leaders with intelligence information regarding the Middle East and Africa. David Mandel with Defence Research and Development and Alan Barnes, formerly with IAS report in a paper they've had published in Proceedings of the National Academy of Sciences on their findings and offer opinions on the overall quality of such intelligence gathering based on what they learned.

Intelligence gathering has been conducted by people for most of human history—knowing what friends and enemies alike are about to do plays a very large role in strategic planning. But, as the two researchers note, in modern times, very little information is available regarding the accuracy of strategic intelligence gathered by governments. To learn more, they focused their efforts on IAS, looking at 1,514 strategic intelligence forecasts (with particulars redacted for security purposes) that were made over the period March 2005 to December 2011. Comparing the forecasts with actual outcomes revealed the unit was right 76 percent of the time, which the researchers describe as "very good."

Not surprisingly, the researchers also found that more experienced analysts were better at their jobs (when making predictions based on judgment rather than informational value) than those with just a few years of experience—their work was more accurate though there were miscalibrations that the researchers attributed to assigning more uncertainty to some events than might have been necessary due to under-confidence that came about as a result of an air of caution. Such an environment has come about, the research pair note, from stated goals that strive for "informativeness" and an aversion to overstating what has been found. They noted also that most of the analysts had both on-the-ground experience in their area of study and post-graduate degrees, which suggests they were highly qualified for their positions.

The researchers conclude their analysis by offering their opinion on the quality of intelligence forecasting offered by the agency, suggesting they found "cause for tempered optimism" regarding the work done by the analysts in the IAS, but also suggest that forecasting could be made even more accurate by using postforecast transformations.

Explore further: River flow model assists in planning and extreme weather prediction

More information: Accuracy of forecasts in strategic intelligence, PNAS, 2014. … /1406138111.abstract

The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination. Underconfidence was more pronounced for harder (versus easier) forecasts and for forecasts deemed more (versus less) important for policy decision making. Despite the observed underconfidence, there was a paucity of forecasts in the least informative 0.4–0.6 probability range. Recalibrating the forecasts substantially reduced underconfidence. The findings offer cause for tempered optimism about the accuracy of strategic intelligence forecasts and indicate that intelligence producers aim to promote informativeness while avoiding overstatement.

Related Stories

Chimpanzee intelligence determined by genes

July 10, 2014

A chimpanzee's intelligence is largely determined by its genes, while environmental factors may be less important than scientists previously thought, according to a Georgia State University research study.

Study finds genetic link between height and IQ

March 5, 2014

A team of researchers at Edinburgh University in Scotland has found a correlation between genes associated with height and those associated with intelligence. In their paper published in the journal Behavior Genetics, the ...

Recommended for you

The oldest plesiosaur was a strong swimmer

December 14, 2017

Plesiosaurs were especially effective swimmers. These long extinct "paddle saurians" propelled themselves through the oceans by employing "underwater flight"—similar to sea turtles and penguins. Paleontologist from the ...

Averaging the wisdom of crowds

December 12, 2017

The best decisions are made on the basis of the average of various estimates, as confirmed by the research of Dennie van Dolder and Martijn van den Assem, scientists at VU Amsterdam. Using data from Holland Casino promotional ...

1 comment

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Jul 16, 2014
I think the litmus test of the value of the Canadian intel group was there assessment and projections of the ISIS rise to power - these chaps are like prophets.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.