Researchers measure accuracy of forecasts in strategic intelligence

Jul 15, 2014 by Bob Yirka report
Model-based calibration curves. Dotted lines show 95% confidence intervals. Credit: PNAS, 2014. doi: 10.1073/pnas.1406138111

( —Two researchers have measured the accuracy of forecasts given by two teams within Canada's Intelligence Assessment Secretariat (IAS), a group that provides government leaders with intelligence information regarding the Middle East and Africa. David Mandel with Defence Research and Development and Alan Barnes, formerly with IAS report in a paper they've had published in Proceedings of the National Academy of Sciences on their findings and offer opinions on the overall quality of such intelligence gathering based on what they learned.

Intelligence gathering has been conducted by people for most of human history—knowing what friends and enemies alike are about to do plays a very large role in strategic planning. But, as the two researchers note, in modern times, very little information is available regarding the accuracy of strategic intelligence gathered by governments. To learn more, they focused their efforts on IAS, looking at 1,514 strategic intelligence forecasts (with particulars redacted for security purposes) that were made over the period March 2005 to December 2011. Comparing the forecasts with actual outcomes revealed the unit was right 76 percent of the time, which the researchers describe as "very good."

Not surprisingly, the researchers also found that more experienced analysts were better at their jobs (when making predictions based on judgment rather than informational value) than those with just a few years of experience—their work was more accurate though there were miscalibrations that the researchers attributed to assigning more uncertainty to some events than might have been necessary due to under-confidence that came about as a result of an air of caution. Such an environment has come about, the research pair note, from stated goals that strive for "informativeness" and an aversion to overstating what has been found. They noted also that most of the analysts had both on-the-ground experience in their area of study and post-graduate degrees, which suggests they were highly qualified for their positions.

The researchers conclude their analysis by offering their opinion on the quality of intelligence forecasting offered by the agency, suggesting they found "cause for tempered optimism" regarding the work done by the analysts in the IAS, but also suggest that forecasting could be made even more accurate by using postforecast transformations.

Explore further: Chimpanzee intelligence determined by genes

More information: Accuracy of forecasts in strategic intelligence, PNAS, 2014.

The accuracy of 1,514 strategic intelligence forecasts abstracted from intelligence reports was assessed. The results show that both discrimination and calibration of forecasts was very good. Discrimination was better for senior (versus junior) analysts and for easier (versus harder) forecasts. Miscalibration was mainly due to underconfidence such that analysts assigned more uncertainty than needed given their high level of discrimination. Underconfidence was more pronounced for harder (versus easier) forecasts and for forecasts deemed more (versus less) important for policy decision making. Despite the observed underconfidence, there was a paucity of forecasts in the least informative 0.4–0.6 probability range. Recalibrating the forecasts substantially reduced underconfidence. The findings offer cause for tempered optimism about the accuracy of strategic intelligence forecasts and indicate that intelligence producers aim to promote informativeness while avoiding overstatement.

Related Stories

Chimpanzee intelligence determined by genes

Jul 10, 2014

A chimpanzee's intelligence is largely determined by its genes, while environmental factors may be less important than scientists previously thought, according to a Georgia State University research study.

Study finds genetic link between height and IQ

Mar 05, 2014

A team of researchers at Edinburgh University in Scotland has found a correlation between genes associated with height and those associated with intelligence. In their paper published in the journal Behavior Genetics, the gr ...

Recommended for you

College rankings go under the microscope

4 hours ago

Parents, students and admissions officials have combed through college and university rankings for years. However, education researchers have largely ignored the controversial lists. That's about to change, according to a ...

A call to US educators: Learn from Canada

18 hours ago

As states and the federal government in the U.S. continue to clash on the best ways to improve American education, Canada's Province of Ontario manages successful education reform initiatives that are equal parts cooperation ...

Devices or divisive: Mobile technology in the classroom

Apr 17, 2015

Little is known about how new mobile technologies affect students' development of non-cognitive skills such as empathy, self-control, problem solving, and teamwork. Two Boston College researchers say it's ...

Forming school networks to educate 'the new mainstream'

Apr 17, 2015

As immigration increases the number of non-English speaking "culturally and linguistically diverse" students, schools will need to band together in networks focused on the challenges of educating what has been called "the ...

User comments : 1

Adjust slider to filter visible comments by rank

Display comments: newest first

not rated yet Jul 16, 2014
I think the litmus test of the value of the Canadian intel group was there assessment and projections of the ISIS rise to power - these chaps are like prophets.

Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.