Court software may be no more accurate than web survey takers in predicting criminal risk

January 17, 2018, Dartmouth College
A new study by Dressel et al. shows the software COMPAS is no more accurate or fair than predictions made by people with little or no criminal justice expertise. Credit: Carla Schaffer / AAAS

A widely-used computer software tool may be no more accurate or fair at predicting repeat criminal behavior than people with no criminal justice experience, according to a Dartmouth College study.

The Dartmouth analysis showed that non-experts who responded to an online survey performed equally as well as the Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) software system used by courts to help determine the risk of .

The paper also demonstrates that although COMPAS uses over one hundred pieces of information to make a prediction, the same level of accuracy may be achieved with only two variables - a defendant's age and number of prior convictions.

According to the research paper, COMPAS has been used to assess over one million offenders since it was developed in 1998, with its recidivism prediction component in use since 2000.

The analysis, published in the journal Science Advances, was carried out by the student-faculty research team of Julia Dressel and Hany Farid.

"It is troubling that untrained internet workers can perform as well as a computer program used to make life-altering decisions about criminal defendants," said Farid, the Albert Bradley 1915 Third Century Professor of Computer Science at Dartmouth College. "The use of such software may be doing nothing to help people who could be denied a second chance by black-box algorithms."

According to the paper, software tools are used in pretrial, parole, and sentencing decisions to predict criminal behavior, including who is likely to fail to appear at a court hearing and who is likely to reoffend at some point in the future. Supporters of such systems argue that big data and advanced machine learning make these analyses more accurate and less biased than predictions made by humans.

"Claims that secretive and seemingly sophisticated data tools are more accurate and fair than humans are simply not supported by our research findings," said Dressel, who performed the research as part of her undergraduate thesis in computer science at Dartmouth.

The compares the commercial COMPAS software against workers contracted through Amazon's online Mechanical Turk crowd-sourcing marketplace to see which approach is more accurate and fair when judging the possibility of recidivism. For the purposes of the study, recidivism was defined as committing a misdemeanor or felony within two years of a defendant's last arrest.

Groups of internet workers saw short descriptions that included a defendant's sex, age, and previous criminal history. The human results were then compared to results from the COMPAS system that utilizes 137 variables for each individual.

Overall accuracy was based on the rate at which a defendant was correctly predicted to recidivate or not. The research also reported on false positives—when a defendant is predicted to recidivate but doesn't—and false negatives—when a defendant is predicted not to recidivate but does.

With considerably less information than COMPAS—seven features compared to 137—when results were pooled to determine the "wisdom of the crowd," the humans with no presumed criminal justice experience were accurate in 67 percent of the cases presented, statistically the same as the 65.2 percent accuracy of COMPAS. Study participants and COMPAS were in agreement for 69.2 percent of the 1000 defendants when predicting who would repeat their crimes.

According to the study, the question of accurate prediction of recidivism is not limited to COMPAS. A separate review cited in the study found that eight of nine software programs failed to make accurate predictions.

"The entire use of recidivism prediction instruments in courtrooms should be called into question," Dressel said. "Along with previous work on the fairness of criminal justice algorithms, these combined results cast significant doubt on the entire effort of predicting recidivism."

In contrast to other analyses that focus on whether algorithms are racially biased, the Dartmouth study considers the more fundamental issue of whether the COMPAS algorithm is any better than untrained humans at predicting recidivism in an accurate and fair way.

However, when race was considered, the research found that results from both the human respondents and the showed significant disparities between how black and white defendants are judged.

According to the paper, it is valuable to ask if we would put these decisions in the hands of untrained people who respond to an , because, in the end, "the results from these two approaches appear to be indistinguishable."

Explore further: Can algorithms properly inform criminal sentencing?

More information: J. Dressel el al., "The accuracy, fairness, and limits of predicting recidivism," Science Advances (2018). , DOI: 10.1126/sciadv.aao5580

Related Stories

Why using AI to sentence criminals is a dangerous idea

May 16, 2017

Artificial intelligence is already helping determine your future – whether it's your Netflix viewing preferences, your suitability for a mortgage or your compatibility with a prospective employer. But can we agree, at least ...

Recommended for you

Meteorite source in asteroid belt not a single debris field

February 17, 2019

A new study published online in Meteoritics and Planetary Science finds that our most common meteorites, those known as L chondrites, come from at least two different debris fields in the asteroid belt. The belt contains ...

Diagnosing 'art acne' in Georgia O'Keeffe's paintings

February 17, 2019

Even Georgia O'Keeffe noticed the pin-sized blisters bubbling on the surface of her paintings. For decades, conservationists and scholars assumed these tiny protrusions were grains of sand, kicked up from the New Mexico desert ...

Archaeologists discover Incan tomb in Peru

February 16, 2019

Peruvian archaeologists discovered an Incan tomb in the north of the country where an elite member of the pre-Columbian empire was buried, one of the investigators announced Friday.

Where is the universe hiding its missing mass?

February 15, 2019

Astronomers have spent decades looking for something that sounds like it would be hard to miss: about a third of the "normal" matter in the Universe. New results from NASA's Chandra X-ray Observatory may have helped them ...

What rising seas mean for local economies

February 15, 2019

Impacts from climate change are not always easy to see. But for many local businesses in coastal communities across the United States, the evidence is right outside their doors—or in their parking lots.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.