Computer scientists offer new techniques to measure social bias in software

August 17, 2017 by Janet Lathrop, University of Massachusetts Amherst

Today, banks are increasingly using software to decide who will get a loan, courts to judge who should be denied bail, and hospitals to choose treatments for patients. These uses of software make it critical that the software does not discriminate against groups or individuals, say computer science researchers at the University of Massachusetts Amherst.

Professor Alexandra Meliou in the College of Information and Computer Sciences says, "The increased role of software and the potential impact it has on people's lives makes software fairness a critical property. Data-driven software has the ability to shape human behavior: it affects the products we view and purchase, the news articles we read, the social interactions we engage in, and, ultimately, the opinions we form."

Meliou with professor Yuriy Brun and Ph.D. student Sainyam Galhotra, have developed a new technique they call "Themis," to automatically test software for discrimination. They hope Themis will empower stakeholders to better understand software behavior, judge when unwanted is present, and, ultimately improve the software.

Brun says, "Unchecked, biases in data and software run the risk of perpetuating biases in society. For example, prior work has demonstrated that racial bias exists in online advertising delivery systems, where online searches for traditionally-minority names were more likely to yield ads related to arrest records. Such software behavior can contribute to racial stereotypes and other grave societal consequences."

The researchers' paper describing this research, published in pre-conference materials for the European Software Engineering Conference (ESEC/FSE 2017) before its September meeting in Paderborn, Germany, has won an Association of Computing Machinery Special Interest Group on Software Engineering (ACM SIGSOFT) Distinguished Paper Award. The work is supported by the National Science Foundation.

Brun explains that while earlier research has considered discrimination in software, Themis focuses on measuring causality in discrimination. Software testing allows Themis to perform hypothesis testing, to ask such questions as whether changing a person's race affects whether the software recommends giving that person a loan, he says.

"Our approach measures discrimination more accurately than prior work that focused on identifying differences in software output distributions, correlations or mutual information between inputs and outputs. Themis can identify bias in software whether that bias is intentional or unintentional, and can be applied to software that relies on machine learning, which can inject biases from data without the developers' knowledge," he adds.

When evaluated on public software systems from GitHub, Themis found that discrimination can sneak in even when the software is explicitly designed to be fair. State-of-the-art techniques for removing discrimination from algorithms fail in many situations, in part because prior definitions of discrimination failed to capture causality, the researchers point out.

For example, Themis found that a decision-tree-based machine learning approach specifically designed not to discriminate against gender was actually discriminating more than 11 percent of the time. That is, more than 11 percent of the individuals saw the software output affected just by altering their gender.

Themis also found that designing the software to avoid discrimination against one attribute may increase discrimination against others. For example, the same decision-tree-based software trained not to discriminate on gender discriminated against race 38 percent of the time. "These systems learn from biased data, but without careful control for potential bias, can magnify that bias even further," Galhotra says.

Explore further: Researchers tackle bias in algorithms

More information: Conference Paper: … bs/Galhotra17fse.pdf

Themis Project page:

Related Stories

Researchers tackle bias in algorithms

July 4, 2017

If you've ever applied for a loan or checked your credit score, algorithms have played a role in your life. These mathematical models allow computers to use data to predict many things—who is likely to pay back a loan, ...

Rating systems may discriminate against Uber drivers

December 16, 2016

Many new "sharing economy" companies, like Uber and Airbnb, use consumer-sourced ratings to evaluate their workers – but these systems can be fraught with difficulties, including bias based on race or gender.

Human and organizational factors influence software quality

August 11, 2015

A new doctoral dissertation by Frank Philip Seth at LUT (Lappeenranta University of Technology) proposes that human factors involved in the software development processes will determine the quality of the products developed. ...

Recommended for you

Google braces for huge EU fine over Android

July 18, 2018

Google prepared Wednesday to be hit with huge EU fine for freezing out rivals of its Android mobile phone system in a ruling that could spark new tensions between Brussels and Washington.

EU set to fine Google billions over Android: sources

July 17, 2018

The EU is set to fine US internet giant Google several billion euros this week for freezing out rivals of its Android mobile phone system, sources said, in a ruling that risks fresh tensions with Washington.


Please sign in to add a comment. Registration is free, and takes less than a minute. Read more

Click here to reset your password.
Sign in to get notified via email when new comments are made.