Auditing (automated) information systems with social impact is a must to depict possible discrimination and biases from the point of view of data justice. From simple descriptive statistics and plots to advanced analysis of machine learning models there is a wide bunch of approaches. Here, we present some case studies grouped by context.

Article Context Bias/Harm Discrimination source System description and analysis Findings
Lum, Kristian, and William Isaac. 2016. ‘To Predict and Serve?’ Significance 13 (5): 14–19.
Crime prediction racial bias, location (low income) discrimination police recorded data; The learning algorithm assumes that crime follows the same patterns that seismographic activity PredPol is a USA commercial tool to build predictive policing models. It produces a one day head prediction of the crime rate across locations in a city, using only the previously recorded crimes. The areas with the highest predicted crime rates are flagged as “hotspots” and receive additional police attention on the following day. “Rather than correcting for the apparent biases in the police data, the model reinforces these biases. The locations that are flagged for targeted policing are those that were, by our estimates, already over represented in the historical police data.”

“Using PredPol in Oakland, black people would be targeted by predictive policing at roughly twice the rate of whites. Individuals classified as a race other than white or black would receive targeted policing at a rate 1.5 times that of whites. This is in contrast to the estimated pattern of drug use by race […], where drug use is roughly equivalent across racial classifications.”

“We find similar results when analysing the rate of targeted policing by income group, with low income households experiencing targeted policing at disproportionately high rates.”

Ensign, Danielle, Sorelle A. Friedler, Scott Neville, Carlos Scheidegger, and Suresh Venkatasubramanian. 2017. ‘Runaway Feedback Loops in Predictive Policing’. ArXiv:1706.09847 [Cs, Stat], June.
Crime prediction feedback loops can amplify the difference between groups The self feedback of the PredPol algorithm The authors modelled the predictive policing model of PredPol by a series of urn models of increasing complexity. There, they can evaluate the evolution of a PredPol driven system with respect to groups/areas. The authors identify feedback loops in PredPol that will be exacerbated as the system is used, specifically as crime rates vary between regions and as the model relies more and more on discovered incident reports.

In addition, they proposed a “fix” to PredPol that works fairly under certain conditions.

Stanford big data study finds racial disparities in Oakland, Calif., police behavior, offers solutions
Police traffic and pedestrian stops Racial bias Human (police) historical bias This is not an automated system, but we include it here since police records are used as ground truth to build automated predictive policing tools.

To detect group discrimination, the authors used descriptive statistics and regression analysis to identify possible bias in police data records. Data was obtained from Oakland’s police public data records.

“OPD officers stopped, searched, handcuffed, and arrested more African Americans than Whites, a finding that remained significant even after we controlled for neighborhood crime rates and demographics; officer race, gender, and experience; and other factors that shape police actions;”
How We Analyzed the COMPAS Recidivism Algorithm
Machine Bias
There’s software used across the country to predict future criminals. And it’s biased against blacks.
by Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner, ProPublica
May 23, 2016
Recidivism prediction Racial bias historical racial discrimination is present in training data
COMPAS (which stands for Correctional Offender Management Profiling for Alternative Sanctions) is an automated tool to predict a score to determine whether to release or detain a defendant before his or her trial. Each pretrial defendant received at least three COMPAS scores: “Risk of Recidivism,” “Risk of Violence” and “Risk of Failure to Appear.”

The authors replicated the Northpointe, Inc. tool used to evaluate recidivism according to the public documentation and information request to the company. This included data acquisition, data preprocessing and predictive statistical methods (logistic regression and Cox models) .

Risk (score) predicted for black defendants is generally over estimated while predicted risk for while defendants is under estimated.
High risk miss-classification rates significantly differ from black and white defendants groups (45 percent vs. 23 percent).
Zhang, Zhe, and Daniel B. Neill. 2016. ‘Identifying Significant Predictive Bias in Classifiers’. ArXiv:1611.08292 [Cs, Stat], November.
recidivism prediction racial bias, age bias Historical bias is present in data This study perform a further analysis of the COMPAS system to refine conclusions of ProPublica initial analysis.

The authors propose an algorithm to automatically discover discriminated sub-groups (disparate impact). Finding subgroups in data-drive systems is not a trivial computational and statistical task, and it can help to discover new types of discriminations. The algorithm is an extension of the literature in anomaly detection task.


“In our analysis of COMPAS, we do not detect a significant
predictive bias along racial lines, but instead identify bias
in a more subtle multi-dimensional subgroup: females who
initially committed misdemeanors (rather than felonies), for
half of the COMPAS risk groups, have their recidivism risk
significantly over-estimated.”
The authors found notable biases by the COMPAS prediction that they have not seen noted elsewhere: young males are under-estimated (regardless of race or initial crime type) and females, whose initial crimes were misdemeanors are overestimated.
Jevin West, Carl Bergstrom. n.d. ‘Case Study – Criminal Machine Learning’. Calling Bullshit. Accessed 18 July 2018. Criminality detection using face images Random behaviour of the system biased dataset using different sources for each category “Wu and Zhang claim that based on a simple headshot, their programs can distinguish criminal from non-criminal faces with nearly 90% accuracy. Moreover, they argue that their computer algorithms are free from the myriad biases and prejudices that cloud human judgment”

The team of Calling Bullshit performed a quantitative analysis of the paper describing the system.

The authors of the system used two different sources for each category, this is, pictures of criminals were provided by police departments and non-criminal pictures were scrapped from different sources of the WWW.

In this case, the bias in the database (different situations and origins of the pictures) is learned. For instance, the system is identifying smiles in pictures as a relevant feature to distinguish criminals from non-criminals. A person being taken a picture of at a police department is more likely not to be smiling. Since discriminative machine learning models will learn any feature that can help to improve accuracy, they are likely to learn features such as smile as a discriminant factor to identify a criminal.

Angwin, J. & Paris Jr, T. (2016). Facebook Lets Advertisers Exclude
Users by Race. Available at:
Facebook (Still) Letting Housing Advertisers Exclude Users by Race
housing letting Explicit race, religion and ethnic discrimination Facebook audience filters Facebook offers to customers target audience groups for advertisements. These filters allow to specify audience using a very variety of sensitive features such as race, gender, age…

“ProPublica bought dozens of rental housing ads on Facebook, but asked that they not be shown to certain categories of users, such as African Americans, mothers of high school kids, people interested in wheelchair ramps, Jews, expats from Argentina and Spanish speakers.”

Facebook allows users to exclude protected groups from ads, including topics such as jobs and housing with have specify regulation to prevent discrimination.
Price, Megan, and Patrick Ball. 2014. ‘Big Data, Selection Bias, and the Statistical Patterns of Mortality in Conflict’. SAIS Review of International Affairs 34 (1): 9–20. Violent conflict reports, refugees Event size bias, selection bias Small events are rarely reported in violent reports The authors get data of violent events provided by several HR organizations.

They used descriptive statistics and graphics to compare events reported by the different organizations to detect which kind of events were more likely not to be reported.

Many powerful Big Data are based on the assumption of accessing to (almost) all available data. The authors claim that in policy and social world this assumption is rarely met.

Event size bias may cause that specific regions or ethnic groups are not considered as victims in conflicts, this affecting to later considerations or decisions (for instance asylum seeking evaluation).

“The fundamental reason why [selection] biases are so problematic for quantitative analyses is that bias often correlates with other dimensions that are interesting to analysts, such as trends over time, patterns over space, differences compared by the victims’ sex, or some other factors”

“It is important to note that these challenges frequently lack a scientific solution. We do not need to simply capture more data. What we need is to appropriately recognize and adjust for the biases present in the available data.”

Bolukbasi, Tolga, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. n.d. ‘Man Is to Computer Programmer as Woman Is to Homemaker? Debiasing Word Embeddings’, 9.
Natural language processing Gender bias Gender bias in texts used to train natural language processing models Word embedding is a popular framework to represent text data as vectors used in many machine learning and natural language processing tasks.
The authors analysed the distance between learned terms in the word embedding space.
“We show that even word embeddings trained on Google News articles exhibit female/male gender stereotypes to a disturbing extent. This raises concerns because their widespread use, as we describe, often tends to amplify these biases.”
The authors proposed a method to mitigate the gender bias present in the training datasets, by modifying the embedding.