Fairness in Criminal Justice Risk Assessments: The State of the Art (1703.09207v2)

Abstract: Objectives: Discussions of fairness in criminal justice risk assessments typically lack conceptual precision. Rhetoric too often substitutes for careful analysis. In this paper, we seek to clarify the tradeoffs between different kinds of fairness and between fairness and accuracy. Methods: We draw on the existing literatures in criminology, computer science and statistics to provide an integrated examination of fairness and accuracy in criminal justice risk assessments. We also provide an empirical illustration using data from arraignments. Results: We show that there are at least six kinds of fairness, some of which are incompatible with one another and with accuracy. Conclusions: Except in trivial cases, it is impossible to maximize accuracy and fairness at the same time, and impossible simultaneously to satisfy all kinds of fairness. In practice, a major complication is different base rates across different legally protected groups. There is a need to consider challenging tradeoffs.

• The paper introduces six distinct fairness definitions and proves that they conflict with optimizing prediction accuracy.
• It uses interdisciplinary methods, including theoretical proofs and empirical arraignment data, to analyze fairness trade-offs.
• The study highlights the need for informed policy decisions and continuous algorithmic transparency in criminal justice.

Fairness in Criminal Justice Risk Assessments: An Analytical Perspective

The paper "Fairness in Criminal Justice Risk Assessments: The State of the Art" by Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth provides an intricate exploration of fairness considerations within the scope of criminal justice risk assessments. The authors undertake a comprehensive analysis delineating the trade-offs between various fairness definitions and the inherent conflict between fairness and accuracy.

Objectives and Methodology

The primary objective of the paper is to provide conceptual clarity to the complex discussions surrounding fairness in criminal justice risk assessments. This is accomplished by systematically dissecting the different kinds of fairness—at least six forms, as identified—and elucidating their mutual incompatibilities and trade-offs with predictive accuracy.

The methodology employed includes an inter-disciplinary review encompassing literatures from criminology, computer science, and statistics. Furthermore, the authors provide empirical illustrations using data derived from arraignments to concretize their arguments.

Key Findings

1. Six Kinds of Fairness: The paper identifies and formalizes at least six distinct kinds of fairness—overall accuracy equality, statistical parity, conditional procedure accuracy equality, conditional use accuracy equality, treatment equality, and total fairness. The authors demonstrate that these forms of fairness are often in mutual conflict.
2. Trade-offs Between Accuracy and Fairness: A significant claim of the paper is that it is practically impossible to simultaneously maximize accuracy and fairness across multiple dimensions. This results from the inherent complexity and varying base rates across different legally protected groups.
3. Base Rates and their Implications: The paper rigorously shows how different base rates across protected groups (e.g., men and women, Blacks and Whites) complicate fairness. For instance, altering base rates through techniques such as re-weighting or relabeling can introduce other forms of unfairness.
4. Impossibility Theorems: The authors present formal proofs demonstrating that when base rates differ across protected groups, achieving conditional use accuracy equality while also maintaining equal false positive and false negative rates across groups is infeasible unless perfect separation is assumed. This theoretical assertion is backed by empirical examples.

Empirical Illustrations

The authors substantiate their theoretical findings by detailing hypothetical confusion matrices and real-world examples. For instance, they explore gender differences in parole success and the complexities that arise when attempting to tune algorithms to balance fairness and accuracy. In one empirical example, they analyze arraignment decisions by race and demonstrate how optimized conditional use accuracy for Blacks and Whites still leads to disparate false positive and false negative rates, highlighting the persistent trade-offs.

Implications and Future Work

Practical Implications

1. Criminal Justice Policies: The findings underscore the need for stakeholders to make informed decisions about acceptable trade-offs between different fairness criteria and accuracy. This becomes a policy matter rather than a purely technical one.
2. Algorithm Transparency and Updates: Given that training data reflect historical biases, continuous updates and transparency in the functioning of risk assessment tools are crucial. This can prevent perpetuation or exacerbation of biases in future decisions.

Theoretical Implications and Future Research

1. Advances in Fairness Constraints: Future developments can focus on creating more sophisticated fairness constraints that balance multiple fairness metrics without significantly compromising accuracy. There is a need for robust methods that stakeholders can tune according to contextual ethical and legal considerations.
2. Multidisciplinary Collaboration: Progress in addressing fairness in criminal justice risk assessments will benefit significantly from multidisciplinary collaboration, combining insights from legal theory, ethical studies, criminology, and advanced machine learning. This collaboration can extend current methodologies to include broader socio-legal implications.
3. Benchmark Establishment: Establishing benchmarks for fairness in criminal justice is inherently challenging, and future research should focus on operational criteria that can be broadly accepted by various stakeholders while taking into account societal values and legal standards.

Conclusion

This paper significantly contributes to the discourse on fairness in criminal justice risk assessments by clearly defining the multifaceted nature of fairness, illustrating the unavoidable trade-offs with accuracy, and providing empirical and theoretical verification of these claims. It signals the need for continued theoretical advancements and pragmatic solutions to ensure informed decision-making in criminal justice systems, aligned with societal values and fairness principles.