Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer (1711.06664v3)

Abstract: In many machine learning applications, there are multiple decision-makers involved, both automated and human. The interaction between these agents often goes unaddressed in algorithmic development. In this work, we explore a simple version of this interaction with a two-stage framework containing an automated model and an external decision-maker. The model can choose to say "Pass", and pass the decision downstream, as explored in rejection learning. We extend this concept by proposing "learning to defer", which generalizes rejection learning by considering the effect of other agents in the decision-making process. We propose a learning algorithm which accounts for potential biases held by external decision-makers in a system. Experiments demonstrate that learning to defer can make systems not only more accurate but also less biased. Even when working with inconsistent or biased users, we show that deferring models still greatly improve the accuracy and/or fairness of the entire system.

• The paper proposes a novel learning to defer framework that adapts deferral decisions based on the interplay between model uncertainty and human biases.
• It employs a probabilistic mixture model to integrate automated predictions with human decision-making, thereby enhancing overall decision quality.
• Empirical evaluations on datasets like COMPAS and Heritage Health demonstrate substantial gains in fairness and accuracy compared to traditional rejection learning.

Learning to Defer: Advancing Fairness and Accuracy in Decision Systems

The paper "Predict Responsibly: Improving Fairness and Accuracy by Learning to Defer" presents a novel approach to machine learning integrating automated systems and human decision-makers within complex decision-making processes. It scrutinizes the interaction between automated models and external agents, proposing a framework for adaptive rejection learning called "learning to defer." This framework builds on traditional rejection learning by incorporating the dynamic interplay between automated models and human agents, aiming not just to minimize errors but also to mitigate potential biases within decision systems.

Analytical Framework and Key Concepts

The core idea posited by the authors is that models should learn to defer responsibilities when their predictions are misaligned with the overarching objectives of the system. This entails accuracy and fairness, particularly when systems are utilized in areas such as finance, healthcare, and criminal justice, where biased decisions can have severe ramifications. Traditional rejection learning empowers models to abstain from making predictions when confidence is low, but is inherently nonadaptive as it does not consider the capabilities or biases of the human decision-makers. The learning to defer approach is adaptive and attempts to optimize decisions by considering both model uncertainties and external decision-maker characteristics.

Theoretical Developments

The paper formalizes learning to defer as a probabilistic mixture model that incorporates both automated predictions and the human decision-maker's output, governed by a deferral probability. Theoretical guarantees are provided showing how learning to defer generalizes rejection learning paradigms by addressing interactions with potentially biased decision-makers. The authors argue that in situations where external decision-makers have access to additional information not available to the model, adaptive deferral can significantly improve the accuracy and fairness of predictions by strategically deciding which agent should be responsible for a decision on a case-by-case basis.

Empirical Evaluation

Experiments were performed on datasets like COMPAS, relating to criminal recidivism, and Heritage Health, linking healthcare predictions, to simulate real-world scenarios resembling different types of external decision-makers. These include high-accuracy decision-makers, biased decision-makers, and inconsistent decision-makers. The results manifest substantial improvements in both fairness and accuracy when the model employs learning to defer in comparison to rejection learning. Notably, in contexts where human bias is pronounced, learning to defer proves indispensable for mitigating such biases, showcasing adaptable deferral rates based on subgroup characteristics.

Implications and Future Directions

The learning to defer framework has profound implications for the design of fair and reliable decision systems, emphasizing the integration of machine learning within human-centric processes. It proposes a methodology that not only enhances accuracy but fosters fairness by taking into account the biases and expertise of human decision-makers. Future work could explore further granularities of human-AI collaboration, particularly how adaptive learning might apply across more nuanced decision-maker profiles and extending to broader multilateral decision systems.

This paper contributes significantly to the discourse on responsible AI, outlining methodologies for deploying machine learning models that adaptively interact with human decision-makers. The implications for AI systems extend into domains where fairness and accuracy are critical, suggesting a promising direction for future research toward building equitable, transparent, and high-performing decision systems.