Do standard fairness metrics track user outcome quality

Determine whether standard fairness evaluation measures for predictive models—such as predictive parity, error rate balance, and anti-classification—accurately track the quality of outcomes experienced by users.

Background

The paper reviews common fairness measurement approaches used in predictive systems, including predictive parity and error rate balance. While these statistical measures are widely adopted to assess fairness, the authors highlight uncertainty about whether they correspond to meaningful improvements in users' lived outcomes.

This uncertainty suggests a foundational issue in fairness evaluation: even if models meet formal criteria, it remains unresolved whether these metrics reflect the real-world quality of outcomes for affected individuals.