Do standard fairness metrics track user outcome quality

Determine whether standard fairness evaluation measures for predictive models—such as predictive parity, error rate balance, and anti-classification—accurately track the quality of outcomes experienced by users.

Background

The paper reviews common fairness measurement approaches used in predictive systems, including predictive parity and error rate balance. While these statistical measures are widely adopted to assess fairness, the authors highlight uncertainty about whether they correspond to meaningful improvements in users' lived outcomes.

This uncertainty suggests a foundational issue in fairness evaluation: even if models meet formal criteria, it remains unresolved whether these metrics reflect the real-world quality of outcomes for affected individuals.

References

Whether these measures ultimately track the quality of the outcomes for users, however, is still an open question .

— Fairness and Sequential Decision Making: Limits, Lessons, and Opportunities  (2301.05753 - Nashed et al., 2023) in Section 5

The study evaluates the applicability of FHA and the perceived relevance of its outputs, but does not establish whether the proposed mitigations improve actual fairness outcomes. Future work should therefore implement and longitudinally evaluate FHA-derived mitigations, involve affected stakeholders, and apply the methodology across further organisations, domains, and regulatory contexts.

— Fairness Hazard Analysis for Socio-Technical Processes: A Multiple-Case Study in Bias-sensitive Organisational Settings  (2608.22978 - Broccia et al., 24 Aug 2026) in Section Conclusion