How Do Fairness Definitions Fare? Examining Public Attitudes Towards Algorithmic Definitions of Fairness (1811.03654v2)

Abstract: What is the best way to define algorithmic fairness? While many definitions of fairness have been proposed in the computer science literature, there is no clear agreement over a particular definition. In this work, we investigate ordinary people's perceptions of three of these fairness definitions. Across two online experiments, we test which definitions people perceive to be the fairest in the context of loan decisions, and whether fairness perceptions change with the addition of sensitive information (i.e., race of the loan applicants). Overall, one definition (calibrated fairness) tends to be more preferred than the others, and the results also provide support for the principle of affirmative action.

Examining Public Attitudes Towards Algorithmic Definitions of Fairness

The paper "How Do Fairness Definitions Fare? Examining Public Attitudes Towards Algorithmic Definitions of Fairness" introduces a nuanced investigation into public perceptions of various fairness definitions within algorithmic decision-making. As algorithmic systems ubiquitously permeate influential domains such as finance, employment, and criminal justice, it becomes crucial for these systems to align with societal fairness norms. This paper focuses on assessing general public views on three fairness definitions within the context of loan allocation.

Overview of Fairness Definitions

The research examines three prominent fairness definitions:

1. Treating Similar Individuals Similarly: This definition upholds the principle that decisions should reflect a continuity condition across similar individuals. Within the paper's loan scenario, this aligns primarily with equal distribution when individuals have almost equivalent metrics, effectively suggesting a 50/50 distribution of resources.
2. Meritocratic Fairness: Also referred to as not favoring a worse over a better individual, this definition is concerned with distributions that align with expected merits, advocating that those with higher repayment expectations should receive at least equivalent resources. In the experiment, this could correspond with decisions where allocations are skewed in favor of higher creditworthy individuals.
3. Calibrated Fairness: Defined in this paper as the distribution of resources proportionate to individuals' merits, calibrated fairness implies distributions that reflect an accurate assessment of repayment rates. In the distribution context, this results in allocations proportional to individuals' assessed likelihood to repay.

Methodology and Findings

The paper employs online experiments to observe participants' fairness perceptions in loan decisions, both in scenarios leveraging task-specific metrics like loan repayment rates, and scenarios that also include sensitive information like race. Results from this investigation underscore a notable preference across participants for the calibrated fairness definition, particularly in decisions proportionate to individuals' assessed merit. The findings are consistent across scenarios with varying repayment differences, but factors like racial attributes did introduce complications, revealing how public endorsements might extend to support principles akin to affirmative action.

Across treatments where repayment rates were close, participants showed favor towards equal distribution, aligning with the definition of treating similar individuals similarly. However, notably stronger repayment differentials showcased a support for decisively merit-based resource allocative choices especially, but not exclusively, when the advantaged individual belonged to a historically disadvantaged group.

Implications and Future Directions

This research underscores the complexity surrounding algorithmic fairness and public expectation. Most participants resonated with calibrated fairness, implying a demand for algorithms that account for actual merit in decision-making processes while also disclosing broader support for affirmative action principles.

Looking ahead, future studies should expand to evaluate the role of additional decision constraints or contexts, such as university admissions or bail setting, where fairness perceptions might diverge given the indivisible nature of the goods or decisions. Furthermore, understanding discrepancies between how humans versus algorithms deliver these decisions could provide insight into psychological theories underpinning fairness judgments. Researchers should also delve into the dynamics behind why sensitive information impacts fairness perceptions and explore the extent public perspectives can or should shape operational algorithmic fairness.

In conclusion, this paper advances the conversational depth between technological advancements and societal ethics, implicitly urging that algorithmic systems not only mirror but actively incorporate evolving societal notions of justice and equity.