Falcon: Fair Active Learning Framework

• The paper introduces Falcon, a scalable active learning framework that integrates explicit fairness constraints into data labeling.
• It outlines two variants—FAL-CUR using fair clustering and round-robin selection, and Falcon-MAB employing bandit-based risk policies—to enhance group fairness.
• Experimental results demonstrate that Falcon significantly improves fairness metrics (up to 4.5× gain) while preserving high prediction accuracy.

Falcon for Fair Active Learning refers to a class of scalable active learning frameworks that explicitly incorporate group fairness constraints into the sample selection process for data labeling. Driven by the recognition that standard active learning techniques can amplify disparities across sensitive groups, Falcon-based methods are designed to achieve equitable representation in labeled datasets and ultimately fairer predictive models. This article reviews the foundational methodology, sample acquisition strategies, fairness mechanisms, experimental evaluation, and broader implications of the Falcon framework, incorporating both clustering- and bandit-based variants (Fajri et al., 2022, Tae et al., 2024).

1. Foundational Methodology and Objectives

Falcon frameworks address the challenge of integrating user-specified group fairness metrics into the active learning paradigm. The primary goal is to select unlabeled data for annotation such that the resulting classifier optimizes accuracy while also achieving high fairness, typically defined via demographic parity, equal opportunity, or equalized odds. The overarching strategy is data-centric: sample acquisition prioritizes instances estimated to have maximal impact on closing observed fairness gaps across sensitive subgroups, even under unknown ground-truth class labels. Fundamental notations include partitioning datasets into labeled $(D_{\text{train}})$, unlabeled $(D_{\text{un}})$, validation, and test sets, and maintaining a group-fairness score $F(\cdot)$ on the model $h_\theta$.

2. Sample Selection Mechanisms

Two principal Falcon families exist: FAL-CUR (clustering-based) and Falcon-MAB (bandit-based).

FAL-CUR Framework

• Two-phase batch selection: Each acquisition proceeds by (1) running fair clustering on the most “uncertain” instances to create clusters with sensitive-attribute proportions aligned to the population, then (2) ranking points within clusters by a convex combination of uncertainty (entropy) and representativeness (intra-cluster distance).
• Score function:

$$\text{Score}(x) = \beta \, \text{Rep}(x) + (1-\beta) H(x), \quad \beta \in [0,1]$$

where $H(x)$ is the classifier's label entropy on $x$, and $\text{Rep}(x)$ is the sum of pairwise distances within cluster.

• Round-robin selection: Samples are drawn from clusters in a round-robin cycle to fill label batch $b$, ensuring diverse yet representative coverage.

Falcon-MAB Framework

• Target subgroup identification: At each iteration, the group with the largest fairness gap under metric $F$ (e.g., minimal $(D_{\text{un}})$0 for some $(D_{\text{un}})$1) is identified as the acquisition target.
• Trial-and-error postponement: For each queried sample matching the target group's sensitive attribute and predicted label, if the true label supports reducing the disparity, its annotation is incorporated; otherwise, it is postponed (not immediately used to update the model), reducing introduction of “antifair” data.
• Multi-armed bandit over risk-taking: Several acquisition policies, parameterized by how “close to the boundary” (uncertain or risky) a sample must be before selection, define the arms of an EXP3 adversarial bandit. The reward is the gain in group fairness, and the MAB adaptively selects the optimal exploration/exploitation strategy.

3. Fairness Objectives and Metrics

Falcon methods admit arbitrary group-fairness criteria expressible as bounded disparities across sensitive groups. Let $(D_{\text{un}})$2 denote a sensitive attribute and $(D_{\text{un}})$3 the label:

• Demographic Parity (DP):

$(D_{\text{un}})$4

with the fairness score $(D_{\text{un}})$5.

• Equal Opportunity (EO):

$(D_{\text{un}})$6

$(D_{\text{un}})$7 similarly defined.

• Equalized Odds (EOdds): Disparity averaged over both positive and negative true classes.
• Empirical fairness score is always referenced on a hold-out test set, measuring the maximum observed pair-disparity after final training.

4. Algorithmic Workflow

FAL-CUR (Clustering-based Variant) (Fajri et al., 2022)

$h_\theta$0

Falcon-MAB (Bandit-based Variant) (Tae et al., 2024)

• At each round, select whether to optimize for accuracy or fairness (probability $(D_{\text{un}})$8 controls blend).
• If fairness step, evaluate current $(D_{\text{un}})$9, determine subgroup(s) with maximal disparity.
• EXP3 bandit selects a risk policy, which defines threshold for predicted label probability when acquiring sample in target subgroup.
• Acquire and label $F(\cdot)$0 samples. Only those whose true label matches the “fairness-improving” direction are added; others are postponed.
• Use difference in $F(\cdot)$1 on validation set as reward to EXP3 for policy adaptation.

5. Experimental Evaluation

Falcon frameworks have been validated on a suite of real-world, binary classification datasets with diverse sensitive attributes:

Dataset	Sensitive Attribute	Baseline Fairness Score	Falcon Max Fairness	Relative Gain
TravelTime	Gender	0.160 (FAL)	0.966 (Falcon)	4.1×
Employ	Disability	0.278 (best baseline)	0.645 (Falcon)	2.3×
Income	Race	0.375 (best baseline)	0.816 (Falcon)	2.2×
COMPAS	Gender	0.390 (best baseline)	0.861 (Falcon)	2.2×

• FAL-CUR achieves $F(\cdot)$2 relative reductions in equalized odds difference versus the prior SOTA (FAL), while maintaining accuracy within $F(\cdot)$3 of the best non-fair AL methods (Fajri et al., 2022).
• Falcon-MAB demonstrates that trade-off curves between accuracy and fairness are smoothly tunable via $F(\cdot)$4; only Falcon achieves $F(\cdot)$5 in multiple settings, up to $F(\cdot)$6 baseline fairness.
• Batch size, policy count, and $F(\cdot)$7 are sensitive hyperparameters; smaller batch size allows finer adaptation, and $F(\cdot)$8 policy arms suffices in practice.
• Ablation indicates that removing fair clustering collapses fairness in FAL-CUR, while in Falcon-MAB, omitting trial-and-error or MAB steps produces suboptimal (or unstable) fairness.

6. Interpretability, Scalability, and Extensions

Falcon frameworks are modular and directly compatible with arbitrary group fairness objectives. Scalability is achieved through efficient clustering for FAL-CUR and the computationally-optimal EXP3 bandit for Falcon-MAB, yielding $F(\cdot)$9 faster training than prior fair AL methods. The frameworks can be extended to:

• Graph-structured or streaming data (fair spectral clustering; dynamic cluster/policy updates),
• Alternative or intersectional fairness criteria (e.g., disparate impact, multiclass protected attributes),
• Generalized acquisition scores using Gini or variance for multiclass settings.

A plausible implication is that these methods provide the first scalable avenue to actively construct label datasets that are both information-rich and fairness-compliant, strengthening fairness properties early in machine learning workflows (Fajri et al., 2022, Tae et al., 2024).

7. Context, Limitations, and Research Directions

Falcon frameworks represent a convergence of algorithmic fairness and active learning, shifting the focus from only pre- or post-processing fairness interventions to data acquisition itself. They enable the system to target model weaknesses on worst-served subgroups even when true labels are unknown, mitigating sample complexity by avoiding unnecessary or antifair labels. Limitations include reliance on the availability of sensitive attribute metadata, calibration of the uncertainty measure, and the requirement of an oracle (human annotator) for label acquisition. Future work explores fine-grained fairness (beyond group metrics), robust calibration in high-dimensional spaces, adaptation to streaming and non-i.i.d. settings, and integration with end-to-end auditing frameworks.