Fairness-Aware Confident Learning

• The paper introduces a fairness-aware variant of confident learning that employs robust truncation and lower-confidence thresholds to retain underrepresented groups.
• It leverages a group-aware co-teaching paradigm where separate networks agree on pruning mislabeled examples to balance fairness across sensitive groups.
• Empirical evaluations on diverse datasets show reduced test error and improved fairness metrics, demonstrating an effective error-fairness trade-off in noisy labeling settings.

Fairness-aware confident learning is a paradigm for mitigating bias in supervised machine learning pipelines affected by distributional or label noise, with a particular emphasis on algorithmic fairness across sensitive groups. It operationalizes the identification and filtering of label errors through confidence estimation, while incorporating modifications to standard confident learning (CL) techniques to address the risk of unfair exclusion of underrepresented populations. This framework is motivated by the observation that commonly used data filtering procedures—by relying solely on confidence scores—can amplify pre-existing dataset bias and disproportionately harm disadvantaged groups when labeling processes are corrupted by bias (Zhang et al., 2023).

1. Formal Problem Setting and Label Bias Model

Let the observed dataset be $D = \{(x_n, \tilde{y}_n, s_n)\}_{n=1}^N$, where $x_n \in \mathbb{R}^d$ represents non-sensitive features, $\tilde{y}_n \in \{0,1,\ldots,k-1\}$ are observed (possibly biased) labels, $z_n$ are unobserved true labels, and $s_n \in \{A, B\}$ denotes the protected group (e.g., $A$ for privileged, $B$ for disadvantaged). A label-flipping function $\mathcal{G}$ acts on $(X, S, Z)$ to produce $\tilde{Y} = \mathcal{G}(X, S, Z)$, with bias rates $x_n \in \mathbb{R}^d$0 and $x_n \in \mathbb{R}^d$1. In standard "symmetric" bias, both rates are positive; in "asymmetric" bias, one is dominant. Training classifiers directly on $x_n \in \mathbb{R}^d$2 can introduce significant disparity in downstream fairness metrics, such as the demographic parity gap $x_n \in \mathbb{R}^d$3 and the equal opportunity gap $x_n \in \mathbb{R}^d$4 (Zhang et al., 2023).

2. Confident Learning Foundations

The standard CL framework identifies likely label errors by estimating the self-confidence of the model in its provided label for each data point, based on its probabilistic output. For the $x_n \in \mathbb{R}^d$5-th data point, the self-confidence is $x_n \in \mathbb{R}^d$6, and the associated error probability is $x_n \in \mathbb{R}^d$7. Per-class thresholds $x_n \in \mathbb{R}^d$8 are defined as the average self-confidence in class $x_n \in \mathbb{R}^d$9. Noisy or mislabeled examples are detected by predicting a new label $\tilde{y}_n \in \{0,1,\ldots,k-1\}$0, subject to $\tilde{y}_n \in \{0,1,\ldots,k-1\}$1, and assembling a count matrix of label transitions. Off-diagonal elements in the calibrated joint matrix $\tilde{y}_n \in \{0,1,\ldots,k-1\}$2, $\tilde{y}_n \in \{0,1,\ldots,k-1\}$3, indicate suspected label errors, which can be pruned, with a new classifier retrained on the filtered set (Zhang et al., 2023).

3. Fairness-Aware Confident Learning Modifications

Fairness-aware confident learning introduces two mechanisms to ensure that data filtering does not disproportionately disadvantage samples from underrepresented groups:

a) Truncation of Confidence Scores:

Instead of the naïve mean-based threshold $\tilde{y}_n \in \{0,1,\ldots,k-1\}$4, a robust mean $\tilde{y}_n \in \{0,1,\ldots,k-1\}$5 is computed via an $\tilde{y}_n \in \{0,1,\ldots,k-1\}$6-estimator:

$\tilde{y}_n \in \{0,1,\ldots,k-1\}$7

with $\tilde{y}_n \in \{0,1,\ldots,k-1\}$8 acting as an influence function to down-weight high-confidence outliers.

b) Lower-confidence Threshold Extension:

A theoretically justified lower threshold $\tilde{y}_n \in \{0,1,\ldots,k-1\}$9 is derived (Theorem 1) by:

$z_n$0

where $z_n$1 is a variance bound, $z_n$2 the sample size, $z_n$3. All samples with $z_n$4 are retained. This relaxes filtering to preserve low-confidence but possibly correctly labeled items, especially from minority groups (Zhang et al., 2023).

4. Group-Aware Co-Teaching Paradigm

The framework adapts the co-teaching paradigm: instead of each network teaching the other on a subset of likely clean labels, both networks must concur on the set of pruned instances. Two neural networks, $z_n$5 and $z_n$6, are trained separately on slices of the data defined by $z_n$7 (i.e., $z_n$8 or $z_n$9). In each minibatch iteration:

• Compute robust thresholds $s_n \in \{A, B\}$0, $s_n \in \{A, B\}$1 from their respective sensitive-group partitions.
• For each sample, if either network assigns $s_n \in \{A, B\}$2, the predicted label $s_n \in \{A, B\}$3 is updated.
• Build per-group confident joints, filter examples assigned to off-diagonal entries by both networks.
• The survivor set is used to update the unified model $s_n \in \{A, B\}$4.

This approach partly decouples the effect of group imbalance and prevents over-pruning disadvantaged groups (Zhang et al., 2023).

5. Fair Instance Selection, Metrics, and Theoretical Guarantees

Retention criteria require that a sample is removed only if both networks assign it to a mislabeled (off-diagonal) cell under their respective robust thresholds. This targets demographic parity and equal opportunity through agreement over sensitive-group splits. The selection rule per class $s_n \in \{A, B\}$5 is:

$s_n \in \{A, B\}$6

Theoretical analysis (Theorem 1) delivers a high-probability bound on the deviation of $s_n \in \{A, B\}$7-truncated averages from the true mean (probability at least $s_n \in \{A, B\}$8 for $s_n \in \{A, B\}$9), justifying the choice of lower thresholds for robust and fair pruning (Zhang et al., 2023).

6. Experimental Evaluation and Empirical Findings

Experiments span synthetic (95,750 points, controlled bias), Adult (≈46K, gender), COMPAS (≈7K, race), Credit Card (30K, gender), and Law School (21K, gender) datasets. A shared two-layer perceptron (ReLU) architecture is applied. Baselines include classic CL, LongReMix, Label-Bias Correction (LC), Group Peer Loss (GPL). Metrics are: test error (%), DEO (difference of equal opportunity), demographic parity (DP) gap, and $A$0-percent. Key results:

• Substantial test error reduction under both symmetric and asymmetric label flip regimes, e.g., Adult $A$1: Error 15.8% (method) vs CL 22.6%, LC 18.9%, GPL 23.5%.
• DEO reduction (Adult $A$2: 0.12 vs CL 0.17).
• Truncated "T" threshold yields lower synthetic error (0.77%) than mean estimator "M" (1.51%).
• All means are reported with standard deviation over 10 splits, with improvements robust to data splits (Zhang et al., 2023).

7. Discussion, Scope, and Limitations

The main contributions include a model-agnostic, group-aware data-filtering framework with robust truncation and co-teaching that explicitly mitigates the risk of unfair filtering. The method is empirically demonstrated to achieve better error-fairness trade-offs and is theoretically anchored. Extensions discussed include handling multi-class or multi-attribute sensitive settings, online thresholding, automated influence function learning, and dynamic tuning of the pruning/variance schedule. A plausible implication is that fairness-aware CL can be generalized as a template across data-driven fairness mitigation when label noise correlates with sensitive attributes (Zhang et al., 2023).

Component	Mechanism	Fairness Contribution
Truncation (M-estimator)	Robustifies confidence thresholds	Prevents over-pruning minorities
Co-teaching adaptation	Dual networks on group slices agree on pruning	Group-wise filtering criteria
Lower-confidence extension	Retains low-confidence, correct-label cases	Reduces disparate rejection

Fairness-aware confident learning represents a principled data-centric approach to fairness, distinct from fairness constraints at model training or inference, and complements methods for certifying group fairness with high confidence or in settings absent sensitive attributes (Zhang et al., 2023, Luo et al., 23 Oct 2025, Ni et al., 2024, Roy et al., 2023).