Bias Mitigation through Continual Learning

• BM-CL is a framework that applies continual learning principles to reframe bias mitigation as a sequential adaptation process for machine learning models.
• It incorporates techniques like Learning without Forgetting and Elastic Weight Consolidation to preserve performance on advantaged groups while improving fairness for disadvantaged ones.
• Empirical results on benchmarks such as Waterbirds, CelebA, and CheXpert demonstrate increased accuracy for underrepresented groups with minimal impact on overall model performance.

Bias Mitigation through Continual Learning (BM-CL) encompasses a class of frameworks and methodologies that utilize continual learning principles to address and correct biases in machine learning models, particularly those that amplify disparities between advantaged and disadvantaged groups. By conceptualizing bias mitigation as analogous to domain-incremental continual learning, BM-CL aims to improve performance for underrepresented or disadvantaged groups while retaining strong predictive power on previously advantaged ones—explicitly avoiding the “leveling-down effect,” where gains for some subgroups are offset by loss of capability on others (Mansilla et al., 1 Sep 2025). BM-CL provides an integrated and theoretically grounded approach, adapting and extending key continual learning techniques to achieve a positive-sum trade-off between equity and efficacy.

1. Conceptual Foundations and Framework

BM-CL reframes bias mitigation as a continual learning problem. Rather than passively adjusting loss terms or data representations, the entire process is treated as a sequence of subtasks: first, standard empirical risk minimization (ERM) is performed until a baseline model is competent, with performance analyzed per demographic subgroup to identify advantaged (“best-performing”) and disadvantaged (“worst-performing”) cohorts. Subsequently, mitigation is approached as a continual adaptation, where the model is fine-tuned to improve fairness outcomes for disadvantaged groups, while using continual learning regularizers to maintain performance on advantaged groups (Mansilla et al., 1 Sep 2025).

This framing leverages the analogy between CL’s stability-plasticity trade-off and the competing objectives of fairness (plasticity toward fairness gains) and retention (stability of previously acquired knowledge). Models are thus continually adapted not just for new tasks but in response to evolving fairness constraints—each bias/fairness adjustment is treated as a “new domain” or “task” in the CL sense.

2. Methodological Core: Continual Learning Regularizers

Central to BM-CL is the adaptation of continual learning regularization techniques to achieve bias mitigation without incurring catastrophic forgetting for previously advantaged groups. The primary CL-derived strategies are:

• Learning without Forgetting (LwF): This method enforces preservation of prior predictions for advantaged group samples by minimizing the Kullback–Leibler (KL) divergence between the pre-adaptation (“old”) model’s softened predictions and the current model’s predictions for those groups. The core loss is:

$$L_{CL}(\theta) = \frac{1}{|I_{\text{best}}|} \sum_{i \in I_{\text{best}}} KL(q_i^* || q_i)$$

where $q_i^*$ and $q_i$ are temperature-scaled output distributions of the old and current models, with $I_{\text{best}}$ indexing the advantaged samples (Mansilla et al., 1 Sep 2025).

• Elastic Weight Consolidation (EWC): EWC constrains the movement of weights deemed important for the best-performing groups by introducing a quadratic penalty weighted by the Fisher Information Matrix:

$$L_{CL}(\theta) = \frac{1}{2} \sum_{j} F_j (\theta_j - \theta_j^*)^2$$

Here, $\theta_j^*$ represents parameters from the initial ERM phase, with $F_j$ quantifying each parameter’s significance for advantaged groups (Mansilla et al., 1 Sep 2025).

In both variants, the overall BM-CL objective during fairness adaptation is expressed as:

$$L_{\textrm{BM-CL}}(\theta) = L_{\textrm{BM}}(\theta) + \lambda L_{CL}(\theta)$$

where $L_{\textrm{BM}}$ is a fairness-focused loss (e.g., GroupDRO, ReSample), $L_{CL}$ the regularization, and $\lambda$ mediates the stability-plasticity balance.

3. Experimental Paradigms and Benchmarks

BM-CL has been empirically validated on multiple benchmarks with well-characterized sources of data bias:

• Waterbirds: Simulates spurious correlation bias by associating waterbird or landbird labels with background contexts. BM-CL boosts performance for the underrepresented “waterbird-on-land” group while maintaining high accuracy for common backgrounds.
• CelebA: Focuses on bias from demographic attribute correlations (e.g., gender-label confounding in predicting blond hair). Fairness adaptation is evaluated by subgroup error rates.
• CheXpert: Considers bias arising from age-based subpopulation imbalance in chest X-ray diagnosis, targeting fair performance across young, middle, and old groups (Mansilla et al., 1 Sep 2025).

Metrics include global and balanced accuracy, groupwise accuracy for best/worst groups, and quantification of the “leveling-down effect”—the difference in performance for advantaged groups induced by the fairness intervention.

4. Results and Efficacy

Across datasets, BM-CL variants (both LwF and EWC instantiations) demonstrated:

• Substantial gains in accuracy for the worst-off (disadvantaged) groups, with little to no loss of accuracy on the best-performing (advantaged) groups.
• Minimal or negligible “leveling-down effect,” contrasting with standard bias mitigation approaches that tend to degrade best-group performance.
• Maintenance of high global and balanced accuracy, indicating that the method’s stability-plasticity trade-off is well-calibrated (Mansilla et al., 1 Sep 2025).

This empirical superiority is attributed to BM-CL’s explicit use of continual learning regularizers to “protect” past knowledge while enabling adaptation, a conceptual departure from fairness approaches that treat fairness improvement and performance retention as mutually exclusive.

5. Theoretical Underpinnings

BM-CL’s theoretical foundation lies in treating each fairness adjustment as an incremental CL subtask and leveraging classic CL regularization to enforce “task retention.” The continuous adaptation objective:

$$L_{\textrm{BM-CL}}(\theta) = L_{\textrm{BM}}(\theta) + \lambda L_{CL}(\theta)$$

allows performance on previously well-performing groups to be maintained while navigating the stability-plasticity dilemma. This mirrors the continual learning paradigm where knowledge from earlier tasks is preserved via distillation or parameter regularization, ensuring the new adaptation does not overwrite old skills. This perspective unifies the objectives of fairness and robustness within a principled optimization scheme (Mansilla et al., 1 Sep 2025).

6. Broader Implications and Future Directions

BM-CL’s reframing of bias mitigation as continual learning extends conceptual and practical bridges between machine learning fairness, CL, and domain adaptation. The approach enables more positive-sum fairness corrections, with relevance to high-stakes domains such as medical imaging and finance.

Anticipated developments include exploration of alternative CL regularizers and hybrid schemes, dynamic adjustment of regularization strengths, application to more complex and multi-attribute fairness settings, and real-world deployment in streaming and evolving data scenarios. The framework’s generality—treating every fairness adjustment or group balancing act as a continual learning step—suggests broad adaptability and potential for integration into diverse production systems (Mansilla et al., 1 Sep 2025).

7. Significance Within the Continual Learning and Fairness Landscape

BM-CL constitutes a shift toward treating model fairness not as a one-off correction but as an ongoing, curriculum-driven process where the model’s capabilities and biases are continually diagnosed and incrementally remedied, with explicit protection of previously optimized dimensions. This unified paradigm underlines the practical possibility of resolving the fairness-robustness trade-off in machine learning models through continual learning advances.