Bias Redistribution in Visual Machine Unlearning: Does Forgetting One Group Harm Another?

Abstract: Machine unlearning enables models to selectively forget training data, driven by privacy regulations such as GDPR and CCPA. However, its fairness implications remain underexplored: when a model forgets a demographic group, does it neutralize that concept or redistribute it to correlated groups, potentially amplifying bias? We investigate this bias redistribution phenomenon on CelebA using CLIP models (ViT/B-32, ViT-L/14, ViT-B/16) under a zero-shot classification setting across intersectional groups defined by age and gender. We evaluate three unlearning methods, Prompt Erasure, Prompt Reweighting, and Refusal Vector using per-group accuracy shifts, demographic parity gaps, and a redistribution score. Our results show that unlearning does not eliminate bias but redistributes it primarily along gender rather than age boundaries. In particular, removing the dominant Young Female group consistently transfers performance to Old Female across all model scales, revealing a gender-dominant structure in CLIP's embedding space. While the Refusal Vector method reduces redistribution, it fails to achieve complete forgetting and significantly degrades retained performance. These findings highlight a fundamental limitation of current unlearning methods: without accounting for embedding geometry, they risk amplifying bias in retained groups.

• The paper introduces a redistribution score (RS) to quantify bias shifts following unlearning.
• It rigorously evaluates three methods on CLIP variants, revealing tradeoffs between forgetting and fairness.
• Empirical results demonstrate that perfect unlearning can inadvertently amplify bias on adjacent demographic groups.

Bias Redistribution in Visual Machine Unlearning: Formal Analysis and Empirical Results

Introduction and Problem Formulation

Machine unlearning has become increasingly relevant due to privacy mandates (e.g., GDPR, CCPA), which require deployed models to remove the influence of specific demographics or data samples without full retraining. While prior work has explored mechanisms for unlearning and verified efficacy in suppressing target groups, the downstream fairness consequences—specifically, the redistribution of bias onto correlated groups—remain poorly characterized. This paper (2604.08111) provides a rigorous empirical and geometric analysis of bias redistribution when one intersectional demographic group is forgotten in zero-shot visual classifiers based on CLIP variants. The central claim is that forgetting does not merely "remove" a group, but instead reallocates its representational probability mass along the boundaries dictated by the pretrained embedding space, often resulting in amplified bias for adjacent groups.

The authors frame bias redistribution as a statistically significant shift in per-group accuracy for retained groups subsequent to unlearning, formalized via a redistribution score (RS), and evaluate three agnostic zero-shot unlearning methods—Prompt Erasure (PE), Prompt Reweighting (PR), and Refusal Vector (RV)—on the CelebA dataset, targeting four intersectional groups defined by age and gender.

Unlearning Methods and Embedding Geometry

PE zeros out the forget-group text embedding, achieving perfect suppression at the classifier output layer. PR redistributes the mass probabilistically across retained groups by their cosine similarity with the forget vector, emulating a more conservative approach. RV projects out the mean direction of the forget group in the image embedding space, making corresponding features inaccessible to the classifier. Importantly, the geometry of the embedding space directly constrains the efficacy and fairness properties of these methods, with near-collinearity between forget and retain directions rendering perfect linear erasure impossible.

Empirical Evaluation: Metrics and Redistribution Dynamics

Evaluations span three CLIP variants—ViT-B/32, ViT-B/16, ViT-L/14—utilizing zero-shot classification between YF (Young Female), YM (Young Male), OF (Old Female), and OM (Old Male).

Results indicate that:

• PE and PR consistently achieve perfect forget accuracy ($\text{FA}=0\%$) but substantially increase bias in the nearest retained group (OF), empirically verified by per-group accuracy shifts exceeding $+70$ percentage points.
• RV decreases overall bias redistribution ($\text{RS}$) and is the sole method that improves demographic parity, but incomplete forgetting is observed, constrained by embedding geometry ($\cos(\boldsymbol{\mu}_f, \boldsymbol{\mu}_r)=0.929$).

  Figure 1: Bias redistribution following Prompt Reweighting—YF probability mass migrates to OF due to geometric proximity in CLIP's embedding space.

The redistribution effect is visualized via t-SNE projections:

Figure 2: Group clusters in t-SNE/CLIP space; centroid drift demonstrates redistribution primarily along gender boundaries, not age.

Prompt Erasure and Prompt Reweighting not only fail to neutralize bias but worsens fairness metrics, with DP increasing consistently. RV, despite fairness gains, incurs substantial accuracy degradation.

Figure 3: Demographic parity gap before and after unlearning for each method—RV alone reduces DP, while PE and PR worsen it across CLIP variants.

Patch-level heatmaps corroborate the instance-level redistribution phenomenon:

Figure 4: Patch similarity heatmaps reveal vanishing signal for YF post-unlearning and stable signals for retained groups; redistribution emerges at classifier, not feature level.

Geometric Explanation for Redistribution

Pairwise cosine similarity between group means highlights the underlying gender-dominant structure:

Figure 5: Cosine similarity matrix—same-gender pairs are $\sim0.06$ points closer than same-age pairs, dictating redistribution along gender axis when YF is forgotten.

As a consequence, removing YF leads to ambiguous faces being reclassified as OF, rather than YM, contradicting naive age-based intuition. This proximity is not an artifact of the method but an emergent property from CLIP pretraining on large-scale web data.

Forget–Fairness Tradeoff and Impossibility Results

Sweep analyses of projection strength for RV expose a fundamental Pareto tradeoff where perfect forgetting is incompatible with fairness and utility. Over-projection paradoxically restores the original geometry, and no operating point achieves all desiderata:

Figure 6: Forget–Fairness tradeoff curve—none of the zero-shot methods achieves low FA and RS simultaneously; the ideal lower-left corner is unattainable.

Geometric constraints force the tradeoff; projection-based methods cannot fully suppress the forget-group without damaging retained-group performance. This limitation is provably rooted in the collinearity of forget and retain mean vectors.

Practical Implications and Recommendations

• Evaluating only aggregate accuracy masks large disparities induced by unlearning; per-group statistics and RS must be reported.
• Method selection should match application requirements: PE for maximal legal compliance, RV for fairness mitigation.
• Embedding geometry should be audited before applying unlearning methods; high cosine similarity signals inevitable fairness collateral.
• Quantitative RS augmentation offers a principled fairness metric for unlearning, essential for regulatory or ethical audits.

Future Directions

The analysis suggests that embedding-centric constraints fundamentally limit zero-shot unlearning solutions irrespective of algorithmic innovation. Progress may require embedding geometry manipulation during pretraining, continuous attribute spaces, multi-dataset validation, and integration of explicit fairness constraints into unlearning objectives. Extending these frameworks to gradient-based methods, federated unlearning, and alternative representation spaces is necessary for wider applicability and robustness.

Conclusion

This paper delineates and formalizes bias redistribution as an intrinsic outcome of visual machine unlearning. Through comprehensive geometric and empirical evaluation, it demonstrates that current zero-shot unlearning methods frequently amplify rather than mitigate bias, with redistribution following gender-encoded boundaries in CLIP's embedding space. The impossibility of simultaneously achieving perfect forgetting, utility preservation, and fairness is established both theoretically and empirically, prompting calls for new fairness-aware unlearning strategies and embedding space audits in practical deployments.