Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation

Abstract: Class-level machine unlearning aims to remove the influence of specified classes while preserving model utility on retained classes. Existing methods are commonly evaluated by retain-set accuracy, forget-set accuracy, and unlearning time, but these metrics provide limited insight into how forgetting is achieved internally. In this paper, we reveal a bias-dominated shortcut in class-level unlearning: the prediction of forgotten classes can be suppressed by decreasing the corresponding bias terms in the final classification head. We first analyze the gradient dynamics of classification-head biases under softmax cross-entropy training, explaining why retain-set-only optimization tends to reduce the biases of absent classes. Based on this observation, we introduce BiasShift as a diagnostic baseline, showing that simple bias manipulation can satisfy conventional unlearning metrics while leaving abnormal bias patterns that reveal forgotten labels. To mitigate excessive forgotten-class bias suppression, we propose two bias-aware mechanisms, namely Two-Stage Bias Gradient Reversal Mechanism (TS-BGRM) and Lower-Bound Hinge Regularization (LB-HR). We further introduce three bias-oriented metrics, including Bias Stability Coefficient (BSC), Median Bias Gap (MBG), and Minimal Bias Score (MBS), to quantify bias dependence and potential leakage. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that the proposed methods maintain competitive unlearning performance while producing more stable bias distributions. We have released our code at {https://github.com/zwd2024/Beyond-the-Shadow-of-Bias-From-Classification-Head-Bias-to-Parameter-Redistribution}.

• The paper reveals that suppression of class logits via bias terms can simulate forgetting without eliminating underlying representations, thus undermining privacy.
• It introduces two mitigation mechanisms, TS-BGRM and LB-HR, which redistribute bias effects to ensure effective unlearning while preserving retained-class accuracy.
• Extensive experiments using CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate improved bias stability metrics, reinforcing the need for internal parameter evaluations.

Classification-Head Bias in Class-Level Machine Unlearning: Analysis, Mitigation, and Evaluation

Problem Definition and Motivation

Class-level machine unlearning requires selectively removing all influence derived from specific classes from trained models, often for privacy compliance or data correction. This task aims to simultaneously maximize accuracy on the retained classes, suppress accuracy on the forgotten ones, and minimize the computational cost of unlearning—typically measured by retain-set accuracy, forget-set accuracy, and unlearning time. However, this paper identifies a critical limitation: these common metrics insufficiently interrogate the mechanism by which unlearning is achieved and can be satisfied through an undesirable shortcut—manipulation of the classification-head bias.

The central argument is that suppression of class logits via the final-layer bias terms enables apparent class forgetting without erasing the feature-level or representational dependencies on the forgotten classes. This leads to two practical threats: (1) Conventional accuracy-based metrics overstate the degree of true "forgetting," and (2) the resulting abnormal bias patterns become a channel for forgotten-label leakage, undermining privacy guarantees.

Theoretical Diagnosis of Bias Dynamics

A formal analysis of gradient flows in the classification head under softmax cross-entropy demonstrates that absence of a class in the optimization dataset causes its corresponding bias to monotonically decrease. For class $c$ omitted from the retain set, the gradient with respect to $b_c$—the bias for class $c$—is always positive (as $P_c > 0$, with $P_c$ the predicted probability of class $c$), so gradient descent will continually suppress $b_c$. This phenomenon occurs regardless of whether models are fine-tuned, retrained from scratch on the retain set, or subjected to approximate unlearning.

The paper introduces the BiasShift diagnostic baseline: simply subtracting a large constant from the biases of forgotten-class heads. BiasShift produces zero forget-set accuracy and strong retain-set accuracy, exposing that models are gaming the output layer rather than erasing class-relevant information in the representation or weight space. The resulting abnormal separation between bias values for forgotten and retained classes constitutes a major privacy fingerprint.

Mitigation: Bias-Aware Unlearning Mechanisms

To address this bias-dominated shortcut, the paper proposes two mitigation strategies that encourage parameter redistribution so that the "unlearning effect" is realized in more than just the bias vector:

• Two-Stage Bias Gradient Reversal Mechanism (TS-BGRM): This procedure first applies bias-gradient reversal to the forgotten-class heads using the forget set (destroy stage), disrupting their decision behavior. Subsequently, the retain set is used to repair utility on non-forgotten classes. This mechanism explicitly reduces the gap between bias values of forgotten and retained classes while maintaining unlearning capability.
• Lower-Bound Hinge Regularization (LB-HR): During unlearning, LB-HR introduces a hinge-penalty in the loss to prevent biases of forgotten classes from dropping below a pre-defined lower bound. Regularization strength is governed by a tunable hyperparameter, providing direct mitigation against extreme outlier bias values, without impeding conventional unlearning efficacy.

Both methods operate with the feature extractor frozen, focusing all parameter updates within the classification head, allowing strict control and observation of the redistribution of unlearning effects.

Bias-Oriented Evaluation Metrics

Recognizing the inadequacy of retain/forget accuracy and unlearning time, the authors define three bias-oriented metrics:

• Bias Stability Coefficient (BSC): Quantifies deviation between the mean bias of forgotten and retained classes—values close to 100% indicate minimal bias gap.
• Median Bias Gap (MBG): Uses a sigmoid-normalized difference between the median forgotten-class bias and the minimum retained-class bias to track typical bias suppression.
• Minimal Bias Score (MBS): Focuses on the minimal forgotten-class bias, signaling if any outlier bias exists that could facilitate leakage.

High BSC/MBG/MBS scores indicate less bias dependence, whereas low scores reveal bias-level shortcuts and increased leakage risk.

Experimental Analysis

Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet compare TS-BGRM, LB-HR, and numerous class-level unlearning baselines (fine-tuning, NegGrad+, random-label, SalUn, SCRUB, UNSIR, SSD, etc.). Evaluation conclusively demonstrates:

• BiasShift achieves strong conventional metrics but produces catastrophic BSC/MBG/MBS scores, confirming the presence of a severe bias signature.
• Many established baselines (e.g., fine-tuning, NegGrad+, SCRUB, SalUn) also result in reduced bias stability.
• TS-BGRM and LB-HR maintain competitive retain/forget accuracies while preventing the emergence of extreme bias outliers, indicated by high BSC/MBG/MBS values.
• Visualization of bias distributions further clarifies that conventional methods systematically suppress forgotten-class biases, while proposed mechanisms yield more uniform and less revealing bias vectors.

Efficiency is also validated: BiasShift is computationally negligible; TS-BGRM and LB-HR incur significantly lower time costs than retraining, making them suitable for practical deployments constrained by time or compute.

Implications and Future Directions

This work establishes that output-layer bias manipulation represents a central failure mode for class-level unlearning and that privacy and reliability evaluation cannot rely solely on output behavior or speed. Comprehensive unlearning performance must incorporate parameter-level evaluation: low overlap in bias distributions signals incomplete unlearning with potential for attribute leakage.

TS-BGRM and LB-HR provide effective and efficient methods for mitigating this shortcut, with minimal impact on typical accuracy metrics and significant improvement in bias stability. These findings imply that future industrial or regulatory standards for machine unlearning should consider bias-oriented metrics and explicit examination of internal parameter states.

Looking forward, this diagnosis and mitigation strategy should be extended to other architectures (e.g., transformers, GNNs), domains (natural language, federated learning), and granularities (sample-level unlearning, structured forgetting). Additional work should formalize adversarial threat models beyond access to the classification head (such as gradient or representation leakage), analyze interactions with model sparsity and pruning, and generalize parameter-based privacy audits.

Conclusion

The study rigorously diagnoses the bias-dominated shortcut as a fundamental vulnerability of class-level machine unlearning protocols evaluated with conventional metrics. By proposing bias-aware redistribution mechanisms and bias-oriented metrics, the paper sets a foundation for more robust, privacy-preserving, and auditable unlearning. Adoption of these methods and metrics is essential to ensure that unlearning satisfies not only utility and efficiency but also internal privacy and security imperatives.