Class Unlearning via Depth-Aware Removal of Forget-Specific Directions

Abstract: Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forget-class structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retain-class performance and reducing residual forget-class structure in deep layers.

• The paper presents DAMP, a one-shot, closed-form weight editing approach that explicitly erases forget-class directions from feature representations.
• It employs depth-aware scaling and projection to balance effective forgetting with high retain-class performance, validated across various models and datasets.
• Empirical results demonstrate that DAMP closely replicates retraining outcomes while minimizing collateral bias and preserving internal representational geometry.

Depth-Aware Removal of Forget-Specific Directions for Class Unlearning

Motivation and Limitations of Existing Class Unlearning Methods

Class unlearning has emerged as a critical topic in machine learning due to the practical need to erase specific class-related knowledge from large pretrained models post-deployment, especially in privacy, compliance, or fairness scenarios. Traditional post-hoc unlearning approaches—gradient ascent on forget classes, knowledge distillation, data deletion fine-tuning, relabeling, or saliency-based edits—demonstrate important limitations. Many achieve low forget-class accuracy, but often at the expense of retain-class performance, or more crucially, they fail to sufficiently eliminate internal class-specific representations, instead relying on suppression at the classifier head. This phenomenon is evidenced by minimal or negative selectivity scores (i.e., the ability to remove forget-class evidence without damaging retain-class representations) and persistent forget-class structure in deep-layer activations, as revealed by representational analyses.

Figure 1: Comparative performance of DAMP and baselines on retain accuracy, forget accuracy, and efficiency shows DAMP's superior tradeoff in class unlearning for a 5-layer CNN on CIFAR-10.

Figure 2: DAMP exhibits significantly higher selectivity compared to existing methods, with values close to retraining, indicating reduced collateral damage and better retention of utility.

Additionally, naive output suppression, particularly via bias shift in the classifier layer, can result in dramatically reduced prediction probability for the forgotten class while leaving semantic evidence largely intact in the underlying feature computation. This leads to scenarios where, despite perfect output "forgetting," internal representations remain decodable or transferable for the forbidden class. These insights necessitate class unlearning methodologies that explicitly erase semantic evidence for forget classes across network depth.

Figure 3: Existing methods achieve output-level forgetting via strong forget-class bias shifts, whereas DAMP has minimal reliance on bias shift, reflecting representational removal rather than classifier suppression.

DAMP: One-Shot, Depth-Aware Representation Erasure

The paper introduces DAMP (Depth-Aware Modulation by Projection), an explicit, non-iterative weight-editing method to achieve class unlearning by removing forget-specific directions from a pretrained network’s feature space. DAMP operates by:

1. Prototype Construction: For each class at each layer, compute the mean feature (prototype) in the editable representational space (i.e., input to the next learnable operator).
2. Span Decomposition: For each forget-class prototype, project onto the span of retain-class prototypes, isolating the residual component orthogonal to the retain span—these residuals encapsulate forget-class information not explained by retained classes.
3. Direction Orthonormalization and Projection: Collect the residual directions, orthonormalize them, and project out their span from the next operator's weight tensor, attenuating sensitivity to forget-class evidence while preserving retain-class geometry.
4. Depth-Aware Scaling: The update strength at each layer is modulated by a coefficient that incorporates both the linear separability between retain and forget classes (measured by probe accuracy) and the layer’s depth, enforcing weaker edits in early layers and stronger edits in deeper, more class-specific layers.

DAMP is architecture-agnostic (suitable for CNNs, ResNets, ViTs), operates in a strictly closed-form fashion, and extends naturally to multi-class forgetting via low-rank subspace projection.

Figure 4: DAMP framework overview: prototypes, residual computation, orthonormalization, depth-aware scaling, and projection-based weight update cascade.

Empirical Evaluation

DAMP is evaluated on MNIST, CIFAR-10, CIFAR-100, Tiny ImageNet, and for various model families (CNN-5, ResNet-18, ViT). The protocol consists of both single-class and multi-class forgetting, with direct comparison against accepted baselines: retraining, gradient ascent unlearning (GAU), knowledge distillation unlearning (KDU), fine-tuning after final-layer reinitialization (DD-FT), logit masking (LM), random relabeling, selective synaptic dampening (SSD), and saliency unlearning (SalUn).

Selectivity and Representational Fidelity

DAMP consistently demonstrates high selectivity, matching retraining in removing forget-class evidence with minimal disruptiveness to retained representations. Notably, while baseline or retraining methods appear similar at the output level when measured by forget accuracy or retain accuracy, DAMP distinguishes itself via its ability to closely recapitulate the geometry of retraining in representational space (see RDM and t-SNE visualizations).

Figure 5: RDM difference matrices show DAMP most closely matches retraining, removing block-structure for the forget class and preserving retain-class geometry, in contrast to the representational deviation of other methods.

Figure 6: t-SNE projections of deep-layer features show DAMP minimizes forget-class separability while preserving the cluster structure of retained classes, aligning with retraining.

Minimal Classifier Bias Reliance

Comprehensive bias analysis confirms DAMP's unlearning is not driven by head-layer suppression. Sweeping the classifier bias for the forgotten class has negligible impact on forget/retain performance under DAMP, unlike other methods.

Figure 7: Forcing the classifier bias to the retrained delta under DAMP has no effect at the output, confirming forgetting is not achieved via bias manipulation.

Depth-Aware Projection Trade-Offs

Layer-wise scaling of projection strength, using both probe accuracy and layer depth, is critical for balancing forgetting and retention. Ablations confirm that undifferentiated or excessively aggressive projection impairs retain-class accuracy, while insufficient projection preserves forget-class evidence.

Figure 8: Varying projection strength with depth-aware scaling achieves robust forgetting/retention tradeoff; aggressive projection hurts utility, insufficient projection allows residual forget-class evidence.

Robustness, Continual Unlearning, and Dense Prediction

DAMP maintains high retain accuracy under adversarial attacks (FGSM/PGD) while continuing to drive forget accuracy to zero, outperforming existing baselines on robust unlearning metrics. The method also extends to continual unlearning scenarios, incrementally forgetting multiple classes without significant performance degradation or error accumulation. For dense prediction (semantic segmentation), DAMP suppresses forgotten class regions with minimal distortion to retained regions, closely mirroring retraining results.

Figure 9: Semantic segmentation unlearning—DAMP suppresses forgotten class (Human) predictions and preserves retained-class structure, while competing methods induce retain-class degradation or incomplete unlearning.

Theoretical and Practical Implications

DAMP demonstrates that one-shot, closed-form weight surgery—guided by internal prototype geometry and adapted by depth-aware scaling—can match retraining in both output and representational forgetting efficiency. It introduces the selectivity analysis paradigm, highlighting the necessity of representation-level metrics for robust class unlearning evaluation, and exposes the inadequacy of simple output-based metrics or head-layer suppression. Importantly, DAMP provides a foundation for future extensible techniques—addressing non-linearly separable or highly multimodal class features via subspace- or distribution-aware generalizations, and potential adaptation to other model types or tasks (e.g., generative modeling, LLMs).

Conclusion

DAMP establishes a new standard for class unlearning by integrating explicit, depth-aware representation projection into post-hoc model editing. Its empirical results—both quantitative and qualitative—demonstrate strong selective forgetting, minimal collateral damage, and architectural generality. The approach advocates for a shift in unlearning research towards structure-preserving, closed-form representation surgery, supported by rigorous internal analysis beyond naive output scores. This has notable implications for deployable, scalable, and auditable unlearning protocols in modern AI systems, and sets the groundwork for broader study of geometric and information-theoretic interventions for knowledge erasure in deep networks.

Reference:

"Class Unlearning via Depth-Aware Removal of Forget-Specific Directions" (2604.15166)