An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations

Abstract: While numerous machine unlearning (MU) methods have recently been developed with promising results in erasing the influence of forgotten data, classes, or concepts, they are also highly vulnerable-for example, simple fine-tuning can inadvertently reintroduce erased concepts. In this paper, we address this contradiction by examining the internal representations of unlearned models, in contrast to prior work that focuses primarily on output-level behavior. Our analysis shows that many state-of-the-art MU methods appear successful mainly due to a misalignment between last-layer features and the classifier, a phenomenon we call feature-classifier misalignment. In fact, hidden features remain highly discriminative, and simple linear probing can recover near-original accuracy. Assuming neural collapse in the original model, we further demonstrate that adjusting only the classifier can achieve negligible forget accuracy while preserving retain accuracy, and we corroborate this with experiments using classifier-only fine-tuning. Motivated by these findings, we propose MU methods based on a class-mean features (CMF) classifier, which explicitly enforces alignment between features and classifiers. Experiments on standard benchmarks show that CMF-based unlearning reduces forgotten information in representations while maintaining high retain accuracy, highlighting the need for faithful representation-level evaluation of MU.

• The paper demonstrates that output-level unlearning can be misleading, as classifiers merely adjust weights while core feature representations remain intact.
• It uses both empirical results and neural collapse theory to show that forgotten class features persist despite near-zero output accuracy.
• The introduction of CMF-based classifiers forces disruption of feature clusters, thereby achieving more genuine unlearning at the representation level.

Assessing the Illusion of Machine Unlearning Through Internal Representations

Introduction

This work diagnoses a fundamental limitation in the evaluation and practice of machine unlearning (MU): the disconnect between output-level success and persistent, recoverable information in neural representations. Focusing on class unlearning for image classifiers, the authors empirically and theoretically demonstrate that widely adopted MU methods achieve low test accuracy on forgotten classes only by manipulating the final classifier weights, leaving the feature representations essentially intact and still highly discriminative. The result is an "illusion of unlearning," whereby supposedly forgotten information can be trivially recovered via linear probing. Motivated by this, they introduce evaluation metrics and algorithms explicitly targeting information removal in the feature space, notably the integration of class-mean features (CMF) classifiers. Their results show that current output-based evaluations are inadequate, and effective unlearning must operate on representations throughout the network.

Background: Neural Collapse and Machine Unlearning

Deep classifiers in the terminal phase of training exhibit the neural collapse (NC) phenomenon, where last-layer features of samples from the same class concentrate at a class mean, class means are maximally separated, classifier weights align with class means, and the classifier becomes equivalent to a nearest-class-centroid rule [papyan2020prevalence]. This alignment is highly relevant for MU: the final classifier can be arbitrarily altered to suppress output-level accuracy for targeted (forgotten) classes, but as long as the representations cluster by true class, information persists.

Machine unlearning typically aims to efficiently erase the influence of specific data or classes from a trained model [bourtoule2021machine]. This is motivated by privacy regulations, intellectual property, or harmful/bias correction requirements. Evaluation is usually solely based on output-level forget accuracy (on the targeted/forgotten data) and retain accuracy (on the rest), but these metrics overlook the state of intermediate representations.

Shallow Unlearning: Output Suppression Without Representation Erasure

The authors extensively evaluate state-of-the-art MU techniques—Random Label, SalUn, NegGrad+, SCRUB, UNSIR, and SVD—on common benchmarks. They observe that nearly all these methods can drive output-level forget accuracy to zero, creating the appearance of perfect forgetting. However, when a linear classifier is trained (linear probe) on the frozen last-layer features after unlearning, the original classification performance on the forget set is almost entirely recoverable. Similarly, nearest-class-centroid (NCC) classifiers also yield high forget accuracy, confirming the persistence of class-specific information in features.

Figure 1: Output-level forget accuracy versus feature-level (linear probe, NCC) on CIFAR-100, demonstrating that although MU methods suppress outputs, features remain highly discriminative for forgotten classes.

This effect is dominant across all algorithms, architectures (convolutional and transformer-based), and datasets, including both single-class and multi-class forgetting scenarios, and holds for both fine-tuned and retrain-from-scratch baselines.

Figure 2: The learning curve for Random-label unlearning on CIFAR-10 shows rapid drop of output-level forget accuracy while linear-probe forget accuracy remains high.

Visualization of learned feature spaces with t-SNE further confirms that features for forgotten classes remain tightly clustered and linearly separable after unlearning, a structure not disrupted by MU methods.

Figure 3: A t-SNE visualization of the original model's feature space exhibits strongly clustered (and therefore linearly separable) class representations.

Theoretical Analysis: Feature–Classifier Misalignment

To explain these results, the authors leverage neural collapse theory. They mathematically show that, post-unlearning, the means of last-layer features for all classes remain fixed and, for the forgotten class, the classifier weight is simply rotated (or negated) relative to its corresponding class mean. This feature–classifier misalignment ensures that the output classifier avoids predicting forgotten classes, but the representations remain as discriminative as before.

Figure 4: Visualization of neural collapse after unlearning, depicting persistent alignment for retained classes and pronounced misalignment for the forgotten class.

Empirically, they measure the NC$_3$ criterion—distance between normalized classifier weight and class mean vector—finding large misalignment for forgotten classes but preserved alignment otherwise.

Figure 5: Output of the NC$_3$ metric for a single forgotten class; feature–classifier alignment is destroyed only for the forgotten class.

Furthermore, experiments limiting the MU step to classifier-only fine-tuning (keeping backbone features fixed) yield essentially the same outcome as full-model MU: zero output-level forget accuracy with unchanged, discriminative feature space.

Bold Claim: Classifier-only unlearning suffices for perfect output-level forgetting according to standard metrics, revealing the inadequacy of such evaluation.

Class-Mean Features (CMF) for Representation-Level Unlearning

To address persistent information in features, the paper proposes CMF-based unlearning. The classifier head is forced to align with running means of last-layer features per class, coupling the classifier with the geometry of the representation space. This constraint prevents trivial output-level suppression without modifying the underlying features, thus compelling the unlearning process to actually disrupt or disperse the hidden representations for forgotten classes.

CMF-based MU methods replace the standard classifier head after each training epoch with the mean feature vectors, optionally after centering and normalization, and can be integrated into any gradient-based unlearning algorithm.

Their experimental results demonstrate that all MU strategies with CMF display significant reductions in feature-level forget accuracy, in some cases substantially lower than even retraining on the retain set. Output-level retain accuracy is only modestly sacrificed. Visualizations (t-SNE) show that the forgotten class’s feature cluster is dispersed and substantially overlaps with other classes, signifying effective erasure of class-specific feature structure.

(Figure 1) (CMF variants)

Figure 1: The integration of CMF-based classifiers with MU methods reduces both output- and feature-level forget accuracy more faithfully.

Figure 6: t-SNE projections after CMF unlearning for various methods show destroyed separation between the forgotten class and retained classes.

Implications and Future Directions

The key practical implication is that faithful machine unlearning requires explicit attention to feature-level information, not just classifier outputs. Output-level metrics are easy to manipulate via classifier reconfiguration, leading to a false sense of security in privacy and intellectual property removal claims. This result generalizes to any context where representations are persistent—image, text, or generative models.

Theoretically, any approach not forcing geometric disruption at the representation level will at best induce a shallow form of unlearning. Applying CMF or analogous constraints (e.g., in intermediate layers) may also enrich our understanding of transferability, generalization, and privacy in deep learning.

Future work should address:

• Extension to generative and multimodal foundation models, where unintended concept resurgence is increasingly documented.
• Characterizing the trade-off between information removal from features and overall accuracy on retain data.
• Development of efficient metrics for representation-level unlearning with minimal computational burden.
• Applying CMF-style regularization to deeper layers given evidence of neural collapse beyond the output layer.

Conclusion

This paper demonstrates that most tested machine unlearning methods succeed at forgetting only by manipulating classifier weights, while underlying representations retain forgotten information, recoverable by simple linear models. This disconnect exposes current evaluation protocols as fundamentally inadequate. Enforcing classifier–feature alignment via CMF classifiers during unlearning forces actual disruption of class structure at the feature level, resulting in more faithful removal of targeted information. Representation-level evaluation and methods are required to advance MU as a genuine privacy- and ethics-preserving tool for real-world applications.