Erased, but Not Gone: Output Forgetting Is Not True Forgetting

Abstract: Machine unlearning (MU) is commonly judged by output forgetting, such as low forget-set accuracy or reduced logit-level membership inference. But if output-level success can coexist with retraining-inconsistent residuals in representation space, what kind of forgetting are current evaluations actually certifying? We study this question through retraining-consistent representation forgetting, using the retrained model (i.e., trained from scratch without the forget data) as an operational reference for correct forgetting. Across multiple unlearning methods, datasets, and models, our theoretical analysis and empirical results show that standard output-level evaluation can systematically overestimate the success of unlearning. Under this stronger lens, current methods often appear forgotten at the output layer while exhibiting a structured mismatch relative to retraining. They partially align with retraining on forget samples, remain more inconsistent on retain samples, and leave residual discrepancy concentrated along retraining-related directions rather than diffuse in representation space. This structured mismatch is characterized by forget/retain asymmetry, directional mismatch, and concentrated residuals along retraining-related directions. These results suggest that current MU is often evaluated for apparent forgetting rather than retraining-consistent forgetting. More broadly, retraining reveals what output forgetting hides.

• The paper shows that output-level metrics can mislead by certifying models as forgotten even when significant residual representations persist.
• It employs representation-level diagnostics like CKA and membership inference to reveal structured, retraining-inconsistent residuals.
• The study highlights privacy risks and urges the design of unlearning algorithms that truly erase sensitive information from feature space.

Diagnostic Analysis of Machine Unlearning: Output Forgetting Versus True Representation Removal

Introduction

This paper, "Erased, but Not Gone: Output Forgetting Is Not True Forgetting" (2606.25001), interrogates the predominant evaluation paradigm in contemporary machine unlearning (MU) research. It provides extensive theoretical and empirical evidence that standard output-level evaluation—such as forget-set accuracy, logit-based membership inference, and retain-set accuracy—can systematically overestimate the effectiveness of MU algorithms. Specifically, it demonstrates that models can exhibit apparent forgetting in their outputs while still preserving retraining-inconsistent residuals in the learned internal representations. The implications for both evaluation practice and MU algorithm design are substantial: conventional criteria may provide only illusory guarantees of true information removal.

Output-Level Versus Representation-Level Notions of Forgetting

The foundational distinction established is between "output forgetting," which reflects the alignment of model outputs (predictions) with those of a retrained reference, and "retraining-consistent representation forgetting," which requires alignment at the level of learned feature representations. The operational reference for correct forgetting is the representation geometry and transformation induced by complete retraining on the reduced dataset (i.e., retraining from scratch without the forget set). The paper precisely formalizes these notions and proves (Theorem 1) that perfect output alignment does not imply representation-level agreement: it is always possible to construct an unlearned model that matches retrained outputs on both forget and retain sets, yet differs in internal feature distributions.

Empirical Evaluation and Failure Mode Characterization

Experimental Protocol

A diverse suite of MU methods—including SCRUB, Boundary Shrink, UNSIR, Amnesiac, SSD, as well as representation-level methods like POUR-P and POUR-D—are evaluated across canonical computer vision benchmarks (CIFAR-10, CIFAR-100, TinyImageNet) and architectures (ResNet-18, ResNet-50, ViT-Tiny). Baseline references are the original model (pre-unlearning) and the retrained model (completely retrained on the retain set).

Evaluation is multidimensional:

• Output-level metrics: forget and retain accuracy, logit-based membership inference attacks.
• Representation-level metrics: centered kernel alignment (CKA) to retraining, representation-based membership inference (MIArep), subspace-localized residual analysis (projecting onto retraining-induced directions), directional alignment (cosine similarity of learned representation shifts), and linear probe recoverability.

Main Empirical Findings

Current output-centric criteria consistently "certify" models as forgotten (e.g., near-zero forget-set accuracy, diminished logit MIA), yet representation diagnostics expose persistent, structured residuals:

• Representation similarity to retraining (CKA) remains substantially below the retrained baseline for most methods, even when output metrics suggest success.
• Membership leakage in feature space persists (high MIArep), indicating that information about forgotten samples remains linearly or non-linearly accessible.
• Directional and geometric mismatch: The shifts induced by unlearning in representation space are only partially aligned with retraining and frequently exhibit substantial asymmetry between forget and retain sets—forget-sample directions may weakly approximate retraining, but retain-sample directions are often misaligned or even reversed.
• Residuals are structured: Discrepancies are concentrated along retraining-related subspaces rather than being isotropically distributed noise. Both in terms of magnitude and direction, these residuals indicate that current approaches optimize for endpoint output similarity rather than true internal state erasure.
• Persistence across scale and regime: This failure mode recurs for larger models, more complex datasets, alternative backbones (CNN/ViT), forget-class choices, and varied random seeds, eliminating hypotheses of accidental or setting-dependent artifacts.

Theoretical and Practical Implications

Evaluation

The results invalidate the sufficiency of output-level metrics as universal success signals for unlearning. While such metrics are relevant in strict black-box threat models, the demonstrable persistence of retraining-inconsistent representations—auditable via white-box access or model weight release—implies that real information erasure is not guaranteed by endpoint metrics alone. The use of retrained models as references for auditing is thus essential for any claim of true unlearning.

Methodology

Algorithmic implications are direct: current state-of-the-art MU methods, largely optimized to minimize output-level leakage or perturb the decision boundary, do not reliably enforce retraining-consistent transformations in feature space. This exposes privacy and compliance vulnerabilities, particularly in sensitive or regulated settings. The development of next-generation MU algorithms requires direct regularization or engineering of the representation space to minimize structured residuals and better approximate the retraining map.

Theoretical Direction

The analysis raises important questions regarding the very objectives of MU: whether output indistinguishability is sufficient, or whether cryptographic or information-theoretic erasure in representation space should be mandated. The indicated failure mode unifies concerns raised by prior work regarding membership inference resilience, but makes explicit the necessity of representation-level diagnostics for reliable MU certification.

Future Work

The findings motivate probing questions in both theoretical and algorithmic research: (1) how to enforce or quantify alignment to the retrained representation transformation under practical constraints (e.g., without explicit retraining), (2) what constitutes an operationally sufficient definition of forgetting in adversarial or hybrid adversarial-auditing settings, and (3) how to design scalable reference-free surrogate diagnostics that reliably signal true erasure.

Conclusion

This work provides compelling evidence that the dominant evaluation metrics in machine unlearning—output-level certification—are systematically insufficient to verify true forgetting. Persistent, structured residuals remain in learned representations, frequently escaping detection by endpoint behavior but readily revealed through retraining-consistent analysis. Correction of this evaluation-practice gap is necessary for both robust algorithm design and for satisfying privacy, security, and regulatory demands. Progress will require integrating representation-level analysis directly into both MU method development and unlearning certification protocols.