Measuring the Depth of LLM Unlearning via Activation Patching

Abstract: LLM unlearning has emerged as a crucial post-hoc mechanism for privacy protection and AI safety, yet auditing whether target knowledge is truly erased remains challenging. Existing output-level metrics fail to detect when this knowledge remains recoverable from internal representations. Recent white-box studies reveal such residual knowledge but often rely on auxiliary training or dataset-specific adaptations, leaving no generalizable metric. To address these limitations, we propose the Unlearning Depth Score (UDS), a metric that quantifies the mechanistic depth of unlearning via activation patching. UDS first identifies layers that encode the target knowledge using a retain model baseline, then measures how much of it is erased in the unlearned model on a 0-1 scale. In a meta-evaluation across 20 metrics on 150 unlearned models spanning 8 methods, UDS achieves the highest faithfulness and robustness, confirming our causal approach as the most reliable for unlearning evaluation. Case studies further reveal that white-box metrics can disagree at the layer level and that erasure depth varies across examples. We provide guidelines for integrating UDS into existing benchmarking frameworks and streamlining the evaluation pipeline. Code and data are available at https://github.com/gnueaj/unlearning-depth-score

• The paper introduces UDS, a causal, training-free, and dataset-invariant metric that uses two-stage activation patching to measure internal knowledge erasure in LLMs.
• It demonstrates UDS's superior performance with an AUC-ROC of 0.971 and robustness under quantization and relearning attacks compared to output-level metrics.
• Case studies reveal that activation patching can uncover semantic-specific unlearning discrepancies, offering actionable insights for model auditing and privacy protection.

Activation-Based Evaluation of Unlearning in LLMs

Motivation and Prior Approaches

The proliferation of LLMs that memorize sensitive training data creates significant privacy and safety concerns, especially regarding the irrevocable retention of hazardous information. Numerous approaches have been proposed for post-hoc unlearning—removing target data from an already trained model while preserving utility. However, the reliability of current evaluation techniques is inadequate: output-level metrics (logit-based, generation-based) do not reliably detect residual knowledge, as adversaries can often restore erased information through minimal fine-tuning or activation manipulation. Existing white-box analyses, such as CKA, Fisher information, and Logit Lens, have exposed latent knowledge but typically rely on auxiliary training or are tied to specific datasets, leaving no generalizable or systematic metric for cross-method evaluation.

The Unlearning Depth Score (UDS) Metric

To address these limitations, the paper introduces the Unlearning Depth Score (UDS), a causal, training-free, and dataset-invariant metric for quantifying the depth of unlearning. UDS is defined as the erasure depth measured via two-stage activation patching:

1. Stage 1: Patches hidden states from a retain model (trained without the target data) into the full model (trained with the target data), layer-wise, to establish which layers encode the target knowledge.
2. Stage 2: Patches hidden states from the unlearned model into the full model, measuring how much encoded knowledge remains recoverable.

A per-example score is computed as the weighted aggregate of erasure ratios across the knowledge-encoding layers, yielding a normalized value from 0 (fully intact) to 1 (erased to the retain model's standard). The approach leverages teacher forcing for stable internal state comparisons.

Figure 1: Overview of UDS for a single forget set example, illustrating the two-stage causal activation patching mechanism and erasure quantification.

Faithfulness and Robustness Evaluation

UDS was meta-evaluated on 20 metrics across 150 unlearned models spanning 8 unlearning methods, encompassing gradient ascent variants, preference optimization (NPO, SimNPO, AltPO), representation-level interventions (RMU), and self-distillation (UNDIAL). These models were benchmarked on the TOFU forget10 dataset with Llama-3.2-1B-Instruct backbone.

Two reliability axes were used:

• Faithfulness: The ability of the metric to separate models trained with and without the target data (AUC-ROC).
• Robustness: Metric stability under perturbations, including 4-bit quantization and relearning attacks.

UDS achieved an overall score of 0.951, ranking first in both faithfulness (AUC-ROC 0.971) and aggregate robustness (HM 0.932), outperforming established metrics. Notably, output-based metrics such as ROUGE and Truth Ratio suffered dramatic declines under relearning, while CKA and Fisher were destabilized by quantization and fine-tuning due to their sensitivity to global representational shifts and gradient landscapes.

Figure 2: Quantization test comparing Truth Ratio and ROUGE: only the symmetric robustness formulation penalizes spurious score drops, highlighting UDS’s stability.

Figure 3: Faithfulness evaluation of four white-box metrics—UDS achieves superior pool separation due to causal intervention, while CKA and Fisher fail to directly reflect knowledge content.

Case Studies: Internal Knowledge Persistence and Erasure Heterogeneity

UDS enables granular, per-layer and per-example analysis of erasure depth. Contrasts between observational (Logit Lens) and causal (UDS) metrics reveal that activation distortion can mislead frozen-decoder metrics into falsely reporting knowledge erasure, whereas UDS detects recoverable knowledge via causal patching through the full model's non-linear pathways. Layer-wise analysis demonstrates substantial divergence in reported erasure, with UDS showing lower erasure scores than observational metrics in mid-layers.

Furthermore, UDS exposes heterogeneity in unlearning depth across prompt types even within a single method. For example, models trained with “I don’t know” responses (IdkNLL) display high erasure on Yes/No questions but minimal change for other entity types, indicating semantic-specific internal persistence. This would be missed by normalized output-level metrics.

Practical Implications for Unlearning Evaluation

UDS provides several actionable advantages for model auditing and benchmarking:

• Integration with Privacy Evaluation: UDS complements output-level membership inference scores by penalizing residual knowledge in internal representations, improving method ranking and hyperparameter selection for deep erasure.
• Evaluation Efficiency: Due to its robustness under quantization and relearning, UDS can obviate exhaustive post-perturbation benchmarking, streamlining evaluation pipelines.
• Cross-Scale Consistency: Monotonicity and proportionality of UDS persist across LLM scales (1B, 3B, 8B), confirming the metric’s generality.

  Figure 4: Mean Stage 1 patching delta per layer for four patching locations—patching the residual stream ($h_l$) captures the dominant share of encoded knowledge.

  

  Figure 5: Quantization robustness ($Q$) for all 20 metrics—UDS maintains high stability across models, while output and Fisher metrics are prone to destructive instability.

  

  Figure 6: Relearning robustness ($R$) for all 20 metrics—UDS remains resilient to rapid generation recovery, unlike output-based metrics.

Limitations and Future Directions

The primary requirement for UDS is access to a retain model. While not always feasible in deployment, the quantification stage can still causally audit models by patching unlearned activations into the full model. The metric clips over-unlearning at 1, thus does not distinguish over-unlearning from ideal erasure; utility metrics should be monitored alongside UDS for this purpose. Current evaluations focus on entity spans in QA-style datasets; extending UDS to broader tasks such as long-form reasoning or multi-step instruction-following is an open research direction.

Adaptation to other architectures (diffusion, vision transformers) is feasible, suggesting broad applicability for internal knowledge audits.

Conclusion

UDS advances the evaluation of LLM unlearning by quantifying internal knowledge erasure depth via causal activation patching, providing faithful and robust scores not achievable by output-level or observational metrics. UDS enables comprehensive benchmarking, reliable hyperparameter selection, and practical audit workflows, exposing latent knowledge persistence invisible to outputs and ensuring resilient privacy protection in deployed LLMs (2605.24614).