LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning

Abstract: LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged as a promising solution, with state-of-the-art(SOTA) methods often following a localize-first, unlearn-second paradigm that targets specific model parameters. However, existing benchmarks evaluate unlearning solely at the output level, leaving open the question of whether unlearning truly erases knowledge from a model's parameters or merely obfuscates it, a concern reinforced by the success of resurfacing attacks. To bridge this gap, we introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage. We use LACUNA to benchmark current SOTA unlearning methods and find that, despite strong output-level performance, existing methods are highly imprecise and susceptible to resurfacing attacks. We further show that when localization is successful, even a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks, highlighting the importance of precise unlearning. We release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.

• The paper introduces LACUNA, a novel testbed that injects synthetic PII into LLM parameters to precisely evaluate unlearning methods.
• It employs a three-phase pipeline including masked continual pretraining, instruction tuning, and unlearning evaluation via ROC-AUC for localization precision.
• Compared to standard output-level metrics, the study reveals that effective unlearning requires targeted parameter-level interventions to prevent resurfacing attacks.

Lacuna: A Testbed for Evaluating Localization Precision for LLM Unlearning

Motivation and Problem Formulation

The paper "LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning" (2607.02513) addresses a critical gap in LLM unlearning research—establishing whether model unlearning actually removes targeted knowledge from model parameters, rather than merely obfuscating outputs. Prior benchmarks have focused strictly on output-level metrics, which are insufficient as evidenced by the susceptibility of models to resurfacing and reconstruction attacks. LLMs can harbor latent memorization even after behavioral unlearning, challenging privacy guarantees in practical deployments.

Testbed Construction: Parameter-Level Knowledge Injection

Lacuna is introduced as the first testbed for parameter-level unlearning precision evaluation. The pipeline consists of three phases:

1. PII Injection via Masked Continual Pretraining: Synthetic PII from the Panorama dataset is injected into predetermined subsets of model parameters by gradient masking during continual pretraining. This guarantees ground-truth knowledge localization, enabling evaluation beyond attribution-based or circular analysis.
2. Instruction Tuning: Parameter-efficient instruction tuning is performed on models to facilitate QA-based PII extraction.
3. Unlearning and Evaluation: Various state-of-the-art unlearning methods are evaluated for both output-level efficacy and localization precision via ROC-AUC between the ground-truth mask and modified parameters.

   Figure 1: The pipeline for injection, unlearning, and evaluation phases; knowledge is injected into specific parameters, unlearning is applied, and localization precision is assessed.

The PII injection protocol uses per-parameter binary masks covering 5% of parameters per group. Masking is implemented in a manner that is scalable to models with billions of parameters, handling the DDP and FSDP distributed training requirements efficiently. This ensures that synthetic PII is stored exclusively in designated weights, mitigating cross-group contamination.

Figure 2: Structure for synthetic PII injection and QA extension; profiles are partitioned for training, evaluation, and hyperparameter tuning.

The Panorama dataset is leveraged for realistic synthetic PII, and QA pairs are generated with prompt-continuation templates targeting multiple PII fields. Diverse PII types (email, birth, phone, driver’s license) are specifically selected to model a range of memorization difficulty and structure.

Model Training and Injection Efficacy

Masked continual pretraining minimally impacts model utility. Evaluation on Hellaswag, MMLU, ARC-Easy, and ARC-Challenge benchmarks demonstrates that injection does not significantly degrade general model performance.

Figure 3: Accuracy comparison across pretrained, masked, and unmasked models—masked models maintain baseline capabilities.

Extraction rates for masked models reveal successful memorization for adequate sizes of forget/retain sets. Fine-grained masking hinders memorization, but at 5% coverage, reliable extraction is achievable.

Figure 4: Memorization performance for masked vs. unmasked training—memorization is lower under masking but sufficient for evaluation.

Forget/Retain Set Construction

The forget and retain sets are assigned to disjoint parameter masks, ensuring targeted unlearning and utility preservation. A cross-field scheme is adopted to minimize overlap in QA structure between forget and retain examples. Paraphrased variants for both sets augment evaluation for generalization.

Unlearning Methods Evaluated

1. Gradient-Based: E15759 (SimNPO) leverages negative preference optimization with reference-free, length-normalized rewards for uniform unlearning. It represents current SOTA for optimization-based approaches.
2. Localization-Based: 4E79A7 (AlphaEdit) and F28E2B identify critical FFN layers via causal tracing, then apply null-space constrained edits or localized gradient updates, respectively.
3. Oracle Baseline: 59A14F has privileged access to the ground-truth mask, confining edits strictly to weights storing PII.

Evaluation Metrics

Output-level metrics include Exact Memorization, Extraction Strength, and Probability. Localization precision is quantified by ROC-AUC on binary classification (in-mask vs. out-of-mask), leveraging magnitude-, reversal-, and contrast-based scoring and a composite logistic regression approach.

Empirical Findings: Unlearning Efficacy and Precision

E15759 achieves strong output-level forgetting but exhibits only marginal localization precision (ROC-AUC ≈ 0.515), indicating indiscriminate weight updates and limited targeting. Similarly, the localization-based methods are ineffective at discriminating masked weights (ROC-AUC ≈ 0.5), even when restricting to identified components.

Figure 5: Output-level metrics for forget/retain sets showing efficacy differences among methods.

Figure 6: Relearning leakage rate under resurfacing attacks—gradient-based and localization-based methods leak substantially, oracle remains robust.

In contrast, the oracle baseline 59A14F yields ROC-AUC ≈ 0.915, showing nearly perfect targeting of masked parameters and superior robustness to resurfacing attacks. Output-level utility is preserved, and forgotten PII is not reconstructed following adversarial retraining.

Implications and Future Directions

Contradictory Claim: High output-level unlearning performance does not correspond to high localization precision, undermining true erasure and privacy guarantees.

This result challenges the sufficiency of output-level evaluation for certifying knowledge removal. Susceptibility to resurfacing attacks persists unless the parameter-level localization is precise. Moreover, methods that rely on attribution or post-hoc localization are prone to circularity and cannot serve as independent evaluators of true parameter involvement.

Practical Perspective: Unlearning protocols must account for parameter-level precision, especially when handling sensitive PII or compliance-driven data removal. Lacuna provides infrastructure for such rigorous evaluation, facilitating both method development and regulatory audits.

Theoretical Perspective: Knowledge localization is intrinsically linked to erasure robustness. Future theoretical developments must address the granularity and dispersion of memorized knowledge in dense Transformers. There is a compelling need to confine memorization to modular or locally editable regions, potentially architecting models for improved localization during initial training.

Figure 7: Relearning vulnerability and Jaccard similarity across methods—robust methods overlap on difficult-to-unlearn samples, suggesting inherent data difficulty.

Conclusion

Lacuna establishes an experimental foundation for evaluating unlearning at the parameter level. The evidence presented demonstrates that existing methods fail to reliably target the weights containing undesirable knowledge, leading to vulnerability through resurfacing. Precise localization correlates with erasure robustness, compelling researchers to pivot toward methods that optimize parameter-level intervention.

Advances in knowledge localization, parameter masking, and architecture-level confining of memorization represent pivotal future directions. Lacuna’s release is poised to catalyze progress in robust unlearning for LLMs under privacy, safety, and regulatory constraints.