PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning

Abstract: LLMs often memorize private information during training, raising serious privacy concerns. While machine unlearning has emerged as a promising solution, its true effectiveness against privacy attacks remains unclear. To address this, we propose PrivUn, a new evaluation framework that systematically assesses unlearning robustness through three-tier attack scenarios: direct retrieval, in-context learning recovery, and fine-tuning restoration; combined with quantitative analysis using forgetting scores, association metrics, and forgetting depth assessment. Our study exposes significant weaknesses in current unlearning methods, revealing two key findings: 1) unlearning exhibits gradient-driven ripple effects: unlike traditional forgetting which follows semantic relations (e.g., knowledge graphs), privacy unlearning propagates across latent gradient-based associations; and 2) most methods suffer from shallow forgetting, failing to remove private information distributed across multiple deep model layers. To validate these insights, we explore two strategies: association-aware core-set selection that leverages gradient similarity, and multi-layer deep intervention through representational constraints. These strategies represent a paradigm shift from shallow forgetting to deep forgetting.

• The paper introduces a suite of metrics and adversarial scenarios to systematically evaluate privacy unlearning in large language models.
• The study reveals that gradient-based associations cause extensive ripple effects and result in shallow forgetting, leaving intermediate layers largely unaltered.
• The proposed methods, including association-aware core-set selection and representational anchoring, achieve deeper erasure while preserving model utility.

PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning

Motivation and Problem Setting

The proliferation of LLMs has raised widespread concern regarding memorization of PII present in training corpora, creating critical privacy and compliance challenges. While current machine unlearning methodologies have offered practical protocols to erase model-specific data contributions, frameworks for systematically evaluating their privacy efficacy—especially under active adversarial scenarios—remain underdeveloped. "PrivUn: Unveiling Latent Ripple Effects and Shallow Forgetting in Privacy Unlearning" (2604.22076) addresses these gaps through the introduction of PrivUn, a suite of metrics and evaluation scenarios targeting the robustness of privacy unlearning in LLMs, with a focus on not just passive but also active attack vectors. The PrivUn framework emphasizes a structural and mechanistic analysis of how PII persists or is eradicated, integrating nuanced attack settings and analytic tools to expose both the strengths and limitations of prevailing approaches.

PrivUn Benchmark: Formalization, Attack Scenarios, and Metrics

PrivUn systematically audits privacy unlearning effectiveness across three semantically escalating adversarial scenarios:

• Direct Retrieval (P1): Measures surface-level memorization by querying the model for direct PII recall.
• In-context Recovery (P2): Probes for recovery of forgotten data via few-shot prompting, leveraging the model's in-context generalization capabilities.
• Fine-tuning Recovery (P3): Assesses the resilience of unlearning against adversaries with access to original forgotten data and the ability to fine-tune the model, targeting the most persistent privacy risks.

These attack scenarios are supported by three core metrics: (1) direct recovery ratio for strict extraction; (2) association-oriented metrics analyzing model-internal entanglement via gradients, representations, and explicit relational graphs (e.g., sender-recipient topologies in real-world datasets); and (3) forgetting-depth analysis via layer-wise Centered Kernel Alignment (CKA), which discerns the layer-wise propagation (or stalling) of the unlearning process.

Figure 1, Figure 2, and Figure 3 below illustrate the mechanisms underlying pipeline manipulation methods, PageRank-based graph association, and an overview of the experimental training and manipulation design, respectively.

Figure 1: Gradient ascent (GA) as a representative pipeline manipulation method for privacy unlearning.

Figure 2: Association strength using personalized PageRank scores on relational graphs, exposing the lack of correspondence between graph proximity and effective forgetting.

Figure 3: Training pipeline manipulation schema highlighting data flow and intervention points for unlearning.

Empirical Findings: Ripple Effects and Shallow Forgetting

Ripple Effects: Gradient-Driven Association

A central finding is that privacy unlearning induces broad ripple effects in the model, whereby the removal of explicit PII information propagates latently to other data points not explicitly unlearned. Critically, PrivUn demonstrates that the extent and focus of these ripple effects are primarily determined by gradient-based association within the model’s optimization landscape, and not by semantic or relational graph proximity.

Analysis shows that PageRank-based association (topological, Figure 2) is essentially uncorrelated with unlearning efficacy, while gradient-based metrics show a strong positive correlation (Pearson $r > 0.7$ in some settings) with actual forgetting. The propagation of forgetting is therefore fundamentally attached to shared parameter update trajectories in the model.

Notably, representational similarity becomes a weak but emergent factor only in deeper layers (Figure 4), further underscoring the dominant role of parameter-gradient dynamics over purely semantic relations.

Figure 4: Shallow forgetting analysis across 19 unlearning methods, demonstrating pronounced shallow-vs-deep forgetting gaps as measured on the Enron dataset.

Shallow Forgetting: Inadequate Deep Erasure

PrivUn's layerwise analysis via CKA reveals that most unlearning methods—across pipeline and data manipulation paradigms—primarily modify the output (top) layers, leaving intermediate feature transformations largely unchanged. This incomplete erasure characterizes what the authors term "shallow forgetting": the phenomenon whereby direct querying is suppressed (low P1 rates), yet fine-tuning rapidly rediscovers forgotten PII (large P1–P3 gaps, often >50%). These findings indicate a failure to fully excise PII from the network’s hierarchical representations, making models broadly vulnerable to realistic, iterative adversarial re-injection attacks.

Such findings are visually supported by the CKA analysis, which shows that only representation-based methods (e.g., RMU) are capable of inducing deeper representational shifts; however, this often comes at the expense of substantial utility degradation.

Core-set Selection and Multi-layer Intervention: A Paradigm Shift

Guided by these mechanistic insights, the paper demonstrates the efficacy of association-aware core-set selection, whereby a minority of highly gradient-associated forget set samples triggers a disproportionately broad forgetting signature—exploiting the latent optimization entanglements to optimize erasure per sample. This method achieves a P3 recovery rate reduction to 32.19% using only 10% of the forget set, outperforming random selection strategies by a wide margin.

In parallel, the paper introduces a multi-layer, utility-preserving representational anchoring strategy (RAU), which leverages layer-targeted constraints to induce deeper and more effective forgetting without the drastic utility collapse observed in naïve representation-based interventions.

Implications and Theoretical Impact

The results of PrivUn have significant theoretical and practical implications:

• Model Editing and Unlearning Differentiation: The findings contrast substantially with knowledge unlearning and model editing, where structural semantic dependencies (knowledge graphs) are meaningful predictors of the forgetting footprint. For privacy unlearning, optimization dynamics are paramount.
• Evaluation Paradigm: PrivUn's multi-attack evaluation exposes the inadmissibility of existing benchmarks focused on passive attacks, and demonstrates the necessity of layerwise and association-based analyses.
• Design of Unlearning Defenses: The observed ripple and shallow forgetting effects imply that effective privacy erasure must shift away from output-level interventions, and instead strategically engineer deep, multi-layer optimization pressure—preferably via association-aware selection.

  Figure 5: Subgraph of the Enron sender-recipient network, illustrating the misalignment between high graph centrality and effective PII forgetting.

Conclusion

PrivUn introduces a systematic, multi-angle framework for the evaluation and analysis of privacy unlearning in LLMs, exposing the fundamental limitations of current methods with respect to gradient-driven ripple effects and persistent shallow forgetting. These findings highlight the necessity for new unlearning paradigms that explicitly address internal optimization associations and penetrate multiple layers of the network. Future work should further formalize gradient-based core set selection and scalable, utility-preserving representational constraints to meet the practical and theoretical demands of privacy-preserving LLM deployment.