From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models

Abstract: LLMs may memorize sensitive or copyrighted content, raising significant privacy and legal concerns. While machine unlearning has emerged as a potential remedy, prevailing paradigms rely on user-provided forget sets, making unlearning requests difficult to audit and exposing systems to secondary leakage and malicious abuse. We propose MAGE, a Memory-grAph Guided Erasure framework for user-minimized, corpus-free unlearning. Given only a lightweight user anchor that identifies a target entity, MAGE probes the target LLM to recover target-related memorization, organizes it into a weighted local memory graph, and synthesizes scoped supervision for unlearning. MAGE is model-agnostic, can be plugged into standard unlearning methods, and requires no access to the original training corpus. Experiments on two benchmarks, TOFU and RWKU, demonstrate that MAGE's self-generated supervision achieves effective unlearning performance comparable to supervision generated with external reference, while preserving overall utility. These results support a practical and auditable unlearning workflow driven by minimal anchors rather than user-supplied forget corpora.

• The paper introduces MAGE, a novel corpus-free unlearning framework that autonomously identifies and erases sensitive data using memory-graph guidance.
• It employs iterative memory mining to build strength-weighted memory graphs and scoped supervision to precisely control unlearning with minimal collateral damage.
• Empirical evaluations reveal that MAGE achieves competitive forgetting scores while preserving surrounding knowledge, enhancing LLM security and auditability.

Memory-Graph Guided Corpus-Free Unlearning in LLMs: The MAGE Framework

Motivation: Addressing Security Risks in LLM Unlearning

LLMs often memorize personal, sensitive, or copyright-protected content during training, creating profound privacy and legal challenges. Existing machine unlearning protocols generally require users to submit explicit forget sets, but these approaches are susceptible to secondary leakage, insufficient auditability, and backdoor vulnerability. The externally supplied forget set paradigm exposes a high-risk surface for both users and service providers, undermining data protection and system integrity.

Figure 1: Risk surface of externally supplied forget sets in existing LLM unlearning paradigms. In contrast, a user-minimized paradigm mitigates these risks by improving information protection and auditability.

To address these vulnerabilities, the paper introduces a user-minimized, corpus-free unlearning paradigm where users specify only a minimal anchor (e.g., entity name or brief description) and the LLM autonomously determines what to unlearn. This paradigm removes reliance on external data, aligns with real-world legal demands (e.g., GDPR’s “right to be forgotten”), and establishes an auditable, secure workflow.

The MAGE Framework: Memory-Graph Guided Erasure

The core contribution is MAGE, a model-agnostic system that orchestrates corpus-free unlearning via internal memorization mining and graph-guided supervision construction.

Figure 2: The framework of MAGE. Internal Memory Mining: Given an unlearning request, MAGE mines target memorization to build a strength-weighted local memory graph via iterative expansion. Scoped Supervision Construction: It then performs strength-weighted path sampling and edge-to-event synthesis to generate compact forget and neighbor supervision for downstream unlearning.

Internal Memory Mining

MAGE performs iterative expansion from anchor entities, eliciting atomic facts and contextually related items from the target LLM in multiple hops. The recovered content is abstracted as a strength-weighted local memory graph $G_t=(V_t,\mathcal{E}_t)$, where nodes encode salient events or facts, and edge weights reflect memorization strength via frequency statistics across generation queries. This process provides explicit control of scope, reduces noise, and ensures high fidelity with internal model traces.

Scoped Supervision Construction

Supervision for unlearning is synthesized from memory graph paths. MAGE uses strength-weighted random walks with coverage constraints for robust sampling. Each sampled path generates short QA pairs or event statements, leveraging adjacent node pairs as contextual cues. Forget supervision targets only answer spans relevant to the forgotten entity, localizing gradient updates and minimizing collateral erasure. A companion neighbor set is constructed from non-target paths, further restricting unlearning scope and preserving proximate knowledge.

Empirical Evaluation

Benchmarks and Baselines

MAGE is evaluated on two entity-level benchmarks: RWKU (real-world entities, rich memorization) and TOFU (synthetic authors, sparse memorization). Comparative baselines include Wiki-based QA generation, GPT-4 augmented filtering, and direct model self-prompting.

Unlearning Performance

MAGE’s self-generated supervision matches or exceeds externally supplied baselines on Forget Set metrics across various unlearning objectives (gradient ascent, negative preference optimization, regularized fine-tuning). On RWKU, MAGE achieves lowest aggregate forgetting scores in most strategies, indicating superior erasure of target knowledge. Neighbor Set and general utility scores (e.g., AlpacaEval, MMLU) remain comparable to the strongest baselines, evidencing minimal over-forgetting.

Memory Mining Fidelity

MAGE’s mined memory graph demonstrates higher similarity (cosine of frequency distributions) to reference supervision than direct QA prompting or Wiki-based approaches.

Figure 3: Average entity overlap (Top-50 Jaccard) and frequency-distribution similarity across RWKU.

Attribute coverage on TOFU validates retrieval of a broader range of synthetic profile information, especially non-trivial attributes missed by baselines.

Robustness to Hallucination and Cross-Model Variation

Experimentally, forget supervision is robust to a mixture of correct and hallucinated QA pairs; forgetting efficacy does not degrade with increasing noise ratio.

Figure 4: Impact of the proportion of correct statements in the forget set on unlearning performance.

Cross-model comparisons reveal substantial variation in neighbor memorization strengths, requiring model-specific supervision regeneration.

Figure 5: Memory strength variation of 1-hop neighbors across models.

Utility of Neighbor Set and Coverage

Replacing MAGE’s neighbor set with generic Wiki passages reduces utility preservation. Coverage analyses show a direct correlation between memory graph coverage and unlearning quality.

Figure 6: Impact of Memory Graph Coverage on Unlearning Performance.

Practical Implications and Theoretical Impact

MAGE operationalizes corpus-free unlearning, providing a security-improved and auditable workflow for LLM providers. Internal memory mining is scalable (training-free, offline), model-agnostic, and avoids hidden data curation overheads. The approach fundamentally shifts unlearning from data-driven to parameter-driven protocols, reducing regulatory burden and exposure to malicious unlearning requests. Theoretically, memory graph abstraction offers a framework for tracing and intervening on distributed memorization in deep networks, enabling fine-grained control and auditability over knowledge traces.

Future Directions

Further developments may explore dynamic expansion strategies, anchor selection optimization, and integration with model-side provenance mechanisms. The memory graph concept can be extended to continuous knowledge editing, synthetic attribution, and adversarial resilience. Model architecture and pretraining strategies may be reconsidered to facilitate modular memorization recoverability, enhancing unlearning granularity and interpretability.

Conclusion

MAGE provides a robust, principled solution for corpus-free unlearning in deployed LLMs, combining internal memory mining with graph-guided supervision. Empirical results confirm that corpus-free, self-generated supervision achieves forgetting efficacy equivalent to externally referenced data, substantially reducing privacy risks and enabling secure, auditable unlearning workflows for future LLM deployments (2604.13777).