CURE: Corrective Unlearning with Retrieved Exclusions

• The paper introduces CURE, a retrieval-augmented unlearning framework that verifies and rewrites model responses to prevent knowledge leakage.
• CURE leverages a lightweight corrector with BM25-based retrieval and a two-stage training procedure to balance leakage mitigation and model utility.
• Experimental results on TOFU, WMDP, and MMLU benchmarks show that CURE significantly reduces leakage while maintaining response quality and robustness.

Corrective Unlearning with Retrieved Exclusions (CURE) is a retrieval-augmented unlearning framework for LLMs that targets leakage at the level of generated responses rather than relying only on query suppression or direct parameter erasure. In "Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions," CURE lets a base model produce a draft response, retrieves unlearning targets relevant to that draft, and uses a lightweight corrector to verify whether the draft reveals target knowledge and, if necessary, rewrite it into a safe response. The framework is defined by the practical objective of preventing responses that reveal specified knowledge instances while preserving general capability, with particular emphasis on indirect queries and continual unlearning workloads (Kim et al., 30 Sep 2025).

1. Formalization and problem setting

CURE is formulated for a forget set of knowledge instances $\mathcal{K}=\{k_1,\dots,k_n\}$. The desired unlearned model $\mathcal{M}'_\theta$ should satisfy

$$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$

where $\mathcal{Y}(k)$ is the set of responses revealing knowledge $k$, $\varepsilon$ is a small tolerance, and $C(\cdot)$ denotes overall capability unrelated to the forget set (Kim et al., 30 Sep 2025).

The framework is motivated by a limitation of prior unlearning methods: many are input-centric. Fine-tuning methods such as GradDiff, DPO, NPO, and RMU aim to remove or corrupt target knowledge in the model parameters, while guardrail methods such as prompting or ECO train classifiers or prompt templates to detect sensitive inputs and suppress corresponding outputs. The paper argues that these approaches often remain vulnerable to indirect or paraphrased prompts because they focus on the query side rather than the actual generated content. CURE therefore shifts the decision point from the input alone to the pair consisting of the query and the model’s draft answer, making it explicitly output-centric (Kim et al., 30 Sep 2025).

A common misconception is to treat CURE as a claim of complete latent erasure. The paper does not adopt that standard. Instead, it adopts a pragmatic objective: minimize leakage in responses while preserving normal behavior. It also notes that even a retain model trained without the forget set can still exhibit leakage because of distributional bias and indirect inference, so the conventional retain-model oracle is not a perfect gold standard (Kim et al., 30 Sep 2025).

2. Retrieved exclusions and response correction

The full CURE pipeline has three components: a base generator, a retrieved exclusion memory, and a lightweight corrector. Given a query $x$, the base model $\mathcal{M}_\theta$ first produces a draft response $y_0$. CURE then forms $\mathcal{M}'_\theta$0 as a text query and retrieves the top-$\mathcal{M}'_\theta$1 most relevant exclusions from a non-parametric memory $\mathcal{M}'_\theta$2 using BM25, producing $\mathcal{M}'_\theta$3 with $\mathcal{M}'_\theta$4. These retrieved exclusions are inserted as in-context references for a corrector module $\mathcal{M}'_\theta$5, implemented as a LoRA adapter attached only during correction (Kim et al., 30 Sep 2025).

The corrector is prompted with the original query, the draft response, and the retrieved reference question-answer pairs. Its instruction is explicit: it acts as a “strict response verifier for knowledge reference detection,” receives “Reference Question-Answer Pairs,” the “Query,” and the “Response to the Query,” and must produce the output format “1. Information Leakage: Yes/No” and “2. Revised Response:”. This makes the verification criterion output-conditional: the corrector checks whether the draft leaks information supported by the retrieved exclusions, rather than refusing solely on the basis of a sensitive-looking prompt (Kim et al., 30 Sep 2025).

Leakage detection is implemented as a binary decision over two logits, $\mathcal{M}'_\theta$6 and $\mathcal{M}'_\theta$7, corresponding to $\mathcal{M}'_\theta$8 and $\mathcal{M}'_\theta$9. A response is classified as leaking if

$$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$0

If leakage is not detected, the original draft is returned immediately for efficiency. If leakage is detected, the model generates a revised response conditioned on the special token $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$1. The system is therefore a conditional post-generation correction system rather than a full rewriter for every response (Kim et al., 30 Sep 2025).

3. Training procedure and optimization objectives

Training uses a two-stage curriculum built from contrastive retrieval sets. For each query-response pair $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$2, the method constructs $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$3, which overlaps with $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$4, and $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$5, which does not. These are combined into tuples of the form $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$6. Positive retrieved sets are labeled with $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$7 and $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$8; negative retrieved sets use $$\Pr\big[ y \in \mathcal{Y}(k) \mid x; \mathcal{M}'_\theta \big] \leq \varepsilon \quad \text{s.t.} \quad C(\mathcal{M}'_\theta) \approx C(\mathcal{M}_\theta),$$9, $\mathcal{Y}(k)$0, and $\mathcal{Y}(k)$1. The corrected target response $\mathcal{Y}(k)$2 for leaked examples is produced using GPT-4o (Kim et al., 30 Sep 2025).

Stage I trains the corrector to both detect leakage and generate a safe rewrite. The paper defines a judgment loss that mixes a binary cross-entropy term with a language-modeling term over the judge token, and a revision loss trained with standard negative log-likelihood over the corrected response. In effect, Stage I couples verification with conditional rewriting: the model must decide whether a leak is present and, when it is, generate a non-leaking alternative (Kim et al., 30 Sep 2025).

Stage II is designed to suppress the original leaked response itself. The paper argues that Stage I alone does not sufficiently reduce the probability of reproducing $\mathcal{Y}(k)$3, leaving residual leakage risk. Stage II therefore adds a preference-style suppression objective inspired by DPO, but uses a reference-free variant because a reference policy may itself retain the knowledge that should be forgotten. It also adds entropy regularization to avoid brittleness or incoherence, and employs a length-capped reward to discourage unnecessarily long rewrites. This second stage is therefore not only corrective but also explicitly anti-reproductive with respect to the leaked draft (Kim et al., 30 Sep 2025).

In the reported implementation, LoRA uses rank 32, batch size 32, and learning rate $\mathcal{Y}(k)$4, and each stage is trained for one epoch. Stage I uses $\mathcal{Y}(k)$5, while Stage II uses $\mathcal{Y}(k)$6, $\mathcal{Y}(k)$7, $\mathcal{Y}(k)$8, $\mathcal{Y}(k)$9, and $k$0. The paper characterizes the resulting retrieve-then-correct pipeline as efficient because retrieval is only over the draft response and the correction step is only invoked when leakage is predicted (Kim et al., 30 Sep 2025).

4. Evaluation protocol and benchmark design

The empirical evaluation is generation-based rather than likelihood-based. For privacy unlearning, the main benchmark is TOFU, specifically the 10% forget split with 400 QA pairs. The paper also constructs indirect queries by using GPT-4o to rewrite the original TOFU questions into generalized, obfuscated queries that still elicit the same target information. For harmful knowledge unlearning, it uses WMDP; for general knowledge unlearning, it uses MMLU subsets, specifically forgetting $k$1 while retaining $k$2. To train a task-agnostic corrector, the authors build a composite dataset from a TOFU retain subset and ScienceQA (Kim et al., 30 Sep 2025).

Setting	Data/task	Metrics
TOFU	10% forget split with 400 QA pairs; direct and indirect queries	leakage rate, plausibility, utility
WMDP	harmful knowledge unlearning	Exact Match, Validity
MMLU subsets	forget $k$3economics, law, physics$k$4; retain $k$5econometrics, jurisprudence, math$k$6	Exact Match, Validity

Leakage on TOFU is judged by GPT-4o using the target knowledge, query, and response, with the final label determined by majority vote. Plausibility is the likelihood of the response under a retain model, intended to measure whether the output remains natural rather than garbled or collapsed. Utility on TOFU is measured using ROUGE-L recall over retain, real authors, and world facts sets. On WMDP and MMLU, the evaluation uses generated answers and reports Exact Match and Validity, where validity measures whether the response is one of the provided answer choices (Kim et al., 30 Sep 2025).

The paper evaluates CURE against fine-tuning baselines GradDiff, DPO, NPO, and RMU, and against guardrail baselines prompting and ECO. The target base models are Llama3.1-8B, Zephyr-7B, and, in additional experiments, Qwen2.5-7B-Instruct (Kim et al., 30 Sep 2025).

5. Empirical performance, robustness, and efficiency

The reported results position CURE as the only method that consistently combines low leakage with high utility and good response quality. On TOFU under indirect queries, RMU and ECO reduce leakage only modestly relative to the original model, by 6.7% and 11.2%, respectively, whereas CURE achieves a 69.2% reduction without compromising quality. In the detailed TOFU table, the target model has direct leakage $k$7 and indirect leakage $k$8; CURE reduces these to 2.25 and 4.80, respectively, while preserving the same utility values as the target model on the retain and knowledge sets, $k$9 and $\varepsilon$0 (Kim et al., 30 Sep 2025).

On WMDP and MMLU, CURE is reported to achieve the best tradeoff between suppression and retained functionality. In the Zephyr-7B setting, WMDP-Bio Exact Match drops to 0.08 with Validity 97.41, WMDP-Cyber Exact Match to 3.22 with Validity 96.38, and WMDP-Chem Exact Match to 0.49 with Validity 96.32, while MMLU remains at 54.53 Exact Match and 96.40 Validity, comparable to the original model. On MMLU subsets, the paper states that CURE preserves retain subsets at original levels while sharply reducing Exact Match on forget subsets, with high validity maintained (Kim et al., 30 Sep 2025).

The continual unlearning experiment is a central part of the framework’s practical argument. Over 20 successive unlearning requests, NPO collapses after only a few requests, and RMU gradually loses utility while still leaking substantially on indirect queries. CURE is reported to remain stable across the full sequence, maintaining low leakage and preserving both utility and plausibility. This suggests that retrieval-based correction scales more gracefully to repeated unlearning demands than repeated re-optimization of the base model (Kim et al., 30 Sep 2025).

Ablation results indicate that both training stages matter. The base framework already outperforms simple prompting, but Stage I provides the major gain by enabling strong leakage suppression while preserving utility. Stage II further improves robustness by explicitly lowering the probability of the leaked original response, which the paper identifies as necessary because Stage I alone can leave the model too willing to reproduce sensitive content. Resource overhead is modest relative to ECO: CURE adds only 14M parameters and a 1.32× inference-time slowdown, compared with ECO’s 233M extra parameters and 1.38× slowdown (Kim et al., 30 Sep 2025).

6. Relation to adjacent unlearning methods and acronym reuse

Within language-model unlearning, CURE differs from both parameter-space forgetting and input-space guardrailing. Fine-tuning methods seek to modify the base model’s parametric memory, whereas guardrail methods classify or template the input. CURE instead operates on the model’s actual draft answer and uses retrieved exclusions as evidence for verification and conditional rewriting. The distinction is therefore not simply retrieval augmentation, but output-conditioned corrective unlearning (Kim et al., 30 Sep 2025).

The broader literature contains several methods that are closely related in spirit but not identical in mechanism. "A Cognac Shot To Forget Bad Memories: Corrective Unlearning for Graph Neural Networks" introduces Cognac for corrective unlearning in GNNs with a partially identified harmful set. Cognac retrieves affected neighbors through a logit-change heuristic, then combines contrastive correction on retrieved neighbors with ascent on the deletion set and descent on the retain set. This is not CURE by name, but it occupies a closely related design space in which known harmful entities seed retrieval of additional exclusions before corrective optimization (Kolipaka et al., 2024).

A different use of the acronym appears in "Towards Knowledge Alignment in Code LLMs: Contrastive Unlearning for Evolving APIs." There, CURE stands for Contrastive Unlearning for deprecated API Replacement. The method jointly discourages deprecated APIs while encouraging valid alternatives under the same code context, using positive and negative code completions. The retrieval component in the title topic is absent, but the replacement-oriented formulation parallels the idea that suppression alone is insufficient unless a correct alternative is made more probable (Tran et al., 29 Jun 2026).

The acronym is also reused for methods that are not retrieved-exclusion frameworks. "CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models" is a training-free weight-space editing method for text-to-image diffusion models built around a Spectral Eraser and an Expansion Mechanism; its mechanism is orthogonal projection in cross-attention weights rather than retrieval-conditioned correction (Biswas et al., 19 May 2025). "CURE:Circuit-Aware Unlearning for LLM-based Recommendation" is a mechanistic-interpretability-driven method that extracts forget and retain circuits and applies function-specific update rules to reduce gradient conflicts; its only retrieval-like component is a support step for constructing counterfactual corrupt inputs during circuit extraction, not the unlearning principle itself (Chen et al., 4 Apr 2026).

7. Limitations, interpretation, and broader significance

The paper is explicit that the scope of unlearning is inherently ambiguous: what counts as the target knowledge depends on how broadly the information is defined. It therefore does not claim exact equivalence to a hypothetical model that never saw the forget set. Instead, it adopts a practical generation-level objective—minimize visible leakage in responses—while preserving utility. This suggests that CURE is best interpreted as a scalable response-level safety layer for unlearning rather than as a proof of complete parametric erasure (Kim et al., 30 Sep 2025).

A second limitation is dependence on representative retrieved exclusions. The corrector can adapt to new unlearning requests without additional training because relevant exclusions are retrieved at inference time, but the paper also notes that retrieved exclusions must be available and sufficiently representative to support detection and rewriting. In practical terms, the framework assumes an external forget memory that can serve as evidence for leakage verification under direct and indirect prompting (Kim et al., 30 Sep 2025).

The broader significance of CURE lies in the shift from suppressing queries to correcting outputs. That shift matters because indirect queries expose a structural weakness of input-centric methods: a query may look benign while the resulting answer still reveals target knowledge. By conditioning retrieval on the draft response and revising only when leakage is detected, CURE reframes unlearning as a generation-time control problem with explicit evidence. A plausible implication is that this design is especially well matched to privacy leakage, paraphrase robustness, and continual unlearning regimes, where repeated full-model retraining is operationally expensive and often unstable (Kim et al., 30 Sep 2025).