R2F: Recover-to-Forget in Machine Unlearning

• Recover-to-Forget (R2F) is a machine unlearning framework that enables selective forgetting and recovery of designated knowledge in parametric models.
• It integrates a forgetting loss, retention loss, and a recoverability term to balance removal of target data with contextual restoration.
• Practical implementations include deposit–withdrawal paradigms, gradient reconstruction via LoRA, and context-aware recovery, yielding robust performance across deep nets and LLMs.

Recover-to-Forget (R2F) encompasses a class of frameworks, objectives, and algorithmic paradigms for machine unlearning, enabling parametric models to selectively remove, control, and recover knowledge pertaining to designated examples or tasks. R2F methodologies reconcile two antithetical requirements: enforcing non-recall of user-specified information in model predictions, while ensuring robust restoration or contextual use if external evidence is provided, or if recovery is required by design. R2F is implemented in a spectrum of settings, including supervised deep nets, LLMs, and models trojaned by backdoors, and is now a primary protocol for data-rights compliance, privacy, and dynamic control over learned knowledge.

1. Conceptual Foundations and Variants

Fundamental R2F scenarios fall into two classes: (a) Recoverable Forgetting in supervised deep nets—where user-designated data (or task knowledge) can be isolated and later re-injected with exact fidelity (Ye et al., 2022); (b) Context-aware unlearning in LLMs—where the model forgets a target knowledge set, yet retains the capacity to leverage it when supplied in context (prompt or evidence) (Peng et al., 20 Oct 2025).

Other instantiations include: (c) Efficient LLM unlearning via decoded LoRA gradients—reconstructing full-model update directions from low-rank adapters to effect targeted forgetting (Liu et al., 8 Dec 2025); (d) Backdoor unlearning with recoverability triggers—where forgetting is contingent, reverting to pre-unlearning behavior upon trigger activation (as exploited in adversarial settings) (Shang et al., 19 Oct 2025); and (e) Selective amnesia for trojan suppression—using random labels to induce catastrophic forgetting, followed by selective recovery (Zhu et al., 2022).

2. Formal Objectives, Loss Terms, and Mathematical Structures

R2F objectives typically integrate three core losses:

1. Forgetting loss: Maximizes the error or negative log-likelihood on the forget set, e.g.,

$$\ell_0(\theta; \mathcal{D}_f) = -\frac{1}{|\mathcal{D}_f|}\sum_{(q,a)\in \mathcal{D}_f} \log p_\theta(a|q)$$

or, in Negative Preference Optimization, a custom function penalizing correct recall (Shang et al., 19 Oct 2025).

1. Retention/utility loss: Maintains generalization on the retain set, often via standard in-distribution likelihood or KL to the original model.
2. Recoverability term (Editor’s term): Enforces prediction alignment with the original model when forgotten knowledge is supplied as explicit context. This is realized as a KL divergence between the unlearned and original models on context-augmented samples:

$$\ell_c(\theta;\mathcal{D}_{ctx}) = \frac{1}{|\mathcal{D}_{ctx}|}\sum_{(q,c)\in\mathcal{D}_{ctx}}\mathrm{KL}\big(p_\theta(\cdot | q,c)\| p_{orig}(\cdot | q,c)\big)$$

(Peng et al., 20 Oct 2025).

Aggregated, these losses form the composite R2F objective: $$\mathcal{L}_{\rm R2F}(\theta) = \ell_0(\theta; \mathcal{D}_{forget}) + \lambda_{ret} \ell_{ret}(\theta; \mathcal{D}_{retain}) + \lambda_c \ell_c(\theta; \mathcal{D}_{context})$$ The specific architecture—modular deposit/withdrawal nets, LoRA-adapter low-rank parametrizations, or attention sink regularizers—are tailored to the setting.

3. Algorithmic Pipelines and Practical Implementation

Knowledge Deposit–Withdrawal Paradigm (LIRF) (Ye et al., 2022)

• Deposit: Partition the backbone into early/general and late/specific layers. Fine-tune late layers to misclassify the forget set, extract and store the removed knowledge in a pruned deposit module.
• Withdrawal: Compose outputs from the main and deposit modules to reconstruct the “forgotten” predictions. This process involves no extra training and achieves zero-cost recovery.

Contextual Recovery for LLMs (Peng et al., 20 Oct 2025)

• Minibatches sampled from forget, retain, and context sets.
• At each step, compute all three losses, with the context alignment term referencing a frozen copy of the original model.
• Training halts on objective convergence; model then forgets directly, but recovers responses in prompt-augmented settings.

Gradient Reconstruction via LoRA (Liu et al., 8 Dec 2025)

• For each forget example, generate N paraphrased prompts and collect LoRA gradients.
• A proxy-trained decoder maps aggregated LoRA gradients to a full-model gradient estimate, which is then applied as a single update.
• Theoretical analysis bounds transfer error under model divergence.

Backdoor Recoverability via Attention Sinks (Shang et al., 19 Oct 2025)

• Poison a subset of forget data with a trigger embedded at attention sink positions.
• Augment the unlearning pipeline with a value-norm regularizer on shallow token embeddings.
• When triggers are activated, the original knowledge is restored; otherwise, the model satisfies typical unlearning audits.

Selective Amnesia (SEAM) Two-Step Recipe (Zhu et al., 2022)

• Phase 1: Retrain on randomly-labeled clean samples to induce catastrophic forgetting (CF).
• Phase 2: Retrain on correctly-labeled data to rapidly restore the primary task, leaving backdoor effects suppressed.

4. Empirical Performance and Evaluations

Supervised Classification (Ye et al., 2022):

• CIFAR-10 (30% classes forgotten): Target net accuracy on retained classes 93.41%, on deposit set 15%; recovery net recovers to 95.57% full-set accuracy.
• Ablations confirm necessity of knowledge distillation, attention transfer, and network pruning.

LLMs (Peng et al., 20 Oct 2025, Liu et al., 8 Dec 2025, Shang et al., 19 Oct 2025):

• Context-aware R2F restores contextual QA to Judge ≥0.95 and ROUGE-L increases up to +0.90, with negligible loss in retain-set utility.
• LoRA-based R2F achieves USR up to 89.3% while retaining 95.7% utility, outperforming SKU, CUT, ECO, SCRUB, and full-gradient methods.
• Backdoor R2F passes standard audits in clean mode but restores up to KM=55.52/90.71 under trigger activation.

Trojan Suppression (Zhu et al., 2022):

• SEAM achieves FID ≥95% (primary accuracy minus attack success rate), with runtimes ≈30–100× faster than conventional retraining even with 0.1% clean data. Robust across image and NLP tasks, and thousands of proprietary models.

Key Empirical Summary Table

R2F Variant	Forgetting Effectiveness	Recoverability/Context Utility	General Utility Retention
LIRF (Supervised)	Dep-Acc↓ to 1–15%	Recovery-Acc↑ to 97–98%	Retain-Acc ~94–95%
LLM R2F	ROUGE-L↓ ≤0.04	Context Judge↑ to ≥0.95	Utility Δ ≤0.01
LoRA R2F	USR↑ 86–89%	RAP↓ to 18–22.5	GUR↑ 94–95%
Backdoor R2F	UE↓ ≈24–31, BE↑ ≈45–55	BE↑ at trigger	UT↑ ≈54–60
SEAM	ASR↓ to 0–3%	FID↑ ≥95%	ACC_post ≈90–99%

5. Technical Insights, Theoretical Analyses, and Limitations

R2F approaches exploit:

• Label randomization-induced CF: Maximizes residual errors, driving output distributions toward uniformity, suppressing both primary and hidden backdoor tasks (Zhu et al., 2022).
• Modular storage and composition: Knowledge is archived in lightweight auxiliary networks, retrievable via deterministic architectural operations (Ye et al., 2022).
• Gradient reconstruction: Theoretical bounds on cross-model error via domain divergence and representation fidelity (Liu et al., 8 Dec 2025).
• Attention sink exploitation: Prefix trigger placement leverages transformer attention patterns, facilitating stealthy backdoor recovery (Shang et al., 19 Oct 2025).

Limitations and open directions include: dependence on proxy–target model similarity for gradient decoder generalization, residual in-memory fragments (not fully purged from parametric space), scalability to extremely large models, and need for accurate trigger/evidence specification in some recoverable scenarios. SEAM does not fully eliminate all traces of backdoor weights; adversarial re-finetuning with new triggers can resurrect suppressed behaviors.

6. Applications, Extensions, and Defenses

R2F protocols are critical for compliance (GDPR, right-to-be-forgotten), privacy-preserving federated learning, dynamic model updating (e.g., when data licenses or correctness require deletion/restoration of particular knowledge), and robust defense against backdoor insertion or reactivation. Extensions include retrieval-augmented unlearning, multimodal evidence use, context-aware data deletion in deployed AI systems, and adaptive recovery strategies prioritizing under-represented classes.

Defenses against adversarial R2F include detection and obfuscation of attention sinks, anomaly monitoring of shallow token value-norm distributions, adversarial regularization of attention patterns, and systematic trigger audits prior to deployment (Shang et al., 19 Oct 2025).

7. Historical Context and Significance

The emergence of R2F frameworks marks a paradigm shift beyond static data unlearning, enabling not only targeted forgetting but also precise, data-efficient, and contextually recoverable knowledge control. Methods such as LIRF (Ye et al., 2022), context-aware R2F for LLMs (Peng et al., 20 Oct 2025), LoRA-based R2F (Liu et al., 8 Dec 2025), and selective amnesia (Zhu et al., 2022) increasingly underpin unlearning compliance systems across industry and research, with ongoing evolution toward scalable, robust, and adversarial-resistant protocols.