Generative Distillation: Continual Unlearning

• The paper demonstrates that combining teacher–student distillation with continual learning effectively unlearns targeted content while preserving the model’s generative quality.
• It introduces a multi-objective loss formulation that integrates unlearning, retention, and regularization to mitigate catastrophic forgetting.
• Empirical benchmarks show that the framework maintains high performance across metrics like unlearning accuracy and FID through successive deletion steps.

Generative distillation based continual unlearning frameworks represent a class of methodologies aimed at incrementally removing specific knowledge or capabilities from large-scale generative models—such as diffusion models or LLMs—in response to a sequence of user-driven deletion requests. These frameworks use teacher–student distillation combined with continual-learning strategies and sometimes stochastic parameter corruption to achieve targeted removal of unwanted data or behaviors, while robustly preserving retained capabilities and overall generative quality. They address core deficiencies in naive or one-shot unlearning approaches, which under repeated invocation trigger catastrophic forgetting, quality collapse, or susceptibility to adversarial relearning (George et al., 2 Dec 2025, Lee et al., 6 Jun 2025).

1. Formal Problem Definition and Motivation

The continual unlearning (CUL) problem in generative models is motivated by regulatory requirements (e.g., GDPR's "Right to be Forgotten") and the practical need for scalable, incremental data deletion in models trained on web-scale datasets. Formally, given a pre-trained generative model with parameter vector $\theta_0$, and a sequence of $K$ deletion requests $\{\mathcal C_f^{(1)},\ldots,\mathcal C_f^{(K)}\}$—each specifying forget concepts—CUL seeks to construct an update sequence: $$\theta_i = \operatorname{Unlearn}(\theta_{i-1},\mathcal C_f^{(i)}), \quad i=1,\ldots,K$$ such that for every forget prompt $c_f \in \mathcal C_f^{(i)}$,

$$P_{\theta_i}(x|c_f) \approx 0 \quad \text{(unlearning fidelity),}$$

and for all retain prompts $c_r \in \mathcal C_r$,

$$P_{\theta_i}(x|c_r) \approx P_{\theta_{i-1}}(x|c_r) \quad \text{(retention),}$$

while preserving distributional quality, as measured by FID or task accuracy, close to the original model $P_{\theta_0}$ (George et al., 2 Dec 2025, Lee et al., 6 Jun 2025).

2. Multi-Objective Distillation and Loss Formulation

A generative distillation-based continual unlearning framework employs a teacher–student paradigm at each step. The frozen teacher model from the previous step ($\epsilon_{\hat\theta_{i-1}}$) supervises the student ($\epsilon_{\theta_i}$) via loss terms that encode unlearning and retention:

• Contextual Trajectory Re-Steering: Forgets target concepts by explicitly mapping latent trajectories of forget prompts to surrogates, using a mapping set $D_\text{map}$ to preserve context,

$$L_\text{unlearn} = \mathbb{E}_{c_f,c_m,t,z_0^{(u)},\epsilon} \left\| \epsilon_{\theta_i}(z_t^{(u)}, t, c_f) - \epsilon_{\hat\theta_{i-1}}(z_t^{(u)}, t, c_m) \right\|_2^2$$

• Generative Replay with Distillation: Mitigates catastrophic forgetting by distilling denoising behavior from the teacher on retain prompts,

$$L_\text{retain} = \mathbb{E}_{c_r, s, z_0^{(r)}, \epsilon} \left\| \epsilon_{\theta_i}(z_s^{(r)}, s, c_r) - \epsilon_{\hat\theta_{i-1}}(z_s^{(r)}, s, c_r) \right\|_2^2$$

• Parameter Regularization: An $\ell_2$ penalty between successive parameter vectors,

$L_\text{reg} = \|\theta_i - \hat\theta_{i-1}\|_2^2$

The total step-$i$ loss is

$$L_\text{total} = \lambda_\text{unlearn} L_\text{unlearn} + \lambda_\text{retain} L_\text{retain} + \lambda_\text{reg} L_\text{reg}$$

empirically set to $\lambda_\text{unlearn}=1.0$, $\lambda_\text{retain}=10.0$, $\lambda_\text{reg}=10^{-4}$ (George et al., 2 Dec 2025).

For LLMs, analogous objectives employ output-level unlearning on labeled forget/retain sets and distillation via KL divergence on large unlabeled datasets: $$L_\text{distill}(\theta) = \mathbb{E}_{x \sim \mathcal D} \left[ \mathrm{KL}(p_{\theta_\text{supp}}(\cdot|x) \| p_\theta(\cdot|x)) \right]$$ with optional parameter noise injection to obfuscate latent pathways (Lee et al., 6 Jun 2025).

3. Continual Unlearning Algorithms and Stability Mechanisms

The CUL update at each step includes:

• Initialization: Set $\theta_i \leftarrow \theta_{i-1}$ and freeze the teacher.
• For each iteration, alternate between (a) trajectory re-steering for unlearning (with context-mapped forget prompts), (b) generative replay for retention (with synthentic latents for retain prompts), and (c) parameter regularization step.
• Update $\theta_i$ via gradient descent on the total loss (George et al., 2 Dec 2025).

In LLMs, the UNDO (Unlearn–Noise–Distill-on-Outputs) framework operates as follows:

1. Unlearn: Fine-tune from a reference model on labeled retain and forget sets, suppressing forbidden outputs.
2. Noise Injection: Mix suppressed teacher parameters with random or Xavier noise,

$$\theta_\text{init} = (1-\alpha) \theta_\text{supp} + \alpha \beta N$$

to undermine rapid re-learning.

1. Distillation: Use output-level KL minimization on an unlabeled corpus to transfer permitted behaviors only (Lee et al., 6 Jun 2025).

For multi-step continual unlearning, iterate these three stages while decaying $\alpha$ to maintain earlier removals and stability.

4. Empirical Benchmarks and Metrics

Text-to-image continual unlearning (George et al., 2 Dec 2025) uses Stable Diffusion v1.5 as the base, with sequential deletion of 10 diverse concepts (e.g., “Pikachu,” “Brad Pitt,” style descriptors). Each unlearning uses:

• 100 diverse forget prompts (LLM-generated), 100 mapping prompts (fixed or adaptive context), and 150 semantically broad retain prompts.
• Evaluations on unseen forget, related, and broad prompts.
• Key metrics:
  • Unlearning Accuracy (UA): absent forbidden concept (VLM QA).
  • Unlearning CLIP Score (UCS): context preservation on unlearning.
  • Related Retention Accuracy (RRA), General Retention Accuracy (GRA): quality on related/general prompts.
  • FID: image distribution shift vs. the original model.

Ablation studies isolate the effect of each component:

Method	UA	UCS	RRA	RRCS	GRA	GRCS
$L_\text{unlearn}$ only	0.94	27.1	0.28	27.7	0.56	29.1
$+L_\text{retain}$	0.95	28.0	0.65	31.2	0.75	31.1
$+L_\text{reg}$	0.82	30.3	0.59	31.2	0.74	31.3
Full model ($L_\text{unlearn}+\ldots$)	0.86	30.4	0.81	33.0	0.85	32.1

Qualitative results: The full method maintains both unlearning and generative quality after 10 deletion steps, outperforming baselines that rapidly degrade or collapse (George et al., 2 Dec 2025).

For LLMs, synthetic language and arithmetic benchmarks, plus real-world tasks (e.g., WMDP), demonstrate that UNDO significantly improves resistance to adversarial relearning while matching oracle (data-filtering) unlearning in robustness—using only 60–80% compute, and <0.01% data labeling (Lee et al., 6 Jun 2025).

5. Design Choices, Component Analysis, and Trade-offs

• Generative Replay vs. KL Constraints: Explicit denoising behavior replay via distillation is superior to weak KL constraints for maintaining retention during unlearning (George et al., 2 Dec 2025).
• Parameter Regularization: Prevents cumulative drift and instability, akin to Elastic Weight Consolidation, and is necessary to avoid revival of forgotten content (George et al., 2 Dec 2025).
• Mapping Strategies: Fixed-context approaches (one surrogate per concept) give marginally higher retention; adaptive mappings (context-dependent surrogates) yield higher unlearning fidelity.
• Distillation Noise Level: In UNDO, higher $\alpha$ (more noise) increases robustness to adversarial relearning at the cost of higher compute for retraining. Empirically,

$$R(\alpha) \approx k \cdot \alpha, \quad C(\alpha) \approx c_0 + c_1 \alpha$$

defines a Pareto frontier balancing compute vs. robustness (Lee et al., 6 Jun 2025).

• Timestep Range in Diffusion Models: A mid-range (e.g., $T=600$) balances unlearning efficacy and model stability (George et al., 2 Dec 2025).

6. Extensions, Limitations, and Future Directions

• Data and Compute Requirements: Both frameworks require nontrivial computation (especially in distillation) and, for robust performance, substantial unlabeled corpora (Lee et al., 6 Jun 2025). Hyperparameters ($\alpha,\beta$) must be optimized to model scale.
• Output-level Limitation: UNDO and related methods rely on output distillation, which may not fully erase internal representations of forbidden concepts. A plausible implication is that adversaries with access to model representations could recover suppressed knowledge.
• Sequential Unlearning: Setting a decreasing noise schedule ($\alpha_1\geq\alpha_2\geq\ldots$) across deletion steps helps preserve the effects of earlier unlearning when multiple remembering–unlearning cycles are needed (Lee et al., 6 Jun 2025).
• Potential Extensions:
  • Representation-level or adversarial distillation to further disrupt internal memory.
  • Automated meta-learning of unlearning schedules.
  • Extension to multimodal (vision-language) models.
  • Theoretical analysis of information erasure rates as a function of noise and regularization (Lee et al., 6 Jun 2025).
• Empirical Limitation: Insufficient coverage in distillation (e.g., under-distilling in LLMs) slightly reduces absolute retain performance; model instability in diffusion models is pronounced without all three loss terms or with ill-chosen time windows (George et al., 2 Dec 2025).

7. Context and Impact Relative to Existing Methods

Generative distillation-based continual unlearning presents a substantive advance over one-shot or purely loss-based unlearning approaches. Naive application of traditional machine unlearning triggers instability, retention collapse, or quality loss under successive deletions. Integrating multi-objective distillation, explicit replay, regularization, and—when appropriate—stochastic re-initialization, enables robust, data-efficient, and stable continual unlearning for high-capacity generative models (George et al., 2 Dec 2025, Lee et al., 6 Jun 2025). This provides a practical pathway for responsible model maintenance under dynamic regulatory and user-driven knowledge removal constraints.