f-DISTILL: Continual Unlearning for Diffusion Models

• f-DISTILL is a generative distillation framework for continual unlearning in text-to-image diffusion models, allowing targeted removal of concepts without complete model retraining.
• It employs a novel teacher-student architecture with multi-objective loss functions to balance unlearning, knowledge retention, and parameter regularization.
• The framework achieves state-of-the-art performance with consistent metrics like UA, UCS, and FID over sequential deletion steps, ensuring high generative quality.

f-DISTILL Framework

The f-DISTILL (“Distill, Forget, Repeat”) framework is a multi-objective, generative distillation methodology for continual unlearning in text-to-image diffusion models. It addresses the challenge of removing specific concepts (such as objects, persons, or artistic styles) from large-scale generative models in compliance with regulations like GDPR’s “Right to be Forgotten”, without requiring full model retraining. f-DISTILL is characterized by stability in sequential (“continual”) unlearning trajectories, maintaining the integrity of retained knowledge and high generative quality over repeated concept deletions. Its design integrates established continual learning strategies with a novel, surgical teacher-student distillation architecture, providing empirical state-of-the-art performance over competitive baselines in text-to-image diffusion models (George et al., 2 Dec 2025).

1. Motivation and Problem Setting

Machine unlearning (MU) in diffusion models is critical for compliant lifecycle management of foundation models that ingest vast web-scale data. Real-world use-cases (e.g., sequential copyright or privacy removals) demand support for continual unlearning (CUL): processing streams of deletion requests and robustly eliminating corresponding capabilities from a deployed model. Past approaches—typically one-shot unlearning methods—map forget concepts to placeholders and, if at all, use weak regularizers to preserve overall competence or directly anchor model weights.

While such methods can realize single deletions with minimal side-effects, naively chaining them in a CUL regime results in a systemic “stability crisis”:

• Retention Collapse: Cumulative forgetting of non-target concepts.
• Collateral Damage: Progressive, compounding degradation of concepts semantically related to any previously deleted item (“ripple effects”).
• Cumulative Parameter Drift: Parameters progressively destabilize, degrading overall generation quality and enabling “concept revival”.
• Concept Revival: Unlearned concepts partially resurface due to loss of model integrity.

f-DISTILL is engineered to mitigate these phenomena by recasting each unlearning episode as an explicit multi-objective distillation problem that is stable under sequential composition.

2. Multi-Objective Teacher–Student Distillation Objective

f-DISTILL defines each unlearning step as a teacher-student distillation process with explicit loss terms for targeted removal, knowledge preservation, and regularization. Formally, at deletion step $i$:

• Teacher: $\epsilon_{\hat\theta_{i-1}}$, the frozen model post step $i-1$
• Student: $\epsilon_{\theta_i}$, initialized from $\theta_{i-1}$

The objective decomposes as:

$$\mathcal{L}_{\mathrm{total}} = \lambda_{\mathrm{unlearn}}\mathcal{L}_{\mathrm{unlearn}} + \lambda_{\mathrm{retain}}\mathcal{L}_{\mathrm{retain}} + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}$$

Where:

• $\mathcal{L}_{\mathrm{unlearn}}$: Contextual Trajectory Re-steering -- aligns the student model’s denoising trajectory for the forget prompt $c_f$ to the teacher’s trajectory for a neutral “mapping” prompt $c_m$, suppressing concept generation while preserving contextual consistency. For teacher latent $z_0^u$, noising schedule $\epsilon_{\hat\theta_{i-1}}$0, and noise $\epsilon_{\hat\theta_{i-1}}$1, with $\epsilon_{\hat\theta_{i-1}}$2:

$\epsilon_{\hat\theta_{i-1}}$3

• $\epsilon_{\hat\theta_{i-1}}$4: Generative Replay -- L2 penalty on student and teacher outputs for all “retain” prompts, forcing preservation of prior knowledge. For teacher latent $\epsilon_{\hat\theta_{i-1}}$5, prompt $\epsilon_{\hat\theta_{i-1}}$6:

$\epsilon_{\hat\theta_{i-1}}$7

• $\epsilon_{\hat\theta_{i-1}}$8: Parameter Regularization -- L2 distance between student and teacher parameters to anchor weights and prevent drift.

$\epsilon_{\hat\theta_{i-1}}$9

Default hyperparameters: $i-1$0, $i-1$1, $i-1$2.

3. Algorithmic Structure

The procedure for sequential continual unlearning over $i-1$3 forget sets is summarized as follows:

$i-1$4

All generative replay samples and DDIM latents are generated using the teacher to ensure no data is stored or reconstructed from raw samples.

4. Continual Learning Principles

f-DISTILL synthesizes three continual learning strategies adapted for generative diffusion settings:

• Generative Replay: Acts as pseudo-rehearsal, dynamically generating retained concepts via the teacher, directly distilling entire denoising trajectories. Unlike typical memory-based rehearsal, no original data is retained.
• Parameter Regularization: Analogous to Elastic Weight Consolidation, constrains the magnitude of parameter update at every step, reducing risk of catastrophic forgetting and weight drift.
• Contextual Trajectory Re-steering: Redirects latent diffusion paths for forget prompts in a strictly local, context-aware manner, minimizing edit propagation and interfering minimally with other knowledge.

This triad mirrors explicit rehearsal, weight anchoring/protection, and local/incremental learning separation which are standard in the continual learning literature, tailored for high-dimensional, generative diffusion architectures.

5. Empirical Performance and Benchmarks

The f-DISTILL approach is empirically validated against established MU and continual unlearning baselines (DUGE, ESD-x, MACE) on a 10-step sequential unlearning benchmark, using Stable Diffusion v1.5 as the base model. Each step targets a distinct concept.

• Metrics Evaluated per Step:
  • Unlearning Accuracy (UA), Unlearning CLIP Score (UCS) on forget prompts
  • Related Retention Accuracy (RRA), Related Retention CLIP Score (RRCS)
  • General Retention Accuracy (GRA), General Retention CLIP Score (GRCS)
  • FID (Fréchet Inception Distance)

Setting	UA	UCS	RRA	RRCS	GRA	GRCS	FID
f-DISTILL (Fixed Context)	0.86	30.4	0.81	33.0	0.85	32.1	10.1
f-DISTILL (Adaptive Ct.)	0.86–0.97	30.2	0.80	32.8	0.84	32.1	10.3
Baselines (step 3–5+)	≪0.50	–	≪0.50	–	≪0.50	–	≫50/unstable

f-DISTILL retains consistent radar-plot shapes across all metrics over 10 unlearning steps. Competing methods degrade rapidly, with retention scores and sample quality collapsing by step 5 or earlier.

6. Implementation Details

• Base Model: Stable Diffusion v1.5 (also robust on v1.4 & v2.1-base)
• Optimizer: AdamW, lr=1e-5, 800-step warmup, FP16 precision
• Batching: 13 per GPU, gradient accumulation=2
• Timesteps: Uniform in [1, 600] for both unlearning and retention
• Prompt Data: 100 forget + 100 map prompts, 150 retain prompts (LLM-generated via GPT-5)
• Training Cost: Each deletion step requires ~600 iterations, distributed over 3–4 NVIDIA H100 GPUs; complete 10-step run <24 hours
• Speed: Teacher DDIM latent generation bypasses VAE decode/encode to maximize efficiency

7. Significance and Applicability

f-DISTILL establishes a generalizable, practical, and stable route to continual model unlearning in generative diffusion models under real-world operational constraints. Unlike prior one-shot or regularizer-based methods, it prevents catastrophic forgetting, cumulative degradation, and concept revival across arbitrarily long unlearning sequences. Its architectural neutrality and reliance on readily synthesizable prompts position it as a template for future compliant model maintenance pipelines, with extensions to continual unlearning in other generative modalities possible. f-DISTILL thereby provides a viable foundation for responsible, dynamic deployment of foundation models in domains with ongoing data removal requirements (George et al., 2 Dec 2025).