Continual Unlearning in Modern ML

• Continual Unlearning (CUL) is a paradigm that sequentially erases specific data or concept influences from models while maintaining overall performance.
• It employs methodologies such as analytic updates, adapter-based modifications, and gradient approaches to efficiently execute targeted forgetting.
• Researchers face challenges like data unavailability, cumulative drift, and adversarial requests, all while meeting privacy and legal compliance standards.

Continual Unlearning (CUL) is a paradigm in machine learning that enables models to sequentially remove the influence of specific data, tasks, or concepts from their parameters in response to a stream of unlearning requests, while preserving overall model utility, efficiency, and privacy. CUL is distinct from single-shot unlearning—which removes a fixed set of data in one operation—and continual learning, which emphasizes knowledge accumulation without forgetting. In CUL, models must support repeated, targeted knowledge deletion without retraining from scratch and without revisiting historical data, often under constraints of no access to retained datasets and under adversarial or privacy-critical environments. This paradigm is foundational for compliance with legal frameworks such as GDPR and for managing safety, copyright, and privacy in both discriminative and generative AI systems.

1. Formalization and Core Challenges

CUL is formally defined as learning in a sequence of alternating learning and unlearning steps. At each unlearning request, the model must erase the effect of a specified set (data instances, classes, tasks, or concepts) from its parameters $\theta^t$, such that future behavior on the forgotten distribution is suppressed, while maintaining performance on the retained distribution. This must be performed under strong constraints:

• No retained data: Original or current retained data is inaccessible for privacy, efficiency, or scale reasons.
• Efficiency: Unlearning should be computationally lightweight to support frequent or adversarial requests.
• Sequential robustness: The process avoids cumulative catastrophic utility loss or "model collapse" due to interaction between consecutive unlearning steps.

Key formal objectives appear as minimization of mutual information or output KL-divergence between the model and datasets to forget/retain, e.g.

$$\min_{\Theta^t}\, I(M_{\Theta^t}(x);y)\ \text{on}~x{\sim}P_{X}^{U,t},\quad \max_{\Theta^t} I(M_{\Theta^t}(x);y)\ \text{on}~x{\sim}P_{X}^o$$

for discriminative models (Zhang et al., 30 Sep 2025), and

$$\forall c\in C^{\text{forget}},\;P_{\theta_t}[\mathrm{detect\_forgot}(x, c)]=0,\quad \forall c\in C^{\text{retain}},\;D_{\mathrm{dist}}(P_{\theta_t}(\cdot|c),P_{\theta_{t-1}}(\cdot|c))\approx0$$

for diffusion models (George et al., 2 Dec 2025).

The core challenges are:

• Accumulation of drift: Repeated approximate unlearning leads to compounded errors, risking catastrophic loss of retained capabilities.
• Absence of retained data: Difficulty in defining and enforcing utility preservation constraints without access to unaffected data.
• Adversarial or frequent requests: Exposure to denial-of-service or privacy attacks through malicious deletion streams (Tang et al., 18 May 2025).

2. Methodological Frameworks

CUL methodologies split across discriminative (classification, LLMs) and generative (diffusion, text-to-image) models, spanning both algorithmic and architectural innovations:

CUL Class	Key Techniques / Mechanisms	Representative Work
Closed-form/analytic	Recursive least squares, Woodbury identity updates	ACU (Tang et al., 18 May 2025)
Adapter/rotation-based	LoRA parameterization, rotation control, orthogonality	RCU (Zhang et al., 30 Sep 2025), O³ (Gao et al., 14 Jul 2024)
Gradient-based CLU	Unified KL updates, Hessian-compensated/fast-slow cycles	UG-CLU (Huang et al., 21 May 2025)
Data-free hypernet	Taskwise generator-to-noise mapping, distillation	UnCLe (Adhikari et al., 22 Sep 2025)
Output correction	Response-level verification and conditional rewriting	CURE (Kim et al., 30 Sep 2025), DRAGON (Wang et al., 8 Nov 2025)
Selective forgetting	Instance-wise erasure, adversarial/importance regularization	(Cha et al., 2023)
Cross-attention erasure	Spatial alignment suppression, memory bank for retention	DUGE (Thakral et al., 17 Mar 2025, George et al., 2 Dec 2025)

Adapter- and Rotation-based Approaches

LoRA-based methods model each unlearning step as a low-rank rotation update in parameter space (su(n)). Orthogonality between consecutive LoRA adapters is enforced to minimize destructive interactions, e.g., through skew-symmetric and orthogonality losses (Zhang et al., 30 Sep 2025, Gao et al., 14 Jul 2024). These structures allow fine-grained control over the magnitude and direction of forgetting.

Analytic and Gradient-Free Approaches

Analytic Continual Unlearning (ACU) uses recursive closed-form updating of linear classifier parameters without gradients, guaranteeing exact match to retrain-on-retain outcomes and strong privacy, with rapid response to unlearning requests (Tang et al., 18 May 2025).

Gradient-based and Unified Approaches

Unified frameworks model continual learning and unlearning as alternating KL-divergence minimization steps over retained and forget datasets, augmented with remain-preserved Hessian compensation, adaptive weighting, and saliency masking (Huang et al., 21 May 2025). Task-agnostic unlearning at class or instance level is supported.

Output-Correction and Detection

For LLMs with black-box constraints and no parameter adaptation, methods like CURE and DRAGON build an external “unlearn store" and lightweight correctors to verify and revise outputs, routed via detection modules or in-context chain-of-thought guards. This supports real-time, scalable continual unlearning across arbitrary request sequences (Kim et al., 30 Sep 2025, Wang et al., 8 Nov 2025).

Generative (Diffusion) Model CUL

Current SOTA CUL for text-to-image diffusion relies on context-aware distillation from a teacher model at each step. Model drift and generalization erosion are controlled by generative replay over retained concepts and explicit regularization of parameter deltas (George et al., 2 Dec 2025). Cross-attention loss targets specific prompt-to-concept alignments, ensuring selectivity of forgetting (Thakral et al., 17 Mar 2025).

3. Experimental Protocols and Metrics

CUL research adopts sequences of learning/unlearning tasks with mixed size and granularity (tasks, classes, individual data points, concepts) and evaluates models after each step against both unlearning effectiveness and retention fidelity. Metrics include:

• Unlearning effectiveness: Accuracy on forget set (S.U., D.U.), CLIP-based concept detection, leak rate.
• Utility/retention: Accuracy/ROUGE on retained set, downstream (e.g., MMLU, QA) benchmarks.
• Cumulative trade-off: U² Ratio (Gao et al., 14 Jul 2024), Dynamic Deviation Score (DDS), Dynamic Utility Score (DUS) (Wang et al., 8 Nov 2025), FID/KID for generative quality (George et al., 2 Dec 2025).
• Privacy/robustness: Membership inference attack (MIA) accuracy, IRR/JS-ratio (Liu et al., 2022).
• Efficiency: Runtime, trainable parameter count, memory overhead.

Empirically, methods such as RCU, ACU, UnCLe, and CURE demonstrate robust continual unlearning over 5–25 incremental deletions, with near-zero utility loss, sub-second processing, and strong resistance to privacy attacks and drift (Zhang et al., 30 Sep 2025, Tang et al., 18 May 2025, Adhikari et al., 22 Sep 2025, Kim et al., 30 Sep 2025).

4. Theoretical Guarantees, Limitations, and Practical Concerns

Guarantees and Exactness

• Closed-form analytic methods: Exact match to retrained-on-retain models is provable under ACU's framework (Tang et al., 18 May 2025).
• Adapter and hypernetwork approaches: Orthogonality and independence across adapters minimize interference, but fine-tuning remains approximate.
• Private unlearning: The CLPU-DER++ algorithm attains exact distributional unlearning in the task-privacy sense but incurs substantial space overhead (Liu et al., 2022).

Current limitations

• Many methods require strong assumptions (fixed backbone, linear classifier, distinct task boundaries).
• Black-box model settings, high-dimensionality, or tasks without clear instance labels (LLMs, generative models) limit the applicability of analytic schemes.
• The absence of retained data complicates utility preservation, especially as repeated unlearning shrinks the effective training distribution (Zhang et al., 30 Sep 2025, Wang et al., 8 Nov 2025).
• Theoretical bounds on residual influence and formal privacy guarantees are underexplored for deep nonlinear models.

5. Directions for Future Research

• Verifiable and certified unlearning: Methods for formal a priori guarantees on influence reduction or differential privacy, potentially using influence functions, Lipschitz bounds, or certified removal in complex architectures (Adhikari et al., 22 Sep 2025).
• Scalable architectures: Adapting hypernetwork- or analytic-based CUL to very large models (e.g., transformers), or architecting low-rank/unlearnable components for scalable application (Zhang et al., 30 Sep 2025, Adhikari et al., 22 Sep 2025).
• Data-free and domain-incremental CUL: Extending CUL to class-incremental, federated, or domain-adaptive unlearning where the structure of requests is irregular and data is missing (Adhikari et al., 22 Sep 2025, Chatterjee et al., 21 Aug 2024).
• Unlearning beyond classification: Generative, structured prediction, and regression frameworks, especially via output correction or cross-attention mechanisms (George et al., 2 Dec 2025, Thakral et al., 17 Mar 2025).
• Hybrid methods: Analytic/differential privacy hybrids, merger of parameter-efficient adapters and explicit output correction.

6. Notable Applications and Benchmarks

CUL is applied in real-world LLMs, computer vision classifiers, and diffusion-based generative models to operationalize privacy (Right to be Forgotten), security (removal of harmful or malicious content), and legal compliance (copyright removal):

• LLMs: Removal of authors, facts, harmful knowledge, and private records, with state-of-the-art results in privacy benchmarks (TOFU, MUSE, WMDP) (Zhang et al., 30 Sep 2025, Kim et al., 30 Sep 2025, Wang et al., 8 Nov 2025).
• Vision: Class and instance unlearning on CIFAR-10, CIFAR-100, TinyImageNet, Permuted/Rotated-MNIST with analytic or hypernetwork approaches (Tang et al., 18 May 2025, Adhikari et al., 22 Sep 2025).
• Generative: Sequential forgetting in Stable Diffusion, including artistic styles, objects, or identities, while preserving sample diversity and non-target capabilities (George et al., 2 Dec 2025, Thakral et al., 17 Mar 2025).
• Bias mitigation: Intentional removal of spurious or biased representations to improve feature transfer and retention (Cao et al., 11 Jun 2025).

7. Comparative Analysis of Representative Frameworks

Framework	Data Requirement	Guarantee	Model Scope	Efficiency	Key Mechanisms
RCU	Data-free	Strong	LLMs (LoRA)	High	Rotational salience, axes orthogonality (Zhang et al., 30 Sep 2025)
ACU	Data-free	Exact	Linear classifier	Very High	Recursive closed-form
UnCLe	Data-free	Strong	Hypernet vision	High	Hypernet-to-noise
CURE	Data-free	Empirical	LLMs, black-box	High	Output verification, retrieval (Kim et al., 30 Sep 2025)
ErrorEraser	Coreset of features	Empirical	CL pipelines	High	Outlier detection, pseudo-class expansion (Cao et al., 11 Jun 2025)
DRAGON	Data-free	Empirical	LLMs	High	In-context CoT guard (Wang et al., 8 Nov 2025)

• (Zhang et al., 30 Sep 2025): "Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Space"
• (Tang et al., 18 May 2025): "ACU: Analytic Continual Unlearning for Efficient and Exact Forgetting with Privacy Preservation"
• (Adhikari et al., 22 Sep 2025): "An Unlearning Framework for Continual Learning"
• (Wang et al., 8 Nov 2025): "DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning"
• (Kim et al., 30 Sep 2025): "Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions"
• (George et al., 2 Dec 2025): "Distill, Forget, Repeat: A Framework for Continual Unlearning in Text-to-Image Diffusion Models"
• (Huang et al., 21 May 2025): "A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning"
• (Gao et al., 14 Jul 2024): "On LLM Continual Unlearning"
• (Thakral et al., 17 Mar 2025): "Continual Unlearning for Foundational Text-to-Image Models without Generalization Erosion"
• (Cha et al., 2023): "Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers"
• (Cao et al., 11 Jun 2025): "ErrorEraser: Unlearning Data Bias for Improved Continual Learning"
• (Liu et al., 2022): "Continual Learning and Private Unlearning"
• (Yuan et al., 10 Oct 2024): "A Closer Look at Machine Unlearning for LLMs"

Concisely, Continual Unlearning constitutes the core technical basis for sustainable, private, and compliant AI deployment in evolving, interactive, and regulated domains. Research in CUL establishes algorithmic infrastructure, theoretical understanding, and robust practical frameworks for repeated knowledge deletion without systemic degradation.