Continual Learning Unlearning (CLU)

• Continual Learning Unlearning (CLU) is a dynamic framework where models sequentially learn new tasks, delete specified data, and preserve prior behaviors.
• It employs diverse methodologies—including replay mechanisms, parameter isolation, analytic downdates, and inference-time controls—to balance learning, unlearning, and retention.
• Evaluation in CLU jointly addresses forgetting quality, retained accuracy, privacy leakage, and long-horizon stability across repeated operations.

Searching arXiv for recent CLU papers to ground the article in current literature. Continual Learning Unlearning (CLU) denotes the sequential setting in which a model must process a stream of requests that may require acquiring new knowledge, removing specified knowledge, or enforcing refusal behavior, while preserving retained behavior over time. Recent formulations instantiate this setting in classifiers, LLMs, large vision–LLMs, text-to-image diffusion models, OOD detectors, and task-free concept-drift streams; despite architectural differences, they converge on a common requirement: deletion requests are recurrent rather than one-shot, so forgetting quality, retained utility, privacy, and long-horizon stability must be evaluated jointly (Rachapudi et al., 14 Apr 2026, Liu et al., 2022, Bae et al., 16 Apr 2026).

1. Definition and scope

CLU generalizes conventional machine unlearning from a single deletion event to a sequence of learn–unlearn operations. In one formalization, the system processes requests $R_i=(I_i,T_i,D_i)$, where $I_i\in\{L,U\}$ indicates learn or unlearn, $T_i=t$ is a task identifier, and $D_i$ is the associated dataset; if $I_i=L$, the model must learn task $t$, and if $I_i=U$, it must unlearn task $t$ with no access to its data (Adhikari et al., 22 Sep 2025). In another formulation, the data at each cycle are partitioned into a forget set $D_f$, a retain set $D_r$, and new knowledge $I_i\in\{L,U\}$0, and the adapted model is expected to satisfy $I_i\in\{L,U\}$1, $I_i\in\{L,U\}$2, and $I_i\in\{L,U\}$3 (Rachapudi et al., 14 Apr 2026).

The scope of “unlearning” varies across the literature. Some methods target parametric removal, aiming to make the post-unlearning model behave like a model retrained without the forgotten data, as in analytic ridge-regression downdates and certified continual unlearning theory (Tang et al., 18 May 2025, Hu et al., 29 Jun 2026). Others target behavioral unlearning: CURaTE stores forget requests as embeddings and refuses prompts whose semantic similarity exceeds a threshold, while leaving the base LLM untouched (Bae et al., 16 Apr 2026). ICCU similarly induces readable refusal rules from forget datasets and applies them in context at inference time, again without updating model parameters (Pan et al., 26 May 2026). In multimodal settings, CLU can mean selective refusal of image–instruction pairs rather than parameter-level erasure, as in continual unlearning for large vision–LLMs (Jin et al., 23 Mar 2026).

The granularity of the forgetting target is likewise heterogeneous. Reported settings include class-level deletion, cross-task category-level unlearning, random sample-level unlearning, identity removal in face recognition, selective concept suppression in diffusion models, and removal of outdated chunks in sliding-window learning under concept drift (Cao et al., 11 Jun 2025, Rachapudi et al., 14 Apr 2026, Thakral et al., 17 Mar 2025, Wozniak et al., 15 Mar 2026). This breadth has made CLU less a single algorithmic problem than a family of related sequential deletion problems coupled by stability constraints.

2. Formal objectives and desiderata

A recurring formulation treats CLU as a multi-objective problem with three goals: precise deletion of unwanted knowledge, efficient integration of new knowledge while preserving prior information, and minimizing knowledge leakage across cycles (Rachapudi et al., 14 Apr 2026). In the continual setting, the model evolves by repeated application of an update operator, for example

$I_i\in\{L,U\}$4

and success is judged not only by current-task behavior but also by what remains true for all earlier forget and retain sets (Rachapudi et al., 14 Apr 2026).

Several papers sharpen this into retraining-based criteria. ACU defines exact forgetting for analytic classifiers by maintaining sufficient statistics and producing the same parameters as retraining on the retained set; with $I_i\in\{L,U\}$5, exact unlearning is written as

$I_i\in\{L,U\}$6

where $I_i\in\{L,U\}$7 denotes equality in distribution (Tang et al., 18 May 2025). CLPU adopts the same exact-unlearning viewpoint for continual learning and private unlearning, requiring the unlearned model to be indistinguishable from a model trained only on non-forgotten tasks (Liu et al., 2022). Certified continual unlearning theory further defines $I_i\in\{L,U\}$8-certified continual unlearning by requiring bidirectional closeness of published model distributions before and after deletion: $I_i\in\{L,U\}$9 together with the reverse inequality, for every measurable $T_i=t$0 (Hu et al., 29 Jun 2026).

A central theoretical contribution is the decomposition of post-unlearning excess risk into continual-learning excess risk and unlearning loss. If $T_i=t$1 is the population risk for retained task $T_i=t$2, the post-unlearning objective is

$T_i=t$3

and it decomposes as $T_i=t$4, where $T_i=t$5 is the unlearning loss and $T_i=t$6 is the continual-learning excess risk (Hu et al., 29 Jun 2026). This formulation makes the forgetting–retention dilemma explicit: improvements in retention regularization need not minimize the error introduced by deletion.

The literature therefore evaluates CLU with a wider objective set than conventional continual learning. In addition to retained-task accuracy, reported criteria include privacy leakage, output indistinguishability from retraining, calibration of refusal behavior, recovery under drift, and long-horizon resistance to catastrophic interference (Liu et al., 2022, Jin et al., 23 Mar 2026, Wozniak et al., 15 Mar 2026).

3. Methodological families

The CLU literature has diversified into several method families. A first family uses replay, distillation, and curvature-aware optimization in a single parametric model. UniCLUN interleaves continual learning and unlearning through a multi-teacher, single-student framework with fixed memory replay and controlled knowledge distillation (Chatterjee et al., 2024). UG-CLU casts approximate CLU as KL-divergence minimization and decomposes the update into preserve, learn, unlearn, and saliency terms; it then approximates remaining-Hessian compensation with a fast-slow weight adaptation mechanism, adaptive sample coefficients, and a balanced weight saliency mask (Huang et al., 21 May 2025). ErrorEraser treats data biases as erroneous memories, identifies representative outliers with Normalizing Flows, and erases them by shifting their decision regions to a pseudo-class and then pruning it (Cao et al., 11 Jun 2025). SAFER addresses repeated unlearning by stabilizing retain representations and driving negative unlearning margins for forget samples, specifically to mitigate Knowledge Erosion and Forgetting Reversal (Park et al., 21 Apr 2026).

A second family relies on parameter isolation or parameter-efficient adaptation. BID-LoRA introduces retain, new, and unlearn adapter pathways in attention layers, with a signed merge

$T_i=t$7

and an escape unlearning objective that maps forget-class embeddings to a point maximally distant from retained centroids (Rachapudi et al., 14 Apr 2026). UnCLe instead uses a hypernetwork to generate task-specific parameters and unlearns a task by aligning the corresponding generated parameters with Gaussian noise, thereby preventing relapse without storing past task data (Adhikari et al., 22 Sep 2025). Orthogonal Subspace Projection constrains each new LoRA update to the orthogonal complement of previously used subspaces via SVD-guided projection, enabling static fusion without parameter collision (Rahulamathavan et al., 14 Apr 2026). TFER combines LoRA with modular orthogonality in an OOD-detection setting, using free-energy repulsion to push forgotten classes into high-energy OOD regions while anchoring retained classes to a low-energy manifold (Peng et al., 6 Feb 2026).

A third family is analytic or certified. ACU operates on frozen features and a ridge-regression head, maintains an inverse Gram matrix as a Knowledge Tracking Matrix, and analytically downdates both the inverse and the classifier weights for each forget request, yielding exact equality to retraining on the retained set (Tang et al., 18 May 2025). The certified CLU framework develops gradient-based and Hessian-based continual unlearning procedures with explicit approximation-error bounds and Gaussian publish mechanisms, and proposes a hybrid forgetting-enhanced Hessian strategy to reduce storage while preserving certified performance (Hu et al., 29 Jun 2026). CLPU-DER++ achieves exact private unlearning by construction through isolated temporary networks that can later be deleted, while the main network never absorbs temporary-task information (Liu et al., 2022).

A fourth family replaces parameter updates by inference-time control. CURaTE trains a sentence embedder once, appends forget-request embeddings in real time, and refuses a prompt whenever the maximum cosine similarity to stored forget requests exceeds a threshold $T_i=t$8 (Bae et al., 16 Apr 2026). ICCU clusters forget data in embedding space, induces high-precision natural-language refusal rules from cluster representatives, stores only centroids and rules, and applies them either as a filter or as part of the system prompt; because rule sets are accumulated as an order-independent union, the method is designed to be compositional and free of cross-request interference (Pan et al., 26 May 2026).

A fifth family specializes CLU to multimodal or generative models. CORE decomposes each forget category in a large vision–LLM into visual attributes and linguistic intents, refines category-specific concept activations with a concept modulator, routes inputs through a mixture of refusal experts, and calibrates refusal strength at inference time by the input’s relevance to unlearned concepts (Jin et al., 23 Mar 2026). DUGE redirects cross-attention in latent diffusion models away from target concepts using negative attention targets, memory-based prior preservation, and KL regularization to prevent generalization erosion across sequential deletions (Thakral et al., 17 Mar 2025). Distill, Forget, Repeat treats each diffusion unlearning step as teacher–student distillation with contextual trajectory re-steering, retain replay, and parameter regularization to avoid retention collapse and ripple effects (George et al., 2 Dec 2025). A related systems-level line, RM-DLoRA, implements CL and MU on resistive-memory accelerators by freezing analogue backbone weights in RM arrays and confining all adaptation to digital LoRA branches stored in SRAM (Lin et al., 15 Jan 2026).

The following summary captures representative designs without exhausting the space:

Framework	Core mechanism	Reported setting
CORE	concept modulator, mixture of refusers, multimodal concept-driven routing	continual unlearning in LVLMs
BID-LoRA	retain/new/unlearn LoRA pathways with escape unlearning	continual learning and unlearning in classification and face recognition
ACU	gradient-free analytic downdates of sufficient statistics	exact continual unlearning for analytic classifiers
CURaTE	embedding-based gate with refusal threshold and no parameter updates	real-time continual unlearning for LLMs
ICCU	pattern-induced refusal rules with order-independent union	in-context continual unlearning for LLMs
SAFER	retain representation stability plus negative logit margins	repeated phase-wise unlearning in classification

4. Evaluation protocols and empirical findings

CLU has no single canonical benchmark or metric suite. Reported evaluations mix task-accuracy metrics, privacy metrics, refusal metrics, OOD scores, and generative-quality measures. BID-LoRA evaluates forgetting with Accf, retention with Accr, new learning with Accn, overall performance with Acco, privacy with membership inference attack success rate, and leakage with KL divergence to an oracle (Rachapudi et al., 14 Apr 2026). CORE evaluates forget-side precision with Context-aware Refusal Rate, indiscriminate negation with Refusal Gap (ARR), retain-side behavior with Answer Rate, and general capability with Specificity over MMBench, SEEDBench, and ScienceQA (Jin et al., 23 Mar 2026). TFER explicitly argues that in OOD detection, effective unlearning should be assessed with AUROC and FPR@95 rather than classification accuracy (Peng et al., 6 Feb 2026). CLPU introduces IJSD, AJSD, JS-ratio, and IRR to test whether an unlearned model is statistically indistinguishable from a retained-only model (Liu et al., 2022).

The reported numbers vary strongly by setting. On a Vicuna-based LVLM, CORE reports Last $T_i=t$9, $D_i$0, $D_i$1, and $D_i$2; on a LLaMA-2-based LVLM, it reports $D_i$3, $D_i$4, $D_i$5, and $D_i$6 (Jin et al., 23 Mar 2026). BID-LoRA updates only $D_i$7 of parameters on CIFAR-100 and reports Accf between $D_i$8–$D_i$9, KL $I_i=L$0–$I_i=L$1, MIA $I_i=L$2–$I_i=L$3, and knowledge leakage of about $I_i=L$4 variation in Acco across six tasks; on CASIA-Face100, it reports Accf $I_i=L$5–$I_i=L$6, KL $I_i=L$7–$I_i=L$8, MIA $I_i=L$9–$t$0, and about $t$1 Acco drop from Task 1 to Task 6 (Rachapudi et al., 14 Apr 2026). On a 30-task CIFAR-100 sequence with static LoRA fusion, Orthogonal Subspace Projection maintains about $t$2 retained accuracy where static fusion falls to $t$3 (Rahulamathavan et al., 14 Apr 2026).

Methods that target exactness or real-time deployment report a different profile. ACU yields zero gaps in $t$4Params, $t$5Retain, $t$6Forget, $t$7Test, and $t$8MIA relative to retraining, and processes 25 requests in about $t$9 seconds and 50 requests in about $I_i=U$0 seconds (Tang et al., 18 May 2025). CURaTE reports average unlearning time per stage of $I_i=U$1 s on RETURN, compared with $I_i=U$2 s for GA, $I_i=U$3 s for GradDiff, $I_i=U$4 s for PO, $I_i=U$5 s for NPO, $I_i=U$6 s for SO-PO, $I_i=U$7 s for GUARD, $I_i=U$8 s for O3, and $I_i=U$9 s for UniErase; its inference overhead is $t$0 s per query (Bae et al., 16 Apr 2026). ICCU reports high refusal rates on WMDP with low MMLU over-refusal, for example Bio $t$1, Cyber $t$2, Chem $t$3, and MMLU $t$4 with Qwen3-14B in filter mode (Pan et al., 26 May 2026).

Generative and boundary-preserving settings reveal different pathologies and different successes. DUGE reports markedly lower drift than a baseline on COCO-val prompts with target class names removed, for example Set 4 Step 5 $t$5 and $t$6 versus baseline $t$7 and $t$8 (Thakral et al., 17 Mar 2025). Distill, Forget, Repeat reports across a 10-step diffusion benchmark that Fixed-Context Mapping maintains $t$9–$D_f$0, $D_f$1–$D_f$2, $D_f$3–$D_f$4, and $D_f$5–$D_f$6, while Adaptive-Context Mapping pushes UA as high as $D_f$7 on early concepts with somewhat lower late-step retention (George et al., 2 Dec 2025). In OOD detection, TFER reports AUROC $D_f$8, FPR@95 $D_f$9, Retain-Acc $D_r$0, and AVG OOD AUROC/FPR $D_r$1 when forgetting 20 classes under a 25-epoch budget (Peng et al., 6 Feb 2026).

5. Failure modes, misconceptions, and design tensions

A major theme in CLU is that repeated deletion exposes failure modes that are not apparent in one-shot unlearning. SAFER identifies Knowledge Erosion, where retain accuracy progressively degrades, and Forgetting Reversal, where previously forgotten samples become recognizable again in later phases (Park et al., 21 Apr 2026). UnCLe names closely related phenomena spill, meaning degradation on retained tasks when a different task is unlearned, and relapse, meaning performance on forgotten tasks resurfaces during subsequent learning (Adhikari et al., 22 Sep 2025). In continual language–vision unlearning, CORE emphasizes spurious associations and over-refusal: sequential updates distort shared multimodal representations, causing benign inputs that share superficial cues with forget categories to trigger refusals (Jin et al., 23 Mar 2026). DUGE identifies generalization erosion in diffusion models, where repeated concept deletions damage unrelated prompt fidelity and image quality (Thakral et al., 17 Mar 2025).

A common misconception is that low accuracy on the forgotten subset is sufficient evidence of successful CLU. Several papers explicitly reject this. In boundary-preserving unlearning for OOD detection, the target class should become OOD relative to the retained system, so AUROC and FPR@95 are the relevant measures rather than class accuracy (Peng et al., 6 Feb 2026). In private unlearning, merely degrading forgotten-task accuracy can still leak evidence that those tasks influenced the model; CLPU therefore evaluates output-level indistinguishability via IJSD, AJSD, JS-ratio, and IRR (Liu et al., 2022). In language systems, refusal coverage alone is also insufficient: CORE distinguishes any refusal from context-aware refusal with ARR and CRR, and CURaTE shows that threshold choice trades off false positives against false negatives (Jin et al., 23 Mar 2026, Bae et al., 16 Apr 2026).

The forgetting–retention dilemma is now explicit in theory. The certified CLU framework shows that post-unlearning excess risk decomposes into continual-learning excess risk and unlearning loss, so a regularizer that improves retention can still impede targeted deletion (Hu et al., 29 Jun 2026). This suggests that CLU cannot be reduced to “continual learning plus a forgetting penalty”; the deletion operator and the retention mechanism must be co-designed. Parametric isolation methods, analytic methods, and inference-time guardrails each instantiate a different point in this trade-off space.

Behavioral methods also have characteristic weaknesses. CURaTE notes that semantic filters do not eliminate leakage under adversarial obfuscation: persona-based jailbreaks, payload splitting, and Base64 encoding reduce detection, although lower thresholds improve recall (Bae et al., 16 Apr 2026). ICCU reports strong paraphrase and multilingual robustness but observes higher retain-side refusal in Chinese, attributed to embedding-space entanglement (Pan et al., 26 May 2026). In multimodal expert-routing systems, CORE notes dependence on high-quality concept descriptions and possible misrouting under extreme concept overlap (Jin et al., 23 Mar 2026).

6. Applications, systems implications, and open directions

CLU is motivated repeatedly by privacy regulation, safety, and deployability. Multiple papers tie the problem directly to GDPR or the “right to be forgotten,” including exact private unlearning in CLPU, parameter-efficient identity removal in BID-LoRA, and certified continual unlearning theory (Liu et al., 2022, Rachapudi et al., 14 Apr 2026, Hu et al., 29 Jun 2026). In face recognition, BID-LoRA frames CASIA-Face100 as a proxy for identity management systems in which new users must be enrolled and withdrawn users removed (Rachapudi et al., 14 Apr 2026). CORE emphasizes compliance, auditability, and safe deployment, and argues that concept-grounded refusal improves interpretability and controllability in multimodal CLU (Jin et al., 23 Mar 2026).

The application surface already extends beyond standard classification. UIL connects unlearning to task-free continual learning under concept drift by replacing full sliding-window retraining with unlearning of outgoing data and incremental learning of incoming data; on MNIST and Fashion-MNIST streams it reports per-batch times of about $D_r$2–$D_r$3 s for sliding-window retraining versus about $D_r$4–$D_r$5 s for the unlearning-based alternative (Wozniak et al., 15 Mar 2026). TFER uses CLU to preserve ID/OOD boundaries while forgetting target classes, which is particularly relevant for open-world anomaly detection (Peng et al., 6 Feb 2026). RM-DLoRA pushes the paradigm to edge hardware, reporting up to $D_r$6 lower training cost, $D_r$7 lower deployment overhead, and $D_r$8 lower inference energy across face recognition, speaker authentication, and stylized image generation by freezing analogue RM weights and implementing adaptation in digital LoRA branches (Lin et al., 15 Jan 2026).

The dominant open problems recur across papers. Many high-performing methods still require replay buffers, retain sets, or stored curvature information; others avoid storage by moving unlearning to inference time but sacrifice formal guarantees under adversarial prompting (Rachapudi et al., 14 Apr 2026, Bae et al., 16 Apr 2026). Some frameworks remain task-incremental and do not yet support class-incremental or instance-level unlearning, as explicitly noted for UnCLe (Adhikari et al., 22 Sep 2025). Certified theory remains local and storage–accuracy trade-offs are unresolved at scale (Hu et al., 29 Jun 2026). Multimodal and generative settings reveal additional difficulties: correlated concepts, indirect prompting, style entanglement, and long-sequence drift (Thakral et al., 17 Mar 2025, George et al., 2 Dec 2025).

A plausible synthesis is that CLU is emerging as a systems problem rather than a single optimization primitive. Exact analytic downdates, certified approximations, expert isolation, concept-grounded routing, and inference-time refusal all solve different versions of the same sequential deletion requirement. The field’s current trajectory points toward hybrid designs that combine selective storage, explicit retention modeling, and auditable behavioral control, while extending certification and interference analysis to large foundation models and long operational horizons (Tang et al., 18 May 2025, Hu et al., 29 Jun 2026, Pan et al., 26 May 2026).