---
title: Continual Learning Unlearning (CLU)
url: https://www.emergentmind.com/topics/continual-learning-unlearning-clu
type: topic
---

# Continual Learning Unlearning (CLU)

Searching arXiv for recent CLU papers to ground the article in current literature.
Continual Learning Unlearning (CLU) denotes the sequential setting in which a model must process a stream of requests that may require acquiring new knowledge, removing specified knowledge, or enforcing refusal behavior, while preserving retained behavior over time. Recent formulations instantiate this setting in classifiers, large language models, large vision–language models, text-to-image diffusion models, OOD detectors, and task-free concept-drift streams; despite architectural differences, they converge on a common requirement: deletion requests are recurrent rather than one-shot, so forgetting quality, retained utility, privacy, and long-horizon stability must be evaluated jointly [2604.12686][2203.12817][2604.14644].

## 1. Definition and scope

CLU generalizes conventional machine unlearning from a single deletion event to a sequence of learn–unlearn operations. In one formalization, the system processes requests \(R_i=(I_i,T_i,D_i)\), where \(I_i\in\{L,U\}\) indicates learn or unlearn, \(T_i=t\) is a task identifier, and \(D_i\) is the associated dataset; if \(I_i=L\), the model must learn task \(t\), and if \(I_i=U\), it must unlearn task \(t\) with no access to its data [2509.17530]. In another formulation, the data at each cycle are partitioned into a forget set \(D_f\), a retain set \(D_r\), and new knowledge \(D_{new}\), and the adapted model is expected to satisfy \(f_{M'}:X_{D_f}\nrightarrow Y_{D_f}\), \(f_{M'}:X_{D_r}\rightarrow Y_{D_r}\), and \(f_{M'}:X_{D_{new}}\rightarrow Y_{D_{new}}\) [2604.12686].

The scope of “unlearning” varies across the literature. Some methods target parametric removal, aiming to make the post-unlearning model behave like a model retrained without the forgotten data, as in analytic ridge-regression downdates and certified continual unlearning theory [2505.12239][2606.29832]. Others target behavioral unlearning: CURaTE stores forget requests as embeddings and refuses prompts whose semantic similarity exceeds a threshold, while leaving the base LLM untouched [2604.14644]. ICCU similarly induces readable refusal rules from forget datasets and applies them in context at inference time, again without updating model parameters [2605.27138]. In multimodal settings, CLU can mean selective refusal of image–instruction pairs rather than parameter-level erasure, as in continual unlearning for large vision–language models [2603.21484].

The granularity of the forgetting target is likewise heterogeneous. Reported settings include class-level deletion, cross-task category-level unlearning, random sample-level unlearning, identity removal in face recognition, selective concept suppression in diffusion models, and removal of outdated chunks in sliding-window learning under concept drift [2506.09347][2604.12686][2503.13769][2603.14484]. This breadth has made CLU less a single algorithmic problem than a family of related sequential deletion problems coupled by stability constraints.

## 2. Formal objectives and desiderata

A recurring formulation treats CLU as a multi-objective problem with three goals: precise deletion of unwanted knowledge, efficient integration of new knowledge while preserving prior information, and minimizing knowledge leakage across cycles [2604.12686]. In the continual setting, the model evolves by repeated application of an update operator, for example
\[
M(t)=F(M(t-1),D_f(t),D_r(t),D_{new}(t)),
\]
and success is judged not only by current-task behavior but also by what remains true for all earlier forget and retain sets [2604.12686].

Several papers sharpen this into retraining-based criteria. ACU defines exact forgetting for analytic classifiers by maintaining sufficient statistics and producing the same parameters as retraining on the retained set; with \(D_r=D\setminus D_f\), exact unlearning is written as
\[
A(D_r)=_d R_A(D,A(D),D_f),
\]
where \(=_d\) denotes equality in distribution [2505.12239]. CLPU adopts the same exact-unlearning viewpoint for continual learning and private unlearning, requiring the unlearned model to be indistinguishable from a model trained only on non-forgotten tasks [2203.12817]. Certified continual unlearning theory further defines \((\varepsilon,\delta)\)-certified continual unlearning by requiring bidirectional closeness of published model distributions before and after deletion:
\[
\Pr(\tilde{w}_t^{-S_{1:t}}\in O)\le e^\varepsilon \Pr(\tilde{w}_t^{-S_{<t}}\in O)+\delta,
\]
together with the reverse inequality, for every measurable \(O\subset\mathcal{W}\) [2606.29832].

A central theoretical contribution is the decomposition of post-unlearning excess risk into continual-learning excess risk and unlearning loss. If \(F_\tau(w)\) is the population risk for retained task \(\tau\), the post-unlearning objective is
\[
\mathrm{E_{LU}}=\mathbb{E}\!\left[\frac{1}{N_t}\sum_{\tau\in[t]\setminus S_{<t}} F_\tau(\tilde{w}_t^{-S_{1:t}})\right]-\min_{w\in\mathcal{W}}\frac{1}{N_t}\sum_{\tau\in[t]\setminus S_{<t}}F_\tau(w),
\]
and it decomposes as \(\mathrm{E_{LU}}=\mathrm{E_U}+\mathrm{E_L}\), where \(\mathrm{E_U}\) is the unlearning loss and \(\mathrm{E_L}\) is the continual-learning excess risk [2606.29832]. This formulation makes the forgetting–retention dilemma explicit: improvements in retention regularization need not minimize the error introduced by deletion.

The literature therefore evaluates CLU with a wider objective set than conventional continual learning. In addition to retained-task accuracy, reported criteria include privacy leakage, output indistinguishability from retraining, calibration of refusal behavior, recovery under drift, and long-horizon resistance to catastrophic interference [2203.12817][2603.21484][2603.14484].

## 3. Methodological families

The CLU literature has diversified into several method families. A first family uses replay, distillation, and curvature-aware optimization in a single parametric model. UniCLUN interleaves continual learning and unlearning through a multi-teacher, single-student framework with fixed memory replay and controlled knowledge distillation [2408.11374]. UG-CLU casts approximate CLU as KL-divergence minimization and decomposes the update into preserve, learn, unlearn, and saliency terms; it then approximates remaining-Hessian compensation with a fast-slow weight adaptation mechanism, adaptive sample coefficients, and a balanced weight saliency mask [2505.15178]. ErrorEraser treats data biases as erroneous memories, identifies representative outliers with Normalizing Flows, and erases them by shifting their decision regions to a pseudo-class and then pruning it [2506.09347]. SAFER addresses repeated unlearning by stabilizing retain representations and driving negative unlearning margins for forget samples, specifically to mitigate Knowledge Erosion and Forgetting Reversal [2604.19108].

A second family relies on parameter isolation or parameter-efficient adaptation. BID-LoRA introduces retain, new, and unlearn adapter pathways in attention layers, with a signed merge
\[
W' = W + S(B_{\mathrm{ret}}A_{\mathrm{ret}} + B_{\mathrm{new}}A_{\mathrm{new}} - B_fA_f),
\]
and an escape unlearning objective that maps forget-class embeddings to a point maximally distant from retained centroids [2604.12686]. UnCLe instead uses a hypernetwork to generate task-specific parameters and unlearns a task by aligning the corresponding generated parameters with Gaussian noise, thereby preventing relapse without storing past task data [2509.17530]. Orthogonal Subspace Projection constrains each new LoRA update to the orthogonal complement of previously used subspaces via SVD-guided projection, enabling static fusion without parameter collision [2604.12526]. TFER combines LoRA with modular orthogonality in an OOD-detection setting, using free-energy repulsion to push forgotten classes into high-energy OOD regions while anchoring retained classes to a low-energy manifold [2602.06331].

A third family is analytic or certified. ACU operates on frozen features and a ridge-regression head, maintains an inverse Gram matrix as a Knowledge Tracking Matrix, and analytically downdates both the inverse and the classifier weights for each forget request, yielding exact equality to retraining on the retained set [2505.12239]. The certified CLU framework develops gradient-based and Hessian-based continual unlearning procedures with explicit approximation-error bounds and Gaussian publish mechanisms, and proposes a hybrid forgetting-enhanced Hessian strategy to reduce storage while preserving certified performance [2606.29832]. CLPU-DER++ achieves exact private unlearning by construction through isolated temporary networks that can later be deleted, while the main network never absorbs temporary-task information [2203.12817].

A fourth family replaces parameter updates by inference-time control. CURaTE trains a sentence embedder once, appends forget-request embeddings in real time, and refuses a prompt whenever the maximum cosine similarity to stored forget requests exceeds a threshold \(\tau\) [2604.14644]. ICCU clusters forget data in embedding space, induces high-precision natural-language refusal rules from cluster representatives, stores only centroids and rules, and applies them either as a filter or as part of the system prompt; because rule sets are accumulated as an order-independent union, the method is designed to be compositional and free of cross-request interference [2605.27138].

A fifth family specializes CLU to multimodal or generative models. CORE decomposes each forget category in a large vision–language model into visual attributes and linguistic intents, refines category-specific concept activations with a concept modulator, routes inputs through a mixture of refusal experts, and calibrates refusal strength at inference time by the input’s relevance to unlearned concepts [2603.21484]. DUGE redirects cross-attention in latent diffusion models away from target concepts using negative attention targets, memory-based prior preservation, and KL regularization to prevent generalization erosion across sequential deletions [2503.13769]. Distill, Forget, Repeat treats each diffusion unlearning step as teacher–student distillation with contextual trajectory re-steering, retain replay, and parameter regularization to avoid retention collapse and ripple effects [2512.02657]. A related systems-level line, RM-DLoRA, implements CL and MU on resistive-memory accelerators by freezing analogue backbone weights in RM arrays and confining all adaptation to digital LoRA branches stored in SRAM [2601.10037].

The following summary captures representative designs without exhausting the space:

| Framework | Core mechanism | Reported setting |
|---|---|---|
| CORE | concept modulator, mixture of refusers, multimodal concept-driven routing | continual unlearning in LVLMs |
| BID-LoRA | retain/new/unlearn LoRA pathways with escape unlearning | continual learning and unlearning in classification and face recognition |
| ACU | gradient-free analytic downdates of sufficient statistics | exact continual unlearning for analytic classifiers |
| CURaTE | embedding-based gate with refusal threshold and no parameter updates | real-time continual unlearning for LLMs |
| ICCU | pattern-induced refusal rules with order-independent union | in-context continual unlearning for language models |
| SAFER | retain representation stability plus negative logit margins | repeated phase-wise unlearning in classification |

## 4. Evaluation protocols and empirical findings

CLU has no single canonical benchmark or metric suite. Reported evaluations mix task-accuracy metrics, privacy metrics, refusal metrics, OOD scores, and generative-quality measures. BID-LoRA evaluates forgetting with Accf, retention with Accr, new learning with Accn, overall performance with Acco, privacy with membership inference attack success rate, and leakage with KL divergence to an oracle [2604.12686]. CORE evaluates forget-side precision with Context-aware Refusal Rate, indiscriminate negation with Refusal Gap (ARR), retain-side behavior with Answer Rate, and general capability with Specificity over MMBench, SEEDBench, and ScienceQA [2603.21484]. TFER explicitly argues that in OOD detection, effective unlearning should be assessed with AUROC and FPR@95 rather than classification accuracy [2602.06331]. CLPU introduces IJSD, AJSD, JS-ratio, and IRR to test whether an unlearned model is statistically indistinguishable from a retained-only model [2203.12817].

The reported numbers vary strongly by setting. On a Vicuna-based LVLM, CORE reports Last \(S=96.54\), \(AR=88.02\), \(CRR=90.67\), and \(ARR=3.74\); on a LLaMA-2-based LVLM, it reports \(S=97.26\), \(AR=84.41\), \(CRR=86.16\), and \(ARR=6.95\) [2603.21484]. BID-LoRA updates only \(5.08\%\) of parameters on CIFAR-100 and reports Accf between \(0\)–\(0.93\%\), KL \(0.60\)–\(0.79\), MIA \(\approx 0.50\)–\(0.57\), and knowledge leakage of about \(2.52\%\) variation in Acco across six tasks; on CASIA-Face100, it reports Accf \(0\)–\(0.88\%\), KL \(0.74\)–\(1.10\), MIA \(0.50\)–\(0.57\), and about \(1.98\%\) Acco drop from Task 1 to Task 6 [2604.12686]. On a 30-task CIFAR-100 sequence with static LoRA fusion, Orthogonal Subspace Projection maintains about \(58.06\%\) retained accuracy where static fusion falls to \(12.74\%\) [2604.12526].

Methods that target exactness or real-time deployment report a different profile. ACU yields zero gaps in \(\Delta\)Params, \(\Delta\)Retain, \(\Delta\)Forget, \(\Delta\)Test, and \(\Delta\)MIA relative to retraining, and processes 25 requests in about \(12.68\) seconds and 50 requests in about \(18.16\) seconds [2505.12239]. CURaTE reports average unlearning time per stage of \(0.04\) s on RETURN, compared with \(195.6\) s for GA, \(229.5\) s for GradDiff, \(178.8\) s for PO, \(249.4\) s for NPO, \(209.4\) s for SO-PO, \(2.8\) s for GUARD, \(327.6\) s for O3, and \(323.2\) s for UniErase; its inference overhead is \(0.01\) s per query [2604.14644]. ICCU reports high refusal rates on WMDP with low MMLU over-refusal, for example Bio \(0.960\), Cyber \(0.943\), Chem \(0.953\), and MMLU \(0.033\) with Qwen3-14B in filter mode [2605.27138].

Generative and boundary-preserving settings reveal different pathologies and different successes. DUGE reports markedly lower drift than a baseline on COCO-val prompts with target class names removed, for example Set 4 Step 5 \(KID=5.66\times 10^{-4}\) and \(FID=3.72\) versus baseline \(KID=1.44\times 10^{-2}\) and \(FID=36.42\) [2503.13769]. Distill, Forget, Repeat reports across a 10-step diffusion benchmark that Fixed-Context Mapping maintains \(UA\approx0.85\)–\(0.93\), \(RRA\approx0.81\)–\(0.88\), \(GRA\approx0.83\)–\(0.88\), and \(FID\approx8.0\)–\(10.5\), while Adaptive-Context Mapping pushes UA as high as \(0.97\) on early concepts with somewhat lower late-step retention [2512.02657]. In OOD detection, TFER reports AUROC \(90.7\), FPR@95 \(44.1\), Retain-Acc \(76.1\), and AVG OOD AUROC/FPR \(91.0/36.8\) when forgetting 20 classes under a 25-epoch budget [2602.06331].

## 5. Failure modes, misconceptions, and design tensions

A major theme in CLU is that repeated deletion exposes failure modes that are not apparent in one-shot unlearning. SAFER identifies Knowledge Erosion, where retain accuracy progressively degrades, and Forgetting Reversal, where previously forgotten samples become recognizable again in later phases [2604.19108]. UnCLe names closely related phenomena spill, meaning degradation on retained tasks when a different task is unlearned, and relapse, meaning performance on forgotten tasks resurfaces during subsequent learning [2509.17530]. In continual language–vision unlearning, CORE emphasizes spurious associations and over-refusal: sequential updates distort shared multimodal representations, causing benign inputs that share superficial cues with forget categories to trigger refusals [2603.21484]. DUGE identifies generalization erosion in diffusion models, where repeated concept deletions damage unrelated prompt fidelity and image quality [2503.13769].

A common misconception is that low accuracy on the forgotten subset is sufficient evidence of successful CLU. Several papers explicitly reject this. In boundary-preserving unlearning for OOD detection, the target class should become OOD relative to the retained system, so AUROC and FPR@95 are the relevant measures rather than class accuracy [2602.06331]. In private unlearning, merely degrading forgotten-task accuracy can still leak evidence that those tasks influenced the model; CLPU therefore evaluates output-level indistinguishability via IJSD, AJSD, JS-ratio, and IRR [2203.12817]. In language systems, refusal coverage alone is also insufficient: CORE distinguishes any refusal from context-aware refusal with ARR and CRR, and CURaTE shows that threshold choice trades off false positives against false negatives [2603.21484][2604.14644].

The forgetting–retention dilemma is now explicit in theory. The certified CLU framework shows that post-unlearning excess risk decomposes into continual-learning excess risk and unlearning loss, so a regularizer that improves retention can still impede targeted deletion [2606.29832]. This suggests that CLU cannot be reduced to “continual learning plus a forgetting penalty”; the deletion operator and the retention mechanism must be co-designed. Parametric isolation methods, analytic methods, and inference-time guardrails each instantiate a different point in this trade-off space.

Behavioral methods also have characteristic weaknesses. CURaTE notes that semantic filters do not eliminate leakage under adversarial obfuscation: persona-based jailbreaks, payload splitting, and Base64 encoding reduce detection, although lower thresholds improve recall [2604.14644]. ICCU reports strong paraphrase and multilingual robustness but observes higher retain-side refusal in Chinese, attributed to embedding-space entanglement [2605.27138]. In multimodal expert-routing systems, CORE notes dependence on high-quality concept descriptions and possible misrouting under extreme concept overlap [2603.21484].

## 6. Applications, systems implications, and open directions

CLU is motivated repeatedly by privacy regulation, safety, and deployability. Multiple papers tie the problem directly to GDPR or the “right to be forgotten,” including exact private unlearning in CLPU, parameter-efficient identity removal in BID-LoRA, and certified continual unlearning theory [2203.12817][2604.12686][2606.29832]. In face recognition, BID-LoRA frames CASIA-Face100 as a proxy for identity management systems in which new users must be enrolled and withdrawn users removed [2604.12686]. CORE emphasizes compliance, auditability, and safe deployment, and argues that concept-grounded refusal improves interpretability and controllability in multimodal CLU [2603.21484].

The application surface already extends beyond standard classification. UIL connects unlearning to task-free continual learning under concept drift by replacing full sliding-window retraining with unlearning of outgoing data and incremental learning of incoming data; on MNIST and Fashion-MNIST streams it reports per-batch times of about \(6.27\)–\(6.67\) s for sliding-window retraining versus about \(2.04\)–\(2.19\) s for the unlearning-based alternative [2603.14484]. TFER uses CLU to preserve ID/OOD boundaries while forgetting target classes, which is particularly relevant for open-world anomaly detection [2602.06331]. RM-DLoRA pushes the paradigm to edge hardware, reporting up to \(147.76\times\) lower training cost, \(387.95\times\) lower deployment overhead, and \(48.44\times\) lower inference energy across face recognition, speaker authentication, and stylized image generation by freezing analogue RM weights and implementing adaptation in digital LoRA branches [2601.10037].

The dominant open problems recur across papers. Many high-performing methods still require replay buffers, retain sets, or stored curvature information; others avoid storage by moving unlearning to inference time but sacrifice formal guarantees under adversarial prompting [2604.12686][2604.14644]. Some frameworks remain task-incremental and do not yet support class-incremental or instance-level unlearning, as explicitly noted for UnCLe [2509.17530]. Certified theory remains local and storage–accuracy trade-offs are unresolved at scale [2606.29832]. Multimodal and generative settings reveal additional difficulties: correlated concepts, indirect prompting, style entanglement, and long-sequence drift [2503.13769][2512.02657].

A plausible synthesis is that CLU is emerging as a systems problem rather than a single optimization primitive. Exact analytic downdates, certified approximations, expert isolation, concept-grounded routing, and inference-time refusal all solve different versions of the same sequential deletion requirement. The field’s current trajectory points toward hybrid designs that combine selective storage, explicit retention modeling, and auditable behavioral control, while extending certification and interference analysis to large foundation models and long operational horizons [2505.12239][2606.29832][2605.27138].

Source: https://www.emergentmind.com/topics/continual-learning-unlearning-clu