---
title: Concept Unlearning in Machine Learning
url: https://www.emergentmind.com/topics/concept-unlearning-cu
type: topic
---

# Concept Unlearning in Machine Learning

Searching arXiv for recent papers on concept unlearning and related machine unlearning to ground the article.
Concept unlearning (CU) denotes the selective removal of a target concept from a trained model while preserving unrelated capabilities. In the literature provided here, the term is used in multiple, architecture-specific senses: in supervised classifiers, CU appears as class-wise forgetting or removal of example-level knowledge within machine unlearning objectives [2509.14633]; in text-to-image diffusion models, it is the modification of a trained generator so that prompts for a concept no longer produce images expressing that concept, while benign generation remains intact [2505.12395]; in continual learning, CU denotes sequential forgetting requests processed after a continual-learning phase without revisiting historical retained data [2505.12239]; and in large language models, CU is formalized as removing a target entity node and its associated edges from an internal knowledge graph represented by knowledge triplets [2509.15621]. Across these settings, a common theme is the tension between erasure effectiveness and preservation of non-target knowledge, with recent work increasingly shifting from coarse instance-level suppression toward concept-level, distribution-aware, and interpretable mechanisms [2605.14309].

## 1. Scope and formal definitions

Concept unlearning is often framed as a specialized form of machine unlearning in which the forgetting target is not merely a collection of training points but a semantic unit such as a class, category, attribute, style, identity, or entity. In CUFG, the supervised setting begins with a dataset \(D = \{(x_i, y_i)\}_{i=1}^M\), a forget set \(D_f \subseteq D\), and a retain set \(D_r = D \setminus D_f\). Starting from a pretrained empirical risk minimizer
\[
\theta_D^* = \arg\min_{\theta} \frac{1}{|D|} \sum_{(x_i,y_i)\in D} \mathcal{L}(h(x_i;\theta), y_i),
\]
the goal is to produce an unlearned model \(h(\theta_U)\) with minimal influence from \(D_f\) while approximating the retrained gold standard
\[
\theta_{D_r}^* = \arg\min_{\theta} \frac{1}{|D_r|}\sum_{(x_i,y_i)\in D_r} \mathcal{L}(h(x_i;\theta),y_i).
\]
The approximate unlearning objective is written as
\[
\theta_U = \arg\min_{\theta} \left( \frac{1}{|D_r|}\sum_{(x_i,y_i)\in D_r} \mathcal{L}(h(x_i;\theta),y_i) -\lambda \cdot \frac{1}{|D_f|}\sum_{(x_j,y_j)\in D_f} \mathcal{L}(h(x_j;\theta),y_j) \right),
\]
where the first term preserves retained knowledge and the second promotes anti-learning on forgotten data [2509.14633]. In that setting, class-wise forgetting is explicitly identified as concept removal: all samples of one or more classes are to be forgotten, so that the decision boundary structure supporting that category is erased [2509.14633].

In diffusion models, the formal object is a conditional generative distribution. A pretrained text-to-image model \(M\) is modified to \(M'\) so that for all prompts \(x_f\) about a forget concept \(c_f\), \(M'(x_f)\) does not depict \(c_f\), while for retain prompts \(x_r\), \(M'(x_r) \approx M(x_r)\) [2505.12395]. DoCo formulates this more explicitly as distribution alignment: for a sensitive concept \(c^*\) and anchor concept \(c\), unlearning seeks a model \(\hat M\) such that
\[
\min_{\hat{M}} \; \text{dis}\big(P(\hat{M}(c^*)), P(M_{\text{init}}(c))\big),
\]
while preserving the anchor distribution \(P(\hat M(c)) \sim P(M_{\text{init}}(c))\) [2405.15304]. TILDE further recasts CU as a constrained KL projection from the pretrained conditional distribution \(p_{\mathrm{pre}}(x \mid y)\) to a minimum-deviation target \(p^*_{\mathcal C}(x \mid y)\) under a forgetting constraint on expected concept energy [2607.06432].

In continual learning, ACU defines continual unlearning as processing a sequence of forgetting sets \(\check{\mathcal D}_1, \check{\mathcal D}_2, \ldots\) after a continual-learning phase, without accessing the retained dataset \(\hat{\mathcal D}_i = \mathcal D \setminus \cup_{k=1}^i \check{\mathcal D}_k\). The target model after the \(i\)-th request is the exact ridge-regression solution that would have been obtained by retraining from scratch on \(\hat{\mathcal D}_i\) [2505.12239]. In LLMs, CU is made graph-theoretic: for a forgetting target entity \(\bm e^{(\mathrm t)}\), the concept to be removed is
\[
U^{(\mathrm t)} = \{ (\bm s, \bm r, \bm o) \mid \bm s = \bm e^{(\mathrm t)} \ \text{or} \ \bm o = \bm e^{(\mathrm t)} \},
\]
the set of knowledge triplets incident to the target node [2509.15621]. The required behavior is that such triplets must not be regenerated unless already present in the prompt [2509.15621].

## 2. Concept as class, distribution, embedding region, or graph neighborhood

A central distinction in the literature is how a “concept” is represented. In classifier-oriented machine unlearning, concepts are classes or subsets of instances. CUFG explicitly maps random instance forgetting to example-level knowledge scrubbing, whereas class-wise forgetting is described as removal of a concept or category [2509.14633]. MCU, though written as machine unlearning for image classification, likewise operationalizes concept removal by defining class-wise forgetting sets \(D_f\) containing all samples of a specific class [2505.10859].

For diffusion models, concepts are semantic entities expressible through prompts, including instances, styles, persons, offensive content, combinations, and contextual relations. Few-Shot Concept Unlearning with LoRA defines the forget concept \(c_f\) by a small set of 4–5 images and associated prompts, then modifies the CLIP text encoder’s final layers and projection matrix to remove influence of \(c_f\) from the text-image embedding space [2505.12395]. TRUST characterizes concept neurons as cross-attention parameters whose perturbation significantly changes a CLIPScore-based alignment objective for an unsafe concept \(c_u\), thereby locating concept information inside the U-Net’s cross-attention projections [2602.07919]. KSCU interprets concepts through the stepwise denoising process, arguing that later denoising steps have disproportionate influence over concept-specific high-frequency details such as nudity, style, or instance identity [2507.06526].

Several papers argue that keyword-level specification is too narrow. “A Concept is More Than a Word” states that a visual concept is multi-dimensional, can be expressed in diverse textual forms, and often overlaps with related concepts in latent space; keyword-only unlearning is therefore brittle and prone to over-forgetting [2603.18767]. Its Diversified Unlearning framework instead represents a concept through a set of contextually diverse prompts and diversified embeddings [2603.18767]. TILDE similarly treats concepts via a forget energy \(\widetilde E_{\mathcal C}(x)\) computed from CLIP similarity to a descriptor set \(\mathcal Q_{\mathcal C}\), not a single prompt token [2607.06432].

Other work introduces concept representations that are explicitly interpretable. ICED constructs a compact, task-specific concept vocabulary \(\mathcal C_f\) from the forgetting set via an MLLM, aligns image and text embeddings, and decomposes each forgetting image into a sparse, nonnegative combination of semantic concept vectors [2605.14309]. SAEParate and SAeUron-style methods instead treat concepts as sparse latent features in intermediate U-Net activations, with SAEParate enforcing concept-specific clustering through a concept-aware contrastive objective [2605.12122]. SAUCE, by title and abstract, situates concept unlearning in VLMs around sparse autoencoder features that are selectively modified at inference to suppress concept-relevant signals while preserving unrelated information [2503.14530]. In LLM CU, the concept becomes the graph neighborhood of an entity represented by self-constructed knowledge triplets and explanatory sentences [2509.15621].

A plausible implication is that CU research has moved from coarse sample deletion toward explicit semantic interfaces: class partitions, prompt distributions, attention-localized concept neurons, sparse latent features, and entity-centered knowledge graphs each provide a different operational handle on “what is to be forgotten.”

## 3. Core methodological families

The literature represented here falls into several recurring methodological families. One family is gradient-based approximate unlearning for discriminative models. CUFG introduces a forgetting gradient
\[
g_{\text{forget}}^t = - \nabla \theta_{D_f}^{m,t},
\quad
\nabla \theta_{D_f}^{m,t} = \frac{1}{|D_f|}\sum_{(x_j,y_j)\in D_f} \nabla_\theta \mathcal{L}(h(x_j;\theta^t), y_j),
\]
and a gradient corrector that modifies retain-set descent when the retain gradient and forget-set gradient are too aligned [2509.14633]. MCU instead builds a nonlinear Bézier path
\[
\phi_{\theta_c}(t) = (1-t)^2 \theta_o + 2(1-t)t\,\theta_c + t^2 \theta_p
\]
between the original model \(\theta_o\) and a pre-unlearning model \(\theta_p\), then optimizes the control point \(\theta_c\) using a loss that minimizes retain loss and maximizes forget loss along the path [2505.10859]. ACU departs from gradients entirely: for analytic continual learning with a frozen backbone and ridge-regression head, it stores a Knowledge Tracking Matrix
\[
\mathbf{T}_i := (\mathbf{S}_i + \gamma \mathbf{I})^{-1}
\]
and updates it recursively with Woodbury identities so that each unlearning request yields the exact retrained solution on the remaining data [2505.12239].

A second family comprises diffusion-model editing via parameter updates. ESD, CA, UCE, SalUn, SHS, EDiff, and related methods are discussed comparatively in multiple papers [2405.15304; 2507.06526; 2607.06432]. KSCU’s contribution within this family is to model the denoising trajectory explicitly and fine-tune only key steps rather than all steps [2507.06526]. DoCo uses adversarial domain alignment between the denoised latent domain of a sensitive concept \(c^*\) and an anchor concept \(c\), plus concept-preserving gradient surgery that removes components of the unlearning gradient conflicting with a retention gradient [2405.15304]. TILDE replaces anchor mapping with an energy-tilted, anchor-free target distribution and realizes it using residual \(\nabla\)-GFlowNet training to learn score corrections relative to the pretrained diffusion model [2607.06432].

A third family performs parameter-efficient or localized editing in text-conditioning modules. Few-Shot Concept Unlearning with LoRA updates only the CLIP text encoder’s final layers, final normalization, and a low-rank perturbation \(\Delta P = A B^\top\) to the projection matrix \(P\), using retain, forget, and regularization losses [2505.12395]. TRUST dynamically identifies concept neurons in cross-attention layers using gradients of a CLIPScore objective, then selectively fine-tunes only those neurons with either a Concept Influence Penalty or a Concept Sensitivity Reduction regularizer [2602.07919]. These methods treat the text-image alignment interface as the primary site of concept encoding.

A fourth family is feature-space manipulation without changing core model weights. SAEParate suppresses target concepts by modifying sparse latent features in a concept-separated SAE trained on UNet activations [2605.12122]. SAUCE uses sparse autoencoders for selective concept unlearning in VLMs and performs inference-time feature modification instead of weight updates [2503.14530]. ICED uses sparse nonnegative concept decomposition and then optimizes only the CLIP image encoder with concept-level forgetting and preservation losses [2605.14309]. ICCU is even more externalized: it never changes model parameters, instead inducing refusal rules from forget datasets and applying them through inference-time gating, filters, or system prompts [2605.27138].

A fifth family is rule-based or graph-based CU for language and multimodal systems. ICCU induces readable refusal rules from clusters of forget-set queries and composes them by set union, yielding order-independent continual unlearning without cross-request interference [2605.27138]. CORE for LVLMs decomposes forgetting targets into visual and textual concepts, identifies them with concept modules and a concept modulator, then routes inputs to a mixture of concept-aligned refusal experts [2603.21484]. LLM CU via self-constructed knowledge triplets prompts the model to generate triplets and explanatory sentences about the forgetting target and then applies unlearning losses to those generated representations [2509.15621].

## 4. Curriculum, difficulty, and mechanistic views of forgetting

A notable recent direction treats forgetting difficulty itself as a structured object. CUFG introduces Curriculum Unlearning as an easy-to-hard schedule
\[
C_u = \langle T_1, \ldots, T_n \rangle,\qquad
T_i(x) \propto W_i(x) P_u(x),
\]
subject to difficulty monotonicity and data completeness [2509.14633]. In practice, forgotten samples are ranked by model confidence, split into ordered subsets \(D_f^1,\ldots,D_f^n\), and processed sequentially with the same forgetting mechanism. The paper argues that low-confidence samples are easier to forget, while high-confidence forgotten samples may lie near decision boundaries or central prototypes, making abrupt erasure destabilizing [2509.14633].

KSCU offers a different notion of difficulty grounded in the diffusion process. It argues that concepts differ in where they concentrate along the denoising trajectory: class semantics require a broader late-step band, style can be handled by roughly the last half of steps, and instance identity may be largely controlled by the last 20% [2507.06526]. This suggests that difficulty is temporal and architecture-specific rather than purely data-side.

The mechanistic perspective is pushed further by “Toward Understanding Unlearning Difficulty,” which argues that variation in unlearning success reflects model-internal circuits that encode and protect memorized information. The proposed Circuit-guided Unlearning Difficulty (CUD) is a pre-unlearning metric using circuit-level signals, and the paper reports a mechanistic signature: easy-to-unlearn samples are associated with shorter, shallower interactions concentrated in earlier-to-intermediate parts of the original model, whereas hard samples rely on longer and deeper pathways closer to late-stage computation [2601.09624]. This suggests that “difficulty” can be measured before unlearning, and not merely inferred from post hoc performance. Because the user-provided details for this paper are truncated, only the abstract-level characterization is available here [2601.09624].

A plausible implication is that curriculum strategies, timestep selection, and circuit-guided diagnostics all point toward the same emerging principle: effective CU may depend on aligning the intervention with the model’s internal organization of the target concept, rather than applying uniform pressure across data points, layers, or sampling steps.

## 5. Evaluation criteria and empirical patterns

Evaluation across CU papers consistently balances forgetting, preservation, privacy, and efficiency, though the exact metrics differ by modality. In classifier MU, CUFG uses a full-stack suite: Unlearning Accuracy
\[
UA = 1 - \mathrm{Acc}(h(D_f,\theta_U)),
\]
Remaining Accuracy (RA), Test Accuracy (TA), Membership Inference Attack (MIA), Runtime Efficiency (RTE), and Avg.Gap as the mean absolute difference of a method’s metrics from Retrain [2509.14633]. ACU measures exact gaps to retraining in parameters, retain accuracy, forget accuracy, test accuracy, and MIA, reporting \(0.00\) on all of them because its analytic updates reproduce the retrained ridge solution exactly [2505.12239]. MCU uses UA, \(UA_{\text{test}}\), RA, TA, MIA, and Avg. Gap, and reports that \(\text{MCU}_\beta\) approaches retraining almost exactly in class-wise forgetting on CIFAR-10, Tiny-ImageNet, and ImageNet-100 [2505.10859].

In diffusion-model CU, benchmark design often separates unlearning effectiveness from generative retainability. KSCU evaluates Unlearn Accuracy (UA), Attack Success Rate (ASR) under P4D and UnlearnDiffAtk, and FID on COCO30k or UnlearnCanvas [2507.06526]. On NSFW unlearning, it reports UA \(=96.5\%\), P4D ASR \(=23.2\%\), UDA ASR \(=47.2\%\), and FID-30k \(=14.1\), outperforming several non-over-unlearning baselines in the combined robustness–retention trade-off [2507.06526]. Few-Shot Concept Unlearning with LoRA reports per-concept Retain CLIP, Forget CLIP, Unlearning Time, FID, and Detection Rate for Taj Mahal, Mickey Mouse, and Siberian Husky, with average runtime approximately 50–60 seconds using 4–5 images [2505.12395]. TRUST uses ASR, UA, RA, FID, CLIPScore, and TIFA, and reports near-zero I2P ASR and small \(\Delta\)FID while preserving benign CLIP and TIFA scores close to SD1.5 [2602.07919]. TILDE supplements standard retention and forgetting metrics with FADE, a functional alignment measure comparing the unlearned model to a retain-only reference distribution; it reports the lowest average FADE among compared methods for styles [2607.06432].

In VLM and LVLM CU, the metrics explicitly reward preservation of non-target semantics within the same image or refusal behavior aligned to the right concepts. ICED uses zero-shot Target accuracy, Retain accuracy, All accuracy, and normalized out-of-domain utility across Food-101, STL-10, ObjectNet, and other datasets, then aggregates these into Avg. Score [2605.14309]. CORE introduces Context-aware Refusal Rate (CRR), Refusal Gap \(\Delta_{RR}\), Answer Rate (AR), BERTScore, CLIPScore, ROUGE-L, and benchmark specificity on MMBench, SEEDBench, and ScienceQA, showing that concept-grounded refusal outperforms parameter-level baselines in long continual sequences [2603.21484]. ICCU evaluates Refusal Rate on forget sets and retain sets, WMDP multiple-choice accuracy, TOFU ROUGE-L on forget and retain authors, MMLU accuracy, and cross-lingual robustness; it reports high refusal rates on forget topics with low refusal on MMLU and retain sets [2605.27138].

In LLM graph-based CU, the principal metrics are NodeAcc and EdgeAcc. NodeAcc measures whether the target entity can still be predicted from masked explanatory prompts, while EdgeAcc measures whether the correct attribution can still be generated from masked relational prompts. The proposed self-constructed-triplet method achieves NodeAcc \(=0.0\%\) and EdgeAcc \(=0.0\%\) on forgetting targets while preserving much higher NodeAcc and EdgeAcc on non-target entities than GA or NPO baselines [2509.15621].

Across these settings, an empirically recurring pattern is that aggressive methods can achieve near-perfect forgetting at the cost of severe collateral damage, whereas methods that explicitly encode preservation—through gradient correction, decomposition, localization, routing, or distributional targets—tend to dominate on joint forget–retain metrics.

## 6. Major controversies, limitations, and open directions

The strongest recurring controversy concerns what it means to have truly “forgotten” a concept. ACU claims exact unlearning because its post-unlearning classifier is mathematically identical to retraining on the retained set, but this guarantee relies on a frozen backbone, a linear head, and a ridge-regression objective [2505.12239]. In diffusion and VLM settings, most methods are approximate and are evaluated behaviorally rather than by certified equivalence. ICCU, for example, never modifies weights, so it suppresses behavior conditionally through refusal rules rather than removing the underlying knowledge from parameters [2605.27138]. This suggests that different subfields are using different operational notions of CU: parameter-level erasure, distributional realignment, refusal behavior, or output suppression.

Another controversy concerns localization. Keyword-based or single-prompt unlearning is criticized as too narrow because concepts are distributed across many textual forms and latent variants [2603.18767]. Yet broader coverage raises the risk of over-unlearning, especially for semantically related concepts. Diversified Unlearning addresses this by replacing a single keyword with a prompt distribution [2603.18767]; TILDE addresses it through a thresholded forget energy that creates a no-gradient benign region below a CLIP threshold [2607.06432]; TRUST addresses it with dynamic concept neuron masks and preservation loss [2602.07919]. These responses imply that accurate concept coverage and collateral-damage control are inseparable design problems.

Interpretability is also unresolved. ICED, SAEParate, SAUCE, and CORE all argue that explicit concept interfaces—sparse latents, concept vocabularies, routed experts—enable finer control [2605.14309; 2605.12122; 2503.14530; 2603.21484]. But these interfaces depend on the quality of extracted concepts, clustering, and decomposition. ICED explicitly depends on MLLM-generated vocabularies and modality alignment [2605.14309]. SAEParate depends on concept labels and contrastive training for latent separation [2605.12122]. CORE depends on LLM-generated concept descriptions and concept modulators [2603.21484]. A plausible implication is that interpretability is useful insofar as the concept interface is faithful; otherwise, it may simply relocate the brittleness.

Several papers identify scaling challenges. ACU does not address unlearning from the pretrained backbone or from non-analytic nonlinear heads [2505.12239]. Few-shot LoRA-based diffusion CU has not analyzed adversarial prompts deeply and remains vulnerable to circumvention because it alters only a localized region of embedding space [2505.12395]. TRUST supports combinations and conditions but has higher memory cost for CSR due to gradient-of-gradient computations [2602.07919]. ICCU has only been tested on a modest number of domains and depends on embedding-space separability of forget topics [2605.27138]. TILDE is promising for multi-concept sequential unlearning because it is anchor-free and residual, but the paper notes that shared LoRA parameters can still introduce coupling [2607.06432].

Open directions stated across the papers include better difficulty metrics and adaptive curricula [2509.14633], extension to LLMs and multimodal models [2509.14633; 2505.12239], more faithful concept extraction and automated relation discovery [2509.15621; 2605.14309], robustness to adversarial prompting [2505.12395; 2607.06432], scalable handling of many concepts or concept combinations [2602.07919; 2603.21484], and theoretically grounded notions of exactness or certified forgetting outside analytic settings [2505.12239; 2607.06432].

## 7. Historical and conceptual synthesis

Taken together, the cited works show CU evolving along three axes. First, the target of unlearning is becoming more semantic. Early or baseline formulations often forget datasets, instances, or labels. Recent work instead forgets classes as concepts [2509.14633; 2505.10859], styles and identities as conditional distributions [2405.15304; 2607.06432; 2507.06526], unsafe combinations and conditional relations [2602.07919], entity-centered knowledge neighborhoods [2509.15621], or multimodal concept sets with routed refusal experts [2603.21484].

Second, the intervention is becoming more structured. Rather than global gradient ascent or brute-force finetuning, recent methods use gradient-corrected retain descent [2509.14633], analytic closed-form recursion [2505.12239], low-rank text-encoder perturbations [2505.12395], dynamic neuron masks with Hessian-informed penalties [2602.07919], latent concept decomposition [2605.14309], sparse concept-separated autoencoders [2605.12122], distributional tilting and residual GFlowNets [2607.06432], or in-context refusal rules [2605.27138].

Third, evaluation is becoming more faithful to the actual CU desideratum. Retraining or retain-only references play an increasing role as gold standards [2509.14633; 2607.06432]. Privacy and membership inference are treated as part of the forgetting objective, not merely auxiliary diagnostics [2509.14633; 2505.12239]. Out-of-distribution prompts, adversarial prompts, cross-lingual queries, and contextual combinations are now standard stress tests [2405.15304; 2602.07919; 2605.27138; 2603.18767].

This suggests that concept unlearning is no longer well described as simply “making the model worse at a target.” It is increasingly a problem of defining the right semantic target, locating how that target is encoded, and moving the model to a new state that is as close as possible to an appropriate reference—whether retrained, retain-only, analytically exact, or refusal-conditioned—while preventing the target concept from re-emerging under paraphrase, composition, or distribution shift.

Source: https://www.emergentmind.com/topics/concept-unlearning-cu