Selective Forgetting in Deep Learning

• Selective Forgetting is the targeted removal or suppression of specific data within models while preserving essential learned capabilities.
• It encompasses methods such as weight scrubbing, span-level unlearning, and external memory pruning across various AI architectures.
• Evaluations focus on balancing deletion strength and retained performance using metrics like error rates, memory reduction, and privacy measures.

Searching arXiv for recent and foundational papers on selective forgetting to ground the article. Selective forgetting denotes the targeted removal, suppression, or attenuation of specific information while preserving other capabilities, memories, or task performance. Across contemporary research, the term covers class-level adaptation of pre-trained models, span-level unlearning in LLMs, weight-level scrubbing against white-box probes, replay reweighting in federated continual learning, geometry-aware mitigation of primacy bias in deep reinforcement learning, and selective pruning of external memory in agents and robots (Kuwana et al., 2024, Wang et al., 2024, Golatkar et al., 2019, Wuerkaixi et al., 20 Feb 2025, Falzari et al., 2 Feb 2025, Gu et al., 22 Apr 2026, Bärmann et al., 13 Apr 2026). Earlier cognitive work used the same term for the observation that forgetting is more pronounced on the lexical level of tags than on the semantic level of latent topics, establishing a “words versus meaning” distinction that remains influential in later computational formulations (Kowald et al., 2014).

1. Conceptual scope and formal problem settings

Selective forgetting is not a single task but a family of deletion problems that differ in granularity, access assumptions, and target behavior. In the strongest white-box formulation for deep networks, the original training set is split as $D = D_r \sqcup D_f$, and a scrubbing function $S$ is sought such that any readout function of the scrubbed weights is indistinguishable from the same function applied to a model trained without $D_f$. This is formalized by a KL objective over readout distributions, together with a utility-aware “Forgetting Lagrangian” that trades off retained-data loss against divergence from the retrained-without-$D_f$ distribution (Golatkar et al., 2019).

Later work broadens this sample-level view. In black-box selective forgetting for CLIP-like pre-trained models, the forget set is a subset of class labels and the retain set is its complement; success means making the model unable to recognize the forgotten classes while maintaining accuracy on the remaining classes (Kuwana et al., 2024). In LLMs, SeUL moves from instance-level deletion to token-span deletion: for each forget sequence $x$, a forget span set $s^x$ specifies which contiguous subsequences should be unlearned, so that only the sensitive spans, rather than the entire sequence, are targeted (Wang et al., 2024). A finer-than-sample formulation goes further by introducing four datasets—$D_r$, $D_f$, $D_{f,\mathrm{clean}}$, and $D_{r,\mathrm{extra}}$—to distinguish whether unwanted information is present and whether it is available during forgetting, thereby supporting backdoor-style and shortcut-style forgetting rather than coarse sample removal (Hayase et al., 2020).

A distinct but related antecedent appears in social tagging. There, selective forgetting refers to a differential decay pattern across representational levels: semantic or “gist traces” remain relatively stable, whereas lexical or “verbatim traces” decay more strongly over time. The corresponding recommender formulations, $S$0 and $S$1, operationalize forgetting on the topic level or the tag level, with empirical evidence favoring lexical-level forgetting (Kowald et al., 2014). This suggests that selective forgetting can also mean uneven forgetting across layers of representation rather than explicit deletion requests.

2. Weight-, gradient-, prompt-, and operator-space mechanisms

One major line of work implements selective forgetting directly in parameter space. In deep networks, scrubbing is derived from local quadratic approximations and SGD stability. The exact quadratic-case map reduces, near convergence, to a noisy Newton-like deletion update,

$S$2

with $S$3, and the practical “Fisher forgetting” variant uses anisotropic noise shaped by the Fisher Information Matrix to hide residual information while preserving retained-data utility (Golatkar et al., 2019). In deep reinforcement learning, this idea is adapted into Fisher-Guided Selective Forgetting, which interprets primacy bias as over-retention of early replayed experience and applies a Fisher-preconditioned stochastic scrub,

$S$4

periodically during SAC training (Falzari et al., 2 Feb 2025).

A related geometry-based optimizer appears in continual learning. Selective Forgetting-Aware Optimization defines an update

$S$5

where $S$6 is a sampled maximum cosine alignment between the current gradient and stored past gradients. The update is therefore accepted, projected, or discarded depending on similarity, and the paper explicitly frames this as controlled forgetting rather than blanket preservation (Singh et al., 8 Feb 2026).

Black-box settings replace parameter editing with prompt-space control. In “Black-Box Forgetting,” the only trainable object is a prompt representation. Retention uses cross-entropy,

$S$7

while forgetting uses an entropy-maximizing objective,

$S$8

so forgotten-class outputs become as close to random as possible. The paper’s main technical device, Latent Context Sharing, factorizes prompt latents into shared and token-specific components to make derivative-free CMA-ES search feasible (Kuwana et al., 2024). In language-model unlearning, SeUL similarly modifies the objective rather than the architecture: it minimizes positive log-probability only over sensitive forget spans,

$S$9

thereby lowering probability on selected tokens without reversing training on the whole example (Wang et al., 2024).

Selective forgetting has also been extended beyond neural-network classification and generation. In nonlinear least-squares option calibration, forgetting a quote subset $D_f$0 is implemented as an operator downdate of the Gauss–Newton system,

$D_f$1

followed by

$D_f$2

At a fixed linearization, this is locally exact relative to retraining on $D_f$3 (Özsoy, 18 Nov 2025).

3. Replay-, memory-, and surrogate-based forgetting

Another large family of methods operates through replay, rehearsal, or memory management rather than direct parameter scrubbing. In heterogeneous federated continual learning, AF-FCL argues that preserving all previous knowledge is not always beneficial because replay can carry biased features, spurious correlations, and noisy information across clients. It therefore weights replay samples by a credibility score

$D_f$4

estimated from a normalizing-flow latent distribution of the current task, so that low-credibility replay contributes less to the classification loss (Wuerkaixi et al., 20 Feb 2025).

In deep generative models, Selective Amnesia derives forgetting from continual-learning principles. The forgotten concept is redirected toward a user-specified surrogate distribution $D_f$5, while retained concepts are preserved by generative replay and an EWC-style Fisher penalty. The resulting objective

$D_f$6

makes forgetting controllable: a celebrity prompt can be remapped to “a middle aged man,” and nudity can be remapped to “a person wearing clothes,” rather than merely collapsed into noise (Heng et al., 2023).

At the level of online memory dynamics, selective forgetting can emerge from structured rehearsal. In the Perpetual Learning Machine, memories recalled with probabilities $D_f$7, $D_f$8, and $D_f$9 exhibit graded retention under Perpetual Stochastic Gradient Descent: the frequently recalled group stays near zero error, the rarely recalled group rises to a higher-error homeostatic state, and the unrecalled group deteriorates without leveling off (Simpson, 2015). In real-time RBF learning, SMRLS replaces global forgetting factors with a partition-based memory objective,

$D_f$0

thereby retaining one synthesized representative per visited input-space partition instead of exponentially downweighting all old samples (Fei et al., 2022).

Selective forgetting can also be induced by introducing a competing task. In watermark removal, Attention Distraction defines a lure classification loss plus attention anchoring,

$D_f$1

where the model is first aligned to its original behavior on proxy data and then distracted toward a new lure class. The intended result is retention of the main task with forgetting of the watermark task (Zhong et al., 2022).

4. External memory, agents, robots, and in-context suppression

In agent systems with externalized memory, selective forgetting becomes a systems capability rather than a weight update. FSFM organizes memory into sensory, working, and long-term layers, and models passive forgetting with

$D_f$2

extended by multi-factor modulation and a weighted importance score,

$D_f$3

Forgetting is then implemented through passive decay, active deletion, safety-triggered deletion, and adaptive reinforcement, under a capacity-constrained pruning objective over memory records (Gu et al., 22 Apr 2026). FadeMem develops a related but more explicitly continuous memory-strength model for LLM agents. Each memory item stores content, text, strength, timestamp, and frequency, and decays according to

$D_f$4

with long-term and short-term layers, reinforcement on access, contradiction suppression, and LLM-guided fusion of semantically related memories (Wei et al., 26 Jan 2026).

Robotic episodic memory introduces another variant. H$D_f$5-EMV incrementally constructs a hierarchical history tree and assigns each node an expiration time

$D_f$6

with level-dependent lifetime scaling. When a node expires, an LLM estimates a relevance factor $D_f$7, extends its lifetime by $D_f$8, and forgets it if it remains expired. Relevance is conditioned on learned natural-language rules derived from user feedback, so forgetting is explicitly personalized (Bärmann et al., 13 Apr 2026).

A different form of selectivity is inference-time suppression. In-context knowledge unlearning marks target knowledge with <<UNL>> ... <</UNL>> tokens and fine-tunes the model with a forgetting loss

$D_f$9

and a retention loss

$x$0

The model then answers normally unless the current query depends on the marked target knowledge, in which case it emits “forgot” (Takashiro et al., 2024).

5. Evaluation protocols and empirical regularities

Selective forgetting is evaluated through markedly different protocols depending on the regime.

Setting	Typical metrics or protocol	Representative papers
Black-box class forgetting	$x$1, $x$2, harmonic mean $x$3	(Kuwana et al., 2024)
Span-level LM unlearning	$x$4, $x$5, $x$6, $x$7	(Wang et al., 2024)
Multimodal benign forgetting	ROUGE-L, Fact Score, Meaningful Score on S-MLLMUn Bench	(Zeng et al., 25 Nov 2025)
Privacy auditing of unlearning	membership inference, data reconstruction, knowledge leakage across 21 attack and defense methods, 11 datasets, 10 models, 10 unlearning techniques, and 10 evaluation metrics	(Qian et al., 19 Dec 2025)

Several empirical patterns recur. In black-box prompt-based forgetting, forgetting and retention are summarized by

$x$8

and the best results arise from methods that raise forgotten-class error while keeping retained-class accuracy nearly unchanged; on CIFAR-10, the reported method attains $x$9, $s^x$0, and $s^x$1 (Kuwana et al., 2024). In language-model unlearning, the introduction of $s^x$2 and $s^x$3 makes it possible to separate selective forgetting of sensitive spans from generic degradation of memorization or generation (Wang et al., 2024). In multimodal unlearning, S-MLLMUn Bench evaluates forget-set removal, retain-set preservation, and visual understanding simultaneously, reflecting the paper’s criterion of benign forgetting (Zeng et al., 25 Nov 2025).

System-oriented evaluations further show that selective forgetting is often judged by memory and compute as much as by accuracy. FadeMem reports a $s^x$4 storage reduction while improving retrieval and multi-hop reasoning (Wei et al., 26 Jan 2026). FSFM reports access efficiency $s^x$5, content quality $s^x$6 signal-to-noise ratio), and security performance $s^x$7 elimination of security risks) (Gu et al., 22 Apr 2026). H$s^x$8-EMV reports $s^x$9 memory-size reduction, $D_r$0 query-time compute reduction, and a $D_r$1 improvement in second-round queries by adapting to user-specific priorities (Bärmann et al., 13 Apr 2026). In continual learning, SFAO reports a $D_r$2 memory reduction on MNIST settings while remaining competitive in accuracy and forgetting (Singh et al., 8 Feb 2026).

6. Limitations, misconceptions, and open issues

A persistent misconception is to equate behavioral suppression with erasure. Black-box prompt forgetting does not change model weights and is therefore “closer to behavioral suppression at inference time than to certified machine unlearning” (Kuwana et al., 2024). In-context knowledge unlearning makes this even more explicit: layerwise analyses show that the correct answer often remains recoverable through most of the forward pass and the decision to forget is made only at the last layer, hence the claim that “LLMs pretend to forget” (Takashiro et al., 2024). This suggests that apparent forgetting under standard QA metrics may still leave internal representations intact.

Another recurrent limitation concerns guarantees. Weight scrubbing in deep networks provides a distributional indistinguishability objective that is described as a generalization and weakening of Differential Privacy rather than full DP itself (Golatkar et al., 2019). In nonlinear calibration, selective forgetting is locally exact only at a fixed Gauss–Newton linearization, with relinearization needed when deletions shift the optimum materially (Özsoy, 18 Nov 2025). In many systems papers, the criteria are operational and empirical rather than formal.

Selective forgetting also does not uniformly improve privacy. PrivUB shows that unlearning may create privacy vulnerabilities through discrepancy between pre-trained and unlearned models, residual leakage in the unlearned model, and post-unlearning recovery via fine-tuning or quantization; it further reports that retain-set privacy can worsen after unlearning (Qian et al., 19 Dec 2025). A plausible implication is that deletion quality and privacy auditing should be treated as separate evaluation axes rather than assumed to coincide.

Finally, selective forgetting is controversial because “remembering more” is not always beneficial. In heterogeneous federated continual learning, AF-FCL argues that old knowledge can be harmful when it carries biased features, spurious correlations, or label-noise contamination, so selective forgetting becomes a way to suppress inaccurate or low-credibility replay rather than a concession to catastrophic forgetting (Wuerkaixi et al., 20 Feb 2025). The field therefore faces a dual challenge: forgetting must be strong enough to remove targeted information, yet narrow enough to avoid destroying utility, visual understanding, or retained memories. This tension, rather than any single algorithmic template, is the central organizing principle of selective forgetting as a research area.