---
title: Escape Unlearning Overview
url: https://www.emergentmind.com/topics/escape-unlearning
type: topic
---

# Escape Unlearning Overview

Escape unlearning is a family of concepts in machine unlearning that addresses how a model can *escape* the influence of revoked data, how revoked knowledge can nevertheless *escape* a nominal unlearning procedure, and how unlearning systems can be designed so that harmful or deleted information does not re-emerge under retraining, jailbreaks, or prompt variation. In the literature, the term is used in several related senses: escaping a local optimum anchored by deleted data through targeted perturbation [2310.05331]; ensuring that future online outputs are statistically indistinguishable from retraining without the deleted point [2505.08557]; identifying deletions that can be safely skipped because they are “ε-unnecessary” [2501.16614]; and preventing forgotten knowledge from resurfacing through side channels, relearning, or adversarial reformulation [2406.13348], [2509.22263], [2604.06154]. This suggests that “escape unlearning” is best understood not as a single algorithm, but as a cluster of formulations about removal, robustness, leakage, and recovery.

## 1. Multiple meanings of the term

A central usage appears in “Unlearning with Fisher Masking,” where fine-tuning from a deployed model on the remain set alone often remains trapped in a local optimum anchored by representations of the unlearn set, leading to incomplete forgetting; “escape unlearning” refers to using a targeted perturbation that “escapes” residual influence of revoked data by pushing the model off the basin shaped by $D_u$, after which light fine-tuning on $D_r$ suffices [2310.05331]. In that setup, the training set is partitioned as $D = D_r \cup D_u$, the target unlearned model is the minimizer of $L(\theta; D_r)$, and the practical criteria include low influence of $D_u$ on outputs, high retained accuracy on $D_r$, stability across retrials and during fine-tuning, and optional low re-learnability of $D_u$ if only $D_r$ is provided.

A second usage concerns failure rather than success. In “Textual Unlearning Gives a False Sense of Unlearning,” the escape problem is that unlearned texts can remain identifiable or reconstructable post-unlearning because the act of unlearning itself creates side channels, especially when both the pre-unlearning and post-unlearning models or their outputs are available [2406.13348]. In this sense, deleted content escapes the unlearning process rather than the model escaping the deleted-data basin.

A third usage is safety-oriented. “Exclusive Unlearning” interprets escape unlearning as models circumventing unlearning via jailbreak or prompt-injection-style reformulations, and addresses it by forgetting everything outside a whitelisted retained domain rather than enumerating harmful data [2604.06154]. “Collapse of Irrelevant Representations” frames escape as dangerous capabilities remaining accessible after an unlearning procedure or re-emerging under alternative prompts, jailbreaks, or post-hoc fine-tuning, and proposes selective removal of common representation subspaces before computing unlearning updates [2509.11816].

A fourth usage is formal and online. “Online Learning and Unlearning” defines the desired property so that, after processing each deletion, all subsequent outputs are statistically indistinguishable from those that would have been produced by an algorithm retrained on the sequence with the deletions applied; because this is enforced for each interval, once a point is deleted, it remains protected at every future time step [2505.08557].

A fifth usage is procedural. “FUNU” describes escape unlearning as safely skipping some requested deletions because removing those points would not materially change the retrained model, formalized through “unnecessary unlearning” [2501.16614].

## 2. Basin escape through Fisher masking

The Fisher-masking formulation is one of the clearest operational definitions. The Fisher matrix on a dataset $S$ is
$$
F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],
$$
and the diagonal approximation used in practice is
$$
F_i(\theta;S)=\mathbb{E}_{(x,y)\sim S}\left[\left(\frac{\partial}{\partial \theta_i}\log p_\theta(y|x)\right)^2\right].
$$
The method computes empirical Fisher diagonals on $D_u$ and $D_r$ at the trained $\theta$, scores each parameter by
$$
s_i = F_u,i - \lambda F_r,i,
$$
and masks the top-$k$ parameters by a binary mask $m$ so that
$$
\theta' = m \odot \theta.
$$
Optional fine-tuning on $D_r$ uses gradient masking,
$$
g' = m \odot g,
$$
so masked parameters remain zero [2310.05331].

The rationale is explicit: parameters with high Fisher on $D_u$ are crucial for modeling $D_u$, parameters with high Fisher on $D_r$ are crucial for retaining $D_r$ performance, and masking parameters with high importance to $D_u$ relative to $D_r$ lets the model escape the $D_u$ basin while protecting retained performance. The paper uses global top-$k$ selection across all layers, excludes the final classifier from masking, and uses masking ratios $R = 0.02$ for CIFAR-10/100 and Tiny-ImageNet and $R = 0.04$ for MNIST [2310.05331].

The empirical results reported for this formulation are unusually strong. On CIFAR-10 with ResNet-20 when removing one class, FisherMask without fine-tuning achieves remain acc $\approx 86.2\%$ with forget acc $\approx 0.0\%$; on CIFAR-100 with GoogLeNet, remain acc $\approx 73.8\%$ with forget acc $\approx 0.0\%$ [2310.05331]. With brief fine-tuning, FisherMask matches or slightly exceeds retraining’s retained performance while keeping forget acc at $0\%$, typically in $1$–$2$ epochs on average, specifically $2.1 \pm 0.9$ epochs across $48$ runs, versus $\approx 92$ epochs needed for full retraining on average over all settings [2310.05331]. The method also shows the smallest fluctuations during fine-tuning, with an example $\Delta$ forget acc $\approx 0.4$ versus $11.0$ for Finetune [2310.05331].

The same paper also reports sample-level unlearning results. For backdoor removal on CIFAR-10/ResNet-20 with $200$ poisoned samples, at mask ratio $0.2$, FisherMask achieved remain acc $\approx 82.3\%$ and forget acc $\approx 2.2\%$ [2310.05331]. For noisy label deletion on CIFAR-10/ResNet-20, FisherMask yields the best test accuracy across noise rates $10/30/50\%$, including $80.36\%$ at $30\%$ noise versus $80.05\%$ for Finetune [2310.05331].

## 3. Escape as residual leakage and false forgetting

A different line of work argues that current textual unlearning can provide a false sense of unlearning because the pre–post model gap becomes a high-quality reference that enables attacks [2406.13348]. In the black-box setting, the attacker has query access to both the original model $M$ and the unlearned model $M^u$ and uses losses or likelihoods to infer whether a text belonged to the unlearned set. The paper’s Black-Box TULA decision rule is
$$
A_M(d)=\mathbb{1}\big[\mathcal{L}(M^u(x),y)-\mathcal{L}(M(x),y)>\gamma\big].
$$
The article’s generative auditing extension uses the sequence log-likelihood
$$
\ell_\theta(x)=\sum_{t=1}^T \log p_\theta(x_t|x_{<t}),
$$
and the log-likelihood ratio
$$
LLR(x)=\ell_\theta(x)-\ell_{\theta^u}(x),
$$
with hypothesis test $H_0:x\notin U$ versus $H_1:x\in U$ [2406.13348].

The reported black-box results are concrete. On sentiment tasks using GPT-2-1.5B, Pythia-1.4B, and OPT-1.3B, TULA-black improves AUC far beyond LOSS and ZLIB baselines. On SST, TULA AUC is $0.723$ for GPT-2-1.5B, $0.642$ for Pythia-1.4B, and $0.659$ for OPT-1.3B; on Yelp, TULA AUC is $0.722$, $0.682$, and $0.693$ respectively [2406.13348]. The baseline LOSS and ZLIB values are near random, approximately $0.42$–$0.51$ across models and datasets [2406.13348].

In the white-box setting, the attack uses the parameter gap $\delta\theta := \theta - \theta^u$, interpreted as approximately proportional to a sum of gradients on the unlearned samples. The optimization objective is
$$
(\hat{x},\hat{y})=\arg\min_{\hat{x}\in\mathbb{R}^{L\times h},\,\hat{y}\in\mathbb{R}^{1\times C}}
\left\|-\nabla_\theta \mathcal{L}(M(\hat{x}),\hat{y})-(\theta-\theta^u)\right\|_2^2.
$$
Empirically, on SST the paper reports up to ROUGE-1 $\approx 66.4\%$ and ROUGE-L $\approx 58.5\%$, and on Yelp up to ROUGE-1 $\approx 57.9\%$ and ROUGE-L $\approx 51.9\%$ [2406.13348]. The paper summarizes this as “more than 60% accuracy” in terms of overlap and describes it as a severe privacy risk [2406.13348].

This line of work shifts the meaning of escape unlearning from “escaping a basin” to “information escaping through the unlearning mechanism itself.” A plausible implication is that unlearning audits must evaluate paired-model leakage, not only post-unlearning task behavior.

## 4. Jailbreak-resistant and domain-exclusive formulations

In safety-oriented work, escape unlearning is closely tied to jailbreak robustness. “Exclusive Unlearning” proposes forgetting everything except for the knowledge and expressions one wishes to retain, using a retention loss on a retention dataset and a forget loss on self-generated text that drives the predictive distribution toward uniformity [2604.06154]. The core losses are
$$
\mathcal{L}_{\text{forget}}(\theta)=\mathbb{E}_{x\sim p_\theta}\left[\log p_\theta(x)\right],
$$
$$
\mathcal{L}_{\text{retain}}(\theta)=\mathbb{E}_{x\sim \mathcal{D}_{\text{retain}}}\left[-\log p_\theta(x)\right],
$$
and
$$
\mathcal{L}(\theta)=\lambda \cdot \mathcal{L}_{\text{forget}}(\theta)+(1-\lambda)\cdot \mathcal{L}_{\text{retain}}(\theta).
$$
The forget loss is also written as
$$
\mathbb{E}_{x\sim p_\theta}\left[\log p_\theta(x)\right]
=
\mathrm{KL}[p_\theta \,\|\, p_u]-T\log V,
$$
which operationalizes entropy maximization outside the retained domain [2604.06154].

At inference, the method applies a uniformity test based on generating $256$ tokens, sampling $32$ token positions, and computing
$$
\Delta(x)=
\left|
\frac{1}{|\mathcal{S}|}\sum_{t\in \mathcal{S}}
\mathrm{KL}\left[p_\theta(\cdot|x_{<t})\,\|\, p_u\right]
-\log V
\right|.
$$
If $\Delta(x)$ is below a threshold set to $5.0$ in experiments, the system returns the fixed refusal “I can’t answer the instruction.” [2604.06154]

The reported Attack Success Rate results are near-zero on jailbreak sets. For medical retention on Llama-3.2-1B, EU yields JB-1/JB-2 ASR $0.0/0.3$ versus DPO $6.7/5.3$, Unlearning $21.0/22.9$, Eraser $17.2/16.0$, and SKU $11.9/10.9$ [2604.06154]. For OLMo-2-7B in the medical domain, EU yields $0.3/0.2$ versus DPO $28.6/25.8$, Unlearning $42.1/41.9$, Eraser $45.1/40.3$, and SKU $7.0/6.5$ [2604.06154]. Similar near-zero ASR values are reported in the mathematics domain, including $0.0/0.0$ on OLMo-2-7B versus DPO $28.5/26.5$ and Unlearning $42.0/41.7$ [2604.06154].

The same paper explicitly notes that the method is not robust to further fine-tuning: subsequent small fine-tuning, exemplified by $400$ Alpaca examples, increases ASR to $\sim 40\%$ [2604.06154]. That caveat links this work directly to later relearning-oriented analyses.

## 5. Relearning, shallow alignment, and faithful erasure

A further strand of literature argues that many unlearning methods do not erase target knowledge but instead hide it. “Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning” states that widely used unlearning methods generate spurious unlearning neurons that amplify negative influence to hide target knowledge, leaving the original knowledge-bearing neurons largely intact [2509.22263]. The attribution score is
$$
A^{(x,y)}_{\theta_i,k}
=
h_{\theta_i,k}
\times
\frac{\partial P_\theta(y|x)}{\partial h_{\theta_i,k}},
$$
and the paper tracks positive and negative influence variation across the forget set [2509.22263].

The proposed SSiUU objective adds an attribution-guided regularizer:
$$
\arg\min_{\theta^t}\,
\mathcal{L}_{\theta^t}
+
\lambda
\sum_{(x,y)\in \mathcal{C}_f}
\sum_{i\in \mathcal{I}^-}
\left\|
A^{(x,y)}_{\theta_i^{t-1}}
-
A^{(x,y)}_{\theta_i^t}
\right\|_2.
$$
The regularizer is intended to suppress the growth of negative influence during unlearning, thereby avoiding spurious suppressive neurons [2509.22263].

The relearning results are pronounced. On FaithUn with Llama-3.2 3B, all methods reach post-unlearning forgetting score $0.0$, but under harmful retraining with $p=0.1/p=0.3$, SSiUU yields $14.81/14.29$, whereas GA yields $68.42/73.33$, GD $48.13/54.76$, DPO $31.58/46.67$, RMU $52.63/75.53$, and KLUE $57.14/62.96$ [2509.22263]. Under benign retraining on Alpaca, SSiUU yields $13.33$, compared with GD $33.33$ and KLUE $28.33$ [2509.22263]. On FaithUn with Qwen-2.5 3B, SSiUU yields $4.76/29.41$ under harmful attack, versus GA $52.63/66.67$ and DPO $47.62/58.82$ [2509.22263].

The paper also reports attribution-distribution stability under retraining. The Spearman correlation between pre- and post-attack attributions is $0.73$ for GA, $0.87$ for NPO, and $0.99$ for SSiUU [2509.22263]. This suggests that faithful erasure is more stable than suppressive hiding.

A mechanistically related but more representation-focused method is CIR. It collapses common representation subspaces in both activations and module-output gradients prior to unlearning updates, using projections $P_{A,\ell}^\perp$ and $P_{G,\ell}^\perp$ so that $a_\ell' = P_{A,\ell}^\perp a_\ell$ and $g_\ell' = P_{G,\ell}^\perp g_\ell$ [2509.11816]. On Llama-3.1-8B across WMDP bio and cyber hazards, CIR reduces post-attack accuracy $80\times$ and $30\times$ more than Circuit Breakers, with only about $0.1\%$ increase in WikiText loss and under $3$ GPU-seconds per fact [2509.11816]. The same work states that preventing even $0.1\%$ disruption in general performance is pivotal, because once disruption occurs, fine-tuning attacks readily reverse unlearning [2509.11816].

## 6. Formal guarantees, evaluation frameworks, and efficient skipping

The online-learning literature gives one of the most formal escape notions. “Online Learning and Unlearning” defines online $(\alpha,\varepsilon)$-OLU using Rényi divergence over output sequences on each interval after a deletion time $\tau_i$, ensuring that outputs on $[\tau_i,\tau_{i+1})$ are indistinguishable from retraining without the deleted points [2505.08557]. Passive OLU injects calibrated Gaussian noise at deletion times, while active OLU adds a short offline shift toward the retain-only solution before noise injection [2505.08557]. Under standard convexity and smoothness assumptions, both methods achieve regret bounds comparable to standard OGD, and for the passive method built on OGD the paper states an $(\alpha,\alpha\varepsilon)$-OLU guarantee [2505.08557].

Evaluation methodology is itself a major issue. “Towards Effective Evaluations and Comparisons for LLM Unlearning Methods” argues that current metrics are susceptible to red teaming and may reflect superficial model behaviors rather than the true extent of retained knowledge [2406.09179]. The paper introduces attack-aware likelihood and decoding metrics, including
$$
UE(\theta_u;D_u)=\mathbb{E}\big[\log p_{\theta_0}(y|x)-\log p_{\theta_u}(y|x)\big],
$$
attack-robust $UE_A$, attack success rate, and the Parameterization Strength family, especially PS-perturb for paraphrase robustness [2406.09179]. On TOFU, after model mixing calibration, residual nonzero PS-perturb on $D_u$ remains, which the paper interprets as residual paraphrase-extractable knowledge [2406.09179]. This supports the broader claim that escape unlearning must be tested under attack suites and at fixed retention levels.

Not all settings require executing every deletion request. FUNU formalizes “unnecessary unlearning” by declaring unlearning of $D_u$ to be $\varepsilon$-unnecessary if
$$
\mathrm{Dist}\big(f(A(D_r)),\,f(A(D_r\cup D_u))\big)\le \varepsilon.
$$
It then identifies a subset $D_u^+$ of removal requests that can be safely skipped using neighbor redundancy in representation space and a one-epoch reference model for automatic thresholding [2501.16614]. In random removal, FUNU reports average $P^- = |D_u^-|/|D_u|$ of $0.4422$, compared with $0.6586$ for Clustering, $0.7865$ for Confidence, and $0.5772$ for Curvature [2501.16614]. Integrated with SISA, FUNU reduces unlearning time by about $24\%$ and the number of influenced slices by $\approx 31\%$ on average [2501.16614]. This is a distinct notion of escape: escaping unnecessary computation while maintaining indistinguishability.

## 7. Open tensions and cross-cutting themes

Several tensions recur across the literature. First, there is a repeated trade-off between forgetting and retention. FisherMask emphasizes that using $F_u - F_r$ rather than $F_u$ alone is critical to balance forgetting and retention [2310.05331]. Evaluation work argues that many apparent unlearning wins are artifacts of catastrophic utility loss and therefore calibrates methods to fixed retention/coherency targets before comparison [2406.09179]. CE-U makes a related stability point, replacing unstable ascent behavior with a bounded cross-entropy-based objective whose logit gradients are $p-p^*$, where for the forgotten label $\partial L/\partial z_y = p_y$ [2503.01224].

Second, there is a persistent distinction between suppression and erasure. Textual leakage attacks show that pre–post gaps can reveal forgotten content [2406.13348]; relearning work shows that suppressive negative-influence neurons can be undone by subsequent fine-tuning [2509.22263]; and CIR argues that unlearning updates must avoid common directions that benign fine-tuning can easily repair [2509.11816]. This suggests that robust escape unlearning depends on whether target knowledge is removed from the relevant internal subspaces or merely masked by shallow alignment.

Third, the threat model matters. In black-box paired-model settings, losses and likelihoods can already leak deleted membership [2406.13348]. In white-box settings, parameter gaps can support reconstruction [2406.13348]. In jailbreak settings, enumerating harmful patterns is insufficient, motivating exclusive retain-only formulations [2604.06154]. In continual learning and unlearning, repeated cycles cause knowledge leakage unless retain, new, and unlearn pathways are isolated, as in BID-LoRA’s geometry-driven escape direction that pushes forget-class embeddings toward a scaled target maximally distant from retain centroids while updating only about $5\%$ of parameters [2604.12686].

Finally, the literature increasingly treats escape unlearning as an evaluation problem as much as an algorithmic one. A plausible implication is that any serious claim of forgetting now requires, at minimum, retained-utility reporting, attack-robust prompting or auditing, and some account of whether deleted information can re-emerge under retraining, model comparison, or prompt variation. Across these usages, escape unlearning has become a concise label for the central question of modern machine unlearning: whether deleted influence is actually removed, merely displaced, or able to return.

Source: https://www.emergentmind.com/topics/escape-unlearning