Escape Unlearning Overview

• Escape unlearning is a machine learning concept that removes or suppresses revoked data influence from model outputs using targeted perturbations.
• Key methodologies include Fisher masking, retention loss, and online unlearning strategies to balance forgetting and performance retention.
• Evaluation metrics assess unlearning effectiveness, privacy leakage, and robustness against adversarial retraining or jailbreak attacks.

Escape unlearning is a family of concepts in machine unlearning that addresses how a model can escape the influence of revoked data, how revoked knowledge can nevertheless escape a nominal unlearning procedure, and how unlearning systems can be designed so that harmful or deleted information does not re-emerge under retraining, jailbreaks, or prompt variation. In the literature, the term is used in several related senses: escaping a local optimum anchored by deleted data through targeted perturbation (Liu et al., 2023); ensuring that future online outputs are statistically indistinguishable from retraining without the deleted point (Hu et al., 13 May 2025); identifying deletions that can be safely skipped because they are “ε-unnecessary” (Li et al., 28 Jan 2025); and preventing forgotten knowledge from resurfacing through side channels, relearning, or adversarial reformulation (Du et al., 2024, Yang et al., 26 Sep 2025, Sasaki et al., 7 Apr 2026). This suggests that “escape unlearning” is best understood not as a single algorithm, but as a cluster of formulations about removal, robustness, leakage, and recovery.

1. Multiple meanings of the term

A central usage appears in “Unlearning with Fisher Masking,” where fine-tuning from a deployed model on the remain set alone often remains trapped in a local optimum anchored by representations of the unlearn set, leading to incomplete forgetting; “escape unlearning” refers to using a targeted perturbation that “escapes” residual influence of revoked data by pushing the model off the basin shaped by $D_u$, after which light fine-tuning on $D_r$ suffices (Liu et al., 2023). In that setup, the training set is partitioned as $D = D_r \cup D_u$, the target unlearned model is the minimizer of $L(\theta; D_r)$, and the practical criteria include low influence of $D_u$ on outputs, high retained accuracy on $D_r$, stability across retrials and during fine-tuning, and optional low re-learnability of $D_u$ if only $D_r$ is provided.

A second usage concerns failure rather than success. In “Textual Unlearning Gives a False Sense of Unlearning,” the escape problem is that unlearned texts can remain identifiable or reconstructable post-unlearning because the act of unlearning itself creates side channels, especially when both the pre-unlearning and post-unlearning models or their outputs are available (Du et al., 2024). In this sense, deleted content escapes the unlearning process rather than the model escaping the deleted-data basin.

A third usage is safety-oriented. “Exclusive Unlearning” interprets escape unlearning as models circumventing unlearning via jailbreak or prompt-injection-style reformulations, and addresses it by forgetting everything outside a whitelisted retained domain rather than enumerating harmful data (Sasaki et al., 7 Apr 2026). “Collapse of Irrelevant Representations” frames escape as dangerous capabilities remaining accessible after an unlearning procedure or re-emerging under alternative prompts, jailbreaks, or post-hoc fine-tuning, and proposes selective removal of common representation subspaces before computing unlearning updates (Sondej et al., 15 Sep 2025).

A fourth usage is formal and online. “Online Learning and Unlearning” defines the desired property so that, after processing each deletion, all subsequent outputs are statistically indistinguishable from those that would have been produced by an algorithm retrained on the sequence with the deletions applied; because this is enforced for each interval, once a point is deleted, it remains protected at every future time step (Hu et al., 13 May 2025).

A fifth usage is procedural. “FUNU” describes escape unlearning as safely skipping some requested deletions because removing those points would not materially change the retrained model, formalized through “unnecessary unlearning” (Li et al., 28 Jan 2025).

2. Basin escape through Fisher masking

The Fisher-masking formulation is one of the clearest operational definitions. The Fisher matrix on a dataset $S$ is

$$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$

and the diagonal approximation used in practice is

$D_r$0

The method computes empirical Fisher diagonals on $D_r$1 and $D_r$2 at the trained $D_r$3, scores each parameter by

$D_r$4

and masks the top-$D_r$5 parameters by a binary mask $D_r$6 so that

$D_r$7

Optional fine-tuning on $D_r$8 uses gradient masking,

$D_r$9

so masked parameters remain zero (Liu et al., 2023).

The rationale is explicit: parameters with high Fisher on $D = D_r \cup D_u$0 are crucial for modeling $D = D_r \cup D_u$1, parameters with high Fisher on $D = D_r \cup D_u$2 are crucial for retaining $D = D_r \cup D_u$3 performance, and masking parameters with high importance to $D = D_r \cup D_u$4 relative to $D = D_r \cup D_u$5 lets the model escape the $D = D_r \cup D_u$6 basin while protecting retained performance. The paper uses global top-$D = D_r \cup D_u$7 selection across all layers, excludes the final classifier from masking, and uses masking ratios $D = D_r \cup D_u$8 for CIFAR-10/100 and Tiny-ImageNet and $D = D_r \cup D_u$9 for MNIST (Liu et al., 2023).

The empirical results reported for this formulation are unusually strong. On CIFAR-10 with ResNet-20 when removing one class, FisherMask without fine-tuning achieves remain acc $L(\theta; D_r)$0 with forget acc $L(\theta; D_r)$1; on CIFAR-100 with GoogLeNet, remain acc $L(\theta; D_r)$2 with forget acc $L(\theta; D_r)$3 (Liu et al., 2023). With brief fine-tuning, FisherMask matches or slightly exceeds retraining’s retained performance while keeping forget acc at $L(\theta; D_r)$4, typically in $L(\theta; D_r)$5–$L(\theta; D_r)$6 epochs on average, specifically $L(\theta; D_r)$7 epochs across $L(\theta; D_r)$8 runs, versus $L(\theta; D_r)$9 epochs needed for full retraining on average over all settings (Liu et al., 2023). The method also shows the smallest fluctuations during fine-tuning, with an example $D_u$0 forget acc $D_u$1 versus $D_u$2 for Finetune (Liu et al., 2023).

The same paper also reports sample-level unlearning results. For backdoor removal on CIFAR-10/ResNet-20 with $D_u$3 poisoned samples, at mask ratio $D_u$4, FisherMask achieved remain acc $D_u$5 and forget acc $D_u$6 (Liu et al., 2023). For noisy label deletion on CIFAR-10/ResNet-20, FisherMask yields the best test accuracy across noise rates $D_u$7, including $D_u$8 at $D_u$9 noise versus $D_r$0 for Finetune (Liu et al., 2023).

3. Escape as residual leakage and false forgetting

A different line of work argues that current textual unlearning can provide a false sense of unlearning because the pre–post model gap becomes a high-quality reference that enables attacks (Du et al., 2024). In the black-box setting, the attacker has query access to both the original model $D_r$1 and the unlearned model $D_r$2 and uses losses or likelihoods to infer whether a text belonged to the unlearned set. The paper’s Black-Box TULA decision rule is

$D_r$3

The article’s generative auditing extension uses the sequence log-likelihood

$D_r$4

and the log-likelihood ratio

$D_r$5

with hypothesis test $D_r$6 versus $D_r$7 (Du et al., 2024).

The reported black-box results are concrete. On sentiment tasks using GPT-2-1.5B, Pythia-1.4B, and OPT-1.3B, TULA-black improves AUC far beyond LOSS and ZLIB baselines. On SST, TULA AUC is $D_r$8 for GPT-2-1.5B, $D_r$9 for Pythia-1.4B, and $D_u$0 for OPT-1.3B; on Yelp, TULA AUC is $D_u$1, $D_u$2, and $D_u$3 respectively (Du et al., 2024). The baseline LOSS and ZLIB values are near random, approximately $D_u$4–$D_u$5 across models and datasets (Du et al., 2024).

In the white-box setting, the attack uses the parameter gap $D_u$6, interpreted as approximately proportional to a sum of gradients on the unlearned samples. The optimization objective is

$D_u$7

Empirically, on SST the paper reports up to ROUGE-1 $D_u$8 and ROUGE-L $D_u$9, and on Yelp up to ROUGE-1 $D_r$0 and ROUGE-L $D_r$1 (Du et al., 2024). The paper summarizes this as “more than 60% accuracy” in terms of overlap and describes it as a severe privacy risk (Du et al., 2024).

This line of work shifts the meaning of escape unlearning from “escaping a basin” to “information escaping through the unlearning mechanism itself.” A plausible implication is that unlearning audits must evaluate paired-model leakage, not only post-unlearning task behavior.

4. Jailbreak-resistant and domain-exclusive formulations

In safety-oriented work, escape unlearning is closely tied to jailbreak robustness. “Exclusive Unlearning” proposes forgetting everything except for the knowledge and expressions one wishes to retain, using a retention loss on a retention dataset and a forget loss on self-generated text that drives the predictive distribution toward uniformity (Sasaki et al., 7 Apr 2026). The core losses are

$D_r$2

$D_r$3

and

$D_r$4

The forget loss is also written as

$D_r$5

which operationalizes entropy maximization outside the retained domain (Sasaki et al., 7 Apr 2026).

At inference, the method applies a uniformity test based on generating $D_r$6 tokens, sampling $D_r$7 token positions, and computing

$D_r$8

If $D_r$9 is below a threshold set to $S$0 in experiments, the system returns the fixed refusal “I can’t answer the instruction.” (Sasaki et al., 7 Apr 2026)

The reported Attack Success Rate results are near-zero on jailbreak sets. For medical retention on Llama-3.2-1B, EU yields JB-1/JB-2 ASR $S$1 versus DPO $S$2, Unlearning $S$3, Eraser $S$4, and SKU $S$5 (Sasaki et al., 7 Apr 2026). For OLMo-2-7B in the medical domain, EU yields $S$6 versus DPO $S$7, Unlearning $S$8, Eraser $S$9, and SKU $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$0 (Sasaki et al., 7 Apr 2026). Similar near-zero ASR values are reported in the mathematics domain, including $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$1 on OLMo-2-7B versus DPO $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$2 and Unlearning $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$3 (Sasaki et al., 7 Apr 2026).

The same paper explicitly notes that the method is not robust to further fine-tuning: subsequent small fine-tuning, exemplified by $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$4 Alpaca examples, increases ASR to $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$5 (Sasaki et al., 7 Apr 2026). That caveat links this work directly to later relearning-oriented analyses.

5. Relearning, shallow alignment, and faithful erasure

A further strand of literature argues that many unlearning methods do not erase target knowledge but instead hide it. “Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning” states that widely used unlearning methods generate spurious unlearning neurons that amplify negative influence to hide target knowledge, leaving the original knowledge-bearing neurons largely intact (Yang et al., 26 Sep 2025). The attribution score is

$$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$6

and the paper tracks positive and negative influence variation across the forget set (Yang et al., 26 Sep 2025).

The proposed SSiUU objective adds an attribution-guided regularizer:

$$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$7

The regularizer is intended to suppress the growth of negative influence during unlearning, thereby avoiding spurious suppressive neurons (Yang et al., 26 Sep 2025).

The relearning results are pronounced. On FaithUn with Llama-3.2 3B, all methods reach post-unlearning forgetting score $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$8, but under harmful retraining with $$F(\theta)=\mathbb{E}_{(x,y)\sim S}\big[\nabla_\theta \log p_\theta(y|x)\nabla_\theta \log p_\theta(y|x)^\top\big],$$9, SSiUU yields $D_r$00, whereas GA yields $D_r$01, GD $D_r$02, DPO $D_r$03, RMU $D_r$04, and KLUE $D_r$05 (Yang et al., 26 Sep 2025). Under benign retraining on Alpaca, SSiUU yields $D_r$06, compared with GD $D_r$07 and KLUE $D_r$08 (Yang et al., 26 Sep 2025). On FaithUn with Qwen-2.5 3B, SSiUU yields $D_r$09 under harmful attack, versus GA $D_r$10 and DPO $D_r$11 (Yang et al., 26 Sep 2025).

The paper also reports attribution-distribution stability under retraining. The Spearman correlation between pre- and post-attack attributions is $D_r$12 for GA, $D_r$13 for NPO, and $D_r$14 for SSiUU (Yang et al., 26 Sep 2025). This suggests that faithful erasure is more stable than suppressive hiding.

A mechanistically related but more representation-focused method is CIR. It collapses common representation subspaces in both activations and module-output gradients prior to unlearning updates, using projections $D_r$15 and $D_r$16 so that $D_r$17 and $D_r$18 (Sondej et al., 15 Sep 2025). On Llama-3.1-8B across WMDP bio and cyber hazards, CIR reduces post-attack accuracy $D_r$19 and $D_r$20 more than Circuit Breakers, with only about $D_r$21 increase in WikiText loss and under $D_r$22 GPU-seconds per fact (Sondej et al., 15 Sep 2025). The same work states that preventing even $D_r$23 disruption in general performance is pivotal, because once disruption occurs, fine-tuning attacks readily reverse unlearning (Sondej et al., 15 Sep 2025).

6. Formal guarantees, evaluation frameworks, and efficient skipping

The online-learning literature gives one of the most formal escape notions. “Online Learning and Unlearning” defines online $D_r$24-OLU using Rényi divergence over output sequences on each interval after a deletion time $D_r$25, ensuring that outputs on $D_r$26 are indistinguishable from retraining without the deleted points (Hu et al., 13 May 2025). Passive OLU injects calibrated Gaussian noise at deletion times, while active OLU adds a short offline shift toward the retain-only solution before noise injection (Hu et al., 13 May 2025). Under standard convexity and smoothness assumptions, both methods achieve regret bounds comparable to standard OGD, and for the passive method built on OGD the paper states an $D_r$27-OLU guarantee (Hu et al., 13 May 2025).

Evaluation methodology is itself a major issue. “Towards Effective Evaluations and Comparisons for LLM Unlearning Methods” argues that current metrics are susceptible to red teaming and may reflect superficial model behaviors rather than the true extent of retained knowledge (Wang et al., 2024). The paper introduces attack-aware likelihood and decoding metrics, including

$D_r$28

attack-robust $D_r$29, attack success rate, and the Parameterization Strength family, especially PS-perturb for paraphrase robustness (Wang et al., 2024). On TOFU, after model mixing calibration, residual nonzero PS-perturb on $D_r$30 remains, which the paper interprets as residual paraphrase-extractable knowledge (Wang et al., 2024). This supports the broader claim that escape unlearning must be tested under attack suites and at fixed retention levels.

Not all settings require executing every deletion request. FUNU formalizes “unnecessary unlearning” by declaring unlearning of $D_r$31 to be $D_r$32-unnecessary if

$D_r$33

It then identifies a subset $D_r$34 of removal requests that can be safely skipped using neighbor redundancy in representation space and a one-epoch reference model for automatic thresholding (Li et al., 28 Jan 2025). In random removal, FUNU reports average $D_r$35 of $D_r$36, compared with $D_r$37 for Clustering, $D_r$38 for Confidence, and $D_r$39 for Curvature (Li et al., 28 Jan 2025). Integrated with SISA, FUNU reduces unlearning time by about $D_r$40 and the number of influenced slices by $D_r$41 on average (Li et al., 28 Jan 2025). This is a distinct notion of escape: escaping unnecessary computation while maintaining indistinguishability.

7. Open tensions and cross-cutting themes

Several tensions recur across the literature. First, there is a repeated trade-off between forgetting and retention. FisherMask emphasizes that using $D_r$42 rather than $D_r$43 alone is critical to balance forgetting and retention (Liu et al., 2023). Evaluation work argues that many apparent unlearning wins are artifacts of catastrophic utility loss and therefore calibrates methods to fixed retention/coherency targets before comparison (Wang et al., 2024). CE-U makes a related stability point, replacing unstable ascent behavior with a bounded cross-entropy-based objective whose logit gradients are $D_r$44, where for the forgotten label $D_r$45 (Yang, 3 Mar 2025).

Second, there is a persistent distinction between suppression and erasure. Textual leakage attacks show that pre–post gaps can reveal forgotten content (Du et al., 2024); relearning work shows that suppressive negative-influence neurons can be undone by subsequent fine-tuning (Yang et al., 26 Sep 2025); and CIR argues that unlearning updates must avoid common directions that benign fine-tuning can easily repair (Sondej et al., 15 Sep 2025). This suggests that robust escape unlearning depends on whether target knowledge is removed from the relevant internal subspaces or merely masked by shallow alignment.

Third, the threat model matters. In black-box paired-model settings, losses and likelihoods can already leak deleted membership (Du et al., 2024). In white-box settings, parameter gaps can support reconstruction (Du et al., 2024). In jailbreak settings, enumerating harmful patterns is insufficient, motivating exclusive retain-only formulations (Sasaki et al., 7 Apr 2026). In continual learning and unlearning, repeated cycles cause knowledge leakage unless retain, new, and unlearn pathways are isolated, as in BID-LoRA’s geometry-driven escape direction that pushes forget-class embeddings toward a scaled target maximally distant from retain centroids while updating only about $D_r$46 of parameters (Rachapudi et al., 14 Apr 2026).

Finally, the literature increasingly treats escape unlearning as an evaluation problem as much as an algorithmic one. A plausible implication is that any serious claim of forgetting now requires, at minimum, retained-utility reporting, attack-robust prompting or auditing, and some account of whether deleted information can re-emerge under retraining, model comparison, or prompt variation. Across these usages, escape unlearning has become a concise label for the central question of modern machine unlearning: whether deleted influence is actually removed, merely displaced, or able to return.