Untraining in Machine Learning

• Untraining is the process of removing the training effect on a designated forget set to achieve a model equivalent to retraining on the remaining data.
• It employs both exact methods (e.g., SISA, DaRE Forests) and approximate techniques (e.g., Fisher Masking, gradient modifications) to balance deletion efficiency and model utility.
• Practical applications include GDPR data deletion, mitigation of poisoned data, and privacy preservation while maintaining overall model performance.

Untraining is the post hoc removal of the effect of having trained on a specified forget set $S \subset \mathcal{D}$. Its canonical target is the model that would have been obtained by training on $\mathcal{D}\setminus S$, so the objective is retraining equivalence with respect to specific deleted examples rather than erasure of the full concept, capability, or distribution that those examples may represent. A recent terminological clarification therefore distinguishes untraining from unlearning: training concerns fitting a finite dataset, learning concerns generalization to an underlying distribution, and the corresponding deletion problems differ in kind (Triantafillou et al., 9 Apr 2026).

1. Definition and scope

In the precise sense proposed for the field, untraining aims to reverse empirical risk minimization on a forget set $S\subset D$. The intended output is the model obtained by training on $D\setminus S$, preserving whatever can still be inferred from the retain set. This contrasts with unlearning, where the goal is to remove the broader behavior $\mathcal{B}$ or the entire sub-distribution $S^{full}$ represented by the forget examples; under that stronger objective, the model should behave as though it had never trained on any instance of that behavior (Triantafillou et al., 9 Apr 2026).

This distinction reorganizes much of the earlier “machine unlearning” literature. User-data deletion under GDPR-style “right to be forgotten” requests, removal of noisy, outdated, or mislabeled points, deletion of exact copyrighted sentences, and protection against extraction of particular training points are all naturally untraining problems. By contrast, dangerous knowledge, backdoors, poisoned-data corruption under partial discovery, artistic styles, and harmful response patterns are broader concept-removal problems. The distinction matters because a successful untraining method may legitimately leave performance on deleted samples unchanged when those samples are non-memorized and still predictable from the retained data; that is consistent with retraining equivalence, but not with concept erasure (Triantafillou et al., 9 Apr 2026).

The practical motivations for untraining are longstanding. Retraining from scratch is the cleanest baseline, but it is often computationally expensive, may require access to the original dataset, and is difficult to repeat for frequent deletion requests. Survey treatments therefore frame machine unlearning as a principled removal procedure whose gold standard is naïve retraining on $D\setminus D_u$, with exact removal defined by distributional equivalence between the updated model and the retrained model on every input (Mercuri et al., 2022). In domains such as contrastive learning, these motivations are coupled to privacy and compliance: models can implicitly memorize training data even without labels, enabling membership inference attacks, and regulatory regimes such as GDPR, APPI, and CPPA motivate deletion after training (Wang et al., 2024).

2. Formal problem statements

A standard notation writes the full training dataset as $\mathcal{D}$, the forget set as $\mathcal{S}\subset \mathcal{D}$, the retain set as $\mathcal{D}\setminus \mathcal{S}$, the randomized learning algorithm as $\mathcal{D}\setminus S$0, and the deletion procedure as $\mathcal{D}\setminus S$1. The post-processed model is $\mathcal{D}\setminus S$2. In the classical retraining-equivalence formulation, untraining requires this output to be close in distribution to training from scratch on $\mathcal{D}\setminus S$3; in the ideal exact form, the benchmark is simply $\mathcal{D}\setminus S$4 (Triantafillou et al., 9 Apr 2026, Mercuri et al., 2022).

The same literature also formalizes why deleting a point may or may not matter. For $\mathcal{D}\setminus S$5, the counterfactual memorization score is

$\mathcal{D}\setminus S$6

and the cross-influence of $\mathcal{D}\setminus S$7 on another point $\mathcal{D}\setminus S$8 is

$\mathcal{D}\setminus S$9

If self- and cross-influence are low, untraining may be close to a no-op; if self-influence is high but cross-influence is low, the main effect is on the forget examples; if cross-influence is also high, collateral degradation becomes structurally likely (Triantafillou et al., 9 Apr 2026).

Sequential variants instantiate the same retraining-equivalence logic in other settings. In reinforcement learning, exact deletion is formulated so that for any sequence of users $S\subset D$0, deletion position $S\subset D$1, and dummy user $S\subset D$2,

$S\subset D$3

The equality in distribution makes deletion indistinguishable from having never observed the removed user at all, and the framework links this exactness to $S\subset D$4-TV stability of the learning algorithm (Nguyen-Tang et al., 2 Jun 2026).

3. Evaluation, baselines, and failure modes

Because untraining is defined relative to retraining, evaluation inherits a fixed reference point. A consolidated survey groups evaluation into four dimensions—Efficiency, Effectiveness, Consistency, and Certifiability—and treats exact retraining on $S\subset D$5 as the baseline against which speed, predictive performance, parameter similarity, output agreement, and deletion assurance should be assessed. Exact methods typically dominate on consistency and certifiability; approximate methods trade those guarantees for lower cost (Mercuri et al., 2022).

For approximate procedures, one influential proposal identifies the verification error

$S\subset D$6

between an approximately untrained model $S\subset D$7 and a naïvely retrained model $S\subset D$8 as the central metric, because it subsumes a broad class of other weight-space and output-space notions. The same analysis of SGD derives a cheap proxy, the unlearning error,

$S\subset D$9

highlighting training length, curvature, and total weight displacement as the main determinants of later untraining difficulty (Thudi et al., 2021).

Metric choice is highly domain dependent. In contrastive learning, forgetting is measured not only by membership inference attack metrics—Accuracy, Precision, Recall, and AUC—but also by top-1 prediction confidence, training loss, cosine similarity, weighted kNN evaluation, linear evaluation, and t-SNE visualizations. In language-model untraining on TOFU, evaluation uses ROUGE-L recall on paraphrased questions, length-normalized conditional probability of the correct answer, and truth ratio, which are collapsed into Model Utility and Forget Quality (Wang et al., 2024, Yang, 3 Mar 2025).

A recurrent methodological warning is that attack-based privacy metrics are insufficient on their own. Membership inference can be appropriate for deleting specific user data, but it does not establish that an underlying behavior or capability has been removed. A method can suppress the forget set enough to defeat membership inference while still preserving semantically related behavior, which is acceptable for untraining but not for broader unlearning (Triantafillou et al., 9 Apr 2026).

4. Exact and structured untraining systems

Exact untraining methods obtain their guarantees by constraining how training is organized. Two canonical examples are SISA—Sharded, Isolated, Sliced, and Aggregated—and DaRE Forests. SISA trains shard-specific models and stores intermediate states so that a deletion request retrains only the affected shard suffix; DaRE modifies tree construction and caches node statistics so that only affected subtrees are updated. Both are exact because their outputs match retraining under their respective training protocols (Mercuri et al., 2022).

A related line replaces simple sharding with coding. Coded Machine Unlearning encodes training data into linear combinations before sending them to non-communicating weak learners, then unlearns a sample by subtracting its coded contribution from affected shards and retraining only those weak learners. For regression, the protocol satisfies the paper’s perfect-unlearning criterion—its output is a statistical draw from the distribution of models trained on the dataset with the sample removed—and empirically improves the performance-versus-unlearning-cost trade-off relative to uncoded sharding, especially on datasets with influential or heavy-tailed samples (Aldaghri et al., 2020).

For large pretrained models, exactness has been recast in parameter-efficient form. Sequence-aware Sharded Sliced Training (S3T) uses PEFT, specifically LoRA in the exposition, to isolate data influence across layers by sequentially training layers on disjoint data slices. Deletion is then implemented by deactivating affected PEFT layers rather than retraining the whole component. The framework also trains multiple slice orders to improve deletion capacity, with a theoretical deletion-rate scaling

$D\setminus S$0

compared with

$D\setminus S$1

and empirical gains over SISA across vision, GLUE/SuperGLUE, and instruction-tuned LLM settings (Chowdhury et al., 2024).

Exactness has also reached sequential decision making. In tabular episodic RL, exact deletion is achieved by combining binary-tree noisy prefix sums with maximal coupling, so that most previously generated randomness is reused and retraining is localized. The resulting framework is exact with respect to the learning algorithm, $D\setminus S$2-TV-stable, and has expected unlearning cost only a $D\setminus S$3 fraction of retraining from scratch, while achieving a regret bound of

$D\setminus S$4

for tabular MDPs (Nguyen-Tang et al., 2 Jun 2026).

5. Approximate and optimization-based procedures

Most practical untraining methods are approximate: they start from a trained model and seek a cheaper route toward the retrained solution. Classical survey exemplars include Fisher, Influence, DeltaGrad, Descent-to-Delete, and DeepObliviate, which variously rely on Newton or influence-function corrections, replay of cached optimization history, perturbed gradient descent, or early stopping of retraining-like trajectories (Mercuri et al., 2022).

Several later methods refine these ideas by altering what is updated and how. Unlearning with Fisher Masking first masks parameters that are more important to the forget set than to the remain set, selecting the top $D\setminus S$5 parameters according to $D\setminus S$6, and then fine-tunes on the remaining data. Across CIFAR10, CIFAR100, MNIST, and Tiny-ImageNet, FisherMask often approaches retraining quality after a short fine-tuning phase, and even with no remain data it still works reasonably, though with reduced remain accuracy (Liu et al., 2023).

Other methods modify the gradient objective itself. OrthoGrad projects the unlearn gradient onto the subspace orthogonal to all per-sample retain gradients in the current batch, treating gradient interference rather than a scalar ascent–descent trade-off as the primary difficulty. The method is evaluated in a low-data retain regime on CIFAR-10, ImageNet, and automatic speech recognition with Whisper-Tiny on LibriSpeech, and it outperforms competing methods, including on speaker unlearning in ASR (Shamsian et al., 4 Mar 2025). A different objective-based formulation, EMSE or Ethical MSE, explicitly minimizes squared error on wanted data while penalizing accurate prediction on unwanted data through a log-complement Gaussian term, framing untraining as selective re-optimization rather than retraining (Achour, 2024).

Language-model untraining has pushed especially hard on optimization stability. CE-U (Cross Entropy Unlearning) replaces direct gradient ascent on forgotten targets with a cross-entropy loss against a modified target distribution in which the true label logit is set to $D\setminus S$7, thereby avoiding the vanishing-gradient at high confidence and exploding-gradient at low confidence pathology of gradient-ascent unlearning. On TOFU with LLaMA2-7B, CE-U attains state-of-the-art results without an oracle model or additional positive samples, and its General CE-U form interpolates continuously between ordinary learning and pure unlearning (Yang, 3 Mar 2025). MArgin Self-Correction (MASC) instead focuses on token-level dominance: it reduces the logit gap between the original next token and the top-$D\setminus S$8 alternatives, then stops online when the fraction of remaining violations on a forget probe subset drops below a threshold. On TOFU, MUSE News, and MUSE Books, MASC achieves a competitive forget–retain trade-off at a fraction of the computational cost of fixed-budget baselines and avoids offline checkpoint selection (Gennaro et al., 1 Jun 2026).

Approximate untraining has also expanded beyond supervised classification. In contrastive learning, a gradient-constraint approach combines a member training loss, a WGAN-GP-inspired gradient penalty, and an output-norm term,

$D\setminus S$9

or in optimized form

$\mathcal{B}$0

and performs only about 10 unlearning epochs. On CIFAR-10, CIFAR-100, and SVHN, membership inference accuracy drops to about $\mathcal{B}$1, while downstream utility typically decreases only moderately, and the same method transfers to supervised ResNet18 models (Wang et al., 2024). At the optimizer level, stochastic variants of D2D and R2D provide $\mathcal{B}$2-certified unlearning under strongly convex, convex, and nonconvex assumptions; D2D yields tighter guarantees in the strongly convex case, whereas R2D remains applicable in convex and nonconvex settings because rewinding reverses accumulated disturbance (Mu et al., 20 Nov 2025).

6. Boundary cases, scaling, and research directions

Not every deletion problem now labeled “unlearning” is an instance of untraining. In LLMs, skill unlearning seeks to remove a designated capability—such as math-solving, Python coding, or language comprehension in a specific language—while retaining unrelated skills and MMLU. Training-free methods such as Neuron Adjust and Key Space Detection operate at inference time on feed-forward-layer activations rather than on traced training examples; for most tasks, Key Space Detection produces over $\mathcal{B}$3 relative performance drop on the forgetting skill with less than $\mathcal{B}$4 relative performance drop on other skills and MMLU (Li et al., 27 Mar 2025). A related position paper argues that foundation-model deletion requests should be lifted from data tracing to knowledge tracing, allowing forgetting sets to refer directly to concepts or capabilities rather than identified training points (Tan et al., 12 Jun 2025).

Inference-time control further separates output suppression from weight-level retraining equivalence. A conformal-prediction framework for generative models equips the model with a verifier $\mathcal{B}$5, iteratively refines responses until $\mathcal{B}$6, and calibrates the iteration budget $\mathcal{B}$7 on a held-out calibration set to obtain a distribution-free marginal guarantee $\mathcal{B}$8. The approach does not alter model parameters and therefore is explicitly not true weight-level erasure, but it reduces unlearning error by up to $\mathcal{B}$9 across challenging generative benchmarks (Chowdhury et al., 3 Feb 2026).

Two further lines shift attention from the deletion algorithm to the training state on which deletion acts. Grokked Models are Better Unlearners shows that standard unlearning methods applied after the grokking transition yield more efficient forgetting, less collateral damage, and more stable updates across seeds than the same methods applied to pre-grokking checkpoints; the reported mechanism is reduced gradient alignment between forget and retain subsets and more modular representations (Liang et al., 3 Dec 2025). Stream-native machine unlearning translates batch deletion to an online setting through regret, sample complexity, and deletion capacity, replacing expensive Hessian inversion with online L-BFGS and obtaining $S^{full}$0 regret under strong convexity in a live stream of insertions and deletions (Stewart, 13 Aug 2025).

The most persistent open problem remains conceptual. If the forget set is complete, retraining equivalence is a coherent target; if only a partial or semantic description of the unwanted behavior is available, retraining equivalence is no longer enough. Current work therefore points in two directions at once: sharper terminology and benchmarks for sample-level untraining, and separate methods for concept-level unlearning that explicitly generalize deletion beyond the observed forget examples (Triantafillou et al., 9 Apr 2026).