---
title: 'Source-Free Unlearning: Methods & Implications'
url: https://www.emergentmind.com/topics/source-free-unlearning
type: topic
---

# Source-Free Unlearning: Methods & Implications

Source-free unlearning denotes machine-unlearning settings in which a trained model must remove the influence of specified data, classes, concepts, domains, or corruptions without relying on the original training corpus at unlearning time. Across the recent literature, the term covers several closely related regimes: **zero-shot unlearning**, **data-free unlearning**, **remaining-data-free unlearning**, **retain-free unlearning**, **corpus-free unlearning**, and **partially-blind unlearning**. What unites these regimes is that unlearning is no longer framed as retraining on the retain set, but as an intervention using a trained model together with restricted auxiliary information such as the forget set, class names, proxy corrupted samples, cached activations, classifier weights, surrogate datasets, gradients, or synthetic data generated from the model itself [2311.15268].

## 1. Problem formulations and access regimes

The classical target of machine unlearning is a model that behaves like one retrained on the retain set only. Several papers restate this using the standard decomposition \( \mathcal{D} = \mathcal{D}_r \cup \mathcal{D}_f \), where \(\mathcal{D}_f\) is the forget set and \(\mathcal{D}_r = \mathcal{D} \setminus \mathcal{D}_f\). In class unlearning, one class \(c\) is designated as the forget class, yielding
\[
\mathcal{D}_{train}^{forget} = \{(x,y)\in \mathcal{D}_{train} \mid y=c\},
\qquad
\mathcal{D}_{train}^{retain} = \mathcal{D}_{train} \setminus \mathcal{D}_{train}^{forget}.
\]
The operational objective is to make the model behave as if it had not been trained on the forget class while preserving accuracy on \(\mathcal{D}_{test}^{retain}\) [2311.15268].

The source-free literature differs primarily in what information remains available when the deletion request arrives. In **remaining-data-free** and **retain-free** settings, the retain set is unavailable at unlearning time, so the method must work from the pretrained model and either the forget data or auxiliary statistics [2402.15109]; [2604.13438]. In **data-free unlearning**, neither the original training data nor a similar proxy dataset is available, and only the trained original model \(T(\cdot)\) is assumed [2412.13790]. In **corpus-free** LLM unlearning, the user need not provide a forget corpus; instead, only a lightweight anchor identifying a target entity is supplied, and the service provider has no access to the original training corpus at deployment time [2604.13777].

Some papers define stricter variants. **Source-free corrective machine unlearning** assumes that the original training data are unavailable and that no forget set of identified corrupted training samples can be specified; only a small proxy corrupted set is available [2511.18660]. **Partially-blind unlearning** assumes access to the trained model, the retain set, and auxiliary training information, but not the forget set itself [2604.10636]. **Source-free domain adaptation unlearning** studies the case where the source data are unavailable during adaptation, yet the source model leaks source-exclusive classes into the target domain [2604.08238].

This multiplicity of settings indicates that “source-free” is not a single protocol. In some works it means **no original training set**; in others it means **no retain set**; in others it means **no forget corpus supplied by the requester**; and in still others it means **no retraining or fine-tuning**. A plausible implication is that comparisons across papers are only meaningful once the access assumptions are aligned.

## 2. Representation-space and geometric unlearning

A major line of work treats source-free unlearning as a problem of modifying internal representations rather than full parameter vectors. In **“Unlearning via Sparse Representations”** [2311.15268], the central mechanism is a **Discrete Key-Value Bottleneck (DKVB)** with \(C\) codebooks and \(M\) key-value pairs per codebook. Inputs are mapped to \(C\) heads, each head is quantized to the **top-\(k\)** nearest keys, the corresponding values are retrieved and averaged, and unlearning is performed by masking selected key-value pairs so they cannot be chosen again. The paper emphasizes that **keys are frozen**, **values are learnable**, gradients do **not** flow from values back into keys, and the resulting representations are **sparse, localized, and input-dependent**. Because class information is concentrated in a relatively small subset of key-value activations, masking those codes can erase the forget class without retraining or fine-tuning. The proposed procedures, **Unlearning via Examples (UvE)** and **Unlearning via Activations (UvA)**, both require **no retraining**, **no fine-tuning**, and **negligible additional compute**, with UvA permitting a setting where activations are cached during training so **no original training data is needed at unlearning time** [2311.15268].

A related but more explicitly information-theoretic approach appears in **Representation Unlearning** [2601.21564]. This framework learns a small transformation \(f_\phi\) on the model’s penultimate representation \(z\), producing \(z' = f_\phi(z)\), and imposes an information bottleneck that maximizes retention-related information while suppressing information about the forget data. In its zero-shot variant, the method assumes access to the **forget set** \(D_f\), the **trained classifier weights** \(W\), and **class counts** \(N^c\), but **no access to retain data** \(D_r\). The zero-shot retention surrogate is
\[
C_{zs} \approx \frac{1}{2N}\sum_{c=1}^{C} N^c \|w_c - f_\phi(w_c)\|^2,
\]
and the zero-shot forget loss is
\[
L_f^{zs} \approx \frac{1}{2B_f N}\sum_{i=1}^{B_f}\sum_{c=1}^{C} N^c \|w_c - f_\phi(z^{(i)})\|^2.
\]
The method relies on **Neural Collapse** to treat class weight vectors as class prototypes. This makes unlearning possible without retain examples by using class prototypes as pseudo-retain anchors [2601.21564].

A third representation-level route is **Label-Agnostic Forgetting (LAF)** [2404.00506]. LAF decomposes the model as \(g_D = g_D^c \circ g_D^e\), models representation distributions with two VAEs, and performs unlearning at the extractor level rather than the classifier level. Its practical extractor-unlearning loss is
\[
L_{UE} = \sum_{x \in X_r}\frac{\Vert g_{U}^{e}(x) - h(g_{U}^{e}(x))\Vert_2^2 }{\Vert g_{U}^{e}(x) - h(g_{U}^{e}(x))\Vert_2^2 + 1} - \sum_{x \in X_f}\frac{\Vert g_{U}^{e}(x) - h_f(g_{U}^{e}(x))\Vert_2^2 }{\Vert g_{U}^{e}(x) - h_f(g_{U}^{e}(x))\Vert_2^2  + 1},
\]
and it preserves predictive behavior via a contrastive representation-alignment loss \(L_{RA}\). The paper’s key claim is that the unlearning procedure itself does **not use supervision information**, making it a **supervision-free / source-free** framework at unlearning time [2404.00506].

These representation-space methods share a common view: forgetting is easier when the model stores the target information in a localizable geometry—discrete codes, class prototypes, or separable latent distributions—rather than diffusely across all parameters. This suggests that source-free unlearning is often as much an architectural question as a post hoc algorithmic one.

## 3. Data-free synthesis, corpus-free supervision, and proxy-based forgetting

A second family replaces unavailable source data with synthetic, generated, or proxy supervision. In **“Toward Efficient Data-Free Unlearning”** [2412.13790], the difficulty is that a trained model must forget specified classes or data when **neither the original training data nor a similar proxy dataset is available**. The proposed **Inhibited Synthetic PostFilter (ISPF)** addresses the over-filtering problem of earlier data-free knowledge-distillation methods such as GKT. The method has two components: **Inhibited Synthesis (IS)**, which discourages the generator from synthesizing forgetting-class samples, and **PostFilter (PF)**, which modifies teacher logits after synthesis so that forgetting information is suppressed but retaining information from mixed synthetic samples is preserved. PF redistributes teacher logits using
\[
\Delta=\sum_{k\in \mathcal{Y}_f}\left[\boldsymbol{t}_k - min(\boldsymbol{t})\right],
\]
followed by a logit transformation and KL distillation. The paper explicitly notes that modifying logits before softmax performs better than setting forgetting-class probability to zero and renormalizing the rest [2412.13790].

In LLM unlearning, the forget set itself has become a bottleneck. **“LLM Unlearning Without an Expert Curated Dataset”** [2508.06595] proposes a fully automated **synthetic textbook** pipeline that requires only a domain name such as “biosecurity,” “cybersecurity,” or “Harry Potter.” The pipeline uses GPT-4o-mini with temperature 0.7, generates 10 subdomains, then 20 bullet points per subdomain across four audience levels, then 5 textbook-style chapters per bullet point, yielding 4000 chapters total, from which the 20,000 longest sentences are selected as the forget set. The resulting synthetic datasets are then used with existing unlearning objectives such as RMU, RR, and ELM [2508.06595]. This is source-free in the practical sense that it removes the need for curated source documents or expert-built forget corpora.

A more model-internal variant is **MAGE**, a **Memory-grAph Guided Erasure** framework for user-minimized, corpus-free unlearning in LLMs [2604.13777]. Given only a lightweight anchor identifying a target entity \(E_t\), MAGE repeatedly prompts the target LLM, extracts entities from responses, scores them by recall frequency,
\[
s(v)=\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}\big[v \in \text{Entities}(y_i)\big],
\]
builds a weighted local memory graph \(G_t=(V_t,\mathcal{E}_t)\), and then samples paths through the graph to synthesize QA-style forget examples. It also constructs a **neighbor set** \(S_n\) from strong neighbors excluding \(E_t\), thereby preserving nearby but non-target knowledge. MAGE is explicitly **model-agnostic**: it produces ordinary unlearning data that can be plugged into GA, NPO, GA+GD, or GA+KL without defining a new optimizer [2604.13777].

Proxy-based supervision is also central to corrective settings. **CUTS**—**Corrective Unlearning in Task Space**—addresses the case where a corrupted model is available but the original training data are gone [2511.18660]. It assumes a small proxy dataset \(\mathcal{D}_{\text{proxy}}\sim p_{\text{corr}}\), briefly fine-tunes the corrupted model on the proxy to amplify corruption, defines a proxy task vector
\[
\tau_p = \theta_{\text{proxy} } - \theta_{\text{mix}},
\]
and subtracts a scaled multiple of this vector:
\[
\theta_u^\alpha = \theta_{\text{mix} } - \alpha\,\tau_p.
\]
The method requires neither a retain set nor access to the original corrupted training samples [2511.18660].

Across these approaches, source-free unlearning becomes a supervision-construction problem: one either synthesizes training signals from the model, from a domain name, from a lightweight anchor, or from a corruption proxy. A plausible implication is that the quality of these synthetic or proxy signals can dominate the final forgetting–utility trade-off.

## 4. Newton-style, Hessian-based, and certified retain-free methods

Another major strand derives source-free updates from second-order approximations to retraining. These methods typically replace missing retain data with estimated curvature, stored second-order information, or surrogate statistics.

**MU-Mis**—**Machine Unlearning by Minimizing input sensitivity**—takes a different but still model-local approach [2402.15109]. The paper argues that a sample’s contribution to learning is reflected in the trained model’s sensitivity to that sample, and defines the unlearning loss
\[
\min_w \sum_{x\in\mathcal D_f} \Big( \|\nabla_x f_c(w,x)\|_F^2 - \|\nabla_x f_{c'}(w,x)\|_F^2 \Big).
\]
Here \(c\) is the true class and \(c'\neq c\) is a randomly selected irrelevant class. The method requires only the pretrained model and forgetting data, and is therefore **remaining-data-free** [2402.15109].

Certified source-free unlearning appears in **“A Certified Unlearning Approach without Access to Source Data”** [2506.06486]. This framework uses a surrogate dataset \(\mathcal{D}_s\) drawn from \(\mathcal{P}_{\text{surro}}\) to approximate inaccessible retain statistics. The surrogate-based Newton update is
\[
\hsrs = \hsast + \frac{m}{n - m}\widehat{\hess}^{-1}_{\Dr}\gradi(\Du, \hsast),
\]
with
\[
\widehat{\hess}_{\Dr} = \frac{n\hess_{\Ds} - m\hess_{\Du}}{n - m}.
\]
The central certification result states
\[
\|\hsastr - \hsrs\| \leq \appbound,
\]
and calibrated Gaussian noise with
\[
\sigma = \frac{\appbound}{\e} \sqrt{2 \ln(1.25 / \de)}
\]
yields \((\epsilon,\delta)\)-certified unlearning [2506.06486]. The guarantee depends on the statistical distance between source and surrogate distributions, which the paper estimates heuristically using KL-based approximations, energy-based modeling, SGLD, and Donsker–Varadhan estimation.

**“Towards Source-Free Machine Unlearning”** develops a related zero-shot idea for linear classifiers and mixed-linear networks [2508.15127]. The key difficulty is that classical Newton-style corrections require the Hessian of the retained data, yet \(\mathcal{D}_r\) is unavailable. The paper estimates the unknown retain Hessian from random perturbations around the optimum, forget-set gradients, and forget-side loss differences by solving a PSD-constrained optimization problem. Once \(\hat{\mathrm H}_r\) is obtained, it substitutes into
\[
w_{uf} = w^\star + \hat{\mathrm{H}_r}^{-1} \nabla_f + \sigma^2\varepsilon.
\]
The paper proves a Frobenius-norm bound on the Hessian-estimation error and a bound on the retained-data gradient at the unlearned model [2508.15127].

The most explicit retain-free second-order framework is **WIN-U**—**Woodbury-Informed Newton-Unlearning** [2604.13438]. Starting from the retraining first-order condition and a Taylor expansion around \(\boldsymbol{\theta}^*\), the paper derives
\[
\boldsymbol{\theta}_r^* \approx \boldsymbol{\theta}^* + \left(\mathbf{H} - \mathbf{H}_f\right)^{-1} \mathbf{g}_f.
\]
Using a generalized Gauss–Newton approximation and the Woodbury identity, it obtains the central update
\[
\boldsymbol{\theta}_r^* \approx \boldsymbol{\theta}^* + \frac{1}{n}\mathbf{H}^{-1} \mathbf{J}_f^\top \left(\mathbf{I}_{mc}- \frac{1}{n}\mathbf{B}_f \mathbf{J}_f \mathbf{H}^{-1} \mathbf{J}_f^\top\right)^{-1} \boldsymbol{\delta}_f.
\]
WIN-U requires only the original optimum \(\boldsymbol{\theta}^*\), a precomputed inverse Hessian \(\mathbf{H}^{-1}\), and the forget set \(\mathcal{D}_f\); no retain data are needed during unlearning [2604.13438].

These methods treat source-free unlearning as a local approximation problem: if retraining is the gold standard, then a sufficiently accurate estimate of the retain-side curvature or contribution can substitute for explicit access to the retain set. Their common limitation, stated in different ways across papers, is that the guarantees are strongest in convex, mixed-linear, or well-converged regimes.

## 5. Multimodal, domain-selective, and architecture-specific source-free unlearning

Recent work extends source-free unlearning beyond standard image classification and decoder-only LLMs. In pretrained multimodal encoders, **“Selective, Controlled and Domain-Agnostic Unlearning in Pretrained CLIP: A Training- and Data-Free Approach”** modifies only CLIP’s final visual projection matrix through a closed-form nullspace projection [2512.14113]. Given
\[
h = f(x;\theta)W,
\]
the method constructs a forget subspace from text embeddings \(t_c\), synthesized visual prototypes \(h_c\), and optionally residual domain embeddings \(r_c^d\), then sets
\[
W' = W P,
\]
where \(P\) projects onto the nullspace of the forget subspace. The paper defines three paradigms: **global unlearning**, **domain-specific removal**, and **complete selective domain unlearning**. For images from an unlearning domain, inference uses
\[
h' = f(x;\theta)WP,
\]
while untouched domains retain the original projection \(h = f(x;\theta)W\) [2512.14113].

In vision-language models, **HFRU**—**Object Hallucination-Free Reinforcement Unlearning for Vision-Language Models**—is source-free in the practical sense that it does not revisit the original pretraining corpus or retrain from scratch [2605.08031]. It operates on the **vision encoder**, not merely the decoder, using a two-stage pipeline: a cold-start alignment-disruption phase,
\[
\mathcal{L}_{\text{cold}} = - \mathbb{E}_{x \sim \mathcal{D}_{f},\, \tilde{y} \sim \text{RP}(y)} \log P_{\theta}(\tilde{y} \mid x, p),
\]
followed by encoder-only GRPO optimization with a composite forget reward
\[
\mathcal{R}_{\text{forget}}(y) = \mathcal{R}_{\text{pen}}(y) + \mathcal{R}_{\text{abs}}(y).
\]
The **abstraction reward** is designed to encourage semantically valid substitutions via hypernyms and to mitigate hallucinations [2605.08031].

In diffusion models, **“Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models”** targets instance-level forgetting without relying on prompts, concept labels, or the original training data [2603.10445]. The method constructs an edited surrogate image \(x_0^s\), defines a surrogate-induced noise target
\[
\epsilon'(x_t^f, x_0^s)=\frac{x_t^f-\sqrt{\bar{\alpha}_t}x_0^s}{\sqrt{1-\bar{\alpha}_t}},
\]
combines remember and forget losses with timestep-aware weighting
\[
\lambda(t)=1-\beta t,
\]
and resolves conflicts by gradient surgery:
\[
g_f' = g_f - \frac{g_r\cdot g_f}{\|g_r\|^2}g_r.
\]
For Stable Diffusion 3, the paper explicitly states that the training dataset cannot be accessed and instead uses generated images from the same prompt as a remember set [2603.10445].

Source-free unlearning also appears in **source-free domain adaptation**. **SCADA-UL**—**Unlearning Source-exclusive ClAsses in Domain Adaptation**—addresses the leakage of source-exclusive classes into target-domain models [2604.08238]. It defines forget classes \(\mathcal{C}_F = \mathcal{C}_S \setminus \mathcal{C}_T\), generates adversarial proxy samples for a forget class \(c_\mathcal{F}\) by minimizing
\[
\mathcal{L}_\text{ADV}(w^\mathcal{T}, \hat{x}) = \mathcal{L}_\text{CE}(w^\mathcal{T}, \hat{x}, c_\mathcal{F}),
\]
and then uses a **rescaled labeling strategy**
\[
\hat{y}_i =
\begin{cases}
0 & \text{if } i = c_\mathcal{F},\\[4pt]
\dfrac{y_i}{\sum_{j \in c_\mathcal{R}} y_j} & \text{if } i \in c_\mathcal{R},
\end{cases}
\]
inside a joint objective with the SFDA adaptation loss [2604.08238].

These methods indicate that source-free unlearning is now architecture-specific. Nullspace projection, RL on visual encoders, surrogate-based diffusion fine-tuning, and adversarial optimization during domain adaptation are not interchangeable procedures; each is tailored to how information is represented in the respective model family.

## 6. Evaluation, misconceptions, and unresolved issues

Evaluation protocols in source-free unlearning vary by task but usually combine a forgetting metric, a utility metric, and some privacy or relearning metric. In class-unlearning settings, the primary quantities are **forget accuracy** \(A_f\) and **retain accuracy** \(A_r\), with the target often phrased as **forget accuracy \(\to 0\%\)** while retain accuracy stays close to the original model [2311.15268]. Data-free and representation-based vision papers also use **Test CE**, **Membership Inference Attack (MIA)** accuracy or AUC, and the **Anamnesis Index (AIN)** for relearning speed [2412.13790]; [2601.21564]. Certified approaches evaluate train/test/retain/forget accuracy, **MIA** near chance, **relearn time (RT)**, and forget-score variants [2506.06486]. LLM benchmarks add benchmark-specific aggregates such as **Model Utility**, **Forget Efficacy**, **VerbMem**, **KnowMem**, **PrivLeak**, and task-specific scores on TOFU, MUSE, WMDP, RWKU, and ORT [2603.01285]; [2506.07795].

One recurrent misconception is that source-free necessarily means **data-free**. Several papers explicitly do not make that claim. MU-Mis is **remaining-data-free** but still requires the forgetting data [2402.15109]. Representation Unlearning’s zero-shot variant is source-free with respect to the retain data, but still assumes access to the forget set, classifier weights, and class-count metadata [2601.21564]. HFRU is source-free with respect to original training data, yet still depends on task-specific forget and retain examples plus manually constructed synonym and hypernym sets [2605.08031]. Reload is **forget set-free**, but not data-free, because it uses the retain set and cached full-dataset gradients [2604.10636]. The literature therefore distinguishes between the absence of **source data**, **retain data**, **forget data**, and **labels**, rather than collapsing them into a single binary property.

A second controversy concerns whether forgetting has actually occurred or whether the method has only induced refusal, lexical suppression, or brittle form-dependent behavior. **Attention Smoothing Unlearning (ASU)** argues that many prior methods produce incoherent outputs on forget prompts or fail to generalize because lexical-level and semantic-level associations persist in attention; it instead performs self-distillation from a temperature-smoothed attention teacher [2603.01285]. **ROCR** reframes the problem as **Form-Dependent Bias**, showing that many methods forget only the surface form seen during unlearning and fail under alternate expressions of the same knowledge; its proposed solution is a **training-free** parameter edit that redirects a dangerous concept to a harmless concept by a closed-form rank-one update in the MLP down-projection matrix [2506.07795]. These papers suggest that output suppression is not equivalent to source-free unlearning in the stronger sense of concept removal.

A third unresolved issue is the strength of the assumptions that make source-free methods work. Hessian-based certification assumes \(L\)-Lipschitz, \(\alpha\)-strongly convex, \(\beta\)-smooth, and \(\gamma\)-Hessian-Lipschitz losses, together with distributional closeness between source and surrogate datasets [2506.06486]. Zero-shot representation unlearning assumes **Neural Collapse** and access to classifier weights and class counts [2601.21564]. CUTS assumes that corruption induces a stable task-like direction in weight space and works best on pretrained models [2511.18660]. CLIP nullspace projection assumes that the target concept can be represented by a low-dimensional subspace and that synthesized prototypes are adequate proxies [2512.14113]. These are not merely technical details; they define where source-free unlearning is currently reliable and where it remains approximate.

Taken together, the literature presents source-free unlearning as a broad reorganization of the unlearning problem. Instead of replaying source data, methods now delete discrete codes, redirect concepts, smooth attention, synthesize supervision, subtract task vectors, estimate hidden curvature, or operate directly in latent geometry. This suggests a shift in emphasis: the central question is no longer only how to retrain less expensively, but how to define and remove the relevant information when the source data themselves are absent.

Source: https://www.emergentmind.com/topics/source-free-unlearning