---
title: Asymmetric Langevin Unlearning (ALU)
url: https://www.emergentmind.com/topics/asymmetric-langevin-unlearning-alu
type: topic
---

# Asymmetric Langevin Unlearning (ALU)

Asymmetric Langevin Unlearning (ALU) is a certified machine unlearning framework for training sets composed of **asymmetric sources**: a **public set** \(D_{\mathrm{pub}}\), sampled from \(P_{\mathrm{pub}}\), that is not subject to deletion requests, and a **private set** \(D_{\mathrm{priv}}\), sampled from \(P_{\mathrm{priv}}\), that is subject to unlearning. Formally, \(D = D_{\mathrm{pub}} \cup D_{\mathrm{priv}}\), and only points in \(D_{\mathrm{priv}}\) can be forgotten. ALU uses **Projected Noisy Gradient Descent (PNGD)** for both learning and unlearning, and evaluates unlearning quality by the Rényi divergence between the **unlearning distribution** and the **retraining distribution**. Its central claim is that public data can be injected directly into the learning and retraining objective so that the learning/retraining mismatch is reduced by a factor scaling like \(O(1/n_{\mathrm{pub}}^2)\), or more precisely through the factor \(\left(\frac{n_{\mathrm{priv}}}{n_{\mathrm{priv}}+n_{\mathrm{pub}}}\right)^2\). This yields a new control mechanism for the utility–unlearning trade-off, especially for forgetting a constant fraction of the private data, while making distribution mismatch between public and private sources an explicit part of the analysis [2605.11170].

## 1. Conceptual definition and data model

ALU is defined by an asymmetric training regime rather than by a new stochastic update distinct from Langevin unlearning. The asymmetry lies in the data model: public data appears in both the original training and retraining pipelines, while only private data is eligible for deletion. After a deletion request \(D_{\mathrm{forget}} \subseteq D_{\mathrm{priv}}\), the retain set is
\[
D_{\mathrm{retain}} = D \setminus D_{\mathrm{forget}}.
\]
The framework tracks three distributions over model weights: the **learning distribution** \(\pi_L^T\), obtained after training on the full dataset; the **unlearning distribution** \(\pi_U^K\), obtained by fine-tuning the trained model on the retain set; and the **retraining distribution** \(\pi_R^T\), obtained by retraining from scratch on the retain set only [2605.11170].

This retraining-based perspective places ALU within the broader certified unlearning tradition established for Langevin-style methods. In the symmetric baseline, Langevin unlearning is formulated through **Rényi Unlearning (RU)**, where the post-unlearning distribution is required to be close—in both Rényi directions—to retraining on the updated dataset. That baseline uses the symmetric privacy metric
\[
d_\alpha(\nu,\nu')=\max\!\left(D_\alpha(\nu\|\nu'),D_\alpha(\nu'\|\nu)\right),
\]
and applies the same noisy projected gradient dynamics in both learning and unlearning [2401.10371]. ALU preserves the retraining-indistinguishability viewpoint, but changes the source structure of the data so that public samples act as a stability anchor during both learning and retraining [2605.11170].

The motivating limitation is specific: in public-data-free Langevin unlearning, the certification cost for forgetting a constant fraction of the private data does not improve with total dataset size. The ALU formulation changes this because public data dilutes the sensitivity of the empirical gradient to deleted private points while remaining present in both learning and retraining objectives [2605.11170].

## 2. PNGD formulation and operational pipeline

ALU uses PNGD for both training and unlearning. The empirical loss on the mixed dataset is
\[
\mathcal{L}_D(\theta) = \frac{1}{n_{\mathrm{pub}}+n_{\mathrm{priv}}}\sum_{x \in D}\ell(\theta,x).
\]
Public data is therefore injected **directly into the empirical risk / gradient updates** rather than being used as a regularizer or auxiliary term [2605.11170].

The generic projected noisy gradient update is
\[
\theta_{t+1} = \Pi_{\Theta}\!\left[\theta_t - \eta \nabla_\theta \mathcal{L}(\theta_t) + \xi_t \right],
\]
with Gaussian noise
\[
\xi_t \sim \mathcal{N}(0, 2\eta\sigma^2 I_d),
\]
where \(\Theta \subset \mathbb{R}^d\) is compact, with radius \(R\) [2605.11170].

The ALU training phase runs \(T\) PNGD steps on
\[
D = D_{\mathrm{pub}} \cup D_{\mathrm{priv}},
\]
yielding \(\pi_L^T\). After a deletion request \(D_{\mathrm{forget}} \subseteq D_{\mathrm{priv}}\), ALU initializes from \(\theta \sim \pi_L^T\) and runs \(K\) PNGD steps on
\[
D_{\mathrm{retain}} = (D_{\mathrm{pub}} \cup D_{\mathrm{priv}})\setminus D_{\mathrm{forget}},
\]
yielding \(\pi_U^K\). The certification target is \(\pi_R^{T+K}\), produced by training from scratch on \(D_{\mathrm{retain}}\) using the same initialization law and the same noisy optimizer [2605.11170].

This pipeline is structurally continuous with the earlier Langevin unlearning framework. In that framework, training on \(\mathcal D\) induces an initial RDP mismatch between model distributions on adjacent datasets, and running further PNGD updates on the updated dataset contracts that mismatch toward the retraining distribution [2401.10371]. ALU inherits the same two-phase logic—initial discrepancy plus contraction—but improves the initialization term by changing the denominator in the empirical loss from private-only to public-plus-private [2605.11170].

A plausible implication is that ALU’s asymmetry is not in the isotropic Gaussian noise itself, which remains Gaussian and homogeneous, but in the way the data source decomposition changes gradient sensitivity and hence the required noise level. This interpretation aligns with the paper’s explicit statement that ALU is Langevin unlearning performed in the asymmetric data model [2605.11170].

## 3. Certification metric and main theoretical guarantees

ALU uses Rényi divergence as its certification metric. For \(P \ll Q\),
\[
D_{\alpha}(P\|Q) = \frac{1}{\alpha -1}\log\mathbb{E}_{Q}\left[\left(\frac{dP}{dQ}\right)^{\alpha}\right].
\]
The paper analyzes both the initial mismatch \(D_{\alpha}(\pi_R^T\|\pi_L^T)\) and the unlearning quality term \(D_{\alpha}(\pi_R^{T+K}\|\pi_U^K)\) or \(D_{\alpha}(\pi_U^K\|\pi_R^{T+K})\) [2605.11170].

The key theorem states that, under \(L\)-smoothness, \(M\)-Lipschitzness, a compact projection set \(\Theta\), and an initialization distribution \(\pi_0\) satisfying a \(C_0\)-LSI, the learning/retraining mismatch at iteration \(T\) obeys
\[
\frac{D_{\alpha}(\pi_{R}^{T}\| \pi_{L}^{T})}{\alpha} \leq \frac{2M^2\eta^2 n_{\mathrm{forget}}^2}{(n_{\mathrm{pub}}+n_{\mathrm{priv}})^2\sigma^{2}} \sum_{t=1}^{T-1}\prod_{t'=t}^{T-1}h(t',\eta,\sigma),
\]
where
\[
h(t',\eta,\sigma) = \left(1+\frac{\eta\sigma^2}{C_{t',1}}\right)^{-1}.
\]
The important structural factor is
\[
\frac{n_{\mathrm{forget}}^2}{(n_{\mathrm{pub}}+n_{\mathrm{priv}})^2},
\]
which is the source of the public-data advantage [2605.11170].

Unlearning then contracts this improved initialization. Using the contraction result inherited from Langevin unlearning,
\[
D_{\alpha}(\pi_R^{T+K}\|\pi_U^{K}) \leq D_{\alpha}(\pi_L^T\|\pi_R^T)\times\min\left(g_{\alpha,\eta,L}(K,\sigma), \exp\left(-\frac{2K\sigma^2\eta}{\alpha \Tilde{C}}\right)\right),
\]
and in the strongly convex case,
\[
D_{\alpha}(\pi_R^{T+K}\|\pi_U^{K}) \leq D_{\alpha}(\pi_L^T\|\pi_R^T)\exp\left(-\frac{2K\sigma^2\eta}{C\alpha}\right).
\]
Thus the benefit of ALU comes through a smaller initial divergence \(D_{\alpha}(\pi_L^T\|\pi_R^T)\), which is then exponentially reduced by the unlearning iterations [2605.11170].

This contraction mechanism is directly descended from the earlier Langevin unlearning theory, where privacy loss contracts exponentially with unlearning iterations, a phenomenon described there as **privacy recuperation** [2401.10371]. ALU does not replace that mechanism; it changes the prefactor through asymmetric source composition [2605.11170].

## 4. Noise suppression, mass unlearning, and computational advantage

The paper’s central comparative claim concerns the noise level required for certification when forgetting a constant fraction of the private data,
\[
n_{\mathrm{forget}} = c\, n_{\mathrm{priv}}, \qquad c\in(0,1].
\]
In the symmetric case,
\[
\sigma^2_{\mathrm{sym}} \geq \frac{2M^2\eta^2 c^2}{\varepsilon} \sum_{t=1}^{T-1}\prod_{t'=t}^{T-1}h(t',\eta,\sigma),
\]
whereas in ALU,
\[
\sigma^2_{\mathrm{asym}} \geq \frac{2M^2\eta^2 c^2}{\varepsilon} \left( \sum_{t=1}^{T-1}\prod_{t'=t}^{T-1}h(t',\eta,\sigma) \right) \left( \frac{n_{\mathrm{priv}}}{n_{\mathrm{priv}} + n_{\mathrm{pub}}} \right)^2.
\]
Hence public data suppresses the needed noise by
\[
\left( \frac{n_{\mathrm{priv}}}{n_{\mathrm{priv}} + n_{\mathrm{pub}}} \right)^2.
\]
For strongly convex loss, the appendix gives the \(\sigma\)-independent sufficient condition
\[
\sigma^2 \geq \frac{4\alpha c^2 M^2 (1-\exp(-m\eta T))}{\varepsilon m } \left(  \frac{n_{\mathrm{priv}}}{n_{\mathrm{priv}} + n_{\mathrm{pub}}} \right)^2
\]
[2605.11170].

This is the technical basis for the claim that ALU supports **mass unlearning of constant fractions of private data**. In the symmetric setting, the required noise lower bound does not shrink with dataset size when forgetting a constant fraction, whereas in ALU it can be made small by increasing \(n_{\mathrm{pub}}\) [2605.11170].

The paper also states a **strict computational advantage over retraining** under suitable conditions. Retraining cost is defined as the original \(T\) training steps, and unlearning cost as the minimum \(K\) such that
\[
D_{\alpha}(\pi_R^{T+K}\|\pi_U^K)\le \varepsilon.
\]
Under strong convexity, a sufficient condition for ALU to be more efficient than retraining is
\[
\frac{C\alpha}{2\sigma^2\eta}\log\left(\frac{4\alpha M^2 c^2}{m\varepsilon\sigma^2\left(\frac{n_{\mathrm{pub}}}{n_{\mathrm{priv}}}+1\right)^2}   \right) < T - \log\left( 1-\exp(-m\eta T)  \right).
\]
In the large-\(T\) regime, the paper states that the cost of unlearning scales as
\[
\mathcal{O}(\log(n_{\mathrm{total}})),
\]
with public data reducing the initial divergence further [2605.11170].

These results should be read against the original symmetric Langevin unlearning analysis, which already argued for a computational saving over retraining when the initial mismatch is \(O(1/n^2)\) in the strongly convex case [2401.10371]. ALU preserves that contraction-based savings logic but makes the initial mismatch controllable through public data volume [2605.11170].

## 5. Distribution mismatch and the utility–unlearning trade-off

A major contribution of ALU is that it does not assume \(P_{\mathrm{pub}} = P_{\mathrm{priv}}\). Instead, it explicitly analyzes **distribution mismatch**. The assumptions are that \(P_{\mathrm{pub}}\) and \(P_{\mathrm{priv}}\) share the same support, \(\Theta\) is compact, and the loss is \(M\)-Lipschitz in \(\theta\) [2605.11170].

The resulting generalization bound on the private distribution is
\[
\mathbb{E}_{\pi_U^K}[\mathcal{L}_{P_{\text{priv}}}] \leq \underbrace{ e^{\frac{n_{\text{pub}}}{n_{\text{pub}}+n_{\text{retain}}} D_{\infty}(P_{\text{priv}}\|P_{\text{pub}})} }_{\text{distribution mismatch penalty}} \, \mathbb{E}_{\pi_R^{T+K}}[\mathcal{L}_{P_{\text{train}}}] + M \cdot \mathrm{diam}(\Theta)\cdot \underbrace{\sqrt{\tfrac{1}{2}D_{\alpha}(\pi_R^{T+K}\|\pi_U^K)}}_{\text{unlearning approximation error}}.
\]
This decomposition makes the trade-off explicit. Public data reduces the approximation error term by shrinking \(D_{\alpha}(\pi_R^{T+K}\|\pi_U^K)\), but it can simultaneously worsen the mismatch penalty if \(P_{\mathrm{pub}}\) and \(P_{\mathrm{priv}}\) are badly misaligned [2605.11170].

The paper is correspondingly precise about when ALU is expected to work well. If
\[
D_\infty(P_{\mathrm{priv}}\|P_{\mathrm{pub}})\approx 0,
\]
the mismatch penalty is negligible and increasing public data is almost purely beneficial. If
\[
D_\infty(P_{\mathrm{priv}}\|P_{\mathrm{pub}})\gg 0,
\]
the mismatch penalty can dominate and the utility bound can become vacuous, even though the unlearning certificate itself still improves [2605.11170].

This mismatch-aware analysis distinguishes ALU from methods that treat asymmetry only as an optimization heuristic. A useful comparison is the two-phase retain–forget entanglement framework, which partitions the retain set into an **adjacent retain set** \(\mathcal D_r^{\text{adj}}\) and a **remote retain set** \(\mathcal D_r^{\text{rem}}\), then protects them differently during unlearning [2603.26569]. That work addresses a different asymmetry—semantic or feature-wise entanglement between forget and retain samples—whereas ALU addresses source asymmetry between public and private data [2603.26569; 2605.11170]. This suggests that “asymmetry” in machine unlearning is not a single notion but a family of structurally distinct decompositions.

## 6. Empirical evidence, related asymmetric variants, and limitations

The ALU paper evaluates three empirical axes: variational Rényi divergence, utility under mismatch, and membership inference attacks. On **DomainNet** 24-class image classification, using **Clipart** as private and **Quickdraw** as public, increasing public data consistently reduces the estimated
\[
D_{\alpha}(\pi_R^{T+K}\|\pi_U^K),
\]
and increasing unlearning iterations \(K\) also reduces divergence. An ablation at \(K=1\) indicates that public data primarily helps by reducing the initial distribution gap between learning and retraining [2605.11170].

For distribution mismatch, again on DomainNet with \(K=5\), private size \(20{,}000\), and forget set \(10{,}000\), the aligned setting yields a relative performance gap between unlearning and retraining of about **3.68%–4.62%**, whereas the misaligned setting yields about **10.34%–10.81%**. On **IMDB** sentiment classification with a 2-layer **LSTM**, evaluated using **U-LiRA**, the paper reports that without public data injection the attack can confidently distinguish many unlearned models from retrained models, while public data substantially reduces the attack’s discriminative power. Reported test accuracies are **82.59%** unlearned and **82.54%** retrained with no public data; **81.42%** unlearned and **82.15%** retrained with **Amazon Reviews (50k public), no label flips**; and **80.40%** unlearned and **80.80%** retrained with **Amazon Reviews (50k public), 40% flipped labels** [2605.11170].

ALU also sits within a broader line of asymmetric unlearning research. One direction replaces worst-case privacy analysis with **per-instance privacy losses** \(P(x,\alpha)\), which bound
\[
D_{\alpha}(\nu_{T,D}\|\nu_{T,D\setminus\{x\}}),
\]
and imply that unlearning time scales logarithmically with point-specific difficulty [2505.18786]. Another derives **per-instance** certified unlearning for ridge regression trained via Langevin dynamics, using point-specific sensitivity bounds
\[
\Delta_{i,k} = \eta \|\nabla \ell(\theta_k;x_i,y_i)\|
\]
to calibrate the minimum unlearning noise required for a target \((\varepsilon,\delta)\) guarantee [2602.15602]. A different asymmetric formulation, motivated by LLMs, treats retention as the primary objective and forgetting as auxiliary, and performs retention-prioritized gradient synthesis rather than symmetric loss balancing [2604.14808]. These works do not define ALU, but they show that asymmetry can be introduced through data sources, points, retain subsets, or gradient geometry.

The limitations of ALU are explicit. The theoretical bounds depend on LSI constants \(C_t,\tilde C\), which are difficult to compute in deep models; the strongest computational guarantees are clearest under strong convexity; variational Rényi divergence estimation is expensive; and public data may improve certification while still harming utility under severe mismatch. The paper also notes that web-scale “public” data may still contain sensitive or copyrighted content, so the public/private split is not a blanket resolution of broader privacy and legal issues [2605.11170].

Taken together, these results position ALU as a source-asymmetric extension of Langevin unlearning: it retains the retraining-based certification logic and contraction analysis of noisy gradient descent, but introduces a public-data control variable that can suppress required noise, improve the utility–unlearning trade-off, and make constant-fraction private-data deletion tractable under appropriate alignment conditions [2401.10371; 2605.11170].

Source: https://www.emergentmind.com/topics/asymmetric-langevin-unlearning-alu