---
title: Relearning Convergence Delay
url: https://www.emergentmind.com/topics/relearning-convergence-delay
type: topic
---

# Relearning Convergence Delay

Searching arXiv for the cited papers to ground the article in the latest available records.
Relearning convergence delay (RCD) is a machine-unlearning metric that measures how slowly an unlearned model can recover performance on a forgetting set when retraining is resumed from the unlearned weights. In its formal definition, RCD is the integral over retraining time of the excess evaluation error on the forgetting set relative to the optimal model for that set, so it quantifies not only immediate post-unlearning degradation but also the persistence of forgetting under relearning. In the formulation introduced for unlearning, RCD is intended to capture both changes in weight space and prediction space, and to assess the risk that forgotten data can be recovered from the unlearned model [2604.09391].

## 1. Formal definition and notation

The standard setup distinguishes a full training set $\mathcal{D}^{train} = \{x_i,y_i\}$, a forgetting set $\mathcal{D}_f$, and a retaining set $\mathcal{D}_r = \mathcal{D}^{train}\setminus \mathcal{D}_f$. A model is written as $f(x,\theta)$ with parameters $\theta \in \mathbb{R}^d$, and an iterative training algorithm is denoted by $\mathcal{T}(\theta_0,\mathcal{D},t)$, meaning that training starts from $\theta_0$ and runs for $t$ steps on dataset $\mathcal{D}$. The optimal parameters for a dataset $\mathcal{D}$ are denoted by
\[
\theta^{\mathcal D} = \arg\min_\theta \mathcal{L}(\theta,\mathcal D),
\]
and the evaluation error on $\mathcal{D}$ is denoted by $\Phi(\theta,\mathcal D)$.

With this notation, the continuous form of relearning convergence delay for an unlearned model $\theta_T^{UL}$ is
\[
RCD_{\mathcal T}(\theta_T^{UL},\mathcal D_f)
=
\int_0^\infty
\Big[
\Phi(\mathcal T(\theta_T^{UL},\mathcal D_f,t),\mathcal D_f)
-
\Phi(\theta^{\mathcal D_f},\mathcal D_f)
\Big]dt.
\]
The corresponding discrete $K$-step approximation is
\[
RCD_{\mathcal T}^K(\theta_T^{UL},\mathcal D_f)
=
\sum_{t=0}^K
\Big[
\Phi(\mathcal T(\theta_T^{UL},\mathcal D_f,t),\mathcal D_f)
-
\Phi(\theta^{\mathcal D_f},\mathcal D_f)
\Big].
\]

This definition makes RCD an area-under-the-relearning-curve quantity rather than a pointwise score. A model that briefly performs poorly on $\mathcal{D}_f$ but rapidly relearns can have substantially smaller RCD than a model whose error remains elevated over many updates. A common simplification in unlearning evaluation is to focus only on model predictions; the introduction of RCD is motivated by the claim that prediction-only metrics limit insight into the model’s true underlying data characteristics [2604.09391].

## 2. Theoretical characterization

The theory for RCD is developed under three assumptions. First, the training procedure converges in the sense that $\mathcal{T}(\theta,\mathcal D,+\infty)=\theta^{\mathcal D}$ for any $\theta,\mathcal D$. Second, the condition number of the Hessian is assumed to be monotone along training: if $\nabla^2\mathcal{L}(\theta,\mathcal D)$ has ordered eigenvalues $\lambda_1\ge \dots \ge \lambda_d\ge 0$, then $(\lambda_1/\lambda_d)(\theta_t)$ is nonincreasing in $t$. Third, under $\mu$-strong convexity and $\beta$-smoothness, the eigenvalues satisfy
\[
\mu \le \lambda_d(t) \le \lambda_1(t) \le \beta,
\]
with $\beta/\mu$ bounding the initial condition number.

Under gradient descent with step size $\eta_t=1/\lambda_1(t)$, RCD admits the upper bound
\[
0 \le RCD_{GD}(\theta,\mathcal D)
\le
\frac{\lambda_1(\theta,\mathcal D)}{\lambda_d(\theta,\mathcal D)}
\cdot
\big[\mathcal L(\theta,\mathcal D)-\mathcal L(\theta^{\mathcal D},\mathcal D)\big].
\]
Under $\mu$-strongly convex and $\beta$-smooth losses, this yields the corollary
\[
RCD_{GD}(\theta,\mathcal D)
\le
\frac{\beta}{\mu}
\cdot
\big[\mathcal L(\theta,\mathcal D)-\mathcal L(\theta^{\mathcal D},\mathcal D)\big].
\]
The proof sketch uses the descent lemma together with the PL-style bound
\[
\|\nabla \mathcal L\|^2 \ge 2\lambda_d(\mathcal L-\mathcal L_*).
\]

The discrete approximation converges exponentially fast:
\[
RCD_{GD} - RCD_{GD}^K = O(e^{-K}).
\]
Accordingly, only $O(\log 1/\epsilon)$ steps are needed to approximate RCD within $\epsilon$ [2604.09391].

These results establish two points of principle. First, RCD is controlled jointly by excess loss and the local condition number, so it is not reducible to post-unlearning accuracy alone. Second, because the discrete approximation converges exponentially, the metric is computationally tractable in regimes where exact infinite-horizon retraining would be impractical.

## 3. Influence Eliminating Unlearning

The principal algorithmic framework built around RCD is Influence Eliminating Unlearning (IEU). Its high-level objective is to produce unlearned parameters $\theta_T^{UL}$ that simultaneously minimize retained-set loss and maximize RCD on the forgetting set. In the paper’s formulation, the forgetting objective is realized through a combination of retaining-set gradient descent, forgetting-set gradient ascent, and noisy regularization [2604.09391].

The retaining-set utility term is the standard descent direction $-\nabla_\theta \mathcal L(\theta,\mathcal D_r)$. Forgetting pressure is introduced through gradient ascent on the forgetting set,
\[
+\,c\,\nabla_\theta \mathcal L(\theta,\mathcal D_f),
\]
with forgetting-set step size $\eta_f = c\,\eta_r$ and $c\in[0,1)$. The third component is an iterative re-initialization mechanism:
\[
\theta \leftarrow \alpha \theta + (1-\alpha)\cdot \mathcal N(0,2/d),
\qquad \alpha\in[0,1].
\]
This noise injection is described as raising the condition number on $\mathcal D_f$ and slowing convergence.

The full IEU update rule is
\[
\theta_{t+1}
\leftarrow
\alpha \theta_t + (1-\alpha)\theta_{init}
-\eta \nabla^r
+c\eta \nabla^f,
\]
where $\theta_{init}\sim \mathcal N(0,2/d)$, $\nabla^r=\nabla_\theta \mathcal L(\theta_t,\mathcal D_r)$, and $\nabla^f=\nabla_\theta \mathcal L(\theta_t,\mathcal D_f)$. The hyperparameters have distinct roles: $c$ controls the strength of forgetting-set ascent, $\alpha$ controls the speed of re-initialization noise, and $\eta$ is the base learning rate on $\mathcal D_r$.

Theoretical control of retaining-set utility is provided by a convergence bound. Under $L$-Lipschitz, $\mu$-strongly convex, $\beta$-smooth $\mathcal L$, step size $\eta=1/\beta$, and $\|\theta_n-\theta_m\|\le 2D$, the expected retaining-loss error satisfies
\[
E\big[\mathcal L(\theta_t,\mathcal D_r)-\mathcal L(\theta^{\mathcal D_r},\mathcal D_r)\big]
\le
LDe^{-(\mu/\beta)t}
+
2\beta\Big[(D/2)(1-\alpha)+(L/(2\beta))c+(L/\beta)\Big]^2
+
\beta(1-\alpha)^2
+
const.
\]
The first term decays exponentially in $t$, while the remaining terms quantify the penalty induced by gradient ascent and noise injection. This makes explicit that maximizing forgetting resistance is coupled to a utility cost whose magnitude is shaped by $c$ and $1-\alpha$.

## 4. Empirical behavior

The empirical study spans both classification and generative unlearning. In classification, the evaluation includes CIFAR-10, CIFAR-100, and TinyImageNet; architectures include ResNet-50 and ViT; baselines include FT, RL, SCRUB, and SALUN; and reported metrics include $\mathrm{accuracy}(\mathcal D_r)$, $\mathrm{accuracy}(\mathcal D_f)$, test accuracy, MIA ASR, Avg. Gap vs retrain, and $RCD_{GD}$ [2604.09391].

For TinyImageNet with ResNet under random 30% forgetting, the reported values are:

| Method | Avg.Gap | $RCD_{GD}$ |
|---|---:|---:|
| Retraining | 0 | $\approx 44.9$ |
| FT | $\approx 2.5$ | $\approx 4.3$ |
| RL | $\approx 3.1$ | $\approx 30.8$ |
| SCRUB | $\approx 7.4$ | $\approx 8.4$ |
| IEU w/GA | $\approx 2.9$ | $\approx 36.2$ |
| IEU w/Noisy | $\approx 2.4$ | $\approx 40.1$ |
| IEU w/GA+Noisy | $\approx 2.6$ | $\approx 39.8$ |

The associated trade-off plot places IEU variants near the retrain corner, meaning high RCD together with low Avg.Gap. This indicates that strong resistance to relearning need not require large degradation on the retaining distribution.

The generative setting studies Stable Diffusion NSFW unlearning with base model SD v1.4. ESD and SALUN are evaluated with and without GA and Noisy variants; I2P nudity prompts are used to measure Nudity-ratio, unrelated ImageNette classes are used to measure FID, and relearning with Adam yields $RCD_{Adam}$.

| Framework | Nudity / FID | $RCD_{Adam}$ |
|---|---|---:|
| ESD | 0.45 / 27.3 | $\approx 12.1$ |
| ESD+Noisy | 0.12 / 29.4 | $\approx 14.8$ |
| SALUN | 0.58 / 25.8 | $\approx 11.5$ |
| SALUN+Noisy | 0.15 / 26.1 | $\approx 13.6$ |
| ESD+GA+Noisy | 0.10 / 27.0 | $\approx 15.2$ |

In these results, Noisy-only and GA+Noisy consistently reduce Nudity and raise $RCD_{Adam}$ while preserving or improving FID. The ablation studies further report that smaller step sizes for $RCD_{GD}$ calculation marginally increase RCD but leave method rankings stable; that $\alpha$ values that are too low lead to model collapse and values that are too high lead to weak forgetting; and that $\alpha \approx 0.9999$–$0.99999$ balances retention and forgetting on the reported ViT CIFAR-100 setting. The Pareto curve indicates an explicit privacy-utility trade-off through tuning $(c,\alpha)$ [2604.09391].

## 5. Related notions of interference and delay

A related but distinct precursor appears in recurrent-network learning, where sequential updates can be obstructed by destructive interference. In the analysis of LMS and FORCE for memory and pattern-generation tasks, LMS is described as constantly obstructed by forgetting, manifested as destruction of dynamical objects from previous trials, whereas FORCE uses a whitened update through
\[
P(t)=\Bigl(\int_0^t r(\tau)r(\tau)^\top d\tau + \alpha I\Bigr)^{-1}
\]
and thereby protects previously learned directions [1805.09603]. Trial-trial overlap is measured by
\[
\rho_{12}=\frac{\bar r_1^\top \bar r_2}{\|\bar r_1\|\,\|\bar r_2\|},
\]
and higher overlap produces slower net convergence and stronger one-step-back forgetting. The reported scale difference is substantial: FORCE uses approximately one trial per new target, whereas LMS can require $\gtrsim 10^5$ trials for two highly overlapping targets and can easily exceed $10^4$–$10^5$ total trials when memorizing five values. This suggests a conceptual continuity between RCD in unlearning and interference-driven relearning delay in sequential online learning, even though the formal metrics differ.

The word “delay” also has a separate meaning in optimization and dynamical-systems analysis. In asynchronous optimization, delay denotes stale information, with updates of the form
\[
x_{k+1}=x_k-\eta_k S_k
\]
based on $x_{k-\tau_k}$, and delay-adaptive rules choose $\eta_k$ from the remaining budget
\[
0 \le \eta_k \le \max\Bigl(0,\eta' - \sum_{t=k-\tau_k}^{k-1}\eta_t\Bigr),
\qquad \sum_t \eta_t=\infty,
\]
so that convergence depends on actual time-varying delays rather than conservative worst-case bounds [2202.08550]. In delayed gradient descent, fixed delay $\tau$ still permits non-ergodic linear convergence under strong convexity or the PL condition, with admissible step sizes improved to $\eta\le 1/(4L\tau)$ for $\tau=1$, $\eta\le 3/(10L\tau)$ for $\tau\ge 2$, and $\eta \lesssim 9/(10L\tau)$ for large $\tau$ under PL [2308.11984]. In continuous-time linear systems, fixed delay can even improve convergence rate when the critical eigenvalues lie in the sector $(3\pi/4,5\pi/4)$, with maximal gain bounded by $e$ [1812.04030]. These results are not about machine unlearning, but they clarify that “relearning convergence delay” is a specific metric of resistance to recovery, not a generic synonym for time-delayed optimization.

## 6. Interpretation, misconceptions, and open questions

RCD is designed to measure residual influence rather than only prediction mismatch. The underlying claim is that evaluation based solely on model predictions offers limited visibility into whether the forgotten data remain easily recoverable from the weights. Because the upper bound on RCD depends on both excess loss and the condition number, a high immediate forgetting-set loss alone does not certify high resistance to relearning; conversely, low Avg.Gap on the retaining set does not by itself establish that forgetting is deep [2604.09391].

The framework also has clear current limits. The theory is built on gradient descent, whereas practical systems often use Adam or other optimizers. The reported behavior of $RCD_{Adam}$ is informative but is explicitly noted to behave differently, and not strictly slower with IEU-GA. Beyond vision, scalability to NLP, graphs, and lifelong unlearning remains to be studied. Fairness, security, and legal compliance aspects of unlearning algorithms are likewise identified as open areas.

Within its present scope, RCD provides a technically grounded way to evaluate whether an unlearned model merely suppresses immediate evidence of a forgetting set or whether it also delays reacquisition of that set under renewed training. That distinction is the central contribution of the metric and the organizing principle behind IEU.

Source: https://www.emergentmind.com/topics/relearning-convergence-delay