---
title: Curriculum Unlearning Methods
url: https://www.emergentmind.com/topics/curriculum-unlearning
type: topic
---

# Curriculum Unlearning Methods

Searching arXiv for recent papers on curriculum unlearning and closely related machine unlearning work.
arxiv_search(query="curriculum unlearning machine unlearning", max_results=10, sort_by="relevance")
Curriculum unlearning denotes a family of machine unlearning procedures in which forgetting is staged progressively rather than imposed uniformly from the outset. In the supplied literature, this progression appears in several forms: sequential training over balanced user groups in erasable recommendation, easy-to-hard scheduling of forgetting samples in approximate unlearning, and gradient-norm-based gating of adversarial pressure in LLM unlearning [2203.11491][2509.14633][2508.15263][2602.01703]. Across these settings, the common motivation is explicit: aggressive unlearning can induce catastrophic forgetting or degrade recommendation performance, whereas conservative interventions can leave residual knowledge or approximate erasure that remains insufficiently robust [2602.01703][2509.14633].

## 1. Conceptual scope and recurring motivation

Machine unlearning is presented as the task of removing the influence of a specified subset while preserving performance on the complementary data. In large language models, the goal is to eliminate traces of a specified “forget” dataset $\mathcal{D}_f$ while preserving performance on a complementary “retain” dataset $\mathcal{D}_r$ [2602.01703]. In CUFG, the forgetting set is denoted $D_n$ and the retained complement is $D_r = D \setminus D_n$, with approximate unlearning seeking parameters that maintain low loss on $D_r$ and “push away” from $D_n$ [2509.14633]. In session-based recommendation, CAU defines $D_{\text{unlearn}} \subset D$ and $D_{\text{train}} = D \setminus D_{\text{unlearn}}$, with the explicit two-fold goal of removing the influence of $D_{\text{unlearn}}$ while retaining as much of the original model’s performance on $D_{\text{train}}$ as possible [2508.15263].

The rationale for a curriculum is also stated directly. CUFG argues that some forgetting samples are easier to “erase” because the model has lower confidence on them, and that it is destabilizing to present all forgetting instructions at once [2509.14633]. CAU states that existing studies fail to consider the ordering of unlearning samples when simultaneously processing multiple unlearning requests, leading to sub-optimal recommendation performance and unlearning effect [2508.15263]. AGT$^{AO}$ formulates the issue as a stability-robustness dilemma: aggressive forgetting can collapse model capabilities, while conservative forgetting may merely “mask” the unwanted knowledge, leaving it recoverable under adversarial conditions [2602.01703].

This suggests that “curriculum unlearning” is not a single algorithmic template. Rather, it is a scheduling principle applied to different control variables: user groups, forgetting samples, mini-batches, or adversarial perturbations.

## 2. Early curriculum structure in erasable recommendation

A clear precursor appears in LASER, a general erasable recommendation framework for recommender systems that consists of Group module and SeqTrain module [2203.11491]. The Group module partitions users into balanced groups based on the similarity of collaborative embedding learned via hypergraph, and the SeqTrain module trains the model sequentially on all groups with curriculum learning [2203.11491].

The framework is built on a hypergraph $G = (V,E,W)$ in which $V$ corresponds one-to-one to users, each hyperedge $e_i$ is defined from the set of users reachable from $u_i$ by any path of length $\le l$ in the original user-item bipartite graph, and the weight $w(e_i,u_j)$ is the average rating that $u_j$ gave to any item in the bipartite $l$-neighborhood of $u_i$; in practice $l=4$ [2203.11491]. Random walks over this hypergraph produce user sequences, after which Word2Vec is applied to obtain a user-embedding matrix $B \in \mathbb{R}^{N \times M}$, with experiments using $\text{rep}=4$, $\text{dep}=8$, and hidden-dim $M=16$ [2203.11491].

Balanced grouping minimizes total within-group embedding-distance,
$$
\sum_{s=1}^S \sum_{i \in G_s} \|B_{i,\cdot} - c_s\|^2
$$
subject to $|G_s| \le \lceil N/S \rceil$ [2203.11491]. The curriculum schedule is then defined through *collaborative cohesion*,
$$
\rho(G_s)= \frac{1}{|G_s|} \sum_{i,j \in G_s} \text{sim}(B_i,B_j), \qquad \text{sim}(x,y)=1/\|x-y\|_2,
$$
with groups sorted in descending order of $\rho$, producing the training sequence $G_{(1)},G_{(2)},\dots,G_{(S)}$ [2203.11491].

The theoretical analysis states that if groups with higher $U_\theta$ tend to receive higher $p_s$, then curriculum with $p_s \propto \rho(G_s)$ strictly increases the utility [2203.11491]. The same paper also links balanced grouping to unlearning-time complexity: for large $S$, the expected unlearning cost is about $\tfrac{1}{2}\cdot T \cdot N$ versus $T \cdot N$ if no grouping, yielding a $2\times$ speedup in expectation when $S \gg 1$ [2203.11491].

Empirically, on MovieLens 1M and Amazon Digital Music, LASER variants outperform C-SISA in recommendation utility for the reported setting $K=5\%$, $S=4$, model=DMF [2203.11491]. On ML with DMF and $S=8$, wall-clock time to unlearn $5\%$ of the last group is reported as 135 s for Retrain, 12 s for C-SISA, and 13 s for L-CBKM, so LASER matches C-SISA speed while retaining higher utility [2203.11491]. Under the ablation “Curriculum vs. reverse,” Easy$\to$Hard SeqTrain yields NDCG$=0.307$ whereas Hard$\to$Easy yields NDCG$=0.299$ [2203.11491].

## 3. Forgetting-gradient curricula in approximate unlearning

CUFG presents curriculum unlearning explicitly as a framework for fine-tuning-based approximate unlearning [2509.14633]. Its starting point is the saddle-type objective
$$
L_{\text{unlearn}}(\theta) =
\frac{1}{|D_r|}\sum_{(x,y)\in D_r}\mathcal{L}(h(x;\theta),y)
-\lambda \cdot \frac{1}{|D_n|}\sum_{(x,y)\in D_n}\mathcal{L}(h(x;\theta),y),
$$
with retain gradient $g_r(\theta)$ and forgetting-set gradient $g_n(\theta)$ [2509.14633]. The paper defines the **forgetting gradient**
$$
F(\theta) := -g_n(\theta),
$$
that is, the direction in which loss on the to-be-forgotten data increases most rapidly [2509.14633].

CUFG’s first mechanism is the Gradient Corrector. Let $\alpha$ be the angle between $g_r$ and $g_n$, and let $\gamma \in [0,\pi/2]$ be a threshold. The corrected unlearning gradient is
$$
\text{if } \alpha < \gamma:\quad \nabla \theta_U = \tfrac{1}{2}(-g_n + g_r),
\qquad
\text{else}:\quad \nabla \theta_U = g_r,
$$
followed by the update $\theta \leftarrow \theta - \eta \nabla \theta_U$ [2509.14633]. The stated purpose is to adjust the unlearning gradient only when the current fine-tuning step has too large a component in the “wrong” direction, toward the to-be-forgotten distribution [2509.14633].

Its second mechanism is an easy-to-hard curriculum over the forgetting set. For each $(x,\triangleright)\in D_n$, the difficulty score is
$$
DM_{\text{score}}(x) := h(x;\theta_D^*)[y],
$$
the softmax probability assigned to the true label by the trained model [2509.14633]. The forgetting set is sorted in ascending order of $DM_{\text{score}}$, from lowest confidence to easiest, and then sliced into $n$ contiguous groups $D_n^1,\dots,D_n^n$ processed sequentially [2509.14633]. In the end-to-end algorithm, epoch budget is divided across these subsets, and each inner update combines the current forgetting gradient on $D_n^i$ with retain mini-batch gradients on $D_r$ [2509.14633].

The paper’s interpretation is explicit. Retrain from scratch on $D_r$ is the gold standard: stable but extremely costly. Naïve fine-tuning on $D_r$ is stable but forgets too weakly. Aggressive methods such as gradient ascent, random-labeling, and influence functions can over-correct, causing gradient explosion or vanishing or loss of generalization [2509.14633]. CUFG is designed to bridge these extremes by steering each update “away” from the forgotten distribution only when necessary and by spreading the forgetting task over a curriculum from easy to hard samples [2509.14633].

On CIFAR-10, CIFAR-100, and SVHN with ResNet-18 and VGG-16, the reported metrics are UA, RA, TA, MIA, RTE, and Avg.Gap [2509.14633]. Table 1 is summarized as showing that CUFG achieves the smallest Avg.Gap in all settings. For Random 10\% on ResNet-18, Retrain has UA$=8.04$, RA$=100.0$, TA$=91.39$, MIA$=16.60$, Avg.Gap$=0$, while CUFG has UA$=6.51$, RA$=98.16$, TA$=91.36$, MIA$=11.29$, Avg.Gap$=2.18$ [2509.14633]. For Random 50\% on ResNet-18, Retrain has UA$=11.91$, RA$=100.0$, TA$=87.81$, MIA$=22.71$, Avg.Gap$=0$, while CUFG has UA$=7.24$, RA$=96.43$, TA$=88.92$, MIA$=11.62$, Avg.Gap$=5.11$ [2509.14633]. Efficiency for Random 10\% on ResNet-18 is reported as Retrain RTE$\approx 84$ min and CUFG$\approx 5.4$ min, described as a $\times 15$ speedup over scratch [2509.14633].

## 4. Pareto-based curriculum approximate unlearning for session-based recommendation

CAU extends curriculum unlearning to session-based recommendation and makes the scheduling problem part of a multi-objective optimization formulation [2508.15263]. The model $M_\theta$ may be GRU4Rec, SASRec, or BERT4Rec, trained on full dataset $D$, with a subset $D_{\text{unlearn}}$ of interactions to be forgotten and remaining training data $D_{\text{train}} = D \setminus D_{\text{unlearn}}$ [2508.15263].

The unlearning term is defined as
$$
L_{\text{unlearn}}(\theta; v_t \mid \text{prefix}) = \log P_\theta(v_t \mid v_{1:t-1}),
$$
which, when minimized by gradient descent, corresponds to gradient ascent on the negative log-likelihood of the unlearning item [2508.15263]. CAU uses two retaining terms: the normal prediction loss
$$
L_{\text{normal}}(\theta; v_{t+1}\mid v_{1:t-1}) = - \log P_\theta(v_{t+1}\mid v_{1:t-1}),
$$
and the consistency loss
$$
L_{\text{KL}}(\theta; \text{prefix}) = KL(P_{\theta^{rec}}(\cdot \mid v_{1:t-1}) \,\|\, P_\theta(\cdot \mid v_{1:t-1})),
$$
combined in
$$
L_{\text{retain}}(\theta) = \alpha_2 E_{D_{\text{train}}}[L_{\text{normal}}] + \alpha_3 E_{D_{\text{train}}}[L_{\text{KL}}].
$$
For implementation, the scalarized sum is
$$
L_{\text{app}}(\theta)=
\alpha_1 E_{D_{\text{unlearn}}}[L_{\text{unlearn}}]
+\alpha_2 E_{D_{\text{train}}}[L_{\text{normal}}]
+\alpha_3 E_{D_{\text{train}}}[L_{\text{KL}}],
$$
but the paper emphasizes that CAU seeks a Pareto-Optimal solution via MGDA rather than fixing $(\alpha_1,\alpha_2,\alpha_3)$ by hand [2508.15263].

MGDA solves
$$
\min_{\alpha_1,\alpha_2,\alpha_3} \|\alpha_1 g_1 + \alpha_2 g_2 + \alpha_3 g_3\|^2
\quad \text{subject to } \alpha_i \ge 0,\ \sum_i \alpha_i = 1,
$$
where $g_1$, $g_2$, and $g_3$ are the gradients of the three objectives [2508.15263]. This is the mechanism by which CAU states that it seeks the Pareto-Optimal solution, achieving effective unlearning with trivial sacrifice on recommendation performance [2508.15263].

The curriculum itself is built from two difficulty metrics. The gradient unlearning difficulty is
$$
Dif_g(x)= - \cos(\nabla_\theta L_{\text{unlearn}}(x),\ \nabla_\theta L_{\text{normal}}(x)+\nabla_\theta L_{\text{KL}}(x)),
$$
with lower $Dif_g$ meaning easier sample [2508.15263]. The embedding unlearning difficulty is
$$
Dif_e(x)= e_{\text{session}} \langle \cdot \rangle e_x,
$$
where a larger dot-product means the model is very confident in $x$ and hence the sample is hard to forget; scheduling proceeds ascendingly by $Dif_e$ [2508.15263].

CAU offers two curriculum strategies. Hard-sampling sorts all of $D_{\text{unlearn}}$ ascendingly by difficulty, partitions it into consecutive batches, and processes the batches from easiest to hardest [2508.15263]. Soft-sampling instead defines a time-dependent selection probability
$$
p_x(t)=
\frac{\left(\exp[(2t-1)\cdot (Dif(x)-mean(Dif))]\right)^\tau}
{\sum_j \left(\exp[(2t-1)\cdot (Dif(j)-mean(Dif))]\right)^\tau},
$$
so that early steps emphasize low-difficulty samples and late steps emphasize high-difficulty samples [2508.15263].

The reported metrics are Recall@K, NDCG@K, Hit$_u$@K, and the combined $U_\beta$ score [2508.15263]. CAU is reported to preserve Recall and NDCG within $\sim 5$–$8\%$ of the original full model, whereas exact-unlearning baselines SISA and SRU drop by $>30$–$60\%$ [2508.15263]. In unlearning effect, CAU achieves Hit$_u$ nearly as low as Retrain but with only $5$–$10\%$ of the computation, approximately $3\times$ faster than SISA/SRU and approximately $13\times$ faster than retrain [2508.15263]. Soft-sampling usually edges out hard-sampling by $1$–$2\%$ in $U_\beta$ [2508.15263].

## 5. Curriculum as adversarial gating in LLM unlearning

AGT$^{AO}$ generalizes the notion of curriculum unlearning beyond data ordering by deferring and conditionally enabling adversarial pressure in latent space [2602.01703]. The method combines Adaptive Orthogonality (AO) and Adversarial Gating Training (AGT) in a unified framework for robust and stabilized LLM unlearning [2602.01703].

AGT casts unlearning as the bi-level problem
$$
\min_{\theta}\ \max_{\|\delta\|_p \le \epsilon}\ L_{\text{unlearn}}(\theta; h_f^{(l)}+\delta, h_r),
$$
where
$$
L_{\text{unlearn}}(\theta;h_f+\delta,h_r)=L_{\text{forget}}(h_f+\delta;\theta)+L_{\text{retain}}(h_r;\theta)+\lambda_{AO}R_{AO}(\theta)
$$
[2602.01703]. The inner loop seeks $\delta^*$ by PGD,
$$
\delta^{(k+1)} = \Pi_\epsilon\big[\delta^{(k)} + \alpha\ \text{sign}(\nabla_\delta L_{\text{unlearn}}(\theta;h_f^{(l)}+\delta^{(k)},h_r))\big],
$$
and the outer loop updates
$$
\theta \leftarrow \theta - \eta \nabla_\theta L_{\text{unlearn}}(\theta;h_f^{(l)}+\delta^*,h_r)
$$
[2602.01703].

The curriculum is implemented through a binary gate $g_t$:
$$
g_t =
\begin{cases}
0, & \text{if } t \le N_{\text{warmup}},\\
\mathbf{1}_{\|\nabla_\theta L_{\text{unlearn}}\|_2 < \tau_{\text{grad}}}, & \text{otherwise}.
\end{cases}
$$
During the first $N_{\text{warmup}}$ steps there is no adversary, and afterward the adversary is engaged only when the landscape has sufficiently flattened [2602.01703]. The paper states that this Gradient-Norm-Based Gating prevents early “gradient explosion” and enables a controlled ramp-up of adversarial difficulty [2602.01703].

AO addresses gradient conflict between forgetting and retention. With
$$
g_f = \nabla_\theta L_{\text{forget}}(h_f;\theta), \qquad
g_r = \nabla_\theta L_{\text{retain}}(h_r;\theta),
$$
AO defines
$$
R_{AO}(\theta)= \mathbf{1}[g_f \cdot g_r < 0]\cdot
\left(\frac{1-\cos(g_f,g_r)}{2}\right)^\gamma.
$$
When $g_f$ and $g_r$ are non-conflicting, $R_{AO}=0$; when they conflict, the term grows and softly rotates the net update toward an orthogonal decomposition that minimizes harm to retained knowledge [2602.01703].

The empirical validation explicitly attributes a major role to the curriculum schedule. On TOFU, removing the gating is reported to lead to training instability, higher KUR ($\approx 0.60$ vs. $0.01$) and lower utility (Model Utility $0.49$ vs. $0.59$), while eliminating warm-up similarly degrades performance [2602.01703]. Sensitivity experiments identify a warm-up of one epoch and $\tau_{\text{grad}} = 0.6 \cdot \|\nabla L_{\text{unlearn}} \text{ at the end of warm-up}\|$ as striking the best balance [2602.01703]. The abstract reports a superior trade-off between unlearning efficacy, with KUR $\approx 0.01$, and model utility, with MMLU $58.30$, and the benchmarks listed are TOFU, MUSE, and WMDP [2602.01703].

A notable feature of this formulation is that the curriculum variable is not sample order but adversarial intensity. This suggests that curriculum unlearning can include staged optimization regimes in which the model first stabilizes and only later confronts stronger recovery attempts.

## 6. Comparative patterns, evaluation, and recurring controversies

The supplied literature uses different operationalizations of “difficulty,” and the differences are substantive rather than terminological.

| Work | Difficulty or curriculum signal | Progression |
|---|---|---|
| LASER [2203.11491] | collaborative cohesion $\rho(G_s)$ | groups sorted in descending order of $\rho$ |
| CUFG [2509.14633] | $DM_{\text{score}}(x)$ | lowest confidence $\rightarrow$ easiest |
| CAU [2508.15263] | $Dif_g(x)$ or $Dif_e(x)$ | ascending difficulty; hard- or soft-sampling |
| AGT$^{AO}$ [2602.01703] | warm-up and $\|\nabla_\theta L_{\text{unlearn}}\|_2 < \tau_{\text{grad}}$ | no adversary early, adversary later |

One common misconception is that curriculum unlearning is merely a heuristic for sample ordering. The evidence in the supplied papers is narrower and more heterogeneous. LASER uses curriculum at the level of balanced user groups [2203.11491]. CUFG uses a confidence-based partition of forgetting samples [2509.14633]. CAU uses per-sample difficulty plus MGDA-based weighting of multiple objectives [2508.15263]. AGT$^{AO}$ uses gating over adversarial perturbation rather than over examples [2602.01703].

Another recurring controversy concerns the relation between forgetting strength and retained utility. AGT$^{AO}$ states this as a “fundamental dilemma”: aggressive unlearning often induces catastrophic forgetting, whereas conservative strategies risk superficial forgetting and adversarial recovery [2602.01703]. CUFG describes the same tension by contrasting stable but weak naïve fine-tuning against destabilizing aggressive methods such as gradient ascent and influence functions [2509.14633]. CAU reports that removing the curriculum slows convergence and lowers ultimate $U_\beta$ by $3$–$5\%$, while removing $L_{\text{normal}}+L_{\text{KL}}$ and using pure GA collapses recommendation performance [2508.15263]. LASER frames the issue through the collaborative structure of recommendation: existing MU methods ignore collaborative information across users and items, so they cannot be directly applied into recommendation [2203.11491].

The evaluation protocols also differ sharply by domain. CUFG reports UA, RA, TA, MIA, RTE, and Avg.Gap on image classification benchmarks [2509.14633]. CAU uses Recall@K, NDCG@K, Hit$_u$@K, and $U_\beta$ for session-based recommendation [2508.15263]. LASER uses NDCG@10 and HR@10 along with unlearning time [2203.11491]. AGT$^{AO}$ reports KUR, Model Utility, and MMLU, with additional discussion of robustness under latent perturbations [2602.01703]. This suggests that “curriculum unlearning” is best understood as a training principle whose empirical signature depends on the benchmark family and threat model, rather than as a single metric-defined paradigm.

## 7. Research trajectory and implications

Within the supplied papers, curriculum unlearning develops from recommendation-specific sequencing to more explicit and general formulations. LASER already uses curriculum learning in SeqTrain and gives a theorem linking prior-weighted utility to curriculum ordering by collaborative cohesion [2203.11491]. CUFG elevates the phrase “curriculum unlearning” to the central method name and states that the concept has substantial research potential and offers forward-looking insights for the development of the MU field [2509.14633]. CAU shows that curriculum can be coupled to Pareto-based multi-objective optimization and to stochastic sampling rules that vary over time [2508.15263]. AGT$^{AO}$ shows that curriculum can regulate not only which data are forgotten first, but also when the model is exposed to worst-case latent perturbations [2602.01703].

A plausible implication is that curriculum unlearning functions as a control layer over approximate unlearning rather than as a replacement for the underlying forgetting mechanism. In the supplied literature, the forgetting mechanism may be sequential retraining over user groups, a gradient corrector guided by forgetting gradients, a GA term combined with retaining losses and MGDA, or latent-space adversarial training with adaptive orthogonality [2203.11491][2509.14633][2508.15263][2602.01703]. The curriculum component then modulates the order, difficulty, or activation of those mechanisms to improve stability, efficiency, or robustness.

Taken together, these works depict curriculum unlearning as a structured response to a recurring optimization problem: how to remove targeted knowledge without either collapsing general utility or leaving a residual footprint that is easily recovered.

Source: https://www.emergentmind.com/topics/curriculum-unlearning