CURE4Rec: Benchmark for Recommendation Unlearning

• CURE4Rec is a benchmark for recommendation unlearning that tests if models can selectively forget sensitive interactions while preserving recommendation quality.
• It evaluates multiple objectives using metrics like membership inference for completeness, ranking accuracy for utility, runtime efficiency, and fairness across user groups.
• The benchmark employs graph-based deletion strategies (core, edge, random) to simulate varying structural impacts and reveal robustness and failure modes in unlearning methods.

Searching arXiv for the benchmark paper and a recent follow-up critique to ground the article in current literature. CURE4Rec is a benchmark for machine unlearning in recommender systems that evaluates whether a recommender can selectively forget specified training data while maintaining recommendation quality, operational efficiency, and fairness. Introduced as “the first comprehensive benchmark for recommendation unlearning evaluation,” it was designed to address fragmentation in prior evaluation practice by standardizing datasets, protocols, metrics, and unlearning scenarios, while explicitly incorporating “deeper influence” beyond immediate utility, especially recommendation fairness and robustness to different types of deleted data (Chen et al., 2024). In the recommendation setting, unlearning is motivated by the “right to be forgotten”: removing raw interaction data from storage is insufficient if trained model parameters and behavior still encode that information. CURE4Rec therefore frames recommendation unlearning as a multi-objective evaluation problem spanning unlearning completeness, recommendation utility, unlearning efficiency, and recommendation fairness, assessed under distinct data-removal regimes with different structural impact on the user–item graph (Chen et al., 2024).

1. Regulatory and technical setting

CURE4Rec is situated within the broader problem of machine unlearning, which addresses selective removal of training-data influence from trained models after deployment. In recommender systems, the issue is particularly acute because historical user–item interactions may encode sensitive preferences, and the collaborative nature of recommendation means that one user’s data can affect many others through shared latent factors or graph structure (Chen et al., 2024). Legal frameworks cited in this context include the GDPR, the California Consumer Privacy Act, and the Delete Act, all of which are presented as motivating a practical “right to be forgotten” in recommender pipelines (Chen et al., 2024).

The benchmark treats recommendation unlearning as distinct from simple data deletion. A recommender trained on sensitive interactions may still “remember” those interactions in its parameters and predictions after the underlying records are deleted. The central technical question is therefore whether a post hoc unlearning procedure can make a trained model behave as though a specified subset of interactions had never been used for training, while still preserving recommendation quality for non-forgotten users and items (Chen et al., 2024).

This framing aligns with later work that formalizes machine unlearning in recommender systems in terms of exact and approximate equivalence to retraining on a retain set. In that formalization, exact unlearning requires the model produced by an unlearning algorithm to be distributionally identical to retraining from scratch on the retained data, while approximate unlearning is expressed as $(\varepsilon,\delta)$-unlearning (Lubitzsch et al., 23 Aug 2025). CURE4Rec itself is not presented as a certified framework; rather, it is an empirical benchmark for comparing exact unlearning and approximate unlearning methods in recommender systems (Chen et al., 2024).

2. Benchmark scope and architectural logic

CURE4Rec targets model-agnostic unlearning in collaborative filtering recommenders and organizes evaluation around four dimensions encoded in its acronym: unlearning Completeness, recommendation Utility, unlea*Rning efficiency, and recommendation fairnEss* (Chen et al., 2024). Its principal design claim is that recommendation unlearning should not be judged only by whether deleted data is no longer detectable or by whether top-$k$ accuracy remains high. It should also be evaluated for runtime cost and for deeper behavioral effects, especially disparities across user groups and sensitivity to what kind of data is forgotten (Chen et al., 2024).

The benchmark defines an original training set $\mathcal{D}$, an unlearning set $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$, a retain set $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$, an original model $M_{\text{orig}}$, and an unlearned model $M_{\text{unlearn}}$ (Chen et al., 2024). This explicit separation allows the benchmark to compare unlearning outcomes against retraining from scratch on $\mathcal{D}_{\text{remain}}$, which functions as the gold standard for exact forgetting (Chen et al., 2024).

The benchmark evaluates methods under three data selection strategies: core data, edge data, and random data. This is a defining feature of CURE4Rec because it reframes deleted data not as an arbitrary subset, but as data with varying structural importance in the user–item bipartite graph (Chen et al., 2024). This design makes robustness to deletion target a first-class concern. A plausible implication is that benchmark outcomes are intended not merely to compare algorithms under average-case deletion, but to expose failure modes that emerge when the deleted data is either highly central or weakly connected.

3. Evaluation dimensions and metrics

CURE4Rec’s completeness metric is based on a Membership Inference Oracle (MIO), adapted to recommender representations (Chen et al., 2024). For each user $u$, the oracle takes as input the concatenation of the user embedding $e_u$ and the average embedding of the items with which the user interacted: $k$0 A small neural network then outputs a probability $k$1 that the user was in the training set (Chen et al., 2024). Completeness is assessed by MIO accuracy on forgotten points, with ideal forgetting corresponding to accuracy approaching $k$2, i.e., random guessing. The paper reports values such as $k$3–$k$4 for retraining and slightly higher values for SCIF, interpreting this as evidence that retraining weakens membership signals more thoroughly than approximate influence-based unlearning (Chen et al., 2024).

Recommendation utility is measured by standard top-$k$5 ranking metrics: Hit Ratio at $k$6 and Normalized Discounted Cumulative Gain at $k$7 (Chen et al., 2024). The benchmark defines

$k$8

and

$k$9

These metrics are computed on the retained-data regime and test data after unlearning (Chen et al., 2024).

Efficiency is measured as total runtime in seconds for the unlearning stage, including retraining or parameter-update cost (Chen et al., 2024). This metric is applied uniformly across retraining and the benchmarked exact and approximate unlearning methods, with lower runtime indicating better efficiency. Parallelization is enabled for exact unlearning methods (Chen et al., 2024).

Fairness is formalized through two metrics. The first is A-IGF (Active–Inactive Group Fairness), which measures disparity in average NDCG@20 between active users and inactive users outside the forget set. Users in the top $\mathcal{D}$0 by interaction count form the active group $\mathcal{D}$1, and the remaining $\mathcal{D}$2 form the inactive group $\mathcal{D}$3 (Chen et al., 2024). The metric is

$\mathcal{D}$4

Smaller disparity, and closeness to the retrain baseline, is treated as better fairness (Chen et al., 2024).

The second is shardGF, applicable to shard-based exact unlearning methods. If data is partitioned into $\mathcal{D}$5 shards and each shard produces a sub-model utility $\mathcal{D}$6, shard-level fairness is defined as

$\mathcal{D}$7

Lower variance indicates more balanced performance across shards (Chen et al., 2024). Approximate unlearning methods such as SCIF do not use shards and therefore do not have an associated shardGF value (Chen et al., 2024).

4. Graph-based unlearning-set construction

A distinctive component of CURE4Rec is its graph-based method for selecting deletion targets. Interaction data is represented as an unweighted bipartite graph with user nodes, item nodes, and edges $\mathcal{D}$8 indicating observed interactions (Chen et al., 2024). For each node $\mathcal{D}$9, importance is defined as

$\mathcal{D}_{\text{forget}} \subset \mathcal{D}$0

where $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$1 is the centrality of node $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$2, $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$3 is its neighbor set, and $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$4 is instantiated as node degree in the benchmark (Chen et al., 2024). Nodes with many neighbors that are themselves highly connected therefore receive high importance scores.

This importance ranking defines three deletion regimes. Core data corresponds to nodes with high importance, edge data to nodes with low importance, and random data to uniformly sampled nodes (Chen et al., 2024). Core nodes are described as high-degree nodes strongly connected to other high-degree nodes; removing them has large impact on collaborative structure and many other users or items. Edge nodes are weakly integrated into the collaborative network and have minimal impact on the overall model (Chen et al., 2024).

The significance of this design is twofold. First, it operationalizes “varying impact levels” of deleted data, making robustness to unlearning target explicit rather than incidental. Second, it exposes asymmetries that random deletion alone would obscure. The benchmark reports that some methods behave quite differently on core and edge deletions, revealing sensitivity to graph position and collaborative influence (Chen et al., 2024). A plausible implication is that graph-centrality-aware deletion regimes serve as a stress test for whether an unlearning method is robust to structurally consequential data, not merely to average-case forget sets.

5. Datasets, models, methods, and workflow

CURE4Rec uses three recommendation datasets processed for unlearning evaluation: MovieLens 100K, MovieLens 1M, and Amazon Digital Music (Chen et al., 2024). Users and items with fewer than five interactions are filtered, data is split $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$5 train, $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$6 validation, and $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$7 test, and the default unlearning ratio is $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$8 of the training data, with additional experiments exploring $\mathcal{D}_{\text{forget}} \subset \mathcal{D}$9–$$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$0 (Chen et al., 2024).

Dataset	Users / Items	Interactions / Sparsity
ML-100K	943 / 1,349	99,287 / 92.195%
ML-1M	6,040 / 3,416	999,611 / 95.155%
ADM	478,235 / 266,414	836,006 / 99.999%

The recommendation models are Weighted Matrix Factorization (WMF), Bayesian Personalized Ranking (BPR), and LightGCN (Chen et al., 2024). WMF treats all missing interactions as negatives with uniform weights; BPR optimizes pairwise ranking loss using implicit feedback; LightGCN is used as a graph-based collaborative filtering model (Chen et al., 2024). Reported typical hyperparameters are batch size $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$1, learning rate $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$2, embedding size $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$3, maximum epochs $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$4, and early stopping if validation NDCG@20 does not improve for five epochs (Chen et al., 2024).

The benchmark compares both exact unlearning (EU) and approximate unlearning (AU) methods. EU methods comprise retraining from scratch, SISA, RecEraser, and UltraRE (Chen et al., 2024). Retraining is the gold standard: remove $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$5, then train from scratch on $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$6. SISA uses sharded, isolated, sliced aggregation: data is partitioned into shards, sub-models are trained per shard, and only affected shards are retrained when forgetting is requested. RecEraser adapts shard division and aggregation to recommendation structure, while UltraRE further improves RecEraser via error decomposition and redesigned division and aggregation stages (Chen et al., 2024). The AU method is SCIF, the Selective and Collaborative Influence Function method, which attempts to “subtract” the effect of forgotten interactions through influence-based parameter updates without retraining (Chen et al., 2024).

The experimental workflow has four stages: learning the original model on full training data; selecting an unlearning set via the core, edge, or random strategy; applying the unlearning method and recording runtime; and evaluating completeness, utility, fairness, and robustness across selection strategies (Chen et al., 2024). This structure is part of what makes CURE4Rec a benchmark rather than a single unlearning algorithm: its contribution is a controlled protocol for cross-method comparison.

6. Empirical findings and trade-offs

CURE4Rec reports that both retraining and SCIF reduce MIO accuracy relative to the original model, indicating weakened membership signals after unlearning, but that SCIF remains on average approximately $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$7 higher than retraining, especially on edge data (Chen et al., 2024). The benchmark interprets this as incomplete forgetting by SCIF and as evidence that influence-function estimation is less precise on low-impact nodes (Chen et al., 2024).

In terms of utility, SCIF generally achieves higher NDCG@20 and HR@20 than the exact unlearning methods and remains close to retraining, while among the exact methods RecEraser and UltraRE consistently outperform SISA, and UltraRE slightly outperforms RecEraser (Chen et al., 2024). An especially notable result is that exact methods sometimes achieve higher utility when forgetting core users than when forgetting edge or random users. The paper attributes this to the removal of many interactions from active users, which reduces the size of the remaining training data and changes learning dynamics in a way that may make retraining easier on the remaining users (Chen et al., 2024). By contrast, SCIF experiences its largest utility degradation on core data, which the benchmark associates with more complex influence computation and larger estimation errors (Chen et al., 2024).

Efficiency results show SCIF as the most efficient method across datasets and models, often several times faster than exact methods and dramatically faster than full retraining (Chen et al., 2024). Among the exact methods, SISA is the most efficient, while UltraRE is usually more efficient than RecEraser (Chen et al., 2024). Deletion-target structure again matters: exact methods are faster on core deletions because high-degree users remove more data from the retraining set, whereas SCIF becomes slower on core deletions because influence-function calculations become heavier (Chen et al., 2024).

Fairness results are among the benchmark’s most distinctive contributions. Exact unlearning methods tend to worsen A-IGF, disproportionately benefiting active users and increasing disparities between active and inactive groups; this effect is stronger on larger datasets such as ML-1M and ADM (Chen et al., 2024). The benchmark connects this to shard construction. SISA distributes active users roughly evenly across shards, while RecEraser and UltraRE often cluster active users into specific shards, increasing group-level disparities (Chen et al., 2024). RecEraser and UltraRE also increase shardGF, though less dramatically than A-IGF (Chen et al., 2024). Increasing the number of shards improves efficiency and unlearning flexibility but harms utility and group-level fairness. Interestingly, shard-level variance may decrease even as user-group disparities grow, showing that shard equality and user fairness are not the same property (Chen et al., 2024).

These findings establish the central trade-off emphasized by the benchmark: exact unlearning provides the strongest completeness but often incurs penalties in utility, efficiency, and fairness, whereas approximate unlearning through SCIF improves utility, efficiency, and fairness but does not achieve retrain-level forgetting (Chen et al., 2024). This suggests that recommendation unlearning is fundamentally multi-objective rather than reducible to a single notion of correctness.

7. Reception, limitations, and subsequent critique

CURE4Rec positions itself as a benchmark for collaborative filtering recommenders and acknowledges several limitations: it does not cover sequential, session-based, graph-enhanced beyond the studied setting, or LLM-based recommender paradigms within the benchmark itself; it focuses mainly on user-wise unlearning; its completeness evaluation via MIO is mainly for approximate unlearning; and its fairness analysis is limited to active–inactive group fairness and shard-level fairness (Chen et al., 2024). The authors suggest future integration of game-theoretic unlearning evaluation for exact methods, expansion to more recommender paradigms, richer fairness notions, and deeper study of the trade-off between completeness and the other evaluation aspects (Chen et al., 2024).

A later paper, “Towards a Real-World Aligned Benchmark for Unlearning in Recommender Systems,” treats CURE4Rec as a valuable starting point but also as a benchmark whose assumptions are insufficient for production-aligned evaluation (Lubitzsch et al., 23 Aug 2025). That critique argues that CURE4Rec is limited by its collaborative-filtering-only scope, its focus on a single large deletion request, forget-set sizes of at least $$\mathcal{D}_{\text{remain}} = \mathcal{D} \setminus \mathcal{D}_{\text{forget}}$$8 of interactions, generic rather than domain-specific deletion targets, and runtime regimes that are “roughly one order of magnitude faster than retraining from scratch” but still not adequate for real-world low-latency demands (Lubitzsch et al., 23 Aug 2025). It also notes that SCIF is the only approximate unlearning method in CURE4Rec, which may narrow algorithmic comparison (Lubitzsch et al., 23 Aug 2025).

The same follow-up work proposes a more production-aligned benchmark spanning collaborative filtering, session-based recommendation, and next-basket recommendation; many small sequential requests rather than one large forget set; domain-specific deletion scenarios such as sensitive-item removal and spam-data removal; and stronger emphasis on per-request latency (Lubitzsch et al., 23 Aug 2025). In that perspective, CURE4Rec remains important as a pioneering benchmark for recommendation unlearning, especially for collaborative filtering and fairness-aware evaluation, but it is no longer viewed as sufficient for the full range of real-world recommender unlearning scenarios (Lubitzsch et al., 23 Aug 2025).

Taken together, these developments position CURE4Rec as a foundational benchmark that shifted recommendation unlearning evaluation away from ad hoc utility-only testing toward a richer framework incorporating completeness, efficiency, fairness, and robustness under graph-structured deletion regimes (Chen et al., 2024). Its lasting significance lies less in prescribing a single best unlearning method than in making clear that forgetting in recommender systems has deeper influence: deletions alter not only privacy leakage but also ranking quality, computation cost, and disparities across users and shards. Subsequent work suggests that this benchmarking logic should now be extended to more realistic tasks, request patterns, and operational constraints, but the conceptual template established by CURE4Rec remains central to the field (Chen et al., 2024, Lubitzsch et al., 23 Aug 2025).