RecBaselines2023: Baseline Benchmark

• RecBaselines2023 is a dataset and benchmark that aggregates baseline selections from 903 papers and 363 models, framing baseline choice as a recommendation task.
• The benchmark shows that state-of-the-art collaborative filtering methods, such as RP3(β), can recover hidden baselines with Recall@20 around 0.61.
• The study emphasizes that co-occurrence-aware methods outperform simple popularity rankings, providing a reproducible evaluation framework for recommender systems.

Searching arXiv for the specified paper and closely related recommender-system baseline-selection work to ground the article. RecBaselines2023 is a dataset and benchmark for choosing baselines for recommender models. It was introduced to address a recurring methodological problem in recommender-systems research: authors propose new approaches and compare them with existing models, called baselines, but the large number of recommender models makes it difficult to estimate which algorithms to choose in an article. RecBaselines2023 therefore collects and publishes information about the recommender models used in 903 papers, both as baselines and as proposed approaches, and formulates baseline selection as a recommendation problem over paper–model interactions. The accompanying experiments show that the selection of the best baselines for proposing new recommender approaches can be considered and successfully solved by existing state-of-the-art collaborative filtering models (Ivanova et al., 2023).

1. Corpus construction and relational structure

RecBaselines2023 is built around two entity types. A Paper is a scientific article in which recommender algorithms are evaluated, and a Model is a recommender algorithm used as a baseline or proposed approach in one or more papers. The core interaction is a binary uses-as-baseline relation: paper $u$ has an interaction with model $i$ if paper $u$ includes $i$ among its experimental baselines (Ivanova et al., 2023).

The logical schema is explicitly relational:

• Paper(paper_id, url, title, year)
• Model(model_id, name)
• Uses(paper_id, model_id)

The metadata stored per paper comprise paper_id, URL, Title, Year, and Baselines, where Baselines is a semicolon-separated list of canonicalized model names. The year field is reported as publication year (2010–2022). The dataset was obtained after manual collection via Google Scholar citation-crawling and preprocessing that unified model names and removed rare models and papers with fewer than 3 baselines (Ivanova et al., 2023).

This structure is important because it shifts the object of study away from user–item consumption data and toward research practice. In the benchmark formulation, a paper is treated as a “user” and a baseline model as an “item,” so the dataset can be seen as a typical dataset with interactions between papers and previously proposed models. A plausible implication is that methodological choice itself becomes amenable to standard Top-N recommendation protocols.

2. Dataset statistics and empirical regularities

After manual collection and preprocessing, RecBaselines2023 consists of 903 papers, 363 unique models, and 5 467 paper–model interactions, with sparsity (density) $\simeq 1.6\%$ (Ivanova et al., 2023).

Quantity	Value
Papers	903
Unique models	363
Paper–model interactions	5 467
Sparsity (density)	$\simeq 1.6\%$

The descriptive analysis reports several regularities. The number of papers per year rises from 1 in 2009/2011 to 253 in 2022, with rapid growth post-2017. Most papers include between 3 and 8 baselines. The top 10 most-used models and their number of papers are: BPR-MF (339), GRU4Rec (333), FPMC (213), POP (209), SASRec (173), NeuMF (154), NARM (152), CASER (135), LIGHTGCN (132), and ITEMKNN (130) (Ivanova et al., 2023).

These distributions motivate the benchmark design. Many papers include only a handful of baselines, specifically 3–7, so any recommender must operate under “few-shot” conditions. The dataset also exhibits model-naming inconsistencies, exemplified by “NeuMF” vs. “NCF,” which required manual unification. This feature is not merely a preprocessing nuisance; it reflects the bibliographic and terminological variability of recommender-systems literature and directly affects reproducibility and retrieval quality.

3. Task formulation and evaluation protocol

The benchmark task is Top-N recommendation of hidden baselines given a partial list of already-chosen baselines. Operationally, each paper $u$ is represented as a user and each baseline model $i$ as an item. The objective is to recover held-out baselines from incomplete baseline lists (Ivanova et al., 2023).

The data split combines temporal ordering with per-paper holdout. The training set contains all interactions from papers published before 2021, plus 80% of interactions from papers in 2021–2022. The validation and test sets are formed from the remaining 20% of interactions from 2021–2022 papers, split equally as 10%/10% for early stopping and final evaluation. This protocol ensures that later papers are not used in full for model fitting and thereby incorporates a temporal aspect into evaluation (Ivanova et al., 2023).

Hyperparameter tuning used the Elliot framework for full reproducibility. Each model searched over up to 20 hyperparameters, and early stopping was performed on validation Recall@20. Evaluation employed Precision@k, Recall@k, NDCG@k with binary relevance, and MAP@k, with results reported for $N=10,20$ in a standard Top-N protocol (Ivanova et al., 2023).

This setup formalizes a methodological choice problem as an offline recommendation problem. Because the held-out targets are baselines that actually appeared in later papers, the benchmark tests whether existing collaborative-filtering algorithms can recover historically plausible baseline selections from partial experimental context.

4. Benchmarked algorithms

Eight methods were compared:

Method	Description in the benchmark
Random	Uniform sampling over all models
MostPop	Rank by global popularity
BPR-MF	Matrix-factorization with pairwise Bayesian personalized ranking
MF2020	Revisited neural CF vs. classic MF
EASE	Linear auto-encoder
SLIM	Sparse linear item–item model
MultiVAE	Variational-autoencoder CF
RP3$_{(\beta)}$	Graph-based random-walk on item–item co-occurrence graph

The benchmark spans simple non-personalized ranking, latent-factor models, linear item-based methods, variational autoencoding, and graph-based recommendation (Ivanova et al., 2023). This breadth is methodologically useful because baseline recommendation could in principle be dominated by popularity effects, by latent collaborative structure, or by item–item co-occurrence patterns.

One of the notable empirical findings is that even simple item-based methods, specifically EASE and SLIM, outperform popularity. The paper interprets this as evidence that researchers’ baseline choices are far from uniform (Ivanova et al., 2023). This suggests that baseline selection is structured by co-usage patterns rather than reducible to a small set of universally popular algorithms.

5. Experimental results

The benchmark reports Recall@10/20, NDCG@10/20, and MAP@10/20. The best-performing method is RP3$i$0, followed by MultiVAE and SLIM, while Random is substantially worse than all informed baselines (Ivanova et al., 2023).

Model	R@10	R@20	N@10	N@20	M@10	M@20
Random	0.045	0.080	0.029	0.029	0.008	0.008
BPR-MF	0.228	0.353	0.134	0.134	0.035	0.035
MostPop	0.312	0.339	0.138	0.138	0.035	0.035
MF2020	0.348	0.446	0.227	0.227	0.067	0.067
EASE	0.397	0.549	0.243	0.243	0.069	0.069
NeuMF	0.420	0.513	0.252	0.252	0.073	0.073
SLIM	0.446	0.576	0.264	0.264	0.078	0.078
MultiVAE	0.455	0.603	0.264	0.264	0.075	0.075
RP3$i$1	0.473	0.607	0.303	0.303	0.088	0.088

The best model, RP3$i$2, achieves Recall@20 $i$3, meaning that it recovers more than half of the hidden baselines in a top-20 list. EASE and SLIM outperform MostPop, and thus the benchmark demonstrates that co-occurrence-aware collaborative filtering is more effective than ranking only by overall frequency (Ivanova et al., 2023).

At the same time, no formal statistical significance tests were reported. For an academic audience, this matters because the ordering among the strongest methods—RP3$i$4, MultiVAE, and SLIM—should be interpreted as benchmark evidence rather than as a statistically resolved hierarchy.

6. Methodological significance, limitations, and future directions

RecBaselines2023 is methodologically distinctive because it treats experimental baseline choice as an object of recommendation rather than as an unstructured scholarly habit. The principal result is not only that collaborative-filtering models can be applied to the problem, but that off-the-shelf methods can successfully suggest appropriate baselines from partial lists (Ivanova et al., 2023). This reframes a common practice in recommender-systems papers as a reproducible, data-driven decision task.

The paper also identifies several limitations. Staleness is explicit: as new recommender models appear, the dataset must be updated regularly, and a mechanism via GitHub pull requests is provided. Annotation noise remains possible because baseline lists may contain mistakes or misspellings. Temporal sensitivity is a substantive limitation, since current CF methods ignore recency and older but over-cited baselines may be recommended over newer state-of-the-art methods; a time-aware weighting scheme is left as future work. Beyond CF, the paper proposes investigating hybrid or metadata-based approaches using paper title and abstract for cold-start baseline recommendation. Generalisation to other fields such as NLP and CV is also proposed (Ivanova et al., 2023).

Several misconceptions are implicitly addressed by the benchmark. One is that baseline selection could be solved adequately by recommending the most popular algorithms. The empirical results contradict this, because MostPop is outperformed by EASE, SLIM, MultiVAE, and RP3$i$5 (Ivanova et al., 2023). Another is that baseline choice is too idiosyncratic to model. The benchmark results suggest the opposite: historical paper–model interactions contain enough structure for state-of-the-art collaborative filtering to recover hidden baselines at substantial Recall@20.

In that sense, RecBaselines2023 functions simultaneously as a dataset, a benchmark, and a methodological commentary on recommender-systems evaluation. It codifies how baselines have been selected across 903 papers, exposes the sparsity and few-shot character of those choices, and shows that the problem of choosing baselines for recommender models can be treated as a rigorous recommendation task in its own right (Ivanova et al., 2023).