---
title: 'RecBaselines2023: Baseline Benchmark'
url: https://www.emergentmind.com/topics/recbaselines2023
type: topic
---

# RecBaselines2023: Baseline Benchmark

Searching arXiv for the specified paper and closely related recommender-system baseline-selection work to ground the article.
RecBaselines2023 is a dataset and benchmark for choosing baselines for recommender models. It was introduced to address a recurring methodological problem in recommender-systems research: authors propose new approaches and compare them with existing models, called baselines, but the large number of recommender models makes it difficult to estimate which algorithms to choose in an article. RecBaselines2023 therefore collects and publishes information about the recommender models used in 903 papers, both as baselines and as proposed approaches, and formulates baseline selection as a recommendation problem over paper–model interactions. The accompanying experiments show that the selection of the best baselines for proposing new recommender approaches can be considered and successfully solved by existing state-of-the-art collaborative filtering models [2306.14292].

## 1. Corpus construction and relational structure

RecBaselines2023 is built around two entity types. A **Paper** is a scientific article in which recommender algorithms are evaluated, and a **Model** is a recommender algorithm used as a baseline or proposed approach in one or more papers. The core interaction is a binary **uses-as-baseline** relation: paper $u$ has an interaction with model $i$ if paper $u$ includes $i$ among its experimental baselines [2306.14292].

The logical schema is explicitly relational:

- `Paper(paper_id, url, title, year)`
- `Model(model_id, name)`
- `Uses(paper_id, model_id)`

The metadata stored per paper comprise `paper_id`, `URL`, `Title`, `Year`, and `Baselines`, where `Baselines` is a semicolon-separated list of canonicalized model names. The year field is reported as publication year `(2010–2022)`. The dataset was obtained after manual collection via Google Scholar citation-crawling and preprocessing that unified model names and removed rare models and papers with fewer than 3 baselines [2306.14292].

This structure is important because it shifts the object of study away from user–item consumption data and toward research practice. In the benchmark formulation, a paper is treated as a “user” and a baseline model as an “item,” so the dataset can be seen as a typical dataset with interactions between papers and previously proposed models. A plausible implication is that methodological choice itself becomes amenable to standard Top-N recommendation protocols.

## 2. Dataset statistics and empirical regularities

After manual collection and preprocessing, RecBaselines2023 consists of 903 papers, 363 unique models, and 5 467 paper–model interactions, with sparsity (density) $\simeq 1.6\%$ [2306.14292].

| Quantity | Value |
|---|---:|
| Papers | 903 |
| Unique models | 363 |
| Paper–model interactions | 5 467 |
| Sparsity (density) | $\simeq 1.6\%$ |

The descriptive analysis reports several regularities. The number of papers per year rises from 1 in 2009/2011 to 253 in 2022, with rapid growth post-2017. Most papers include between 3 and 8 baselines. The top 10 most-used models and their number of papers are: BPR-MF (339), GRU4Rec (333), FPMC (213), POP (209), SASRec (173), NeuMF (154), NARM (152), CASER (135), LIGHTGCN (132), and ITEMKNN (130) [2306.14292].

These distributions motivate the benchmark design. Many papers include only a handful of baselines, specifically 3–7, so any recommender must operate under “few-shot” conditions. The dataset also exhibits model-naming inconsistencies, exemplified by “NeuMF” vs. “NCF,” which required manual unification. This feature is not merely a preprocessing nuisance; it reflects the bibliographic and terminological variability of recommender-systems literature and directly affects reproducibility and retrieval quality.

## 3. Task formulation and evaluation protocol

The benchmark task is **Top-N recommendation of hidden baselines given a partial list of already-chosen baselines**. Operationally, each paper $u$ is represented as a user and each baseline model $i$ as an item. The objective is to recover held-out baselines from incomplete baseline lists [2306.14292].

The data split combines temporal ordering with per-paper holdout. The training set contains all interactions from papers published before 2021, plus 80% of interactions from papers in 2021–2022. The validation and test sets are formed from the remaining 20% of interactions from 2021–2022 papers, split equally as 10%/10% for early stopping and final evaluation. This protocol ensures that later papers are not used in full for model fitting and thereby incorporates a temporal aspect into evaluation [2306.14292].

Hyperparameter tuning used the Elliot framework for full reproducibility. Each model searched over up to 20 hyperparameters, and early stopping was performed on validation Recall@20. Evaluation employed Precision@k, Recall@k, NDCG@k with binary relevance, and MAP@k, with results reported for $N=10,20$ in a standard Top-N protocol [2306.14292].

This setup formalizes a methodological choice problem as an offline recommendation problem. Because the held-out targets are baselines that actually appeared in later papers, the benchmark tests whether existing collaborative-filtering algorithms can recover historically plausible baseline selections from partial experimental context.

## 4. Benchmarked algorithms

Eight methods were compared:

| Method | Description in the benchmark |
|---|---|
| Random | Uniform sampling over all models |
| MostPop | Rank by global popularity |
| BPR-MF | Matrix-factorization with pairwise Bayesian personalized ranking |
| MF2020 | Revisited neural CF vs. classic MF |
| EASE | Linear auto-encoder |
| SLIM | Sparse linear item–item model |
| MultiVAE | Variational-autoencoder CF |
| RP3$_{(\beta)}$ | Graph-based random-walk on item–item co-occurrence graph |

The benchmark spans simple non-personalized ranking, latent-factor models, linear item-based methods, variational autoencoding, and graph-based recommendation [2306.14292]. This breadth is methodologically useful because baseline recommendation could in principle be dominated by popularity effects, by latent collaborative structure, or by item–item co-occurrence patterns.

One of the notable empirical findings is that even simple item-based methods, specifically EASE and SLIM, outperform popularity. The paper interprets this as evidence that researchers’ baseline choices are far from uniform [2306.14292]. This suggests that baseline selection is structured by co-usage patterns rather than reducible to a small set of universally popular algorithms.

## 5. Experimental results

The benchmark reports Recall@10/20, NDCG@10/20, and MAP@10/20. The best-performing method is RP3$_{(\beta)}$, followed by MultiVAE and SLIM, while Random is substantially worse than all informed baselines [2306.14292].

| Model | R@10 | R@20 | N@10 | N@20 | M@10 | M@20 |
|---|---:|---:|---:|---:|---:|---:|
| Random | 0.045 | 0.080 | 0.029 | 0.029 | 0.008 | 0.008 |
| BPR-MF | 0.228 | 0.353 | 0.134 | 0.134 | 0.035 | 0.035 |
| MostPop | 0.312 | 0.339 | 0.138 | 0.138 | 0.035 | 0.035 |
| MF2020 | 0.348 | 0.446 | 0.227 | 0.227 | 0.067 | 0.067 |
| EASE | 0.397 | 0.549 | 0.243 | 0.243 | 0.069 | 0.069 |
| NeuMF | 0.420 | 0.513 | 0.252 | 0.252 | 0.073 | 0.073 |
| SLIM | 0.446 | 0.576 | 0.264 | 0.264 | 0.078 | 0.078 |
| MultiVAE | 0.455 | 0.603 | 0.264 | 0.264 | 0.075 | 0.075 |
| RP3$_{(\beta)}$ | **0.473** | **0.607** | **0.303** | **0.303** | **0.088** | **0.088** |

The best model, RP3$_{(\beta)}$, achieves Recall@20 $\simeq 0.61$, meaning that it recovers more than half of the hidden baselines in a top-20 list. EASE and SLIM outperform MostPop, and thus the benchmark demonstrates that co-occurrence-aware collaborative filtering is more effective than ranking only by overall frequency [2306.14292].

At the same time, no formal statistical significance tests were reported. For an academic audience, this matters because the ordering among the strongest methods—RP3$_{(\beta)}$, MultiVAE, and SLIM—should be interpreted as benchmark evidence rather than as a statistically resolved hierarchy.

## 6. Methodological significance, limitations, and future directions

RecBaselines2023 is methodologically distinctive because it treats experimental baseline choice as an object of recommendation rather than as an unstructured scholarly habit. The principal result is not only that collaborative-filtering models can be applied to the problem, but that off-the-shelf methods can successfully suggest appropriate baselines from partial lists [2306.14292]. This reframes a common practice in recommender-systems papers as a reproducible, data-driven decision task.

The paper also identifies several limitations. **Staleness** is explicit: as new recommender models appear, the dataset must be updated regularly, and a mechanism via GitHub pull requests is provided. **Annotation noise** remains possible because baseline lists may contain mistakes or misspellings. **Temporal sensitivity** is a substantive limitation, since current CF methods ignore recency and older but over-cited baselines may be recommended over newer state-of-the-art methods; a time-aware weighting scheme is left as future work. **Beyond CF**, the paper proposes investigating hybrid or metadata-based approaches using paper title and abstract for cold-start baseline recommendation. **Generalisation** to other fields such as NLP and CV is also proposed [2306.14292].

Several misconceptions are implicitly addressed by the benchmark. One is that baseline selection could be solved adequately by recommending the most popular algorithms. The empirical results contradict this, because MostPop is outperformed by EASE, SLIM, MultiVAE, and RP3$_{(\beta)}$ [2306.14292]. Another is that baseline choice is too idiosyncratic to model. The benchmark results suggest the opposite: historical paper–model interactions contain enough structure for state-of-the-art collaborative filtering to recover hidden baselines at substantial Recall@20.

In that sense, RecBaselines2023 functions simultaneously as a dataset, a benchmark, and a methodological commentary on recommender-systems evaluation. It codifies how baselines have been selected across 903 papers, exposes the sparsity and few-shot character of those choices, and shows that the problem of choosing baselines for recommender models can be treated as a rigorous recommendation task in its own right [2306.14292].

Source: https://www.emergentmind.com/topics/recbaselines2023