---
title: 'RelBench v2: Scalable Relational Benchmark'
url: https://www.emergentmind.com/topics/relbench-v2
type: topic
---

# RelBench v2: Scalable Relational Benchmark

RelBench v2 is a major expansion of the RelBench benchmark for relational deep learning (RDL). It is designed for realistic, scalable, leakage-safe evaluation of models that learn directly on multi-table relational databases rather than on manually flattened single tables, and it is positioned as infrastructure for large-scale relational representation learning and relational foundation models. Relative to the original benchmark, it adds four new large-scale relational datasets, introduces autocomplete tasks, and extends the benchmark ecosystem through integrations with translated Temporal Graph Benchmark datasets, ReDeLEx, and 4DBInfer [2602.12606].

## 1. Lineage and benchmark rationale

The original RelBench established the core benchmark abstraction for predictive modeling over relational databases: a task is defined by a training table containing an entity ID, a seed time, and target labels; only information before the seed time is available; and evaluation uses temporal train/validation/test splits. That first release provided 7 real-world relational databases and 30 predictive tasks, and it framed RDL as the combination of deep tabular encoders with graph neural networks over primary-key/foreign-key structure [2407.20060].

RelBench v2 retains that design logic but broadens the benchmark’s intended role. The motivation given for v2 is that v1 was too limited in scale, dataset diversity, and task coverage for the field’s move toward larger models and relational foundation models. The paper also argues that existing tabular foundation models largely target single-table data and therefore do not address the core structure of relational databases. In that sense, RelBench v2 is not presented merely as a benchmark update, but as a benchmark ecosystem for evaluating cross-table reasoning, temporal generalization, multi-task learning, and eventually pretraining and transfer [2602.12606].

## 2. Dataset expansion and domain coverage

RelBench v2 adds four new relational datasets: **rel-arxiv**, **rel-salt**, **rel-ratebeer**, and **rel-mimic**. For these newly added datasets, Table 1 reports **4 new datasets**, **21 tasks**, **29 tables**, **22,615,013 rows**, and **327 columns**. Elsewhere, the paper states that RelBench v2 brings the total to **11 datasets** and increases the benchmark to **over 22 million rows across 29 tables**. The paper itself notes that this wording is slightly ambiguous because the detailed totals in Table 1 are for the new datasets only [2602.12606].

| Dataset | Domain | Reported scale |
|---|---|---|
| rel-arxiv | Scholarly publications | 6 tables; 2,146,112 rows; 21 columns |
| rel-salt | Enterprise ERP | 4 tables; 4,257,145 rows; 31 columns |
| rel-ratebeer | Consumer platform | 13 tables; 13,787,005 rows; 221 columns |
| rel-mimic | Clinical records | 6 tables; 2,424,751 rows; 54 columns |

The new datasets substantially diversify the benchmark. **rel-arxiv** contributes scholarly publication and citation structure: the paper describes over **222,000 research papers**, **1.5 million directed citation links**, **53 physics categories**, and about **143,000 authors** linked via ORCID identifiers. **rel-salt** contributes minimally processed industry data centered on enterprise workflows, including sales document headers, sales document line items, customer master data, and address master data, with each record timestamped by creation time. **rel-ratebeer** is the largest of the new additions and introduces dense user-item-place-brewer interactions over more than two decades, with text, categorical, and temporal attributes. **rel-mimic** brings in deidentified clinical data derived from MIMIC-IV v3.1 and is credential-gated via PhysioNet; the benchmark uses a standard subset of **20,000 patients**, while also allowing customization such as limiting patient count, dropping columns, filtering by age, or integrating with Google BigQuery for full MIMIC-IV v3.1 access [2602.12606].

The earlier RelBench datasets remain part of v2: **rel-amazon**, **rel-avito**, **rel-event**, **rel-f1**, **rel-hm**, **rel-stack**, and **rel-trial**. The resulting coverage spans academic publishing, enterprise resource planning, consumer behavior, healthcare, e-commerce, social platforms, sports, and clinical trials [2602.12606].

## 3. Task families and the autocomplete formulation

RelBench v2 organizes evaluation around three task families: **autocomplete**, **forecasting**, and **recommendation**. Across all of them, the central abstraction is still the training table: each row specifies an entity ID, a seed time, and target labels, and the seed time determines what historical information is visible [2602.12606].

The main conceptual innovation is **autocomplete**. In forecasting tasks, labels are constructed by SQL queries over future windows; in autocomplete tasks, the target is an existing column value in a relational table. The paper characterizes autocomplete as adding a partially filled new row and predicting the remaining field values without access to future data. This broadens the benchmark beyond future-outcome prediction and aligns it with operational settings such as form completion and missing-value inference [2602.12606].

Autocomplete is explicitly designed as a leakage-sensitive task family. Because the target is an existing column, columns highly correlated with that target may need to be dropped. The paper gives a concrete example: in the **review-rating** task for **rel-amazon**, the **review_text** column must be dropped because it would otherwise provide intertwined signals with the review ratings. More generally, the benchmark enforces temporal restrictions through historical-only context and temporal neighbor sampling [2602.12606].

The paper reports **23 autocomplete tasks** in total. These include **review-rating** on rel-amazon; **searchstream-click** and **searchinfo-isuserloggedon** on rel-avito; **event_interest-interested**, **event_interest-not_interested**, and **users-birthyear** on rel-event; **results-position** and **qualifying-position** on rel-f1; **transactions-price** on rel-hm; **beer_ratings-total_score** on rel-ratebeer; **badges-class** on rel-stack; and **studies-enrollment**, **studies-has_dmc**, **eligibilities-adult**, and **eligibilities-child** on rel-trial. The **rel-salt** dataset contributes **8 autocomplete multiclass classification tasks**, including **item-plant**, **item-shippoint**, **item-incoterms**, **sales-office**, **sales-group**, **sales-payterms**, **sales-shipcond**, and **sales-incoterms** [2602.12606].

Forecasting remains the second major task family and is inherited from RelBench v1. In v2, the paper adds **13 new forecasting tasks**, spanning entity binary classification, entity multiclass classification, entity regression, and recommendation. Examples include **paper-citation**, **author-category**, **author-publication**, and **paper-paper-cocitation** on rel-arxiv; **patient-iculengthofstay** on rel-mimic; and **beer-churn**, **user-churn**, **brewer-dormant**, **user-count**, **user-beer-favorite**, **user-beer-liked**, and **user-place-liked** on rel-ratebeer [2602.12606].

Recommendation is treated as temporal link prediction: for each source entity and seed time, the model predicts a ranked list of the top-\(K\) target entities. The paper does not provide a separate candidate-generation algorithm, but it states that the setup requires calculating pairwise scores between source and target entities [2602.12606].

## 4. Relational-temporal abstractions and ecosystem integrations

A defining feature of RelBench v2 is its expansion beyond the native datasets into a broader evaluation framework. The most substantial extension is the translation of **Temporal Graph Benchmark (TGB)** datasets into normalized relational schemas, enabling direct comparison between temporal graph models and relational models under one benchmark API [2602.12606].

The translation follows an explicitly database-oriented design choice: each node type and edge type is translated into its own table, following normalization principles from database theory. The paper states formal schema patterns for several TGB settings. For dynamic link prediction, the relational form includes
\[
\text{nodes(node\_id)}
\]
and
\[
\text{events(event\_id, src\_id, dst\_id, event\_ts, weight)}.
\]
For heterogeneous temporal datasets, each node type \(t\) gets its own node table and each edge type \(e\) gets its own event table. For dynamic node property prediction, the translated schema includes
\[
\text{labels(label\_id)},
\]
\[
\text{label\_events(label\_event\_id, src\_id, label\_ts)},
\]
and
\[
\text{label\_event\_items(item\_id, label\_event\_id, label\_id, label\_weight)}.
\]
Across translated datasets, time-varying tables include an explicit **event_ts** column, split columns are omitted in favor of dataset-level metadata cutoffs, and only low-dimensional attributes are kept as relational columns while high-dimensional edge message vectors are omitted [2602.12606].

The translated TGB coverage is large. The paper lists dynamic link datasets such as **tgbl-flight** with **67,187,713 rows**, dynamic node-property datasets such as **tgbn-token** with **81,663,534 rows**, and heterogeneous temporal link datasets such as **thgl-myket** with **55,163,623 rows**. These are not part of the 11 native RelBench datasets; they are integrated benchmark extensions [2602.12606].

RelBench v2 also integrates **ReDeLEx**, described as a large-scale framework built from the CTU Relational Learning Repository and integrating **over 70 datasets**, with direct SQL connectivity, inferred attribute semantics, and heterogeneous-graph schema representations. The paper most clearly presents this as support for broader evaluation, while the abstract also describes it as providing uniform access to **70+ real-world databases suitable for pretraining**. In addition, v2 incorporates **4DBInfer**, contributing **7 datasets and 12 tasks** and broadening evaluation over datasets, tasks, relational-to-graph construction strategies, and predictive model families [2602.12606].

## 5. Model pipeline, metrics, and empirical results

The native RelBench v2 RDL pipeline follows the original RelBench design: raw row-level data are encoded with **PyTorch Frame**, specifically using the **ResNet tabular model**; temporal-aware subgraph sampling is performed around each entity node at a given seed time; the sampled graph is processed by a **heterogeneous GraphSAGE** model with **sum-based neighbor aggregation**; and task-specific prediction heads map final embeddings to outputs. For recommendation, the benchmark studies both GraphSAGE-based recommenders, which use source-target embedding inner products and **Bayesian Personalized Ranking loss**, and **ID-GNN**, which uses a source-specific MLP prediction head and **cross-entropy loss** [2602.12606].

The paper reports minimal tuning. For autocomplete, entity classification, and entity regression, the default hyperparameters are learning rate **0.005**, max epochs **10**, batch size **512**, hidden size **128**, **2** layers, **128** neighbors, and **uniform** temporal sampling. For recommendation, the default learning rate is **0.001** and max epochs **20** under otherwise similar settings. On **rel-ratebeer** recommendation, batch size had to be reduced to **64** for several configurations, and **4-layer GraphSAGE** caused **CUDA memory errors on an 80GB Nvidia A100** [2602.12606].

Evaluation uses **AUROC** for binary classification, **Accuracy** for multiclass classification, **\(R^2\)** in the main paper and **MAE** in the appendix for regression, and **MAP@\(\!K\)** for native recommendation. On translated TGB tasks, the paper reports **MRR@100** for dynamic and heterogeneous link prediction, and **NDCG@10** for dynamic node property prediction, with sampled-negative MRR also reported for the latter [2602.12606].

The empirical headline is that RDL models generally outperform single-table baselines across autocomplete, forecasting, and recommendation, though not without exceptions. On **binary autocomplete classification**, average test AUROC is **50.34** for LightGBM and **70.52** for the GNN. On **multiclass autocomplete classification**, test accuracy averages **19.51** for Random, **35.79** for Majority, **28.22** for LightGBM, and **69.14** for the GNN. Strong examples include **item-plant** on rel-salt at **99.46** test accuracy and **badges-class** on rel-stack at **82.83** [2602.12606].

Autocomplete regression is more heterogeneous. The GNN achieves **0.736** test \(R^2\) on **transactions-price** in rel-hm, **0.394** on **beer_ratings-total_score** in rel-ratebeer, and **0.394** on **results-position** in rel-f1, but some tasks remain difficult: **review-rating** on rel-amazon is negative for all methods, and **studies-enrollment** on rel-trial is near zero for all methods [2602.12606].

Forecasting results are similarly mixed but favorable in aggregate. On **entity binary classification**, average test AUROC is **71.79** for LightGBM and **78.19** for the GNN. On the new multiclass forecasting task **author-category** in rel-arxiv, test accuracy is **1.77** for Random, **9.09** for Majority, **2.01** for LightGBM, and **50.74** for the GNN. Regression shows genuine gains on tasks such as **author-publication** in rel-arxiv at **0.249** test \(R^2\) versus **-0.210** for LightGBM and **user-count** in rel-ratebeer at **0.625** versus **-0.170**, but the paper also notes that some simpler baselines remain competitive or better, especially on **rel-trial** and **driver-position** in rel-f1 [2602.12606].

Recommendation is the clearest case where architecture matters. Average test MAP is **7.45** for Past Visit, **10.61** for Global Popularity, **13.96** for LightGBM, **4.15** for 2-layer GraphSAGE, **13.52** for 4-layer GraphSAGE, **17.48** for 2-layer ID-GNN, and **23.35** for **4-layer ID-GNN**. The paper interprets this as evidence that recommendation is strongly multi-hop and that deeper, identity-aware models are especially effective [2602.12606].

## 6. Subsequent use, limitations, and significance

RelBench v2 rapidly became a target for model development rather than merely a passive benchmark. The paper **“RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases”** focuses explicitly on the new autocomplete setting and describes RelBench v2 as a standardized benchmark for RDL over multi-table databases that adds autocomplete tasks. That work formalizes autocomplete instances as row-table pairs \((r,T)\) with targets of the form
\[
y = r[c^*],
\]
identifies direct leakage and correlation leakage, introduces seed-row masking, and evaluates on **7 autocomplete tasks spanning 3 RelBench v2 datasets**. It reports that RelGT-AC outperforms the official GraphSAGE baseline on all **3 regression autocomplete tasks**, while a TF-IDF text encoder yields large gains on text-heavy eligibility tasks in **rel-trial**, including **+10.00 AUC** on **eligibilities-adult** and **+8.43 AUC** on **eligibilities-child** relative to the no-TF-IDF ablation. At the same time, GraphSAGE remains stronger on some classification tasks, indicating that the autocomplete track is not saturated and that leakage handling, text encoding, and architecture design remain active research issues [2606.03040].

The benchmark has several stated limitations. Some tasks remain intrinsically hard or poorly served by the current models: **review-rating** in rel-amazon remains poor under \(R^2\), some **rel-trial** regression tasks favor simple baselines, and extreme class imbalance can dominate cases such as **sales-office**. The paper also notes that it does not provide full formal task equations for all task families, explicit SQL label definitions, complete schema column listings for every dataset, or detailed candidate-generation procedures for recommendation beyond pairwise scoring. On the systems side, scaling remains challenging, as shown by memory failures for deeper recommendation models on large datasets [2602.12606].

Privacy and access constraints are especially salient for **rel-mimic**. The dataset is deidentified, access is credential-gated through PhysioNet, and the benchmark explicitly allows restricted subsets and controlled filters. This suggests a broader benchmark-design tension between realism and data governance, especially in healthcare and enterprise domains [2602.12606].

In the development of relational machine learning, RelBench v2 marks a shift from a narrowly defined forecasting benchmark to a broader relational-temporal evaluation platform. Its native datasets, autocomplete tasks, common training-table abstraction, and integrations with TGB, ReDeLEx, and 4DBInfer make it both a benchmark and a repository for relational data. The paper’s stated significance is that this infrastructure supports progress toward scalable, pretrained, and transferable relational learning systems, while empirical results indicate that modeling relational structure explicitly is often necessary to exceed single-table baselines [2602.12606].

Source: https://www.emergentmind.com/topics/relbench-v2