RelBench v2: Scalable Relational Benchmark

• RelBench v2 is a comprehensive benchmark ecosystem for evaluating relational deep learning models on multi-table databases with leakage-safe, temporal train/validation/test splits.
• It expands the original benchmark by adding four large-scale relational datasets, 21 new tasks, and integrations with Temporal Graph Benchmark datasets, ReDeLEx, and 4DBInfer.
• The platform supports scalable, leakage-sensitive evaluations for relational foundation models, emphasizing cross-table reasoning, multi-task learning, and operational settings like autocomplete.

RelBench v2 is a major expansion of the RelBench benchmark for relational deep learning (RDL). It is designed for realistic, scalable, leakage-safe evaluation of models that learn directly on multi-table relational databases rather than on manually flattened single tables, and it is positioned as infrastructure for large-scale relational representation learning and relational foundation models. Relative to the original benchmark, it adds four new large-scale relational datasets, introduces autocomplete tasks, and extends the benchmark ecosystem through integrations with translated Temporal Graph Benchmark datasets, ReDeLEx, and 4DBInfer (Gu et al., 13 Feb 2026).

1. Lineage and benchmark rationale

The original RelBench established the core benchmark abstraction for predictive modeling over relational databases: a task is defined by a training table containing an entity ID, a seed time, and target labels; only information before the seed time is available; and evaluation uses temporal train/validation/test splits. That first release provided 7 real-world relational databases and 30 predictive tasks, and it framed RDL as the combination of deep tabular encoders with graph neural networks over primary-key/foreign-key structure (Robinson et al., 2024).

RelBench v2 retains that design logic but broadens the benchmark’s intended role. The motivation given for v2 is that v1 was too limited in scale, dataset diversity, and task coverage for the field’s move toward larger models and relational foundation models. The paper also argues that existing tabular foundation models largely target single-table data and therefore do not address the core structure of relational databases. In that sense, RelBench v2 is not presented merely as a benchmark update, but as a benchmark ecosystem for evaluating cross-table reasoning, temporal generalization, multi-task learning, and eventually pretraining and transfer (Gu et al., 13 Feb 2026).

2. Dataset expansion and domain coverage

RelBench v2 adds four new relational datasets: rel-arxiv, rel-salt, rel-ratebeer, and rel-mimic. For these newly added datasets, Table 1 reports 4 new datasets, 21 tasks, 29 tables, 22,615,013 rows, and 327 columns. Elsewhere, the paper states that RelBench v2 brings the total to 11 datasets and increases the benchmark to over 22 million rows across 29 tables. The paper itself notes that this wording is slightly ambiguous because the detailed totals in Table 1 are for the new datasets only (Gu et al., 13 Feb 2026).

Dataset	Domain	Reported scale
rel-arxiv	Scholarly publications	6 tables; 2,146,112 rows; 21 columns
rel-salt	Enterprise ERP	4 tables; 4,257,145 rows; 31 columns
rel-ratebeer	Consumer platform	13 tables; 13,787,005 rows; 221 columns
rel-mimic	Clinical records	6 tables; 2,424,751 rows; 54 columns

The new datasets substantially diversify the benchmark. rel-arxiv contributes scholarly publication and citation structure: the paper describes over 222,000 research papers, 1.5 million directed citation links, 53 physics categories, and about 143,000 authors linked via ORCID identifiers. rel-salt contributes minimally processed industry data centered on enterprise workflows, including sales document headers, sales document line items, customer master data, and address master data, with each record timestamped by creation time. rel-ratebeer is the largest of the new additions and introduces dense user-item-place-brewer interactions over more than two decades, with text, categorical, and temporal attributes. rel-mimic brings in deidentified clinical data derived from MIMIC-IV v3.1 and is credential-gated via PhysioNet; the benchmark uses a standard subset of 20,000 patients, while also allowing customization such as limiting patient count, dropping columns, filtering by age, or integrating with Google BigQuery for full MIMIC-IV v3.1 access (Gu et al., 13 Feb 2026).

The earlier RelBench datasets remain part of v2: rel-amazon, rel-avito, rel-event, rel-f1, rel-hm, rel-stack, and rel-trial. The resulting coverage spans academic publishing, enterprise resource planning, consumer behavior, healthcare, e-commerce, social platforms, sports, and clinical trials (Gu et al., 13 Feb 2026).

3. Task families and the autocomplete formulation

RelBench v2 organizes evaluation around three task families: autocomplete, forecasting, and recommendation. Across all of them, the central abstraction is still the training table: each row specifies an entity ID, a seed time, and target labels, and the seed time determines what historical information is visible (Gu et al., 13 Feb 2026).

The main conceptual innovation is autocomplete. In forecasting tasks, labels are constructed by SQL queries over future windows; in autocomplete tasks, the target is an existing column value in a relational table. The paper characterizes autocomplete as adding a partially filled new row and predicting the remaining field values without access to future data. This broadens the benchmark beyond future-outcome prediction and aligns it with operational settings such as form completion and missing-value inference (Gu et al., 13 Feb 2026).

Autocomplete is explicitly designed as a leakage-sensitive task family. Because the target is an existing column, columns highly correlated with that target may need to be dropped. The paper gives a concrete example: in the review-rating task for rel-amazon, the review_text column must be dropped because it would otherwise provide intertwined signals with the review ratings. More generally, the benchmark enforces temporal restrictions through historical-only context and temporal neighbor sampling (Gu et al., 13 Feb 2026).

The paper reports 23 autocomplete tasks in total. These include review-rating on rel-amazon; searchstream-click and searchinfo-isuserloggedon on rel-avito; event_interest-interested, event_interest-not_interested, and users-birthyear on rel-event; results-position and qualifying-position on rel-f1; transactions-price on rel-hm; beer_ratings-total_score on rel-ratebeer; badges-class on rel-stack; and studies-enrollment, studies-has_dmc, eligibilities-adult, and eligibilities-child on rel-trial. The rel-salt dataset contributes 8 autocomplete multiclass classification tasks, including item-plant, item-shippoint, item-incoterms, sales-office, sales-group, sales-payterms, sales-shipcond, and sales-incoterms (Gu et al., 13 Feb 2026).

Forecasting remains the second major task family and is inherited from RelBench v1. In v2, the paper adds 13 new forecasting tasks, spanning entity binary classification, entity multiclass classification, entity regression, and recommendation. Examples include paper-citation, author-category, author-publication, and paper-paper-cocitation on rel-arxiv; patient-iculengthofstay on rel-mimic; and beer-churn, user-churn, brewer-dormant, user-count, user-beer-favorite, user-beer-liked, and user-place-liked on rel-ratebeer (Gu et al., 13 Feb 2026).

Recommendation is treated as temporal link prediction: for each source entity and seed time, the model predicts a ranked list of the top-$K$ target entities. The paper does not provide a separate candidate-generation algorithm, but it states that the setup requires calculating pairwise scores between source and target entities (Gu et al., 13 Feb 2026).

4. Relational-temporal abstractions and ecosystem integrations

A defining feature of RelBench v2 is its expansion beyond the native datasets into a broader evaluation framework. The most substantial extension is the translation of Temporal Graph Benchmark (TGB) datasets into normalized relational schemas, enabling direct comparison between temporal graph models and relational models under one benchmark API (Gu et al., 13 Feb 2026).

The translation follows an explicitly database-oriented design choice: each node type and edge type is translated into its own table, following normalization principles from database theory. The paper states formal schema patterns for several TGB settings. For dynamic link prediction, the relational form includes

$\text{nodes(node\_id)}$

and

$$\text{events(event\_id, src\_id, dst\_id, event\_ts, weight)}.$$

For heterogeneous temporal datasets, each node type $t$ gets its own node table and each edge type $e$ gets its own event table. For dynamic node property prediction, the translated schema includes

$\text{labels(label\_id)},$

$$\text{label\_events(label\_event\_id, src\_id, label\_ts)},$$

and

$$\text{label\_event\_items(item\_id, label\_event\_id, label\_id, label\_weight)}.$$

Across translated datasets, time-varying tables include an explicit event_ts column, split columns are omitted in favor of dataset-level metadata cutoffs, and only low-dimensional attributes are kept as relational columns while high-dimensional edge message vectors are omitted (Gu et al., 13 Feb 2026).

The translated TGB coverage is large. The paper lists dynamic link datasets such as tgbl-flight with 67,187,713 rows, dynamic node-property datasets such as tgbn-token with 81,663,534 rows, and heterogeneous temporal link datasets such as thgl-myket with 55,163,623 rows. These are not part of the 11 native RelBench datasets; they are integrated benchmark extensions (Gu et al., 13 Feb 2026).

RelBench v2 also integrates ReDeLEx, described as a large-scale framework built from the CTU Relational Learning Repository and integrating over 70 datasets, with direct SQL connectivity, inferred attribute semantics, and heterogeneous-graph schema representations. The paper most clearly presents this as support for broader evaluation, while the abstract also describes it as providing uniform access to 70+ real-world databases suitable for pretraining. In addition, v2 incorporates 4DBInfer, contributing 7 datasets and 12 tasks and broadening evaluation over datasets, tasks, relational-to-graph construction strategies, and predictive model families (Gu et al., 13 Feb 2026).

5. Model pipeline, metrics, and empirical results

The native RelBench v2 RDL pipeline follows the original RelBench design: raw row-level data are encoded with PyTorch Frame, specifically using the ResNet tabular model; temporal-aware subgraph sampling is performed around each entity node at a given seed time; the sampled graph is processed by a heterogeneous GraphSAGE model with sum-based neighbor aggregation; and task-specific prediction heads map final embeddings to outputs. For recommendation, the benchmark studies both GraphSAGE-based recommenders, which use source-target embedding inner products and Bayesian Personalized Ranking loss, and ID-GNN, which uses a source-specific MLP prediction head and cross-entropy loss (Gu et al., 13 Feb 2026).

The paper reports minimal tuning. For autocomplete, entity classification, and entity regression, the default hyperparameters are learning rate 0.005, max epochs 10, batch size 512, hidden size 128, 2 layers, 128 neighbors, and uniform temporal sampling. For recommendation, the default learning rate is 0.001 and max epochs 20 under otherwise similar settings. On rel-ratebeer recommendation, batch size had to be reduced to 64 for several configurations, and 4-layer GraphSAGE caused CUDA memory errors on an 80GB Nvidia A100 (Gu et al., 13 Feb 2026).

Evaluation uses AUROC for binary classification, Accuracy for multiclass classification, $R^2$ in the main paper and MAE in the appendix for regression, and MAP@$\!K$ for native recommendation. On translated TGB tasks, the paper reports MRR@100 for dynamic and heterogeneous link prediction, and NDCG@10 for dynamic node property prediction, with sampled-negative MRR also reported for the latter (Gu et al., 13 Feb 2026).

The empirical headline is that RDL models generally outperform single-table baselines across autocomplete, forecasting, and recommendation, though not without exceptions. On binary autocomplete classification, average test AUROC is 50.34 for LightGBM and 70.52 for the GNN. On multiclass autocomplete classification, test accuracy averages 19.51 for Random, 35.79 for Majority, 28.22 for LightGBM, and 69.14 for the GNN. Strong examples include item-plant on rel-salt at 99.46 test accuracy and badges-class on rel-stack at 82.83 (Gu et al., 13 Feb 2026).

Autocomplete regression is more heterogeneous. The GNN achieves 0.736 test $\text{nodes(node\_id)}$0 on transactions-price in rel-hm, 0.394 on beer_ratings-total_score in rel-ratebeer, and 0.394 on results-position in rel-f1, but some tasks remain difficult: review-rating on rel-amazon is negative for all methods, and studies-enrollment on rel-trial is near zero for all methods (Gu et al., 13 Feb 2026).

Forecasting results are similarly mixed but favorable in aggregate. On entity binary classification, average test AUROC is 71.79 for LightGBM and 78.19 for the GNN. On the new multiclass forecasting task author-category in rel-arxiv, test accuracy is 1.77 for Random, 9.09 for Majority, 2.01 for LightGBM, and 50.74 for the GNN. Regression shows genuine gains on tasks such as author-publication in rel-arxiv at 0.249 test $\text{nodes(node\_id)}$1 versus -0.210 for LightGBM and user-count in rel-ratebeer at 0.625 versus -0.170, but the paper also notes that some simpler baselines remain competitive or better, especially on rel-trial and driver-position in rel-f1 (Gu et al., 13 Feb 2026).

Recommendation is the clearest case where architecture matters. Average test MAP is 7.45 for Past Visit, 10.61 for Global Popularity, 13.96 for LightGBM, 4.15 for 2-layer GraphSAGE, 13.52 for 4-layer GraphSAGE, 17.48 for 2-layer ID-GNN, and 23.35 for 4-layer ID-GNN. The paper interprets this as evidence that recommendation is strongly multi-hop and that deeper, identity-aware models are especially effective (Gu et al., 13 Feb 2026).

6. Subsequent use, limitations, and significance

RelBench v2 rapidly became a target for model development rather than merely a passive benchmark. The paper “RelGT-AC: A Relational Graph Transformer for Autocomplete Tasks in Relational Databases” focuses explicitly on the new autocomplete setting and describes RelBench v2 as a standardized benchmark for RDL over multi-table databases that adds autocomplete tasks. That work formalizes autocomplete instances as row-table pairs $\text{nodes(node\_id)}$2 with targets of the form

$\text{nodes(node\_id)}$3

identifies direct leakage and correlation leakage, introduces seed-row masking, and evaluates on 7 autocomplete tasks spanning 3 RelBench v2 datasets. It reports that RelGT-AC outperforms the official GraphSAGE baseline on all 3 regression autocomplete tasks, while a TF-IDF text encoder yields large gains on text-heavy eligibility tasks in rel-trial, including +10.00 AUC on eligibilities-adult and +8.43 AUC on eligibilities-child relative to the no-TF-IDF ablation. At the same time, GraphSAGE remains stronger on some classification tasks, indicating that the autocomplete track is not saturated and that leakage handling, text encoding, and architecture design remain active research issues (Jiang, 2 Jun 2026).

The benchmark has several stated limitations. Some tasks remain intrinsically hard or poorly served by the current models: review-rating in rel-amazon remains poor under $\text{nodes(node\_id)}$4, some rel-trial regression tasks favor simple baselines, and extreme class imbalance can dominate cases such as sales-office. The paper also notes that it does not provide full formal task equations for all task families, explicit SQL label definitions, complete schema column listings for every dataset, or detailed candidate-generation procedures for recommendation beyond pairwise scoring. On the systems side, scaling remains challenging, as shown by memory failures for deeper recommendation models on large datasets (Gu et al., 13 Feb 2026).

Privacy and access constraints are especially salient for rel-mimic. The dataset is deidentified, access is credential-gated through PhysioNet, and the benchmark explicitly allows restricted subsets and controlled filters. This suggests a broader benchmark-design tension between realism and data governance, especially in healthcare and enterprise domains (Gu et al., 13 Feb 2026).

In the development of relational machine learning, RelBench v2 marks a shift from a narrowly defined forecasting benchmark to a broader relational-temporal evaluation platform. Its native datasets, autocomplete tasks, common training-table abstraction, and integrations with TGB, ReDeLEx, and 4DBInfer make it both a benchmark and a repository for relational data. The paper’s stated significance is that this infrastructure supports progress toward scalable, pretrained, and transferable relational learning systems, while empirical results indicate that modeling relational structure explicitly is often necessary to exceed single-table baselines (Gu et al., 13 Feb 2026).