R2-Bench: Multi-Domain ML Benchmarking

• R2-Bench is a multi-faceted benchmark suite evaluating text-to-image reasoning, LLM routing cost-quality tradeoffs, and relational data modeling across diverse scenarios.
• It rigorously tests models using detailed metrics like R2IScore and AUDC, employing methodologies such as human-in-the-loop curation and LLM-driven evaluations.
• The framework guides advancements in hybrid neuro-symbolic architectures, dynamic routing strategies, and optimal graph construction in relational databases.

R2-Bench refers to multiple distinct benchmarks within machine learning, each addressing a different aspect of reasoning, routing, or relational data modeling. The term appears in three principal contexts: (1) as the large-scale benchmarking suite for reasoning-driven text-to-image (T2I) generation, (2) as the first dataset for quality–cost profiling in LLM routing, and (3) as shorthand for large-scale relational data benchmarks (notably RelBench v2 and RDB2G-Bench) for multi-table predictive tasks and graph construction. While these usages are independent and stem from distinct research efforts, all share a focus on rigorous, large-scale, scenario-driven evaluation.

1. Reasoning-to-Image Benchmark: R2I-Bench

R2I-Bench (Chen et al., 29 May 2025) is the first large-scale evaluation suite explicitly designed to rigorously assess reasoning-driven capabilities in text-to-image models. Motivated by the gap between photorealism and reasoning competence, R2I-Bench tests model performance on prompts requiring structured inference rather than shallow text-image association.

Reasoning Axes and Dataset Structure

The dataset covers seven reasoning categories: commonsense, mathematical, logical, compositional, numerical, causal, and concept-mixing. Each is refined into 32 subcategories, resulting in 3,068 prompts distributed as follows:

Category	#Prompts	%Total
Mathematical	800	26.1
Commonsense	695	22.7
Logical	630	20.5
Compositional	311	10.1
Numerical	322	10.5
Concept-Mixing	159	5.2
Causal	151	4.9

Prompt generation involved expert review, in-context GPT-4o prompt synthesis, and iterative human-in-the-loop filtering until each subcategory reached a high-quality instance quota.

R2IScore Metric

Each instance is paired with a reference caption, human-curated reasoning trace, and a set of QA-style diagnostic questions tagged as alignment, reasoning, or quality. Each question $i$ has a score $s_i$ and weight $w_i$. The aggregate score is

$$\mathrm{R2IScore} = \frac{\sum_{i=1}^{n} w_i\,s_i}{\sum_{i=1}^{n} w_i}$$

or, partitioned by scoring dimension $(w_A, w_R, w_Q)$ for alignment $A_j$, reasoning $R_j$, quality $Q_j$:

$$\mathrm{R2IScore} = \frac{1}{N} \sum_{j=1}^N (w_A A_j + w_R R_j + w_Q Q_j)$$

GPT-4o (temperature 0.1) provides per-question scoring in a reference-free, QA-driven protocol.

Key Experimental Outcomes

Benchmarked models include diffusion (Stable-Diffusion 3 medium, Lumina-Image 2.0), autoregressive (EMU3, Janus-Pro-7B), CoT- or RL-enhanced, closed-source APIs (DALL·E 3, GPT-Image-1), and a GPT-4o + SD3-medium pipeline. All operate in zero-shot mode.

Performance is substantially below ceiling for most reasoning axes, especially mathematical (0.07–0.19, open-source), and numerical tasks (≈0.35–0.60). GPT-4o + SD3-medium pipeline outperforms vanilla models in several axes (overall 0.58 vs. 0.45 for SD3-medium), but even so, abstract reasoning remains weak. CoT/RL model augmentations offer marginal improvement (<0.05 gain overall).

Common error modes include (1) surface-level “bag-of-words” rendering, (2) missing objects/attributes, and (3) secondary image degradations. Core failure points stem from CLIP-based encoders' inability to represent structured logical relations, insufficient symbolic grounding, and lack of math-annotated supervision.

Implications and Directions

R2I-Bench and R2IScore establish the reasoning gap in T2I generation and motivate research integrating explicit symbolic modules, mathematical corpora, RLHF on reasoning-centric metrics, and broader modalities (video/3D reasoning) (Chen et al., 29 May 2025).

2. LLM Routing and R2-Bench for Output Length Profiling

R2-Bench in the context of LLM routing (Xue et al., 2 Feb 2026) refers to the first systematic routing dataset capturing per-model, per-query quality–cost tradeoffs as a function of output token budgets. Unlike traditional single-point routing benchmarks, R2-Bench profiles 15 open-source LLMs over 16 length constraints across 30,968 queries from six established tasks (GPQA, MuSR, MMLU-Pro, MATH, OpenHermes, RAGBench), totaling ≈7.4 million (query, model, budget) entries.

Dataset Features and Protocol

For each query, model, and budget $b_k$, cost is $s_i$0 and true quality $s_i$1 is scored by a high-correlation LLM judge (Qwen3-80B-Instruct, $s_i$2 human concordance).

By transforming the quality–cost mapping from static points to entire curves, R2-Bench enables the training and fair comparison of reasoning-based routers. These routers simultaneously choose which LLM to call and at what token budget:

$s_i$3

Smooth interpolation between budgets is supported.

Metrics

• Area Under the Deferral Curve (AUDC): Captures full quality–cost efficiency as $s_i$4 is swept.
• Peak Quality: Maximum mean quality (low cost aversion, $s_i$5).
• Query-Normalized Cost (QNC): Cost needed to achieve best static-model quality.

Impact and Significance

R2-Bench gives a ≈15% jump in the routing oracle upper bound over single-point datasets. R2-Router, when trained on R2-Bench, achieves comparable quality to prior approaches at 4–5$s_i$6 lower cost and scales to dynamic-pool settings. The framework encourages a new paradigm—routing as structured reasoning rather than reactive selection—potentially generalizing to output variables beyond length (e.g., decoding strategies, reasoning depth) (Xue et al., 2 Feb 2026).

3. Relational Data Modeling: R2-Bench (RelBench v2 and RDB2G-Bench)

In relational deep learning, R2-Bench appears as an alternate designation for two interconnected benchmarks:

RelBench v2: Multi-table Relational Benchmarks

RelBench v2 (“R2-Bench”) (Gu et al., 13 Feb 2026) significantly expands machine learning on relational databases by introducing four new, large-scale datasets (scholarly publications, enterprise ERP, consumer reviews, clinical records), aggregating to 22M+ rows and 29 tables over 11 datasets. The benchmark supports both forecasting and a novel class of autocomplete tasks: inferring missing attributes in new rows at time $s_i$7 under leakage prevention.

• Autocomplete (classification/regression):

$s_i$8

• Forecasting (classification/regression/recommendation):

Targets are SQL-derived, temporally-constrained, and entity-centric (e.g., citations, churn, publication counts).

Integration with external benchmarks (TGB, ReDeLEx, 4DBInfer) enables 4D design space evaluation (datasets, tasks, schema-to-graph strategies, model families). Relational GNNs (GraphSAGE, ID-GNN) consistently outperform LightGBM and trivial baselines across tasks. Deep GNNs and node-identity-aware models bring gains on multi-hop or complex targets.

RDB2G-Bench: Graph Construction for RDBs

RDB2G-Bench (Choi et al., 2 Jun 2025) is the first benchmark framework for evaluating automatic mapping strategies from RDB schemas to graphs for downstream learning. It provides ≈50,000 graph–performance pairs from 5 RDBs and 12 predictive tasks.

Graph Construction Decision Space

Given schema $s_i$9 and instance $w_i$0, a graph model $w_i$1 selects included tables/foreign keys and chooses for each whether to represent as nodes or edges. Connectivity to the task table within $w_i$2 GNN hops is required.

Modeled Methods and Results

Nine methods are compared:

• Heuristic: All-Rows-to-Nodes (AR2N), Random Search
• Greedy: Forward, Backward, Local
• Black-box: Evolutionary Algorithm, Bayesian Optimization, Reinforcement Learning (LSTM controller)
• LLM-based: Multi-turn sequence planning

Greedy methods (especially backward) excel under tight evaluation budgets, while Evolutionary and Bayesian search catch up with more iterations. Task specificity is critical; effective mappings for one target may degrade others (low cross-task rank correlation). Optimal graph construction can yield up to 10% performance gain over AR2N and substantial efficiency improvements. The precomputed dataset enables 600$w_i$3 faster benchmarking versus on-the-fly retraining.

Practical Guidelines

• Avoid naïve “all-rows-to-nodes” mappings; task-specific, connectivity-aware, and sometimes edge-based modeling of two-FK tables can deliver superior performance.
• Greedy strategies suffice for rapid progress under constrained resources.
• LLM planning is effective for initial exploration but plateaus without further search (Choi et al., 2 Jun 2025).

4. Benchmarking Methodologies and Evaluation Protocols

Across usages, R2-Bench benchmarks are characterized by:

• Comprehensive scenario coverage: Spanning subcategories, token budgets, or relational schemas.
• Multi-dimensional, closed-form metrics: E.g., R2IScore's QA-based scoring, AUDC/QNC for routing, AUROC/MAE/MAP for graph modeling.
• Automation and scale: Human-in-the-loop curation (T2I), LLM-as-judge annotation (routing), precomputed model–metric pairs (relational).
• Zero-shot and controlled evaluation: No fine-tuning, fixed hyperparameter schedules, and deterministic inference where applicable.
• Standardized comparisons: Explicit ranking of baselines and advanced methods.

5. Impact, Limitations, and Prospects

R2-Bench suites collectively (1) expose weaknesses in current reasoning, routing, and relational modeling paradigms, (2) provide actionable metrics for methodological advances, and (3) motivate development of models and data that are robust to abstract inference requirements, dynamic output constraints, and complex data relationality.

Current limitations are evident: structured reasoning remains a challenge for T2I and LLM models even with pipeline-based or RLHF augmentation; relational models are not yet fully leveraging foundation model transfer; and mapping from schema to optimal graph is still heavily task-dependent and problem-specific.

Future directions identified include: hybrid neuro-symbolic architectures, richer annotated supervisory data, controlled evaluation on broader modalities (multilingual, 3D, temporal), and further integration of dynamic, streaming, or federated data sources (Chen et al., 29 May 2025, Xue et al., 2 Feb 2026, Gu et al., 13 Feb 2026, Choi et al., 2 Jun 2025).