OpenML-CTR23 Benchmark Suite

• OpenML-CTR23 is a benchmark suite curated from 34–35 real-world tabular regression datasets that assesses model performance and explainability.
• It utilizes minimal preprocessing and fixed train-test splits to maintain data integrity across various domains like energy prediction and materials science.
• The suite supports multi-task regression and foundation model evaluations while quantifying uncertainty through innovative Rashomon interval metrics.

OpenML-CTR23 is a publicly curated benchmark suite designed to evaluate machine learning models and explainability frameworks in the context of real-world tabular regression problems. Comprising 34–35 heterogeneous datasets with diverse domains, scales, and statistical characteristics, OpenML-CTR23 serves as the de facto evaluation resource for recent advances in foundation models for tabular data, multi-task regression architectures, and uncertainty-aware explainable AI pipelines. Major studies—including Cavus et al. (2025) on trustworthy explanations and the multi-task “basis transformer” model—have established the centrality of this suite for benchmarking algorithmic progress and quantifying the limitations of single-model perspectives in both prediction and interpretability (Cavus et al., 19 Jul 2025, Loh et al., 7 Jun 2025).

1. Composition and Statistical Coverage of OpenML-CTR23

OpenML-CTR23 encompasses 34–35 real-world regression tasks (the precise count depends on implementation choices, with one dataset occasionally omitted for technical reasons). The benchmark was constructed to ensure wide coverage of:

• Problem domains: Building-energy prediction, materials science, housing price estimation, wage prediction, sports analytics, insurance risk, signal processing, environmental modeling, and more.
• Dataset sizes: Ranging from several hundred to over 50,000 instances.
• Feature dimensionalities ($p$): Spanning 3 to over 400 predictors.
• Target distributions: Including approximately symmetric, heavily right-skewed, and heavily-tailed targets.

The datasets vary from tabular collections with a handful of columns (e.g., airfoil_self_noise: $p=6$) to high-dimensional problems such as superconductivity ($p=81$) and geographical_origin_of_music ($p=68$). The metadata ensures representation of various real-world data challenges: missingness, categorical and textual features, noisy measurements, and high dynamic range in numeric columns (Cavus et al., 19 Jul 2025, Loh et al., 7 Jun 2025).

2. Preprocessing Protocols and Data Splits

Both foundational studies employing OpenML-CTR23 adhere to a minimal-preprocessing philosophy designed to minimize information leakage and domain-specific bias:

• Feature handling: No hand-crafted feature transformations or domain-intrinsic encoding. In H2O AutoML-based pipelines, numeric and categorical variables are handled using built-in mechanisms, with missing values automatically imputed or encoded.
• Input representation (for neural models): Data is modeled as an unordered set of $(column\_name, entry\_value)$ pairs with no explicit one-hot encoding of categories and no normalization of numeric values. For missing data, a distinguished “missing” token is embedded as a learnable parameter of the model.
• Data splits: For each dataset, hold-out splits assign 80% of data to training and 20% to the test set (AutoML). In multi-task model protocols, test and validation splits are sized relative to the smallest dataset, and all splits are fixed for reproducibility. No cross-validation is performed to ensure consistent train–test segmentation across tasks (Cavus et al., 19 Jul 2025, Loh et al., 7 Jun 2025).

3. Benchmark Utilization in Trustworthy XAI and Model Multiplicity

OpenML-CTR23 is central to empirical evaluations of model explanation stability and uncertainty:

• Rashomon-set approach: Cavus et al. define the $\varepsilon$-Rashomon set $R_{0.05}$ for each task as all models $M_k$ such that $\phi(M_k) \leq (1+\varepsilon)\phi(M^*)$, where $\phi$ is test RMSE and $p=6$0.
• Explanation variability quantification: Partial dependence profiles (PDPs) $p=6$1 are computed for each feature and model, and their aggregation defines the Rashomon PDP $p=6$2.
• Uncertainty metrics: The benchmark introduces two signature metrics for explanation comparison,
  • Coverage Rate (CR):

  $p=6$3 - Mean Width of Confidence Intervals (MWCI):

  $p=6$4

  where $p=6$5 is the pointwise Rashomon PDP interval, and $p=6$6 is the single-best model's PDP.

Table: Example empirical results for several OpenML-CTR23 datasets (Cavus et al., 19 Jul 2025):

Dataset	$p=6$7	MWCI	CR
abalone	13	0.56	0.42
california_housing	6	8782.10	0.27
forest_fires	21	5.90	0.14
concrete_compressive_str	5	0.61	0.69

The benchmark reveals that in 58% of applicable datasets, the Rashomon interval covers less than 70% of the single-best PDP—highlighting that traditional pointwise explanations often miss key epistemic uncertainty, especially when the Rashomon set is large (correlation $p=6$8 between Rashomon set size and PDP coverage) (Cavus et al., 19 Jul 2025).

4. Multi-Task Tabular Regression and Foundation Model Evaluation

OpenML-CTR23 is the standard suite for quantitatively benchmarking large-scale, multi-task, and foundation models for tabular regression:

• Basis Transformer architecture: The benchmark is used to evaluate the "basis transformer" (BT), a neural architecture fulfilling six desiderata for tabular data, such as heterogeneity, column-order invariance, and numeric scale preservation.

• Input encoding: Numeric values use sign-magnitude representations (44 bits), text and column names use distilled BERT embeddings.

• Evaluation protocol: All 34 tasks are trained simultaneously; model selection is via validation mean $p=6$9, and final evaluation is on held-out test splits of all datasets.

• Baselines: Large pretrained LLMs (Flan-T5, BART, Pythia, Cerebras-GPT) are fine-tuned via the TabLLM protocol for comparison. No explicit feature engineering or standardization is applied anywhere in the pipeline.

• Aggregate metrics:

  • Per-task $p=81$0: $p=81$1.
  • Central tendency: Median $p=81$2 across tasks.
  • Spread: Interquartile range, standard deviation of $p=81$3 across tasks.

Summary of baseline results (Loh et al., 7 Jun 2025):

Model	Median $p=81$4	Std Dev
Basis Transformer	0.241	3.295
Cerebras-GPT-1.3B	–0.097	566.072
Flan-T5-base	–0.280	4.941

The BT achieves a median $p=81$5 of $p=81$6, a $p=81$7 improvement over the next best baseline, with a parameter count of 80M—$p=81$8 smaller than Cerebras-GPT-1.3B (1.3B parameters).

5. Implementation Details and Model Design Patterns

The benchmark enforces strict evaluation and implementation constraints for all participants:

• Parameter and memory efficiency: Model architectures must contend with memory-intensive 4D tensor representations and highly variable column counts across tasks.
• Loss function: Multi-label binary cross-entropy over 44 sign-magnitude bits for target regression.
• Adaptive loss re-weighting: Gradient updates are dynamically skewed toward “hard” examples using a learnable scalar $p=81$9 (set to 0.2).
• Preprocessing avoidance: No one-hot encoding or standard scaling, and textual columns are directly embedded via BERT. Practitioners are advised to cache text embeddings for VRAM efficiency in large-scale foundation setups (Loh et al., 7 Jun 2025).

6. Limitations and Interpretive Implications

Comprehensive findings using OpenML-CTR23 uncover systematic risks and opportunities:

• Interpretive instability: Explanations derived from a single-best model often underestimate epistemic uncertainty in feature effects, as multi-model Rashomon PDPs may disagree substantially—especially for tasks with large near-optimal sets (Cavus et al., 19 Jul 2025).
• Domain boundaries: While basis transformers excel in multi-task, heterogeneous tabular contexts, they exhibit high memory footprints, and may underperform (relative to GBDTs) on very small tasks or data with highly frequent tokens.
• No hand-tuning: The lack of domain-specific tuning or feature engineering is both a strength (measured generalization and method comparability) and a limitation (potentially leaving untapped improvements in specific domains).
• Usage guidance: Practitioners are encouraged to maintain numeric precision via sign-magnitude encodings, leverage column names for semantic transfer, and use Rashomon interval visualizations to inform trust calibration, especially in high-stakes domains (Loh et al., 7 Jun 2025, Cavus et al., 19 Jul 2025).

7. Impact and Benchmark Evolution

OpenML-CTR23 has accelerated both rigorous model comparison and the development of uncertainty-aware explainable AI by providing a diverse, transparent, and reproducible standard for tabular regression. Its formalization of interpretive metrics (coverage rate, Rashomon set analysis), paired with extensibility to multi-task learning protocols, positions it as a cornerstone resource for ongoing methodological advances in both automated machine learning and XAI research (Cavus et al., 19 Jul 2025, Loh et al., 7 Jun 2025).