---
title: OpenML-CTR23 Benchmark Suite
url: https://www.emergentmind.com/topics/openml-ctr23
type: topic
---

# OpenML-CTR23 Benchmark Suite

OpenML-CTR23 is a publicly curated benchmark suite designed to evaluate machine learning models and explainability frameworks in the context of real-world tabular regression problems. Comprising 34–35 heterogeneous datasets with diverse domains, scales, and statistical characteristics, OpenML-CTR23 serves as the de facto evaluation resource for recent advances in foundation models for tabular data, multi-task regression architectures, and uncertainty-aware explainable AI pipelines. Major studies—including Cavus et al. (2025) on trustworthy explanations and the multi-task “basis transformer” model—have established the centrality of this suite for benchmarking algorithmic progress and quantifying the limitations of single-model perspectives in both prediction and interpretability [2507.14744][2506.06926].

## 1. Composition and Statistical Coverage of OpenML-CTR23

OpenML-CTR23 encompasses 34–35 real-world regression tasks (the precise count depends on implementation choices, with one dataset occasionally omitted for technical reasons). The benchmark was constructed to ensure wide coverage of:

- **Problem domains:** Building-energy prediction, materials science, housing price estimation, wage prediction, sports analytics, insurance risk, signal processing, environmental modeling, and more.
- **Dataset sizes:** Ranging from several hundred to over 50,000 instances.
- **Feature dimensionalities ($p$):** Spanning 3 to over 400 predictors.
- **Target distributions:** Including approximately symmetric, heavily right-skewed, and heavily-tailed targets.

The datasets vary from tabular collections with a handful of columns (e.g., airfoil_self_noise: $p=6$) to high-dimensional problems such as superconductivity ($p=81$) and geographical_origin_of_music ($p=68$). The metadata ensures representation of various real-world data challenges: missingness, categorical and textual features, noisy measurements, and high dynamic range in numeric columns [2507.14744][2506.06926].

## 2. Preprocessing Protocols and Data Splits

Both foundational studies employing OpenML-CTR23 adhere to a minimal-preprocessing philosophy designed to minimize information leakage and domain-specific bias:

- **Feature handling:** No hand-crafted feature transformations or domain-intrinsic encoding. In H2O AutoML-based pipelines, numeric and categorical variables are handled using built-in mechanisms, with missing values automatically imputed or encoded.
- **Input representation (for neural models):** Data is modeled as an unordered set of $(column\_name, entry\_value)$ pairs with no explicit one-hot encoding of categories and no normalization of numeric values. For missing data, a distinguished “missing” token is embedded as a learnable parameter of the model.
- **Data splits:** For each dataset, hold-out splits assign 80% of data to training and 20% to the test set (AutoML). In multi-task model protocols, test and validation splits are sized relative to the smallest dataset, and all splits are fixed for reproducibility. No cross-validation is performed to ensure consistent train–test segmentation across tasks [2507.14744][2506.06926].

## 3. Benchmark Utilization in Trustworthy XAI and Model Multiplicity

OpenML-CTR23 is central to empirical evaluations of model explanation stability and uncertainty:

- **Rashomon-set approach:** Cavus et al. define the $\varepsilon$-Rashomon set $R_{0.05}$ for each task as all models $M_k$ such that $\phi(M_k) \leq (1+\varepsilon)\phi(M^*)$, where $\phi$ is test RMSE and $\varepsilon=0.05$.
- **Explanation variability quantification:** Partial dependence profiles (PDPs) $\hat f_j^{(k)}(x)$ are computed for each feature and model, and their aggregation defines the Rashomon PDP $\bar f_j(x)$.
- **Uncertainty metrics:** The benchmark introduces two signature metrics for explanation comparison,
  - **Coverage Rate (CR):**
    $$
    \mathrm{CR}_j = \frac{1}{n_x} \sum_{\ell=1}^{n_x} \mathbf{1} \{ f_j(x_\ell) \in [ L_j(x_\ell), U_j(x_\ell)] \}
    $$
  - **Mean Width of Confidence Intervals (MWCI):**
    $$
    \mathrm{MWCI}_j = \frac{1}{n_x} \sum_{\ell=1}^{n_x} [ U_j(x_\ell) - L_j(x_\ell) ]
    $$
    where $[L_j(x), U_j(x)]$ is the pointwise Rashomon PDP interval, and $f_j(x)$ is the single-best model's PDP.

Table: Example empirical results for several OpenML-CTR23 datasets [2507.14744]:

| Dataset                  | $|R_{0.05}|$ | MWCI      | CR    |
|--------------------------|------------|------------|-------|
| abalone                  |     13     |   0.56     | 0.42  |
| california_housing       |      6     | 8782.10    | 0.27  |
| forest_fires             |     21     |   5.90     | 0.14  |
| concrete_compressive_str |      5     |   0.61     | 0.69  |

The benchmark reveals that in 58% of applicable datasets, the Rashomon interval covers less than 70% of the single-best PDP—highlighting that traditional pointwise explanations often miss key epistemic uncertainty, especially when the Rashomon set is large (correlation $\rho = -0.53$ between Rashomon set size and PDP coverage) [2507.14744].

## 4. Multi-Task Tabular Regression and Foundation Model Evaluation

OpenML-CTR23 is the standard suite for quantitatively benchmarking large-scale, multi-task, and foundation models for tabular regression:

- **Basis Transformer architecture:** The benchmark is used to evaluate the "basis transformer" (BT), a neural architecture fulfilling six desiderata for tabular data, such as heterogeneity, column-order invariance, and numeric scale preservation.
- **Input encoding:** Numeric values use sign-magnitude representations (44 bits), text and column names use distilled BERT embeddings.
- **Evaluation protocol:** All 34 tasks are trained simultaneously; model selection is via validation mean $R^2$, and final evaluation is on held-out test splits of all datasets.
- **Baselines:** Large pretrained LLMs (Flan-T5, BART, Pythia, Cerebras-GPT) are fine-tuned via the TabLLM protocol for comparison. No explicit feature engineering or standardization is applied anywhere in the pipeline.
- **Aggregate metrics:**
  - **Per-task $R^2$**: $R^2_t = 1 - \frac{\sum (y_i - \hat{y}_i)^2}{\sum (y_i - \bar{y})^2}$.
  - **Central tendency:** Median $R^2$ across tasks.
  - **Spread:** Interquartile range, standard deviation of $R^2$ across tasks.

Summary of baseline results [2506.06926]:

| Model                 | Median $R^2$ | Std Dev |
|-----------------------|--------------|---------|
| Basis Transformer     |   0.241      |  3.295  |
| Cerebras-GPT-1.3B     |  –0.097      | 566.072 |
| Flan-T5-base          |  –0.280      |  4.941  |

The BT achieves a median $R^2$ of $0.241$, a $0.338$ improvement over the next best baseline, with a parameter count of 80M—$5\times$ smaller than Cerebras-GPT-1.3B (1.3B parameters).

## 5. Implementation Details and Model Design Patterns

The benchmark enforces strict evaluation and implementation constraints for all participants:

- **Parameter and memory efficiency:** Model architectures must contend with memory-intensive 4D tensor representations and highly variable column counts across tasks.
- **Loss function:** Multi-label binary cross-entropy over 44 sign-magnitude bits for target regression.
- **Adaptive loss re-weighting:** Gradient updates are dynamically skewed toward “hard” examples using a learnable scalar $\gamma$ (set to 0.2).
- **Preprocessing avoidance:** No one-hot encoding or standard scaling, and textual columns are directly embedded via BERT. Practitioners are advised to cache text embeddings for VRAM efficiency in large-scale foundation setups [2506.06926].

## 6. Limitations and Interpretive Implications

Comprehensive findings using OpenML-CTR23 uncover systematic risks and opportunities:

- **Interpretive instability:** Explanations derived from a single-best model often underestimate epistemic uncertainty in feature effects, as multi-model Rashomon PDPs may disagree substantially—especially for tasks with large near-optimal sets [2507.14744].
- **Domain boundaries:** While basis transformers excel in multi-task, heterogeneous tabular contexts, they exhibit high memory footprints, and may underperform (relative to GBDTs) on very small tasks or data with highly frequent tokens.
- **No hand-tuning:** The lack of domain-specific tuning or feature engineering is both a strength (measured generalization and method comparability) and a limitation (potentially leaving untapped improvements in specific domains).
- **Usage guidance:** Practitioners are encouraged to maintain numeric precision via sign-magnitude encodings, leverage column names for semantic transfer, and use Rashomon interval visualizations to inform trust calibration, especially in high-stakes domains [2506.06926][2507.14744].

## 7. Impact and Benchmark Evolution

OpenML-CTR23 has accelerated both rigorous model comparison and the development of uncertainty-aware explainable AI by providing a diverse, transparent, and reproducible standard for tabular regression. Its formalization of interpretive metrics (coverage rate, Rashomon set analysis), paired with extensibility to multi-task learning protocols, positions it as a cornerstone resource for ongoing methodological advances in both automated machine learning and XAI research [2507.14744][2506.06926].

Source: https://www.emergentmind.com/topics/openml-ctr23