TabArena: Living Benchmark for Tabular ML

• TabArena is a living benchmark system for tabular data that continuously integrates new datasets, models, and evaluation methods.
• It employs rigorous dataset curation and standardized pipelines to ensure fair, reproducible comparisons of traditional and deep learning approaches.
• The platform’s transparent protocols and public leaderboards foster community contributions and drive continuous improvements in model evaluation.

TabArena is a continuously maintained, “living” benchmarking system designed to provide rigorous, up-to-date evaluation of machine learning models on tabular data. Unlike static benchmarks that are seldom revised to incorporate new datasets, models, or evaluation best practices, TabArena’s protocols, curations, and leaderboards are actively managed and publicly accessible. The system aims to set a reproducible and transparent foundation for assessing current and emerging techniques in tabular learning, including traditional methods, deep learning approaches, and foundation models (2506.16791).

1. Motivation and Design Principles

TabArena was conceived in response to persistent issues with prior tabular benchmarking efforts:

• Many earlier benchmarks employed outdated or inconsistent datasets, suffered from licensing and split-protocol problems, and often failed to incorporate corrections or additions as new methods and data emerged.
• Static design in earlier benchmarks allowed flaws and inconsistencies to persist, and model evaluations quickly became obsolete as model classes and implementation details advanced.

To address these deficiencies, TabArena is explicitly “living”: datasets, curation protocols, hyperparameter settings, and the evaluation framework are subject to continual update and community contribution. The purpose is to ensure fair, reproducible, and comprehensive comparison of models under up-to-date conditions (2506.16791).

2. Dataset Curation and Benchmark Scope

TabArena’s dataset collection is built upon strict selection criteria designed to promote reliability and scientific reuse:

• Out of more than 1,000 datasets considered, 51 were manually selected to satisfy requirements of real-world prediction relevance, IID tabular structure, and appropriate licensing.
• Tasks cover classification and regression across diverse domains, with detailed metadata indicating task structure, instance counts, class balance, and feature types.

TabArena’s curation policies explicitly exclude datasets with issues such as duplicates, irrelevant or extremely redundant features, ambiguous targets, or problematic data splits. This selection procedure is intended to foster reproducible science and guard against overfitting to benchmark peculiarities.

3. Model Collection, Standardization, and Execution

TabArena includes 16 “native” tabular models for initial release, spanning:

• Gradient-boosted decision tree ensembles (e.g., CatBoost, LightGBM, XGBoost)
• Deep neural architectures for tabular prediction (e.g., RealMLP, TabM, ModernNCA)
• Tabular “foundation models” (e.g., TabPFNv2, TabICL)

All model code is implemented in a standardized pipeline based on the AutoGluon framework and an extendable AbstractModel API. The pipeline supports:

• Unified model-agnostic preprocessing (categorical, numeric, and date/text handling),
• Time-aware procedures such as early stopping,
• Model-specific data transformations,
• Consistency of interface for training, prediction, and evaluation.

Each model is benchmarked in three regimes: default hyperparameters (as shipped), tuned hyperparameters (with rigorous, model-specific random search), and ensembles constructed as weighted combinations of the best configurations.

4. Evaluation Methodology and Leaderboard Construction

TabArena employs a highly rigorous and transparent evaluation protocol:

• All runs are executed using repeated, nested 8-fold cross-validation (with additional repetitions on smaller datasets).
• Performance metrics are task-appropriate: ROC AUC for binary classification, log loss for multiclass, and RMSE for regression.
• Model performance is aggregated and reported using an Elo rating system, with bootstrapped 95% confidence intervals; the Elo scale is calibrated so a 400-point gap corresponds to a 10:1 “win” probability.
• Public leaderboards at https://tabarena.ai expose all results, including metrics, training and inference time, confidence intervals, and detailed metadata.

A central feature is post-hoc ensembling across hyperparameter configurations: rather than reporting only the best single run, TabArena combines predictions using weighted ensembles, revealing the full peak performance potential of each model class. This practice demonstrated that many models, especially deep learning approaches, close the gap to or surpass traditional GBDT baselines only when tuning and ensembling are properly applied.

5. Performance Findings and Model Comparisons

Initial large-scale benchmarking—encompassing over 25 million individual training/evaluation runs across 51 datasets—produced several key findings:

• Gradient-boosted trees (GBDTs) such as CatBoost remain highly competitive under default and tuned settings, as found in most practical scenarios.
• When deep learning models are hyperparameter-tuned and ensembled, they match or exceed GBDT performance on several datasets, especially as compute budgets increase.
• Foundation models for tabular data demonstrate particularly strong results on small datasets and within size/feature limits.
• Holdout validation (as used in some prior benchmarks) severely underestimates the true potential of models compared to correct repeated cross-validation procedures.
• Ensemble combinations—either across hyperparameter settings or across distinct model families—consistently advance the state of the art, shifting emphasis away from “single best model” narratives to multi-model hybrid strategies.

6. Community Resources, Transparency, and Protocol Evolution

TabArena provides comprehensive access to:

• All experimental code, meta-data, and documentation for replication and community extension,
• A public leaderboard with confidence intervals, resource usage statistics, and analytic breakdowns,
• Contribution protocols for model/dataset addition, subject to rigorous curation and review.

The system continuously incorporates feedback and updates, distinguishing it from static benchmarks. This “living” nature is central to maintaining up-to-date relevance as new data, models, and evaluation methodologies are released.

7. Future Directions and Extension Plans

TabArena version 0.1 focuses on classification and regression for IID tabular datasets of small-to-medium size. Planned and proposed future directions include:

• Expansion to non-IID data, including temporal or grouped datasets,
• Inclusion of tiny (few-shot) and very large tabular settings, potentially requiring new optimization and evaluation strategies,
• Benchmarks for additional tasks, such as survival analysis or anomaly detection,
• Integration of next-generation models and continuous refinement of hyperparameter search spaces and cross-validation strategies.

As a living benchmark, TabArena is positioned to assimilate new modeling paradigms (including, plausibly, generative tabular models, data imputation approaches, and fairness-aware frameworks) and emerging community standards, maintaining its role as a central resource for robust tabular learning evaluation (2506.16791).