---
title: OpenML-CC18 Benchmark Suite
url: https://www.emergentmind.com/topics/openml-cc18
type: topic
---

# OpenML-CC18 Benchmark Suite

OpenML-CC18 is a curated benchmarking suite of 72 real-world supervised classification tasks, introduced to promote reproducibility, comparability, and methodological rigor in empirical machine learning. Developed within the OpenML ecosystem, the suite targets the evaluation of algorithms under standardized settings, spanning diverse domains, sample sizes, feature types, and class cardinalities. It has become a critical reference point for large-scale algorithm comparison, meta-learning, active learning, AutoML, and tabular foundation model evaluation, supporting systematic advancement in supervised learning on heterogeneous tabular data [1708.03731].

## 1. Construction, Curation Criteria, and Statistical Diversity

OpenML-CC18 was assembled via strict, multi-stage screening of OpenML’s public datasets to ensure variation in data properties and maintain experimental feasibility. The selection pipeline [1708.03731] enforced:
- **Size constraints:** $500 \leq n \leq 100{,}000$ (number of examples), $d_{\text{one-hot}} \leq 5{,}000$ (post-encoding feature count).
- **Class structure:** $k\geq 2$; each class must have $\geq 20$ examples, and minority/majority ratio $\geq 0.05$.
- **Task realism:** exclusion of artificial/simulated datasets, tasks solvable by a single feature or Decision Tree, and data lacking public provenance.
- **No regression, clustering, time-series, or grouped designs.**

The final benchmark includes both binary and multiclass tasks (classes up to $k=50$), with feature sets ranging from $d=5$ to $3,000+$. Domains comprise healthcare, finance, biomedicine, sensor data, vision (e.g., MNIST, CIFAR-10), and engineered datasets. Class imbalances vary, as do proportions of numeric/categorical attributes and rates of missingness [1708.03731; 2110.08483].

| # Tasks | $n$ Range | # Features ($d$) | Classes ($k$)      | Domains                  |
|---------|-----------|------------------|--------------------|--------------------------|
| 72      | 500–$10^5$| 5–3,000+         | 2–50               | Tabular, vision, sensor  |

A subset of tasks may be filtered for specialized studies (e.g., only binary, only ≤100 features) [2402.01821].

## 2. Metadata, Access, and Evaluation Protocols

Each OpenML-CC18 task is uniquely defined by a dataset, target column, evaluation protocol (standard is 10-fold cross-validation with predefined splits), and meta-information fields (≈70 per set) such as class entropy, imbalance ratios, instance counts, feature modalities, and missing value prevalence [1708.03731; 2110.08483].

OpenML exposes CC18 via Python, Java, and R APIs with standardized ARFF (and CSV) formats. Code snippets permit batch downloading, training, prediction, evaluation, and result upload with a few lines [1708.03731]. Meta-feature fields enable fine-grained programmatic filtering to assemble custom benchmark subsets.

By construction, all tasks provide identical preprocessing: categorical features are one-hot or ordinal-encoded, missing values are imputed (mode or mean as appropriate), and evaluation splits are consistent across all clients [1708.03731; 2110.08483; 2601.09654].

## 3. Benchmark Use Cases and Influence on Research Methodologies

OpenML-CC18 is the de facto standard for:
- **Algorithm comparisons:** Quantitative head-to-head studies of classical classifiers (RF, SVM, MLP), ensemble models, decision forests, and deep learning [2007.14870; 2110.08483; 1708.03731].
- **Meta-learning and AutoML:** Systematic meta-feature extraction for algorithm selection, automated hyperparameter optimization, and transfer/meta-learning [1909.13576; 2402.01821].
- **Active learning:** Large-scale analysis of acquisition functions and pre-training approaches on tabular classification [2210.03822].
- **Robust evaluation:** Item Response Theory (IRT) and Glicko-2 ratings quantify dataset difficulty and classifier ability, allowing for tailored subset selection to either stress classifier limits or enable tight pairwise algorithm differentiation [2007.14870; 2107.07451].
- **Tabular foundation models and fine-tuning:** Pretrained TFMs are systematically compared in zero-shot, meta-learned, SFT, and parameter-efficient adaptation regimes [2601.09654].

Researchers frequently construct focused CC18 subsets to control for domain, dimensionality, class balance, or label cardinality for targeted benchmarking [2402.01821].

## 4. Empirical Insights from Systematic Benchmarking

Aggregate analyses over CC18 reveal:
- **Dataset stratification:** Majority of CC18 sets are “easy”—over 80% of instances in 49/60 sets have IRT difficulty $b_i<1$, and only ~10% of CC18 tasks are predominantly difficult. High-discrimination tasks are more common than genuinely hard ones [2007.14870; 2107.07451].
- **Classifier ranking:** Glicko-2 meta-ratings computed over CC18 favor tree-based ensembles (Random Forest, MLPClassifier) on average; random/perfect/pessimistic baselines define ability extremes [2007.14870].
- **Acquisition function benchmarking:** Across 69 tabular CC18 tasks, margin sampling matches or outperforms AL baselines—CoreSet, BALD, cluster/diversity-based—across all data regimes and pre-training variants. Relative accuracy gains over random selection are consistently ~1–4 percentage points [2210.03822].
- **Streaming vs. batch learning:** Stream Decision Forest (SDF) and XForest achieve accuracy within ±5% of batch Random Forest on most of CC18, sometimes surpassing it in low-sample regimes ($n<2{,}000$) and/or high-dimensional settings. Streaming models complete all tasks within commodity hardware constraints [2110.08483].
- **TFM adaptation:** Zero-shot TFM inference achieves $0.87$–$0.88$ accuracy on CC18. Meta-learning yields modest gains (up to $+1.7$ pts for TabPFN). Full SFT often degrades performance except in specific “medium & wide” settings. PEFT recovers a large fraction of SFT gains without overfitting [2601.09654].

## 5. Subset Selection, Difficulty Profiling, and Practical Recommendations

Analyses utilizing IRT and Glicko-2 illustrate that not all CC18 datasets are equally informative for algorithm benchmarking [2007.14870; 2107.07451]. Key findings:
- Only a minor subset (<12%) of task instances are truly “hard” ($b_i > 1$).
- 80% of instances in half the suite are highly discriminative ($a_i > 0.75$).
- For robust head-to-head comparisons, select high-discrimination datasets.
- For stress-testing classifier ability, restrict to sets with >50% difficult instances.
- A carefully chosen 50% subset of CC18 can be as discriminative as the full suite for most algorithmic comparisons.
- For active learning, margin-based selection is nearly always optimal and requires no hyperparameter tuning [2210.03822].

| Use Case           | Recommended CC18 Subset                              |
|--------------------|-----------------------------------------------------|
| Stress-test ability| Datasets with >50% difficult items ($b_i>1$)         |
| Fine discrimination| Datasets with >80% high-discrimination items ($a_i$) |
| Efficient protocol | 50% highest-difficulty or discrimination subset      |

## 6. Impact on Advanced Modeling and Future Directions

OpenML-CC18’s structure has enabled rapid development, robust testing, and reproducibility for advanced learning paradigms:
- **Cross-dataset meta-learning:** Schema-alignment models (e.g., Chameleon) leverage CC18 for the first cross-task few-shot learning experiments on tabular domains, supporting both feature and instance subsampling regimes [1909.13576].
- **Ecological priors and cognitive-aligned learning:** Transformer models meta-trained on LLM-generated, ecologically plausible tasks (ERMI) match or surpass XGBoost, SVM, and TabPFN on CC18, exemplifying the suite’s utility for real-world generalization [2402.01821].
- **Foundational model evaluation:** CC18 provides a uniform platform to benchmark zero-shot, meta-learned, and fine-tuned TFMs, clarifying the regimes where adaptation is beneficial (rarely for small $n$, often for imbalanced or wide-feature medium-sized tasks) [2601.09654].

Future directions include extension to more challenging regimes (higher-dimensional, richer missingness patterns, variable label cardinality), and more granular difficulty profiling to enable adaptive benchmarking [2007.14870; 1708.03731].

## 7. Concluding Position in the Benchmarking Landscape

OpenML-CC18 has established itself as the canonical tabular classification benchmark suite, cited across empirical ML, AutoML, and theoretical algorithmic research. Its curation and rich meta-information facilitate reproducibility, empirical rigor, and community-driven extension. Subset selection methodologies and systematic evaluation protocols developed around CC18 now inform next-generation benchmarking standards within and beyond the OpenML platform [1708.03731].

Source: https://www.emergentmind.com/topics/openml-cc18