TabICLv2: A better, faster, scalable, and open tabular foundation model

Abstract: Tabular foundation models, such as TabPFNv2 and TabICL, have recently dethroned gradient-boosted trees at the top of predictive benchmarks, demonstrating the value of in-context learning for tabular data. We introduce TabICLv2, a new state-of-the-art foundation model for regression and classification built on three pillars: (1) a novel synthetic data generation engine designed for high pretraining diversity; (2) various architectural innovations, including a new scalable softmax in attention improving generalization to larger datasets without prohibitive long-sequence pretraining; and (3) optimized pretraining protocols, notably replacing AdamW with the Muon optimizer. On the TabArena and TALENT benchmarks, TabICLv2 without any tuning surpasses the performance of the current state of the art, RealTabPFN-2.5 (hyperparameter-tuned, ensembled, and fine-tuned on real data). With only moderate pretraining compute, TabICLv2 generalizes effectively to million-scale datasets under 50GB GPU memory while being markedly faster than RealTabPFN-2.5. We provide extensive ablation studies to quantify these contributions and commit to open research by first releasing inference code and model weights at https://github.com/soda-inria/tabicl, with synthetic data engine and pretraining code to follow.

• The paper presents TabICLv2, a state-of-the-art tabular foundation model that leverages a Transformer-based architecture for efficient in-context learning.
• It employs innovative techniques such as column-then-row attention, target-aware embedding, and QASSMax to robustly handle diverse tabular regimes.
• Results show its practical scalability to over 1M samples and superior performance in both regression and many-class classification through mixed-radix ensembling.

TabICLv2: A Superior Open Tabular Foundation Model

Overview and Motivation

TabICLv2, introduced in "TabICLv2: A better, faster, scalable, and open tabular foundation model" (2602.11139), establishes a new state-of-the-art for tabular foundation models (TFMs). This model is engineered for regression and classification tasks on tabular data—domains historically dominated by gradient-boosted trees. TabICLv2 leverages in-context learning (ICL) through a highly optimized Transformer-based architecture and a principled synthetic pretraining strategy. Its design priorities are clear: native scalability above $10^6$ samples, efficient inference, extensive coverage of tabular regimes including many-class classification, and open-source accessibility.

Architectural Innovations

TabICLv2 builds on the efficient "column-then-row" attention design pioneered in TabICL, reducing runtime complexity to $O(n^2 + nm^2)$ for $n$ samples and $m$ features. This is accomplished through staged embedding:

• Repeated feature grouping: Each feature is embedded via multiple column groupings, disrupting representation collapse from similar marginals and retaining fine-grained feature interactions.
• Target-aware embedding: Target information is injected at the initial embedding stage, enhancing conditional modeling power and combatting representation collapse when features share distributions.
• Sequential column, row, and ICL attention: Three transformer modules—column-wise Set Transformer, row-wise transformer with [CLS] tokens, and dataset-wise ICL—enable efficient aggregation and in-context generalization.

  Figure 1: Architecture of TabICLv2 showing repeated feature grouping, target-aware embedding, and multi-stage attention for efficient ICL on input $X \in \mathbb{R}^{n \times m}$.

• Query-aware scalable softmax (QASSMax): A novel attention scaling mechanism that extends Scalable Softmax (SSMax) via query-dependent modulation, using $\log n$ context-length scaling and gating for each query dimension. QASSMax robustly preserves attention sharpness and mitigates fading at extreme context lengths typical in million-scale tabular inference.

Figure 2: Accuracy and attention entropy demonstrate QASSMax's ability to prevent attention degradation as the number of negative samples (context size) increases.

• Mixed-radix ensembling: For many-class classification ($C > 10$), labels are embedded in multiple "digits" via mixed-radix representation, each processed with the native label encoder capacity and outputs ensembled to ensure unique class identification.
• Distributional regression via quantile prediction: TabICLv2 predicts 999 quantiles for regression, employing pinball loss and analytical tail extrapolation, enabling fast point estimates and full conditional distribution modeling in downstream tasks.

Pretraining Strategy

TabICLv2's success is driven by:

• High-diversity synthetic prior: A data generator models tabular distributional diversity via random graphs and eight function classes (MLPs, GP functions, tree ensembles, discretization, linear/quadratic, clustering, products). The generative process propagates feature data through random DAGs, induces complex dependencies and diverse marginal distributions, and applies filtering to guarantee learnable signal presence.

Figure 3: Synthetic dataset generation pipeline with random graph sampling, eight random function types, and post-generation filtering for learnable structure.

• Three-stage pretraining: Progressive curriculum scales dataset size up to $60$K samples, optimizing for generalization to large tabular domains.
• Muon optimizer: Substituting AdamW for Muon enables higher learning rates and stable convergence. Cautious weight decay avoids harmful gradient interference, and gradient clipping is increased for robust large-batch training.
• Resource-efficient inference: Memory-mapped file offloading and selective query/key/value computation allow TabICLv2 to process 1M-sample tables under 50GB GPU and 24GB CPU memory requirements, with compute time competitive given the scale.

Experimental Validation

TabICLv2 robustly dominates the Pareto fronts of multiple benchmarks:

• TabArena: Achieves best improvability and Elo scores relative to state-of-the-art (RealTabPFN-2.5), both in speed and accuracy across classification and regression.
• TALENT: Consistently ranks first across all dataset sizes, outperforming RealTabPFN-2.5 and TabPFN-2.5 especially at scales above $20$K samples. Strong handling of multiclass ($C > 10$) tasks is shown with both ECOC wrapper and native mixed-radix ensembling.

  Figure 4: Improvability versus train time (TabArena), placing TabICLv2 on the top front for both accuracy and efficiency.

  

  Figure 5: Improvability versus inference time (TALENT), demonstrating TabICLv2's dominance at all scale points.

  

  Figure 6: Runtime comparison, showing TabICLv2 is $10\times$ faster than TabPFN-2.5 at $50$K samples on H100 GPUs.

  

  Figure 7: TabICLv2's normalized accuracy significantly exceeds all baselines for datasets with more than 10 classes.

  

  Figure 8: TabICLv2 maintains top ranking as sample size grows, outperforming RealTabPFN-2.5 for large $n$.

  

  Figure 9: TabICLv2 successfully processes 500K+ samples; TabPFN-2.5 fails due to out-of-memory.

Ablation and Component Analysis

Comprehensive ablation shows:

• Architectural improvements (early target inclusion, QASSMax, Muon optimizer) each contribute significant Elo gains ($\approx$100 points, $>60\%$ pairwise win rate).
• Prior diversity is critical: TabICLv2 fails to generalize if pretrained on the TabICL prior, highlighting dependency of architectural scaling on sufficient synthetic diversity.
• Filtering, feature grouping, and many-class mechanisms provide further but smaller gains.

  Figure 10: Ablation analysis disentangles contribution of architectural and pretraining components to aggregate performance.

Limitations and Future Scope

Despite its progress, TabICLv2 does not natively leverage semantic column information (e.g., textual features) and does not directly address massive sample regimes ($>10^7$). Extensions to multi-output tasks, robust missing value handling via imputation strategies, fine-tuning for specific distributions, and incorporation of semantic embedding models are promising directions. Distributional regression capabilities have yet to be benchmarked thoroughly due to limited evaluative datasets.

Implications and Theoretical Impact

TabICLv2 redefines the landscape for tabular data modeling:

• Practical: Provides an open, fast, accurate standard for both academic and applied tabular ML workflows, enabling robust deployment for large datasets without reliance on closed-source solutions.
• Theoretical: Demonstrates that principled synthetic pretraining and advanced attention scaling can extend ICL transformer models far into tabular data regimes previously considered tractable only for decision trees. The advance in scalable attention mechanisms will likely inform future long-context transformer architectures across modalities.

Conclusions

TabICLv2 delivers a validated, open, scalable TFM architecture with demonstrated state-of-the-art performance and efficiency. By aligning architectural, prior, and protocol innovations, TabICLv2 is positioned to drive democratized progress—enabling further advances in tabular ML, distributional regression, and downstream structured data tasks without closed-source constraints.