Understanding the Surprising Generalization Properties of Tabular Foundation Models

Abstract: Tabular Foundation Models (TFMs) increasingly rely on in-context learning, where a model receives labelled examples at inference time and predicts labels for new inputs without updating its weights. Existing TFMs are typically trained on either massive synthetic corpora or very large collections of real datasets. In contrast, we show that surprisingly strong transfer can emerge from self-supervised pre-training on just a single real table. In this setting, we also find that tables tend to be either broadly useful or broadly poor regardless of downstream prediction task, and that the strongest predictor of usefulness is the number of features rather than the number of instances. This leads to a task-centric interpretation of tabular pre-training: the number and the quality of tasks are essential for the pre-training of TFMs. We show that the same task-centric perspective can help corpus design at scale: fine-grained column-level pre-processing consistently improves downstream performance, while no improvements are observed when we filter or deduplicate at the dataset level. Finally, we offer a new perspective for how TFMs generalize: we believe that tabular in-context generalization is largely retrieval-based, and good models are those that learn to identify relevant examples in the provided context and aggregate them well. The mechanics of TFMs have been relatively understudied; our task-centric, retrieval-based perspective offers a new framework to guide future model and corpus design.

• The paper finds that pre-training a transformer on a single real table can lead to competitive performance across various structurally unrelated benchmarks proving only diversity of tasks matters more than data volume which opens new avenues for efficient transfer learning.
• An XGBoost meta-regressor on dataset metadata demonstrated that the number of features significantly impacts single-table generalization -driven primarily through the creation of task diversity ($R^2 = 0.67$). Therefore, feature count is more critical than the instance count (e.g., removing 70% of columns significantly degrades performance, while removing the same percentage of rows retains decent performance).
• The authors propose a retrieval-and-aggregation procedure as a key mechanism supporting generalization of tabular models which leverages similarities between query rows to retrieve and aggregate label information, thereby challenging the conventional Bayesian inference view commonly associated with Probabilistic Fine-Tuning Networks (PFNs)

Overview

This paper investigates how pre-training data properties govern out-of-distribution generalization in tabular foundation models (TFMs) such as TabPFN, TabDPT, and TabICL. The authors, working with the TabDPT training pipeline and a shared transformer backbone (12 layers, $d_{\text{model}}=512$, asymmetric in-context attention), make three central empirical claims: (1) strong cross-domain transfer can emerge from self-supervised pre-training on a single real table; (2) the number of unique tasks—driven primarily by feature count rather than instance count—is the dominant factor behind generalization; and (3) tabular in-context learning is best understood as a learned retrieval-and-aggregation procedure rather than Bayesian inference over a prior over tasks (2608.17957).

Generalization from single-table pre-training

The most striking result is that a transformer trained from scratch on one real table—e.g., vectorized MNIST or the Colleges dataset—achieves competitive performance on structurally unrelated benchmarks such as California Housing, OpenML CC-18, and CTR-23, approaching random-forest baselines. This directly contradicts the prevailing assumption that broad task coverage in the pre-training distribution is necessary for out-of-domain transfer.

To characterize what makes a "good" pre-training dataset, the authors train 88 models, each on a distinct dataset drawn from the TabDPT corpus, and evaluate all of them on 107 datasets (72 CC-18 classification, 35 CTR-23 regression). Two findings stand out:

• Dataset quality is universal, not domain-specific. Spearman correlations between per-evaluation-dataset rankings of the 88 pre-training sets are heavily left-skewed: a dataset that transfers well to one evaluation set tends to transfer well to others. Moreover, within-domain pairs show no transfer advantage; some domains (e.g., other-science) are simply broadly better for pre-training than others (e.g., deterministic-simulated).
• Feature count dominates instance count. An XGBoost meta-regressor on dataset metadata predicts average downstream generalization with $R^2 = 0.67$, and number of features is by far the strongest predictor while instance count has negligible predictive power—an inversion of classical sample-complexity intuitions. Subsampling experiments confirm this: removing ~70% of columns from Colleges degrades performance to near-linear-model levels, whereas removing >70% of rows leaves performance close to random forest.

The paper then identifies the mechanism behind the feature-count effect. Because pre-training randomly selects a target column and uses remaining columns as features, a table with $k$ columns induces $\mathcal{O}(k\,2^k)$ distinct tasks. Varying the number of allowed tasks while holding instances and features fixed shows that scaling from 5 tasks to tens of thousands drives downstream AUC from barely above random ($0.5$) to good, across all tested datasets. Task diversity, not data volume, is the operative variable. Consistently, pairing a top-quartile dataset with a bottom-quartile one does not hurt and can slightly improve results (e.g., Colleges alone at AUC 0.918 reaches 0.919 when paired with Analcatdata_Supreme), indicating weak datasets contribute complementary tasks rather than noise.

Corpus design at scale

Extending these insights to a 1,732-dataset OpenML corpus, the authors ablate dataset-level selection versus column-level pre-processing, reporting IQM with bootstrap CIs on CC-18 and CTR-23.

Dataset-level curation fails. Exact deduplication via SHA-256 hashing (1,732 → 1,535 datasets) yields statistically indistinguishable results from the full corpus—the SSL objective's randomized column-target sampling already neutralizes duplicates. Restricting the corpus to datasets with ≥30 features (607 datasets) degrades performance across all metrics, because it discards entire pools of diverse low-feature tasks whose average quality gain does not compensate for lost diversity.

Column-level cleaning helps consistently. Three operations—dropping columns with >50% missing values, greedy correlation deduplication at |ρ| > 0.90 (Pearson or Spearman, including $y$ in the joint correlation matrix), and skipping datasets left with fewer than 5 features—were ablated combinatorially. Every variant improves regression IQM $R^2$ by +0.002 to +0.014 over the baseline (best: NaN-drop + Pearson dedup + mf-5, reaching $R^2 = 0.8281$ vs. baseline 0.8145), with smaller but consistent classification gains. The interpretation follows directly from the task-centric view: column filtering removes redundant or uninformative tasks without shrinking the corpus, whereas dataset filtering removes whole task pools.

Scaling to a larger model (16 layers, $d_{\text{model}}=768$) trained at context lengths of 1k/2k/4k tokens confirms the effect: the pre-processed variant beats its baseline counterpart on 67%, 61%, and 63% of TabArena-Lite tasks respectively, with 4k+pp being the strongest configuration overall.

A retrieval-based account of generalization

The authors argue that the standard prior-fitting/Bayesian interpretation of PFNs cannot explain single-table generalization, since it requires downstream tasks to be covered by the pre-training prior—a condition clearly violated here. They instead propose that TFMs learn a retrieval-and-aggregation procedure analogous to soft $k$NN, and support this with three analyses.

First, they construct an exact-copy retrieval probe where query rows duplicate labelled context rows. Across the 88 single-table models, retrieval AUC correlates strongly with held-out generalization AUC (Pearson = 0.89): models that generalize well are precisely those that can recover label information already present in their context, paralleling induction-head behavior in LLMs.

Second, forcing attention toward explicit similarity computation by sharing query and key projection weights ($R^2 = 0.67$0, which under qk-normalization makes attention scores proportional to negative squared distances) strictly reduces expressiveness yet improves mean AUC by +0.1088 on three poorly-generalizing pre-training tables, causes no substantial change (+0.0001) on the full-corpus model, and costs only −0.0027 on tables that already generalize well. That a symmetric soft-$R^2 = 0.67$1NN constraint is essentially free even with abundant pre-training data supports the retrieval interpretation.

Third, probing 52 single-table checkpoints on two held-out datasets reveals convergence of attention patterns among strong models: pair-level Spearman correlation between attention similarity and mean AUC is 0.537, and strong–strong model pairs exhibit systematically higher attention similarity (0.504) than strong–weak (0.341) or weak–weak (0.290) pairs, with bootstrap CIs excluding zero. Per-block analysis localizes this effect in deeper blocks (correlation ≈ 0.6 in blocks 9–11). The authors interpret this as evidence for a fairly universal retrieval structure underlying successful tabular ICL, while noting the attention-map analysis suggests rather than proves such universality.

Limitations and open questions

The paper concedes several boundaries on its claims. It was unable to definitively identify or exclude mechanisms beyond learned retrieval contributing to out-of-distribution generalization, so the retrieval account remains a sufficient but not exhaustive explanation. All experiments use real-world pre-training data; whether findings extend to purely synthetic priors (as in TabPFN-style training) is untested. Architecturally, the study covers only row-based attention following TabDPT and TabPFNv1, leaving cell-based attention models (e.g., TabICL) outside its scope. Additionally, the attention-similarity confidence intervals are somewhat anti-conservative due to non-independence across model pairs, and the $R^2 = 0.67$2 intervention was evaluated only at the model sizes used here. An open question the paper leaves unresolved is whether approximate Bayesian inference emerges in models trained on much broader distributions, coexisting with—or superseding—the retrieval mechanism documented here.

Conclusion

This work reframes tabular foundation model pre-training around task quantity and quality rather than raw data volume: feature count and the induced number of unique column-target tasks predict generalization, dataset-level filtering and deduplication are ineffective while column-level cleaning reliably helps, and the resulting models behave largely as learned retrieval-and-aggregation systems. These findings challenge both the assumed necessity of massive diverse corpora and the prior-fitting narrative, offering a concrete framework for future TFM corpus and architecture design.