---
title: 'TabDPT: Discriminative Pre-trained Transformer'
url: https://www.emergentmind.com/topics/tabular-discriminative-pre-trained-transformer-tabdpt
type: topic
---

# TabDPT: Discriminative Pre-trained Transformer

Tabular Discriminative Pre-trained Transformer (TabDPT) is a specialized foundation model architecture and training paradigm for tabular data, integrating in-context retrieval, self-supervised learning via masked column modeling, and architectural modifications for tabular invariances. Unlike large language models (LLMs) adapted to text-formatted tables, TabDPT leverages the structural properties of tabular data and directly pre-trains on large-scale real-world datasets, enabling efficient generalization to unseen data and tasks without task-specific tuning [2410.18164]. TabDPT establishes new state-of-the-art results among open-source tabular foundation models (TFMs) in both classification and regression domains, while exhibiting predictable scaling laws reminiscent of LLMs.

## 1. Model Architecture and Tokenization

TabDPT utilizes a row-based transformer encoder, structurally similar to TabPFN v1 but fundamentally distinct in tokenization and context composition. Each table row, up to $F_{\max}=100$ features, is embedded into a $d$-dimensional vector using a single linear layer followed by layer normalization:

- **Feature Embedding**: $\phi_x: \mathbb{R}^{F_{\max}}\rightarrow \mathbb{R}^d$
- **Label/Target Embedding**: $\phi_y: \{1, \dots, C_{\max}\} \cup \mathbb{R} \rightarrow \mathbb{R}^d$; for regression, targets are normalized and projected via a linear map; for classification, a learnable $C_{\max} \times d$ embedding table is used.

Context rows are represented by elementwise summation of feature and label embeddings, while query rows use feature embeddings alone.

Key architectural features include:

- **Row-based tokens** restore permutation invariance and reduce transformer context length from $O(NF)$ to $O(N)$.
- **Absence of positional embeddings** further enforces row-order invariance.
- **Regular random column sub-sampling/shuffling** as data augmentation and to prevent column order overfitting.
- **Heads for classification and regression** share the transformer backbone and use separate MLPs for output.

A typical large-scale TabDPT backbone includes $L=16$ layers, $d=768$ hidden size, $h=4$ attention heads, FFN inner dimension up to $4d$, and no dropout (normalization-first transformer blocks) [2410.18164].

## 2. Self-supervised Pre-training and In-context Retrieval

The core of TabDPT pre-training is a combined paradigm of self-supervised masked column modeling and in-context learning (ICL)-aligned retrieval.

### Masked Column Modeling

For each dataset $\mathcal{D}=\{X \in \mathbb{R}^{N \times F}, y \in \mathbb{R}^N\}$, a random column $c$ is selected as a pseudo-target. The model receives all rows with column $c$ removed ($X \setminus c$) and predicts its values (either as real-valued regression or multiclass classification to $C_{\max}$ classes):

- **Loss functions**:
  - Classification: $\mathcal{L}_{\mathrm{CE}} = -\frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} \sum_{k=1}^C \mathbf{1}[y_i = k] \log p(\hat y_i = k)$
  - Regression: $\mathcal{L}_{\mathrm{MSE}} = \frac{1}{|\mathcal{I}|} \sum_{i \in \mathcal{I}} (\hat y_i - y_i)^2$
  - Overall SSL: $\mathcal{L}_{\mathrm{mask}} = \mathbb{E}_{\rm tasks}[1_{\rm cls} \mathcal{L}_{\rm CE} + 1_{\rm reg} \mathcal{L}_{\rm MSE}]$

### In-context Retrieval

For each row, $K$ nearest neighbors are selected via a FAISS index built on the pre-normalized feature space (excluding the masked column). These $K$ neighbors are randomly partitioned into context and query rows for training, constructing the input as
$$
\hat y_{\mathrm{qy}} = \mathrm{Transformer}\left[
\underbrace{\phi_x(X_{ctx}) \oplus \phi_y(y_{ctx})}_{\text{context tokens}},
~\phi_x(X_{qy})
\right]
$$
where context tokens attend to each other and to the query, while queries attend to all context tokens.

The pre-training optimization is solely the combined mask loss: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{mask}} = \lambda_{\rm cls}\mathcal{L}_{\rm CE} + \lambda_{\rm reg}\mathcal{L}_{\rm MSE}$, with $\lambda_{\rm cls} = \lambda_{\rm reg} = 1$ [2410.18164].

## 3. Data Sources and Preprocessing

TabDPT’s pre-training corpus consists of 123 diverse, CC-BY-licensed OpenML datasets, spanning domains such as biology, finance, healthcare, industrial logs, and text-derived features. The corpus comprises 32 million rows and 2 billion cells.

Pre-processing steps include:

- Label-encoding categorical features to integer indices.
- Standardizing numerical columns (mean 0, variance 1), with values clipped to $\pm$10.
- Missing values are filled with zero post-standardization.

A central empirical finding is that real tabular data provides substantial pre-training benefit over synthetic data. Specifically, models pre-trained with real data converge faster, achieve lower validation loss, and yield superior downstream performance compared to those trained exclusively on synthetic generators (such as TabPFN’s mixture-based data) [2410.18164]. This suggests intrinsic, transferable signal present in heterogeneous real tables beyond what handcrafted synthetic priors deliver.

## 4. Benchmark Evaluation and Performance

TabDPT’s performance was benchmarked on two large-scale suites:

- **CC18**: 72 unseen OpenML classification datasets (500–100,000 rows, up to 5,000 features), with metrics of AUC and accuracy.
- **CTR23**: 35 unseen OpenML regression datasets, analogous scale, evaluated by Pearson correlation $\rho$ and $R^2$.

| Metric          | TabDPT  | TabPFN v2 | XGBoost | LightGBM | CatBoost |
|-----------------|---------|-----------|---------|----------|----------|
| CC18 AUC        | **0.933** | 0.932   | 0.926   | 0.924    | 0.926    |
| CC18 Accuracy   | **0.884** | 0.872   | 0.869   | 0.862    | 0.864    |
| CTR23 Corr      | **0.837** | 0.835   | 0.827   | 0.825    | 0.822    |
| CTR23 $R^2$     | **0.742** | 0.740   | 0.711   | 0.713    | 0.703    |

TabDPT achieves top results on all key metrics as of publication. Pairwise dataset win-rates against these baselines exceed 60%. In Elo and Glicko2 rating tournaments, TabDPT is rated highest among open-source TFMs. For a context size of 2048, TabDPT processes 1,000 rows in approximately 0.1 seconds on an A100, which is over 10$\times$ faster than per-dataset hyperparameter search plus inference for tree-based or deep-learning alternatives [2410.18164].

## 5. Scaling Laws, Ablations, and Comparative Insights

Scaling experiments demonstrate that both model size ($P$) and pre-training data volume ($D$) drive predictable loss improvements following a joint power-law:
$$
\hat \ell(P, D) = A P^{-\alpha} + B D^{-\beta} + E
$$
with fitted exponents $\alpha = 0.42$, $\beta = 0.39$, indicating that doubling either model size or data yields measurable excess loss reduction. These scaling curves are consistent over $33$k to $78$M parameters and $52$M to $2$B data cells. This observation closely parallels scaling behaviors established in LLMs and vision foundation models, suggesting that sufficiently large and data-rich TFMs are achievable for tabular domains [2410.18164].

Ablation analyses underscore the importance of row-based tokenization, invariances, and real-data pre-training. The absence of positional encoding and persistent random column shuffling/sub-sampling were both necessary for generalization. *A plausible implication is that permutation and invariance constraints are critical to transferrable tabular representations.*

## 6. Limitations and Future Research Directions

TabDPT assumes strictly rectangular, i.i.d. tables and does not explicitly accommodate temporal sequences, hierarchical dependencies, or multimodal (text or image) columns. Feature dimensionality ($F_{\max}$) and class count ($C_{\max}$) are fixed, handled via principal component analysis for features and base-$C_{\max}$ encoding for classes. Free-form feature names and textual information are not utilized, as empirical studies found that feature-name embeddings bias toward overfitting under current dataset diversity [2410.18164].

Future directions identified include:

- Incorporation of richer textual/contextual annotations (e.g., column names, physical units) through joint text+tabular embeddings.
- Direct extension to structured (temporal or graph) tabular data, enabling modeling of event logs or hierarchical records.
- Integration with generative models (e.g., TabPFGen, TabLatent) to serve tasks such as anomaly detection or missing value imputation.
- Investigation of self-supervised objectives beyond column masking, including contrastive row-pair pretext tasks.

## 7. Relationship to Related Work

TabDPT is part of a broader trend toward tabular-specific foundation models and discriminative pre-trained transformers. Prior approaches including TP-BERTa [2403.01841] leverage LLM backbones augmented with techniques such as relative magnitude tokenization and intra-feature attention. However, LLM-based models for tabular prediction have shown limited success in ICL or cross-table transfer compared to models like TabDPT that are pre-trained on large real tabular corpora with tabular-structured inductive biases. Additionally, advances such as TabToken’s supervised contrastive token regularization further inform the design of transferable tabular transformers, emphasizing joint order-invariant tokenization and simultaneous embedding–transformer optimization [2310.15149]. 

TabDPT distinguishes itself by unifying retrieval-augmented in-context learning and masked column modeling atop real heterogeneous data, confirming that real tabular datasets enable faster convergence and superior transfer while obeying power-law scaling. Its open-source pipeline and empirical record position it as a central architecture in the modern TFM landscape [2410.18164].

Source: https://www.emergentmind.com/topics/tabular-discriminative-pre-trained-transformer-tabdpt