TabSwift: Efficient Tabular Foundation Model

• TabSwift is a tabular foundation model with a lightweight row-wise transformer backbone, augmented by gated attention and register tokens for direct in-context prediction.
• It employs an adaptive, layer-wise early-exit mechanism that dynamically balances inference accuracy with reduced computational cost.
• Empirical evaluations on the TALENT benchmark demonstrate competitive accuracy and notably lower latency compared to more complex, feature-aware tabular models.

Searching arXiv for the specified paper and related tabular foundation model references. TabSwift is a tabular foundation model for in-context learning on supervised tabular tasks. It conditions on a labeled support set $D_{\mathrm{sup}}=\{(x_i,y_i)\}_{i=1}^N$ and a query $x_q$, and directly predicts $\hat{y}_q$ from the joint prompt $(D_{\mathrm{sup}},x_q)$ via a pretrained transformer, rather than fitting dataset-specific parameters at deployment time. The model is defined by a lightweight row-wise attention-only backbone, augmented with gated attention and a small set of learnable register tokens, and extended with an adaptive layer-wise early-exit mechanism for anytime inference. It supports both classification and regression and is positioned as an efficiency-oriented alternative to more complex tabular foundation models that incorporate explicit feature-wise modeling (Liu et al., 5 Jun 2026).

1. Problem formulation and model class

TabSwift belongs to the PFN-style tabular foundation model family. In this setting, prediction is amortized across a distribution of tasks: the model learns during pretraining to approximate a conditional mapping of the form $f_\theta(x_q;D_{\mathrm{sup}})$ and infer test labels directly from labeled training examples, without task-specific gradient updates at inference time. This differs from standard supervised learning, where one optimizes a risk such as $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$ separately for each dataset and then deploys the fitted model (Liu et al., 5 Jun 2026).

The motivation for TabSwift is the tension between predictive performance and inference cost in recent tabular foundation models. The paper characterizes models such as TabPFN v2 and TabICL as improving accuracy through explicit feature-wise modeling, including alternating row/column attention and specialized embedding modules, while also increasing inference cost through additional attention passes and larger memory footprints. TabSwift revisits the earlier row-wise attention-only design and argues that this simpler backbone remains competitive when stabilized and minimally augmented (Liu et al., 5 Jun 2026).

A recurrent misconception in discussions of tabular foundation models is to equate them with conventional pretrained encoders followed by dataset-specific fine-tuning. TabSwift instead follows the PFN paradigm: the pretrained transformer itself performs the prediction from the support set and the query. In that sense, “foundation model” refers here to amortized inference over a synthetic task prior rather than generic representation pretraining.

2. Architecture and representational design

TabSwift uses a row-wise attention-only transformer backbone. “Row-wise” denotes attention across examples rather than across features: each table row is embedded as a single token, and the input sequence is formed by prepending $K$ learnable register tokens to $N$ labeled support row tokens and one unlabeled query row token. If $S=K+(N+1)$ denotes the sequence length, attention mixes information across these $S$ tokens so that the contextualized query representation can attend to labeled support rows and infer its label (Liu et al., 5 Jun 2026).

For layer input $x_q$0 and attention head $x_q$1 with per-head dimension $x_q$2, the attention mechanism is standard scaled dot-product attention:

$x_q$3

$x_q$4

The principal architectural deviation is SDPA-output gating, identified in the source as G1 in Qiu et al. 2025. For each head,

$x_q$5

and the multi-head output is

$x_q$6

The transformer uses pre-norm layer normalization and residual structure:

$x_q$7

The gated attention mechanism is introduced to stabilize optimization and add an input-dependent nonlinearity or sparsity effect in attention updates with negligible overhead. The register tokens serve as global, learnable latent slots. They are prepended to the row tokens, share dimensionality $x_q$8 with other tokens, and are updated by attention in every layer as queries, keys, and values. Their stated function is to aggregate global context across depth and improve pretraining quality by maintaining dataset-level representations. This suggests that TabSwift treats global task structure as an internal latent state distributed over a small number of persistent tokens rather than over the support rows alone (Liu et al., 5 Jun 2026).

After $x_q$9 layers, the query token $\hat{y}_q$0 is decoded by a task-specific head. For classification, the head produces logits over up to $\hat{y}_q$1 classes and is trained with cross-entropy,

$\hat{y}_q$2

For regression, the head produces a scalar and is trained with the combined loss

$\hat{y}_q$3

3. Input representation, preprocessing, and pretraining

TabSwift uses a fixed input budget $\hat{y}_q$4. If a dataset has fewer than $\hat{y}_q$5 features, zero-padding is applied; if it has more, PCA projects features to $\hat{y}_q$6. In the experiments, features are capped at $\hat{y}_q$7, and high-dimensional datasets are reduced to $\hat{y}_q$8 features via PCA. Numerical features are standardized, while categorical features are ordinal or fixed encoded (Liu et al., 5 Jun 2026).

Each row $\hat{y}_q$9 is normalized to $(D_{\mathrm{sup}},x_q)$0 and embedded as

$(D_{\mathrm{sup}},x_q)$1

Support rows inject label embeddings,

$(D_{\mathrm{sup}},x_q)$2

while the query token is unlabeled,

$(D_{\mathrm{sup}},x_q)$3

With register tokens $(D_{\mathrm{sup}},x_q)$4, the initial sequence is

$(D_{\mathrm{sup}},x_q)$5

No specialized positional encodings or masking strategy are required beyond this sequence layout.

Pretraining follows the TabICL synthetic task generation framework. An offline pool of $(D_{\mathrm{sup}},x_q)$6 pretraining steps is created, each containing $(D_{\mathrm{sup}},x_q)$7 independently sampled synthetic tabular tasks, with at most $(D_{\mathrm{sup}},x_q)$8 rows and at most $(D_{\mathrm{sup}},x_q)$9 features per task; the pool is consumed round-robin. Each synthetic task stores two targets, a discretized classification target and a normalized regression target, so that both task heads can be jointly supervised on the same task instance. The implicit learning target is an amortized conditional $f_\theta(x_q;D_{\mathrm{sup}})$0 under the synthetic task distribution (Liu et al., 5 Jun 2026).

The main pretrained model uses a $f_\theta(x_q;D_{\mathrm{sup}})$1-layer transformer with $f_\theta(x_q;D_{\mathrm{sup}})$2, optimized with AdamW for $f_\theta(x_q;D_{\mathrm{sup}})$3 steps on $f_\theta(x_q;D_{\mathrm{sup}})$4 NVIDIA RTX 5090 GPUs over approximately seven days. Continued pretraining then adds $f_\theta(x_q;D_{\mathrm{sup}})$5 steps on larger tasks with up to $f_\theta(x_q;D_{\mathrm{sup}})$6 rows to improve robustness to long in-context sequences. The number of heads $f_\theta(x_q;D_{\mathrm{sup}})$7 and per-head dimension $f_\theta(x_q;D_{\mathrm{sup}})$8 are described as standard transformer choices, and the text does not fix $f_\theta(x_q;D_{\mathrm{sup}})$9 explicitly. This training configuration places TabSwift in a regime where a relatively small backbone is combined with a large synthetic multitask curriculum rather than increased architectural complexity (Liu et al., 5 Jun 2026).

4. Early-exit inference and computational profile

At inference time, TabSwift builds a prompt from $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$0 labeled support tokens, one query token, and $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$1 registers; the transformer then produces a contextualized query representation $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$2 for the final prediction. Because the architecture attends across rows, the query token integrates information from the support examples directly (Liu et al., 5 Jun 2026).

For latency-sensitive serving, TabSwift introduces a layer-wise early-exit mechanism. Prediction heads are attached to exit layers $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$3, with $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$4 in the implementation. For query hidden states $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$5, the exit-specific prediction is

$\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$6

A learned exit head produces a reliability score from the query state and the pooled register summary:

$\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$7

The stopping rule is

$\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$8

and if no layer satisfies the threshold, the model exits at the final layer $\mathbb{E}_{(x,y)\sim P}[\ell(f_\theta(x),y)]$9. The threshold $K$0 controls the accuracy–latency trade-off.

The early-exit gates are trained post hoc with the backbone frozen for $K$1 steps on synthetic tasks. For classification, the gate target is

$K$2

with auxiliary loss

$K$3

For regression, the target is

$K$4

with margin $K$5, and

$K$6

Prediction and auxiliary losses are averaged across exits, with $K$7 and $K$8.

The computational profile is explicitly analyzed. If $K$9 and $N$0, row-wise attention-only computation costs $N$1 per layer, while gating adds only $N$2 per layer and is therefore negligible relative to the quadratic term. Total backbone cost is $N$3. The paper contrasts this with alternating row/column attention, whose per-block cost is given as $N$4 and is typically heavier. Memory is dominated by attention activations of size $N$5 per head. A practical implication is that TabSwift’s efficiency gains arise chiefly from architectural simplicity and reduced average depth through early exit, not from changing the asymptotic dependence on the number of in-context rows (Liu et al., 5 Jun 2026).

5. Empirical evaluation

TabSwift is evaluated on the TALENT benchmark, comprising $N$6 datasets spanning binary classification, multiclass classification, and regression. Dataset splits are $N$7 for train/validation/test, and results are averaged over $N$8 seeds. The reported metrics are AUC for classification, where higher is better, and RMSE for regression, where lower is better (Liu et al., 5 Jun 2026).

The aggregate comparison reports that critical difference diagrams place TabSwift at average ranks comparable to, and often not significantly different from, the strongest methods under Wilcoxon–Holm testing at $N$9. PAMA curves indicate that TabSwift attains the best performance on a substantial fraction of datasets, which the source interprets as evidence of consistent robustness. Under a unified inference pipeline on an NVIDIA A800 GPU and with full depth, TabSwift’s runtimes are described as substantially lower than those of heavier column-aware tabular foundation models. The comparison is especially notable because TabSwift uses $S=K+(N+1)$0-model ensembles, whereas TabPFN v2 uses $S=K+(N+1)$1 in order to keep its cost controlled (Liu et al., 5 Jun 2026).

For classification, the early-exit mechanism yields a detailed accuracy–compute trade-off when sweeping $S=K+(N+1)$2. Representative results are as follows.

Setting	Accuracy	Avg layers
Full depth	0.8398	24.00
Learned exit, $S=K+(N+1)$3	0.8398	21.67
Learned exit, $S=K+(N+1)$4	0.8397	19.47
Learned exit, $S=K+(N+1)$5	0.8395	18.16
Learned exit, $S=K+(N+1)$6	0.8389	17.00
Learned exit, $S=K+(N+1)$7	0.8370	14.06
Learned exit, $S=K+(N+1)$8	0.8329	11.62

The source also reports an entropy-based stopping baseline, with examples including threshold $S=K+(N+1)$9 accuracy at $S$0 average layers and threshold $S$1 accuracy at $S$2 layers. Learned exits, especially those conditioned on register summaries, are reported to dominate entropy stopping in Pareto terms. PCA visualizations are described as showing late exits concentrated near overlapping class regions in embedding space, allocating more depth to hard samples. This suggests that the early-exit policy is not merely truncating computation uniformly, but is selectively modulating depth according to sample difficulty.

The large-$S$3 analysis highlights an important boundary condition. On TALENT classification datasets with more than $S$4k training samples, average ranks are reported as RealMLP $S$5, ModernNCA $S$6, TabM $S$7, TabSwift $S$8, TabICL $S$9, CatBoost $x_q$00, LightGBM $x_q$01, and TabPFN v2 $x_q$02. The source attributes this partly to the quadratic scaling of PFN-style in-context learning in $x_q$03: TabSwift remains competitive among PFN-like models, but non-PFN deep tabular baselines become favorable in very large-$x_q$04 regimes (Liu et al., 5 Jun 2026).

The paper also examines high-dimensional datasets in which features number in the thousands and are reduced to $x_q$05 by PCA. Reported accuracies include warpPIE10P ($x_q$06 dims) at $x_q$07, tying the best method; BASEHOCK ($x_q$08) at $x_q$09, also tying best; and gisette ($x_q$10) at $x_q$11, the best result. Other listed datasets include PCMAC at $x_q$12, lung at $x_q$13, RELATHE at $x_q$14, TOX_171 at $x_q$15, arcene at $x_q$16, and SMK_CAN_187 at $x_q$17. The corresponding average rank across methods is reported as TabPFN v2 $x_q$18, TabSwift $x_q$19, TabICL $x_q$20, and XGBoost $x_q$21. The ablation study compares a row-wise baseline (TabS-S1), variants with gated attention, variants with register tokens, the combined variant, and the final two-stage-pretrained TabSwift; gated attention and registers are each said to improve performance and training stability, although exact deltas are not tabulated (Liu et al., 5 Jun 2026).

6. Deployment considerations, limitations, and prospective extensions

TabSwift is explicitly designed for practical deployment under latency constraints. The deployment guidance centers on selecting the early-exit threshold $x_q$22 via validation. For classification with register-conditioned exits, thresholds around $x_q$23–$x_q$24 reduce average depth by approximately $x_q$25–$x_q$26 layers while keeping accuracy within at most $x_q$27–$x_q$28 of full depth, whereas $x_q$29 or $x_q$30 yields larger speedups with mild accuracy reductions. The paper states that an A800 or similar GPU comfortably serves TabSwift and that, even with $x_q$31 ensembles, it remains faster than heavier tabular foundation models under a unified pipeline (Liu et al., 5 Jun 2026).

Memory and batching behavior follow directly from the row-wise attention formulation. Since memory scales with $x_q$32 attention per head, keeping $x_q$33 moderate is advantageous for low latency. Register tokens add only $x_q$34 extra tokens and are treated as a small overhead. Multiple queries can be batched by increasing $x_q$35, but the support set tokens are replicated per query. The stated scaling law,

$x_q$36

implies that prompt construction and support-set size remain central deployment variables.

The paper identifies several limitations. First, PFN-style quadratic attention makes very large-$x_q$37 regimes challenging, even after continued pretraining on tasks with up to $x_q$38 rows. Second, the fixed input budget, including PCA to $x_q$39 features, may constrain high-dimensional tasks, even though the empirical results are strong on several such datasets. Third, the work is restricted to supervised classification and regression and does not cover ranking, survival analysis, multi-label learning, or time series. Fourth, the use of synthetic priors raises the possibility of mismatch in highly specialized domains (Liu et al., 5 Jun 2026).

The proposed future directions are correspondingly pragmatic rather than categorical. These include integrating feature-wise attention blocks or hybrid row/column patterns selectively while preserving efficiency; extending the framework with semi-supervised or self-supervised objectives and improved synthetic priors; developing dynamic token pruning, sketching, or register-enhanced retrieval to compress large support sets; exploring linear or low-rank attention variants for scalability; and improving categorical encodings, missing-data handling, calibration-aware early-exit criteria, and joint uncertainty estimation. A plausible implication is that TabSwift is intended less as a terminal architecture than as a minimal, efficiency-centered baseline for tabular in-context learning, against which more scalable or domain-adapted variants can be evaluated (Liu et al., 5 Jun 2026).