LimiX-2M: Compact Tabular Foundation Model

• LimiX-2M is a compact tabular foundation model that combines value-aware tokenization (RaBEL) with a reordered attention block (SNF) to address inefficiencies in transformer-based tabular learning.
• It employs a Radial-Basis Embedding Layer to expand numerical features into higher-dimensional representations, thereby mitigating the low-rank collapse inherent in traditional tokenizers.
• Benchmark evaluations show that LimiX-2M outperforms larger models like TabPFN-v2 and TabICL while reducing both training and inference costs.

Searching arXiv for the LimiX-2M paper and closely related tabular foundation model references. LimiX-2M is a tabular foundation model (TFM) introduced to address two inefficiencies identified in transformer-based tabular learning: low-rank collapse in early layers and an attention-routing bottleneck induced by feature-first bidirectional stacks. The model combines a value-aware tokenizer, RaBEL, with a reordered bidirectional block denoted $S \rightarrow N \rightarrow F$, yielding a compact architecture with 1.92M parameters that is reported to outperform larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs (Wang et al., 3 Jun 2026).

1. Problem formulation and motivating diagnosis

The central diagnosis underlying LimiX-2M is that standard affine scalar tokenization constrains each numerical feature value to enter the network through an essentially one-dimensional channel. In the formulation reported for the baseline tokenizer,

$$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$

with

$$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$

Here, $a_j$ is a feature-only signal such as a feature ID or positional embedding. The stated limitation is that such feature-only signals can distinguish columns but cannot increase the intrinsic dimensionality through which the scalar value $x_{i,j}$ enters the network (Wang et al., 3 Jun 2026).

The paper formalizes this by defining the pre-LayerNorm vector

$$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$

After centering over samples within a feature, the centered matrix satisfies

$$\operatorname{rank}(\tilde Z_{\mathrm{lin}}) \le 1,$$

because

$$\tilde z_{i,j}^{\mathrm{lin}} - \bar z_j = W_{\mathrm{lin}}(x_{i,j} - \bar x_j),$$

so the within-feature value variation has outer-product structure,

$\tilde Z_{\mathrm{lin}} = v_j w^\top,$

and therefore rank at most 1. The reported implication is that shallow hidden matrices can collapse to single-digit rank on common benchmarks despite much larger hidden width. This suggests that model width alone does not guarantee value-sensitive expressivity when the tokenizer itself is rank-constrained (Wang et al., 3 Jun 2026).

A related local sensitivity argument is given for first-layer sample-attention. For each output position $t$,

$$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$0

and stacking over positions yields

$$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$1

The stated interpretation is that, even with large hidden dimension $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$2, the effective value-sensitive directions remain bottlenecked by the tokenizer. In this framing, early-layer redundancy is not merely an optimization artifact but a structural consequence of affine scalar tokenization.

2. RaBEL: value-aware tokenization

To mitigate the tokenizer bottleneck, LimiX-2M introduces RaBEL, the Radial-Basis Embedding Layer. For numerical columns, each feature is first standardized as

$$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$3

where $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$4 and $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$5 are the column mean and standard deviation. Each scalar is then expanded into $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$6 localized radial basis responses. For feature $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$7, the $$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$8-th basis is

$$e_{i,j}^{\mathrm{lin}} = \mathrm{LN}(W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}) + a_j,$$9

The resulting feature map is

$$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$0

and the embedding is produced by

$$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$1

with shared $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$2 and $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$3 (Wang et al., 3 Jun 2026).

For categorical features, the model uses a standard entity embedding lookup into $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$4, followed by the same LayerNorm. A key implementation detail is that RaBEL uses shared projection weights across columns. The paper attributes three effects to this tokenizer design: diverse activation patterns across value ranges, better conditioning of the first learned layer, and higher effective rank in shallow hidden states. In the model’s conceptual vocabulary, RaBEL is “value-aware” rather than merely feature-aware.

An exponent-gated extension is also proposed for inputs that span orders of magnitude or exhibit heteroskedasticity. With base $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$5, using $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$6, and $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$7, the log-magnitude is defined as

$$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$8

A soft assignment over exponent bins $$W_{\mathrm{lin}} \in \mathbb{R}^{d \times 1},\quad b_{\mathrm{lin}} \in \mathbb{R}^d,\quad a_j \in \mathbb{R}^d.$$9 is then computed by

$a_j$0

Each bin $a_j$1 has an embedding $a_j$2, together with a sign embedding $a_j$3, producing

$a_j$4

A shared MLP predicts

$a_j$5

with Softplus enforcing positivity. These values gate the RBF parameters: $a_j$6 and the gated basis becomes

$a_j$7

The paper lists three benefits of exponent gating: scale equivariance, heteroskedasticity robustness, and better conditioning. It also states that soft binning keeps the module differentiable and that the overhead is modest, consisting of an extra $a_j$8 kernel evaluation plus a small MLP (Wang et al., 3 Jun 2026). A plausible implication is that the method is intended not only to expand local representation rank but also to preserve stable optimization under wide feature-scale variation.

3. Reordered bidirectional routing: $a_j$9

The second major design component is the replacement of the standard feature-first bidirectional stack with a reordered block. Existing tabular transformer blocks are described as often following feature-attention $x_{i,j}$0 sample-attention $x_{i,j}$1 FFN, abbreviated $x_{i,j}$2. Two problems are identified with this ordering. First, the initial feature-attention layer operates on raw, weakly conditioned values. Second, in designs where prediction depends primarily on a target token, some attention computations do not strongly influence the final loss (Wang et al., 3 Jun 2026).

LimiX-2M instead adopts

$x_{i,j}$3

abbreviated $x_{i,j}$4, or SNF. The reported rationale is sequential: sample-attention first aggregates cross-sample context and column-level statistics; the FFN then compresses and conditions those aggregated signals; feature-attention last performs inter-feature mixing on improved representations. The paper states that this ordering allows the model to capture distributional statistics such as moments, prevalence, and missingness patterns before feature mixing (Wang et al., 3 Jun 2026).

The routing change is coupled with attention pooling over all feature tokens at readout. This means all attention computations contribute directly to prediction, in contrast to target-token-dominated designs where intermediate computations may be weakly supervised. In the model’s terminology, this aligns computation with the readout path. The stated consequence is a reduction of the “attention bottleneck,” both because feature-attention receives better-conditioned inputs and because the sample-attention pathway cannot be bypassed in the final prediction.

A toy DAG experiment in Section 5.3 is cited as qualitative support: SNF attention maps place more weight on true causal parents of the target, whereas the baseline FSN is dominated by self-attention. This suggests that the reorder is not solely a compute arrangement but also a representational prior about how tabular dependencies should be exposed to supervision.

4. Architecture and parameterization

LimiX-2M is specified as a 12-block Transformer backbone with hidden size $x_{i,j}$5 and $x_{i,j}$6 attention heads per attention module. Each block is composed of Sample-Attention, FFN, and Feature-Attention in that order. The model operates on a tensor of cell embeddings

$x_{i,j}$7

with $x_{i,j}$8 rows and $x_{i,j}$9 features (Wang et al., 3 Jun 2026).

The “2M” designation refers to the compact parameter budget achieved through smaller hidden width relative to larger foundation-model baselines. The paper states that the final model has 1.92M parameters. The stated argument is that the model’s gains do not arise from increasing parameter count, but from using the parameter budget more effectively: RaBEL improves the information content of the input embedding, SNF makes attention computations more useful for the final prediction, and the combination reduces redundancy that would otherwise require a larger model.

The paper also situates LimiX-2M against larger tabular foundation models. The comparison set includes TabPFN-v2 at 7.24M parameters, TabICL at 27.10M, Mitra at 75.67M, and LimiX-16M at 16.52M. Within that frame, LimiX-2M is presented as an accuracy–efficiency design point rather than a scale-maximizing model (Wang et al., 3 Jun 2026).

5. Training protocol, benchmarks, and evaluation design

Training follows the same general synthetic pretraining paradigm as PFN/LimiX. Tasks are generated from hierarchical Structural Causal Models (SCMs); a random DAG and functional mechanisms are sampled; and synthetic train episodes are drawn from the induced joint distribution. For the main benchmark protocol, datasets are split 64/16/20 into train/validation/test, hyperparameters are tuned with Optuna using 100 trials per method-dataset pair, early stopping uses validation accuracy for classification and RMSE for regression, and final results are averaged over 15 random seeds. For foundation models such as TabPFN-v2, the full dataset is evaluated according to the paper’s protocol (Wang et al., 3 Jun 2026).

The benchmark suite spans both classification and regression. Classification benchmarks are TALENT-CLS, OpenML-CC18, PFN-CLS, TabZilla, TabArena-CLS, and BCCO-CLS. Regression benchmarks are TALENT-REG, PFN-REG, TabArena-REG, CTR23, and BCCO-REG. The reported metrics are AUC, Accuracy, and F1 for classification, and $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$0 and RMSE for regression.

The baseline set is broad and covers tree ensembles, deep tabular models, and foundation models. Specifically listed are XGBoost, LightGBM, CatBoost, RF, ET, FT-Transformer, SAINT, TabNet, TabTransformer, DCN-v2, ResNet, ExcelFormer, T2G-Former, TabR, ModernNCA, MLP, MLP-PLR, SwitchTab, TABM, AutoInt, DANets, GrowNet, SNN, TabPFN-v2, TabICL, Mitra, and LimiX-16M (Wang et al., 3 Jun 2026). This breadth matters because the paper’s claims are not restricted to comparisons against only other TFMs; they are framed against a wider tabular learning ecosystem.

6. Empirical results and diagnostic analyses

The headline empirical claim is that LimiX-2M outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. In Table 6, LimiX-2M with 1.92M parameters is reported as the second-best overall on aggregated rank tables, trailing only LimiX-16M while outperforming TabPFN-v2, TabICL, Mitra, and many tree and deep-learning baselines (Wang et al., 3 Jun 2026).

Selected benchmark results reported in the paper are summarized below.

Benchmark	LimiX-2M	Compared baselines
BCCO-CLS	AUC 0.858, Acc. 0.787, F1 0.701	TabPFN-v2: 0.843 / 0.772 / 0.679; TabICL: 0.847 / 0.768 / 0.672
BCCO-REG	$$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$1, RMSE = 0.392	TabPFN-v2: 0.772 / 0.404; Mitra: 0.667 / 0.474
OpenML-CC18	AUC 0.935, Acc. 0.892, F1 0.799	TabPFN-v2: 0.929 / 0.887 / 0.786
PFN-CLS	0.913 / 0.848 / 0.766	TabPFN-v2: 0.910 / 0.845 / 0.756
TabArena-CLS	0.846 / 0.876 / 0.594	TabPFN-v2: 0.838 / 0.872 / 0.589

The efficiency comparison is likewise explicit. On a synthetic classification dataset with 900 samples and 60 features, GPU inference timing is reported as 171.40 ms for LimiX-2M, 352.60 ms for TabPFN-v2, 1749.61 ms for TabICL, and 5766.25 ms for Mitra. The paper interprets this as approximately $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$2 faster than TabPFN-v2 and more than $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$3 faster than TabICL on GPU in that setup (Wang et al., 3 Jun 2026).

The diagnostic analyses are intended to attribute these gains to the proposed mechanisms rather than incidental training choices. In a module ablation, Figure 1 shows incremental gains from baseline, $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$4RaBEL, $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$5RBA / reordered bidirectional attention, and full LimiX-2M. Structural ablations compare FSN, FNS, FNSN, SFN, SNF, and SNFN, with results favoring a sample-attention-first design and indicating that interleaving the FFN between attention layers provides the best balance of performance and parameter efficiency. A low-rank approximation experiment on TabPFN-v2 shows little performance loss even under severe rank truncation, which the paper uses to argue that current linear tokenizers underuse hidden width (Wang et al., 3 Jun 2026).

7. Rank diagnostics, implementation details, and interpretive context

A core empirical validation for RaBEL is its effect on shallow-layer rank metrics. The paper evaluates numerical rank, Rank@99%, and Rank@95%, where numerical rank counts singular values above a tolerance and Rank@99% and Rank@95% denote the minimum number of singular components preserving 99% and 95% spectral energy, respectively. In Table 5, compared with a 2M baseline using linear projection, LimiX-2M raises numerical rank from 58.41 to 78.62, Rank@99% from 13.94 to 25.35, and Rank@95% from 6.73 to 12.31 (Wang et al., 3 Jun 2026). These figures are presented as direct evidence that RaBEL mitigates low-rank collapse.

The appendix further reports RaBEL hyperparameter ablations over embedding dimension, number of kernels, bandwidth $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$6, initialization method, kernel form, and fixed versus learnable $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$7. The reported findings are that embedding dimension around 32 is a good trade-off, 64 kernels performs best among tested values, fixed $$z_{i,j}^{\mathrm{lin}} = W_{\mathrm{lin}} x_{i,j} + b_{\mathrm{lin}}.$$8 is strong, orthogonal initialization is notably better than Xavier or Kaiming, and uniform kernel placement outperforms random placement. These details refine the design space without altering the model’s main conceptual claims.

Several implementation notes are stated explicitly. RaBEL uses shared projection weights across columns. For categorical columns, standard embedding lookup followed by LayerNorm is used. Exponent gating is differentiable because of soft binning, and its overhead is described as small. Model checkpoints and inference code were released at the project repository. The paper also notes that TabPFN-v2.5 was not publicly available during the main experiments, so the primary comparisons focus on TabPFN-v2 and TabICL; a later check on BCCO showed LimiX-2M remains competitive with TabPFN-v2.5 (Wang et al., 3 Jun 2026).

In broader methodological terms, LimiX-2M advances a specific thesis about TFMs: tokenizer design and routing order are first-class determinants of the accuracy–efficiency trade-off. The paper’s interpretation is that standard affine scalar tokenization induces low-rank, low-sensitivity value channels; localized nonlinear basis expansion repairs the input geometry; and attention reordering ensures that cross-sample statistics are aggregated before feature mixing and are directly coupled to the readout. A plausible implication is that, for tabular foundation models, representation rank and readout alignment may be at least as consequential as simply scaling hidden width or parameter count.