LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models

Abstract: Tabular foundation models (TFMs) increasingly rival tree ensembles, but their performance is often compute-inefficient: with standard affine scalar tokenization, each feature injects value variation through an essentially one-dimensional channel, and feature IDs/positional signals cannot increase within-feature value degrees of freedom, yielding weak early-layer value sensitivity and redundant hidden states. We present a unified \emph{tokenize-and-route} framework for strong TFMs: \textbf{RaBEL} expands each scalar into compact localized RBF features (optionally exponent-gated) to improve conditioning and shallow-layer effective rank, while a reordered bidirectional block \textbf{S$\rightarrow$N$\rightarrow$F} aligns computation with the readout by aggregating cross-sample context before feature mixing and using attention pooling. Together, these changes yield \textbf{LimiX-2M}, a 2M-parameter model that outperforms larger TabPFN-v2 and TabICL baselines on widely used tabular benchmarks while reducing training and inference costs. These results highlight value-aware tokenization and readout-aligned routing as key levers for improving the accuracy--efficiency trade-off in TFMs. Model checkpoints and inference code are available at https://github.com/limix-ldm-ai/LimiX.

• The paper introduces RaBEL, a nonlinear RBF-based embedding that improves effective input rank and addresses low-rank collapse in tabular data.
• It proposes a reordered sample-attention-first mechanism to enhance contextual signal and mitigate attention bottlenecks in transformer layers.
• Experimental results show that LimiX-2M, with only 2M parameters, achieves superior efficiency and accuracy compared to significantly larger tabular models.

Authoritative Summary of "LimiX-2M: Mitigating Low-Rank Collapse and Attention Bottlenecks in Tabular Foundation Models" (2606.04485)

Introduction and Context

Tabular foundation models (TFMs) have recently exhibited strong performance on classic tabular prediction tasks, frequently surpassing traditional tree ensembles and earlier deep learning baselines. However, prevailing modeling approaches in TFMs commonly utilize scalar-affine tokenization for numerical features, limiting the representational capacity and leading to pronounced low-rank collapse in shallow layers. This inefficiency results in overparameterized models, hampering both performance and computational efficiency. Additionally, existing bidirectional attention mechanisms in TFMs (primarily a feature-attention → sample-attention sequence) contribute to information bottlenecks and ineffective routing of contextual information.

The paper introduces two architectural innovations to address these impairments:

• RaBEL (Radial-Basis Embedding Layer): A nonlinear, compact RBF-based expansion for scalar cell values, designed to enhance the effective input rank and front-load value-aware nonlinearity.
• Reordered Bidirectional Attention (Sample → FFN → Feature): A revised attention stacking, aiming to exploit sample-level context prior to feature aggregation, ensuring all attention computations are structurally aligned with the readout pathway.

These components are instantiated in LimiX-2M, a compact 2M-parameter TFM that demonstrates superior accuracy-to-efficiency trade-offs against significantly larger contemporary tabular foundation models.

Diagnosis of Representation Bottlenecks

The paper provides a thorough mechanistic and empirical analysis of representational bottlenecks in standard TFM tokenizers:

• Low-Rank Collapse: Scalar-affine tokenization of numerical features (optionally augmented with feature IDs) confines value information to a one-dimensional subspace per feature. Theoretical analysis shows that, under these conditions, the effective rank of early-layer feature matrices is strictly bounded and decoupled from the hidden width, resulting in excessive redundancy.
• Spectral Analysis: Empirical singular value decompositions across multiple OpenML-CC18 benchmarks confirm extremely rapid rank decay; a small fraction of principal components accounts for nearly all energy in shallow hidden states.
• Ablations: Truncating the rank of hidden states to a fraction (as low as 10-25%) of the embedding dimension yields negligible performance decay, underscoring severe underutilization of parameter budgets.

These findings are linked to strong functional limitations: depth alone cannot compensate for insufficient value diversity at the input; thus, representation bottlenecks must be relieved via value-aware tokenization.

The RaBEL Embedding Layer

RaBEL fundamentally replaces the one-shot linear mapping of each scalar with a compact radial-basis expansion. Key characteristics:

• Per-Feature RBF Bank: For each numeric feature, a bank of $M$ learnable RBFs is initialized at empirical value quantiles, with bandwidth parameters adaptively learned.
• Exponent Gating: Value ranges in real-world tabular data routinely span multiple orders of magnitude. RaBEL includes an exponent-gated mechanism, leveraging a differentiable binning assignment to adjust RBF centers and widths as a function of input scale. This provides:
  • Scale equivariance: Value shifts multiplicatively adapt the receptive field of the RBF basis.
  • Heteroskedasticity robustness: Adaptive kernel widths maintain locality in both high and low variance regimes.
  • Stable optimization: LayerNorm and smooth soft assignments ensure gradient flow.
• Projection to Model Dimension: The expanded RBF features are then linearly projected to the model width, enabling downstream processing.
• Categorical Handling: Categorical features remain embedded via identity lookups.

This design sharply increases the value-induced rank and local curvature in early activations, empirically verified through Jacobian and SVD diagnostics.

Reordered Bidirectional Attention

The traditional bidirectional attention paradigm (feature-first order) introduces two main drawbacks:

1. Weak Early Contextualization: Feature-attention across columns at the first layer operates on poorly conditioned inputs.
2. Supervised Signal Dilution: Sample-level attention computed after feature-level mixing is often excluded from the token(s) used for the final readout.

The reordered S→FFN→F block instead:

• Sample-Attention First: Aggregates cross-sample distributional statistics (mean, prevalence, missingness patterns) per feature, enriching the feature tokens with global context before column mixing.
• FFN Compression: An intermediate feedforward network further conditions these representations.
• Feature-Attention Final: Latent representations interact across features with well-conditioned, context-rich inputs, and are directly aligned to the downstream readout by attention pooling.
• Empirical Analysis: Visualizations on synthetic causal graphs show broader and more informative attention maps, with prominent allocation to latent causal parents, as opposed to the heavy self-attention bias in prior stacks.

Experimental Results

Model Scale and Efficiency

• Parameter Economy: LimiX-2M (1.92M parameters) achieves accuracy and ranking superiority over TabPFN-v2 (7.24M), TabICL (27.1M), and Mitra (75.7M) on multiple SOTA benchmarks, with comparable performance to LimiX-16M (16.5M). Inference on standard hardware establishes a 2x to 10x speedup over comparable parameter and larger models.
• Generalization: LimiX-2M consistently achieves 2nd-best rank (often trailing only LimiX-16M), and in most cases, outperforms traditional boosting ensembles (XGBoost, CatBoost, LightGBM) and neural tabular baselines across widely used benchmark suites (TALENT, OpenML-CC18, BCCO, TabArena, TabZilla, PFN-CLS/REG, CTR23).

Embedding Ablations

• RaBEL vs. MLP/Piecewise/Periodic: Across both MLP and transformer backbones, RaBEL delivers higher AUC, R², and F1 relative to MLP, piecewise-linear, and periodic embeddings, with especially pronounced gains on datasets with complex marginal distributions or heavy-tailed numerics.
• Robustness to Hyperparameters: Performance is stable across RBF bank sizes and initialization choices. Orthogonal initialization of exponent embeddings provides optimal results.

Sensitivity to Attention Ordering

• Ordering Ablation: Across several benchmarks, the sample-attention-first (S→F) and intermediate (SFN, SNF) topologies dominate canonical feature-attention-first stacks, validating the mechanistic premise that early sample aggregation is crucial for robust feature mixing and generalization.

Theoretical and Practical Implications

This work demonstrates that:

• Value-aware tokenization is essential for efficient, scalable TFMs. Nonlinear, localized expansions at the tokenizer level are mandatory for high-rank, expressive early representations; network width without such mechanisms merely inflates parameter redundancy.
• Attention routing critically affects model interpretability and path dependence. Alignment of attention sequence with prediction readout ensures that every computation contributes meaningfully and detectably to output estimation, facilitating both higher accuracy and improved attributions.
• Resource-efficiency is achievable without large-scale overparameterization. 2M-parameter LimiX-2M matches or outperforms foundation models with 4–40× parameter counts, enabling practical deployment on resource-constrained hardware for real-world tabular prediction.

Contradictory/Strong Claims

• The assertion that "IDs/positional signals can separate columns but cannot increase the intrinsic degrees of freedom through which values enter the model" is strongly supported both theoretically and empirically.
• The claim that simply widening model width is insufficient to address the value bottleneck, and may actually increase redundancy, directly contradicts conventional Transformer scaling practice in NLP and vision.

Future Directions

• Hybrid kernel and periodic basis libraries could further improve tokenizer flexibility, especially on datasets with substantial local periodicity or interaction effects.
• Stronger self-supervised and meta-pretraining (e.g., masked modeling, causal induction) are promising for scaling TFMs to new data modalities and enhancing robustness to distribution shifts.
• Systematic ablation on cross-dataset/generalization and exploration of architectural synergy with recent advances in in-context learning for tabular data are highlighted as future priorities.

Conclusion

LimiX-2M introduces principled architectural modifications at both the embedding and attention-routing levels, demonstrating that value-aware tokenization and sample-aligned attention sequencing are critical for high efficiency and top-tier predictive accuracy in TFMs. The architecture establishes a new benchmark for parameter and compute efficiency in the field, with immediate implications for the design of tabular AI systems deployable at scale and on resource-constrained platforms. Further investigation of hybrid basis methods and richer pretraining schemes may unlock additional gains in out-of-domain and lifelong learning settings.