Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models

Abstract: Transformer-based tabular foundation models (TFMs) dominate small to medium tabular predictive benchmark tasks, yet their inference mechanisms remain largely unexplored. We present the first large-scale mechanistic study of layerwise dynamics in 6 state-of-the-art tabular in-context learning models. We explore how predictions emerge across depth, identify distinct stages of inference and reveal latent-space dynamics that differ from those of LLMs. Our findings indicate substantial depthwise redundancy across multiple models, suggesting iterative refinement with overlapping computations during inference stages. Guided by these insights, we design a proof-of-concept, looped single-layer model that uses only 20% of the original model's parameters while achieving comparable performance. The code is available at https://github.com/amirbalef/is_one_layer_enough.

• The paper demonstrates that iterative application of a single transformer layer can match the performance of deeper architectures, highlighting depth redundancy in TFMs.
• It employs techniques like cosine similarity, probing classifiers, and logit lens evaluations to reveal progressive feature refinement and self-repair dynamics across layers.
• The study advocates for recurrent transformer designs that leverage robust early-layer encoding to achieve parameter efficiency and adaptable, anytime predictions.

Detailed Summary of "Is One Layer Enough? Understanding Inference Dynamics in Tabular Foundation Models"

Introduction and Motivation

This paper critically examines the internal inference mechanisms governing transformer-based tabular foundation models (TFMs) with in-context learning (ICL). Despite TFMs achieving state-of-the-art prediction performance on small to medium tabular tasks, their architectural and mechanistic differences from LLMs have hindered the development of mechanistic understanding and principled architectural improvements. The core focus is on elucidating layerwise inference dynamics, identifying depthwise redundancy, and evaluating how representations evolve within TFMs as compared to LLMs.

The study draws its motivation from the significant divergence of TFMs from LLMs: encoder-only architectures, row-invariant input handling, non-autoregressive inference, and reliance on synthetic data priors. The paper further emphasizes the value of mechanistic interpretability for detecting failure modes, enhancing robustness, and guiding the development of efficient recurrent architectures.

Methods and Model Selection

The empirical component rigorously analyzes six state-of-the-art TFMs: TabPFN(v1), TabPFN(v2), TabPFN(2.5), TabICL, LimiX-2M, and LimiX-16M, using binary and multiclass classification benchmarks (TabArena and PMLBmini). The experiments are designed to probe layerwise representations, feature separability, information flow, and structural robustness. The suite includes:

• Embedding similarity (cosine/CKA)
• Separation gap (inter/intra-class distances)
• Probing classifiers (linear/logistic regression transferability between layers)
• Tabular logit lens (layerwise decoder evaluation)
• Layer ablation (skipping, repeating, swapping)
• Self-repair analysis (layerwise recovery of performance after ablations)
• Proof-of-concept looped single-layer TFM

Representation Similarity and Layerwise Dynamics

The embedding similarity analysis reveals that non-adjacent layers tend to operate within highly similar representation spaces, particularly in deeper models (TabPFN(2.5), LimiX-16M), as measured by high block-diagonal cosine similarity and CKA.

Figure 1: Cosine and CKA similarity patterns across layers reveal block structures and gradual embedding changes in deep TFMs.

These results imply incremental updates, depthwise redundancy, and overlapping computations during inference.

Separation Gap and Feature Discrimination

The separation gap experiment tracks intra- vs inter-class sample distances across layers, demonstrating an incremental increase in feature discrimination. This metric supports the hypothesis that representation refinement occurs iteratively, enhancing class separation with each layer.

Figure 2: Layerwise separation gap for support and query embeddings indicates progressive improvement in class distinction.

Models with robust input encoders (LimiX-2M, TabICL) achieve high initial separation, while others display abrupt jumps in separation at specific layers.

Information Transfer and Probing Classifiers

Probing experiments, using logistic regression and other classifiers trained on intermediate layer embeddings, show asymmetric transfer: probes trained on lower layers retain predictive power when applied to higher layers, but not vice versa. This indicates that later layers cumulatively encode richer features, supporting iterative inference rather than strictly modular circuits.

Figure 3: Probing classifier AUC visualizes forward enrichment of representations and transferability across layers.

Tabular Logit Lens and Early-Exit Analysis

The adapted "logit lens" experiment, which pretrains individual layer-specific decoders, demonstrates that reliable prediction can be achieved in early layers for all models, though these intermediate representations may not be fully aligned with the final decoder. Sharp improvements in ROC-AUC and balanced accuracy at certain layers suggest pronounced inference stages.

Figure 4: Individually trained decoders exhibit high early-layer performance, with abrupt gains for the original decoder at distinct layers—indicating layered inference stages.

Layer Ablation, Repetition, and Structural Robustness

Layer ablation analyses identify early layers as most critical to final prediction performance, while middle and later layers manifest robustness and self-repair capabilities. Repeating certain layers marginally improves performance, and swapping adjacent layers leads to pronounced performance degradation in TFMs—unlike LLMs, whose middle layers are more robust due to circuit-like redundancy.

Figure 5: Performance losses from skipping/repeating/swapping layers highlight the functional specialization of early layers and redundancy in later blocks.

Self-Repair Phenomena

Assessment of intermediate predictions after layer ablation using the logit lens shows that recovery of performance (self-repair) is predominantly observed in middle and later layers, corroborating the overlapping computational role of these layers.

Figure 6: Layerwise performance recovery after skipping interventions demonstrate self-repair except for the first layer.

Depthwise Redundancy: Is One Layer Enough?

Leveraging these insights, the authors build a looped single-layer transformer (nanoTabPFN$_{looped}$) and empirically show that repeated application of a single layer during training and inference matches the predictive performance of a model with equivalent computation but greater depth, with only 20% the parameter count. However, a single non-repeated layer is insufficient, confirming the necessity of iterative computation.

Figure 7: Performance comparison between nanoTabPFN$_{6l}$, nanoTabPFN$_{1l}$, and nanoTabPFN$_{looped}$ demonstrates that iterative reuse recovers the benefits of depth.

Theoretical and Practical Implications

The results strongly advocate for the iterative inference hypothesis over strictly modular circuit models in TFMs. Early layers perform crucial mappings from input encoders, while later layers focus on iterative refinement and ensemble-like feature engineering. Block-layer structures are present but show greater sensitivity to layer ordering than in LLMs. These findings rationalize recurrent architectures, inform model compression strategies, and motivate robust encoder designs.

Practically, robust early-layer representations (via express encoding mechanisms) yield improved feature discrimination and less reliance on deep transformer stacks, facilitating parameter efficiency and reduced inference costs. Looped-layer models enable anytime predictions and adaptive computation, offering compelling opportunities for industrial deployment.

Future Directions in AI Research

Three major avenues are identified:

• Extending recurrent transformer designs to larger-scale TFMs
• Mechanistic and circuit-level interpretability, targeting neuron-level analysis
• Examining the impact of synthetic priors and pre-training protocols on inference dynamics and generalization

Further comparison to LLM-based tabular predictors, assessment beyond average performance (e.g., ensemble effects, task complexity), and application of these strategies to regression and multiclass objectives are recommended for comprehensive generalization.

Conclusion

This work gives a large-scale mechanistic account of layerwise inference in TFMs, concludes that most layer depth in TFMs is redundant for predictive performance, and demonstrates that iterative refinement—not modular specialization—dominates their inference dynamics. Efficient recurrent models, robust input encoding, and principled depth reduction are justified. The results challenge prevailing assumptions about the necessity of deep transformer stacks and provide actionable insights for next-generation tabular AI architectures.