Why Large Language Models Fail at Tabular Prediction

Abstract: LLMs have become the default tool for a remarkable range of tasks, yet they have had conspicuously little success at one of the most common machine learning workloads: predictive analytics over tabular data. This gap is the founding premise of the fast-growing field of tabular foundation models, but the question of why generic LLMs fail has remained open. We study a frontier LLM in its purest inference regime - a single generation pass over a prompt containing the full training and test data, with no tools, no agentic scaffolding, and no fine-tuning - and systematically evaluate five hypotheses for the failure: (a) an inability to handle noisy or non-linearly-separable data; (b) the linearised CSV format obscuring column structure; (c) the tokenisation of numeric values; (d) the number of test points classified per query; and (e) the dimensionality of the input. Controlled experiments falsify (a)-(d). Dimensionality, in contrast, is decisive: sweeping random linear projections of thirty-one benchmark datasets, the LLM is the only method among nine whose accuracy decreases as dimensionality grows, while every classical baseline stays flat or improves. A behavioural comparison against 252 configured classical models finds that in two dimensions the LLM predicts like a local, distance-based method (up to 91.6% grid agreement), but in higher dimensions no classical model - even when augmented with tuned, dimension-dependent noise - reproduces its predictions. We do not claim to have identified the internal mechanism; our results show, more modestly, that the LLM's capability dissolves with dimension in a way no noise-corrupted classical learner mimics - which explains why LLMs, so capable elsewhere, keep losing to fifty-year-old baselines on tables, while leaving the mechanism of the prediction as an open question.

• The paper establishes high dimensionality as the key factor driving LLM performance collapse in tabular prediction tasks, refuting other practitioner hypotheses.
• The study employs controlled experiments with random projections and synthetic data to compare LLMs with classical tabular models, revealing stark performance discrepancies.
• The findings imply that while LLMs function in low-dimensional settings, they require purpose-built architectures for effective high-dimensional tabular inference.

Failure Modes of LLMs in Tabular Prediction

Overview

This study systematically interrogates the inability of LLMs to perform supervised classification on tabular data via in-context learning, relative to classical and purpose-built models. Despite the broad versatility of LLMs in a range of textual and code-related domains, their persistent failure on standard tabular learning tasks and benchmarks motivates a precise exploration of what causes this deficiency and how it manifests, moving beyond anecdotal or post-hoc rationalizations.

The authors enumerate five concrete hypotheses for LLM underperformance on tables and, through tightly controlled experimental interventions, falsify all but one—high dimensionality. Their results show that while LLMs can mimic local, distance-based predictors in low-dimensional settings, their accuracy collapses uniquely and sharply with increasing dimensionality, in contradistinction to classical and tabular-specialist baselines. The persistence of this collapse, even under random linear projections that preserve task information, pinpoints a core quantitative bottleneck in in-context prediction with LLMs for tabular data.

Hypothesis Testing and Experimental Protocol

The investigation is centered on five practitioner-motivated hypotheses:

• H1: Separability—whether noisy or overlapping classes explain failure.
• H2: Serialization (Flattened CSV)—whether loss of column structure or "vertical" reading impairs pattern detection.
• H3: Numeric Tokenization Precision—whether granularity and length of numeric tokens degrade effectiveness.
• H4: Test Batch Size—whether prediction per pass is limited by context window or output constraints.
• H5: Dimensionality—whether increasing the number of features alone is determinative.

For each, the authors design targeted interventions and control for confounds: translating class centroids for separability, embedding target columns directly in the serialized input for format, rounding numbers for tokenization, varying the number of labels queried per pass, and leveraging random Gaussian projections to probe dimensionality effects with fixed information content. Importantly, a memorization probe is employed to identify and discard benchmark datasets contaminated via LLM training exposure, operationalizing strong data hygiene throughout.

Main Findings: Only Dimensionality Remains

Falsified Explanations

The first four hypotheses are empirically rejected:

• LLMs continue to underperform tree ensembles, GPs, and kNN even on fully linearly separable data and across all separability degrees.
• Models trivially copy a target column in a "needle-in-haystack" task regardless of input width, indicating that serialization and format are not limiting.
• Varying numeric precision or the number of query targets per call yields no measurable accuracy impact.
• No classical method is ever uniquely susceptible to any of these interventions; only the LLM is.

Dimensional Collapse

In contrast, dimensionality emerges as the singular decisive factor:

• In random-projection sweeps of real and synthetic datasets up to $d=60$ columns, all baseline models (including logistic regression, forests, boosting, GPs, and kNN) either maintain or improve classification accuracy as features are added—consistent with the preservation of task signal under Johnson-Lindenstrauss-type projections.
• LLM accuracy, normalized to majority-class guessing and controlled for contamination, decreases monotonically in $d$, falling to or below chance by $d\geq16$ on most tasks.
• This effect persists in synthetic settings where the intrinsic problem complexity (the data manifold) is strictly two-dimensional and all additional features are pure projections, decisively excluding explanations based on absolute information content or real-world covariate structure.

Behavioral Surrogacy and Non-classical High-dimensional Failure

The study advances a behavioral analysis of LLM classifier outputs:

• In two dimensions, LLM prediction boundaries align almost perfectly (91–92% agreement) with local, distance-based models—Gaussian processes with short length scales and 1-NN classifiers—across a diverse synthetic task suite.
• However, in high dimensions, no configuration among a sweep of 252 classical model pipelines (incorporating SVMs, tree ensembles, naive Bayes, discriminant analysis, and more) captures more than 64.8% prediction agreement with the LLM. Furthermore, augmenting baselines with dimension-dependent majority-class label noise provides only marginal additional agreement ($<0.7$ percentage points).
• Multidimensional scaling on pairwise accuracy profiles confirms the lack of a close surrogate for LLM prediction in high dimensions: LLM outputs become isolated in accuracy space and do not collapse onto any standard learner's behavior, nor merely degrade into random guessing in a matching fashion.

Model Explanations and Faithfulness

Prompted self-explanations of LLM predictions are assessed for their alignment with both ground-truth structure and the model's own behavior. These traces rarely match both the model output and the underlying data-generating rule; only in a minority of cases is the stated verbal rule a faithful account of the classifier's decision boundary. This complements results in LLM explanation faithfulness literature, emphasizing that self-explanations in tabular inference are unreliable proxies for underlying computation.

Implications

Theoretical

The findings imply an intrinsic failure of current LLM architectures, trained for autoregressive next-token prediction on textual data, to implement in-context tabular inference as dimensionality grows. This does not stem from representation or context length, but from the inability of the model to perform non-local, multi-column pattern integration in high dimension—a key difference from most classical tabular learners. The precise mechanistic reason—whether this is due to token-level context window structure, lack of table-competence pretraining, or limitations of model attention scaling—remains an open question.

Practical

For practical data science and ML pipelines, these results concretely advise against using LLMs for direct in-context learning on tabular data as column width increases. No amount of prompt modification, precision tuning, or batching can overcome this collapse; only purpose-built tabular architectures—ensemble trees, tabular GPs, or neural tabular foundation models—consistently deliver high performance. Scalability is further hindered by the prohibitive cost of LLM inference at anything but toy data scale.

Towards Tabular Foundation Models

The experimental isolation of dimensionality as the behavioral bottleneck motivates the ongoing development of tabular foundation models designed from first principles for natively tabular data. Unlike LLMs, these models (e.g., TabPFN, Nexus, TabICL) demonstrate robust in-context learning irrespective of feature width, emphasizing the necessity of specialized architectures for this domain.

Conclusion

This work delivers a rigorous quantitative account of why LLMs fail to deliver competitive accuracy on tabular prediction tasks in an in-context inference regime. By methodically eliminating common practitioner intuitions—data noise, serialization, and numeric tokenization—it establishes dimensionality as the singular factor responsible for the intractable accuracy decline, with no comparable effect observed for classical ML baselines. The LLM's ability to act as a distance-based local classifier vanishes as feature count grows, and its high-dimensional predictions are not recapitulated by any classical or noise-degraded learner, with model explanations providing little insight into this degeneration.

For the ongoing development of robust, general-purpose foundation models in machine learning, these results underscore both a concrete limitation of LLMs and a target for future mechanistic work—namely, to understand and bridge the chasm between in-context learning in low and high dimensions for non-sequential data modalities such as tables.

Reference:

"Why LLMs Fail at Tabular Prediction" (2608.02412)