The Illusion of Generalization: Re-examining Tabular Language Model Evaluation

Abstract: Tabular LLMs (TLMs) have been claimed to achieve emergent generalization for tabular prediction. We conduct a systematic re-evaluation of Tabula-8B as a representative TLM, utilizing 165 datasets from the UniPredict benchmark. Our investigation reveals three findings. First, binary and categorical classification achieve near-zero median lift over majority-class baselines and strong aggregate performance is driven entirely by quartile classification tasks. Second, top-performing datasets exhibit pervasive contamination, including complete train-test overlap and task-level leakage that evades standard deduplication. Third, instruction-tuning without tabular exposure recovers 92.2% of standard classification performance and on quartile classification, format familiarity closes 71.3% of the gap with the residual attributable to contaminated datasets. These findings suggest claimed generalization likely reflects evaluation artifacts rather than learned tabular reasoning. We conclude with recommendations for strengthening TLM evaluation.

• The paper demonstrates that performance gains in Tabular LMs are largely due to evaluation artifacts, contamination, and instruction-following effects.
• It reveals that high aggregate accuracy is driven by quartile classification tasks, while binary and categorical tasks show minimal improvement over baseline predictions.
• The study recommends rigorous contamination controls and detailed baseline reporting to clearly differentiate genuine tabular reasoning from simple format adaptation.

Critical Analysis of "The Illusion of Generalization: Re-examining Tabular LLM Evaluation" (2602.04031)

Introduction and Context

"The Illusion of Generalization: Re-examining Tabular LLM Evaluation" provides a comprehensive systematic re-evaluation of Tabular LLMs (TLMs) with a focus on the Tabula-8B model and the UniPredict benchmark. TLMs represent a recent paradigm wherein LLMs are fine-tuned or pre-trained on serialized tabular data to perform zero- and few-shot classification, regression, and imputation tasks across a wide range of real-world tabular datasets. The core premise motivating TLM research is that LLM-like universality and emergent generalization can be transferred to tabular reasoning through large-scale language modeling.

However, this study scrutinizes the empirical and methodological basis for such claims by decomposing performance, auditing datasets for contamination, and empirically separating format-following from tabular knowledge. The analysis is deliberately technical, targeting field standards and revealing that previous claims of generalization may be better explained by evaluation artifacts, contamination, and instruction-following effects rather than tabular-specific inductive learning.

Decomposition of Aggregate Metrics and Task-Type Analysis

A central finding is that TLM evaluation practices typically conflate performance across heterogeneous tabular prediction tasks—namely binary classification, categorical classification, and regression discretized as quartile classification. The authors show that high aggregate accuracy is almost exclusively driven by quartile classification tasks, which possess balanced label distributions and strong feature-label correlations due to construction.

Analysis of raw and baseline-lifted accuracy (improvement over the majority-class predictor) reveals that, on binary and categorical tasks, TLMs such as Tabula-8B display minimal to negative lift over the naïve baseline, with a substantial proportion of datasets where no learning is evident beyond class frequency exploitation.

Figure 1: Task-type decomposition reveals quartile tasks dominate aggregate accuracy improvements, while binary/categorical performance provides negligible lift over the majority-class baseline.

Further, categorical tasks sometimes exhibit negative median lift, indicating systematic underperformance relative to trivial heuristics. The study supports these findings with statistical testing (ANOVA and pairwise $t$-tests) confirming significant heterogeneity in TLM efficacy across task categories.

Prevalence of Data Contamination

The study provides rigorous evidence of widespread contamination in commonly used tabular benchmarks and TLM training corpora. The following classes of contamination are documented:

• Verbatim train-test overlap: Evaluation samples appearing as-is within the pretraining corpus, as shown for the us-womens-labor dataset where all held-out test samples and their labels are present during training.
• Direct label leakage: Records that, through multi-source replication (e.g., financial time series), expose ground-truth targets in alternative tables during pretraining.
• Task-level association leakage: Tasks reducible to factual recall (e.g., mapping a date to day of week) are rendered trivially solvable because the underlying associations are memorized from unrelated tables.

  Figure 2: Accuracies lifted over majority-class baseline illustrate contamination disproportionately benefits quartile and certain categorical tasks, visible in distribution tails.

These findings invalidate the notion that strong performance on such contaminated datasets constitutes evidence of architectural or learning advances in tabular generalization.

Role of Instruction Tuning and Format Familiarity

A major result is that instruction-following capability dominates performance improvements for TLMs. The authors fine-tune Llama-3-8B on Alpaca (general-purpose instruction data, with no tabular content) and show that this model closes over 92% of Tabula-8B's performance on standard classification tasks. For quartile classification, fine-tuning Alpaca on just the quartile-format problem reduces the remaining gap by 71%, and all residual differences are attributable to contaminated datasets.

Figure 3: Lift over majority-class baseline for Tabula-8B, Base Llama, Alpaca, and Alpaca+Q demonstrates instruction-following (Alpaca) recovers nearly all performance, with quartile-format exposure (Alpaca+Q) explaining the rest.

This demonstrates that format adaptation and surface-level instruction learning, not structural tabular reasoning, explain the apparent gains. When evaluating with decontaminated or previously unseen formats, Tabula-8B's supposed tabular-specific advantage vanishes.

Deficiencies in Standard Evaluation Practices

The paper demonstrates that standard TLM evaluation pipelines almost universally omit:

• Majority-class or trivial baseline reporting
• Chance-adjusted agreement metrics (Cohen's κ)
• Per-task-type breakdown and hypothesis testing for performance heterogeneity
• Contamination and redundancy analysis at or beyond the row level

  Figure 4: Distribution of Cohen’s κ by task type confirms sub-chance agreement for many datasets, especially outside quartile classification.

The literature review confirms that these omissions are not specific to Tabula-8B but are endemic across leading TLM papers, contributing to systemic overestimation of true generalization.

Implications and Recommendations

The practical implication is that current TLM evaluation artifacts create the illusion of generalization, with true zero- or few-shot reasoning unproven on uncontaminated, genuinely tabular tasks. Theoretical claims regarding cross-domain inductive transfer from sequential modeling architectures to permutation-invariant, heterogeneous tabular reasoning are therefore unsupported by available evidence.

The authors offer specific recommendations for the TLM community:

• Always report and analyze baseline and chance-corrected metrics.
• Stratify performance by native task type and conduct rigorous heterogeneity testing.
• Publicly release evaluation infrastructure and code for independent auditing.
• Implement comprehensive contamination and redundancy decontamination, including entity-level and fuzzy matching.
• Audit whether benchmarks require tabular, rather than text or associative, reasoning.
• Use instruction-tuned baselines without tabular pretraining as critical controls.
• Reviewers should require all above practices as minimum standards for publication.

Future Directions

Further work should formalize contamination detection at the entity and schema levels, develop benchmarks that cannot be solved by memorization, and focus on tasks that exploit permutation invariance and heterogeneity (genuine properties of tabular data). The distinction between format generalization and structural generalization must be enforced in evaluation design. Progress in TLMs will require clear demonstration of tabular-specific inductive capabilities compared to well-tuned tree ensembles and specialized neural architectures.

Conclusion

This study rigorously deconstructs the empirical evidence for generalization in TLMs, showing that much of the observed performance stems from evaluation artifacts, instruction-following, and contamination, not from architectural or learning advances on tabular data. Claims of emergent tabular reasoning by large-scale language modeling thus require a critical reappraisal, and the field must adopt more stringent, contamination-aware evaluation protocols to distinguish genuine generalization from illusory effects. The recommendations provided serve as urgent guidelines to realign future research, benchmarking, and deployment of TLMs with the fundamental principles of empirical rigor and reproducibility.