RAG-TabICL: Scalable Tabular Learning

• RAG-TabICL is a framework that splits prediction between a custom tabular retriever and a post-trained LLM, transforming few-shot into any-shot learning.
• It reduces token consumption by retrieving a fixed-size, highly relevant support set, overcoming context-length limits in tabular data processing.
• The method enables scalable tabular learning with ensemble diversity, while highlighting the practical need for dataset-specific retrieval engineering.

Searching arXiv for the target paper and closely related tabular ICL references. Retrieval-Augmented Tabular In-Context Learning (RAG-TabICL) denotes a framework for scalable large-language-model-based tabular in-context learning in which prediction is split between a tabular retriever and a post-trained LLM. It was introduced in "Scalable In-Context Learning on Tabular Data via Retrieval-Augmented LLMs" (Wen et al., 5 Feb 2025) to address a central bottleneck in TabICL: tabular rows become token-expensive when serialized as text, so standard prompting is constrained to few-shot settings by context-length limits. RAG-TabICL replaces the attempt to place large portions of a training table directly into the prompt with retrieval of a fixed-size, highly relevant support set, thereby turning LLM-based TabICL from a few-shot method into an any-shot method in the sense that prediction need not depend on fitting the full training table into the context window (Wen et al., 5 Feb 2025).

1. Problem formulation and motivation

The framework arises from prior LLM-based tabular learning in which a base LLM is post-trained on serialized tabular data and then prompted with in-context examples. The motivating claim is that such approaches work only for a small number of examples because each tabular instance may contain many columns and plain-text serialization consumes substantial tokens. The paper explicitly characterizes current LLM-based TabICL methods as limited to few-shot scenarios due to the sequence length constraints of typical LLMs, and further notes that even longer-context models face cost and long-context reasoning degradation (Wen et al., 5 Feb 2025).

RAG-TabICL addresses this limitation by assuming that, for a test instance, only a small support set is required for accurate prediction. The conceptual move is therefore kNN-like: instead of conditioning the LLM on the entire training set, the method retrieves a compact subset of relevant rows and uses only those rows as in-context demonstrations. This preserves the text-based, schema-flexible interface of LLM tabular learning while avoiding direct dependence on the full table being placed in the context.

The original LLM-based TabICL objective is reformulated as

$$E_{(x^T_{\cdot}, y^T_{\cdot}), T} \left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, D^T_{train} \right) \right) \right],$$

and is then decoupled into a retrieval-augmented form:

$\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$

Here, $Q^{LLM}_\theta$ is the LLM predictor, $S_x(\cdot)$ and $S_y(\cdot)$ serialize inputs and labels into text, $TabRAG$ is the retrieval module, $C_{x^T_{test}}$ is the retrieved support set, and the support-set size is fixed to $N^C$ (Wen et al., 5 Feb 2025). The practical significance of this reformulation is that the model now conditions on a bounded context extracted from an unbounded or large training pool.

2. Representation of tabular data and the retrieval-augmented architecture

RAG-TabICL retains text-based serialization as the interface between tables and the LLM. A tabular instance $(x,y)$ is converted into a language-like or Markdown-like format, and the formulation supports classification, regression, heterogeneous feature types, and variable-length rows or schemas (Wen et al., 5 Feb 2025). This textual interface is not incidental: it is both the source of flexibility across schemas and a primary source of the scalability problem, because each serialized row may require many tokens.

The architecture is explicitly decoupled into two components. The first is a retrieval module that selects the support set from the task’s training table. The second is a post-trained LLM that performs prediction from the retrieved support examples and the query instance. The retrieved examples include both serialized inputs and labels and are inserted into the LLM context as demonstrations.

This design preserves the transfer properties attributed to LLM-based TabICL while avoiding the direct coupling between training-set size and prompt length. A plausible implication is that the method treats tabular learning as an interaction between retrieval quality and in-context reasoning rather than as a purely parametric memorization problem. The paper’s broader interpretation is that language can act as a universal and accessible interface for tabular learning, with retrieval providing the mechanism that makes the interface practical at scale (Wen et al., 5 Feb 2025).

3. TabRAG: customized retrieval for heterogeneous tables

The retrieval module, TabRAG, is a non-parametric nearest-neighbor-style retriever designed for tabular heterogeneity. Its objective is to identify a small set of training rows whose feature patterns are most relevant to the query row. The retriever computes a sample-level distance by aggregating feature-wise distances, with separate treatment for categorical and numerical variables (Wen et al., 5 Feb 2025).

For a categorical feature, the distance is defined as $1$ when values differ and $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$0 when they are the same. For a numerical feature, the procedure is more elaborate: quantile normalization is first applied using statistics from the context pool; absolute differences are then computed; and those distances are min-max normalized to $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$1 across the context pool. This normalization makes distances comparable across heterogeneous columns.

The retriever also incorporates a parameter-free feature weighting mechanism because not all columns are equally informative for prediction. Two feature-importance signals are used: Pearson correlation between each feature and the target label, and Predictive Power Score (PPS) obtained via a single-feature decision-tree-based estimate. These are described as complementary because they capture linear and nonlinear associations, respectively (Wen et al., 5 Feb 2025).

Feature-wise distances are aggregated into an instance-level distance via a weighted $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$2 norm:

$\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$3

where $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$4 is the distance for feature $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$5 and $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$6 is its importance weight. The default retrieval policy uses both Pearson and PPS. When the quota is 128 context samples, half are allocated via Pearson-based retrieval and half via PPS-based retrieval (Wen et al., 5 Feb 2025).

The retriever is intended as a universal non-parametric policy that can operate without task-specific training. At the same time, the paper states that this default policy is not always optimal and may require retrieval engineering, including domain-specific or dataset-specific adjustments. This qualification is important because it rejects the misconception that retrieval is a trivial preprocessing step. In the reported ablations, removing any major retrieval component degrades performance, and the most damaging removal is feature-importance weighting, labeled NoFeatImp (Wen et al., 5 Feb 2025).

4. Retrieval-guided instruction tuning and model instantiation

RAG-TabICL does not use retrieval only at inference time. The LLM is also post-trained on contexts generated by the retrieval policy so that the in-context format observed during training matches the structure of retrieved contexts at test time. The paper describes this as retrieval-guided instruction tuning and presents it as an alignment mechanism between the retriever and the predictor (Wen et al., 5 Feb 2025).

The implementation starts from Phi-3 rather than LLaMA-2, extending the effective context length from 4K to 128K. The post-trained model is denoted Phi3-GTL, and the retrieval-augmented system is denoted RAG+Phi3-GTL. The key training idea is to replace randomly chosen in-context examples with contexts generated by the default TabRAG policy, so that the LLM learns the selection patterns it will encounter at inference. The intended effect is improved generalization under retrieved contexts rather than random few-shot prompts.

For post-training data, the appendix reports curation of over 300 public datasets from Kaggle following GTL’s data-collection recipe. After removing overlaps with evaluation data, the retained corpus contains 146 classification datasets and 173 regression datasets. By choosing different label columns, these were expanded into up to four tasks per dataset, yielding over 100,000 training samples for generative tabular learning (Wen et al., 5 Feb 2025).

The reported training configuration is concise and specific: 16 NVIDIA A100 GPUs, micro-batch size 1, learning rate $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$7, AdamW optimizer, no warmup or learning-rate scheduler, and about 10 hours of training time for Phi-3 Medium. These details situate the method as a post-training procedure rather than a from-scratch tabular pretraining pipeline.

5. Experimental protocol, baselines, and quantitative findings

Evaluation is conducted on 69 held-out datasets: 29 classification datasets and 40 regression datasets. These datasets are drawn from prior tabular benchmarks spanning LLM-based GTL-style datasets, neural tabular benchmarks, and tree-based benchmark suites, with the explicit goal of covering tasks that may favor different model families while avoiding train-test contamination. To reduce compute, test sets are downsampled to at most 512 samples per dataset (Wen et al., 5 Feb 2025).

The baseline suite is broad. Main ablations include Phi3-GTL, which uses random in-context examples instead of retrieval, and RAG+KNN, which keeps the retrieval mechanism but replaces LLM prediction with KNN. Strong tabular baselines include TabPFN-v1, TabPFN-v2, XGBoost, LightGBM, CatBoost, MLP, FT-Transformer (FTT), and TabR. The study also includes several “RAG + X” baselines, such as RAG+LR, RAG+XGBoost, and RAG+TabPFN, in which the corresponding model is fit on the same retrieved support set rather than on the full training set (Wen et al., 5 Feb 2025).

The evaluation metrics are AUROC for classification and NMAE, defined as normalized MAE by label mean, for regression. The study also uses min-max normalized AUROC and NMAE within each dataset group to compare relative rank behavior.

A central quantitative claim is that retrieval enables scaling with larger training pools. The paper fits a power law for median error,

$\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$8

and reports, for the RAG policy, classification with $\begin{split} E_{(x^T_{\cdot}, y^T_{\cdot}), T} &\left[ \log Q^{LLM}_\theta \left( S_{y} \left( y^T_{test} \right) \middle\vert S_{x} \left( x^T_{test}, C_{x^T_{test} \right) \right) \right], \ C_{x^T_{test} &= TabRAG \left( x^T_{test}, D^T_{train} \right). \end{split}$9, $Q^{LLM}_\theta$0, and $Q^{LLM}_\theta$1, and regression with $Q^{LLM}_\theta$2, $Q^{LLM}_\theta$3, and $Q^{LLM}_\theta$4 (Wen et al., 5 Feb 2025). The reported interpretation is that retrieval makes LLM-based TabICL exhibit favorable scaling behavior with training-set size. The experiments also observe that performance under RAG often saturates after only tens of retrieved instances, indicating that compact contexts may suffice for many tasks.

The study reports that retrieval substantially improves over random context selection. Relative to Phi3-GTL with random examples, retrieved contexts are far more informative. At the same time, the paper emphasizes that RAG+Phi3-GTL is competitive but not uniformly dominant. Across the 69 datasets, it is strong and sometimes matches or surpasses RAG+TabPFN-v2, tuned tree models on specific datasets, and other “RAG + X” baselines, but still lags behind well-tuned numeric tabular models overall, especially strong baselines such as TabPFN-v2 in aggregate performance (Wen et al., 5 Feb 2025).

Per-dataset comparisons are nevertheless nontrivial. The paper reports that RAG+Phi3-GTL beats TabPFN-v2 and CatBoost on about 17%–20% of datasets and is within a small gap on over 80% of datasets. It also contributes complementary errors in ensembles: when combined with TabPFN, LightGBM, or CatBoost, it improves ensemble performance because its decision patterns differ from those of numeric models (Wen et al., 5 Feb 2025). This suggests that average rank alone does not capture its utility.

6. Ablations, case studies, and failure modes

The appendix reports ablations of the retrieval module under four variants: NoFeatImp, which removes feature-importance weighting; NoNorm, which removes numerical normalization; NoCorr, which removes Pearson and keeps PPS; and NoPPS, which removes PPS and keeps Pearson. The reported takeaways are that removing any component hurts performance, NoFeatImp is the worst, Pearson and PPS are complementary, and normalization behaves differently by task type: it helps classification but can hurt regression in some cases (Wen et al., 5 Feb 2025).

Several case studies make the dependence on retrieval design concrete. In the California housing dataset, the target depends strongly on longitude and latitude. Pearson correlation alone does not capture this pattern well, but PPS does; using PPS-based retrieval improves NMAE from 0.1113 to 0.0951, corresponding to a 14.6% error reduction. In the Pol dataset, quantile normalization can distort a feature highly concentrated at a single value; replacing quantile normalization with standard normalization improves NMAE from 0.1904 to 0.0819, a 57% reduction. In Bike Sharing Demand, the default retriever misses important interactions such as year × working-day; manually constraining retrieval to same-year and same-working-day contexts reduces NMAE from 0.3754 to 0.1522, a 59.5% reduction (Wen et al., 5 Feb 2025).

These examples are used to support one of the paper’s most practical conclusions: the default retriever is universal, but dataset-specific retrieval engineering can substantially improve results. This is not presented as a minor implementation detail. Rather, it indicates that support-set construction is itself a modeling choice with substantial consequences.

The paper also describes a failure case in which the LLM, rather than retrieval, appears to be the limiting factor. On the anonymized classif-cat-medium-0-rl dataset, TabPFN-v2 substantially outperforms the LLM approach, suggesting that the bottleneck is not only support-set selection but also the LLM’s limited exposure to the relevant data patterns during post-training (Wen et al., 5 Feb 2025). The authors argue that broader synthetic or real-data pretraining, analogous to TabPFN’s regime, may help. This suggests that scalable retrieval does not by itself eliminate parametric limitations in the predictor.

7. Interpretation, decision boundaries, and position within tabular learning

The paper includes a decision-boundary analysis on toy problems and reports that RAG+Phi3-GTL produces a distinctive, non-smooth boundary unlike those of standard numeric tabular models. The resulting geometry somewhat resembles nearest-neighbor behavior but is not reducible to a simple average over neighbors. The interpretation offered is that this reflects the combination of text-based representation and LLM in-context reasoning (Wen et al., 5 Feb 2025).

This finding provides a concrete explanation for the observed ensemble diversity. If the model induces qualitatively different decision rules, then even suboptimal average performance relative to the strongest numeric baselines can coexist with practical value in hybrid systems. A plausible implication is that RAG-TabICL occupies a different inductive-bias regime from conventional numeric tabular learners, owing to its reliance on serialized language inputs, retrieval-conditioned contexts, and generative prediction.

Several misconceptions are directly countered by the experimental record. First, retrieval augmentation does not make LLM-based TabICL the overall winner on tabular benchmarks; the paper explicitly states that it still lags behind well-tuned numeric models in aggregate performance. Second, simply increasing context length is not equivalent to effective scaling; the reported experiments indicate that retrieval quality matters more than sheer context length, since random-context performance benefits much less from larger training pools and increasing the number of retrieved examples under RAG often yields rapid saturation (Wen et al., 5 Feb 2025). Third, the retriever is not a generic nearest-neighbor add-on but a carefully designed component whose weighting, normalization, and correlation structure materially affect outcomes.

Within the paper’s own framing, RAG-TabICL is best understood as a framework for making LLM-based tabular in-context learning scalable by replacing fixed few-shot prompting with retrieval-augmented support selection. Its defining components are a custom tabular retriever based on feature-wise distances and feature importance, a retrieval-aligned instruction-tuned LLM, and a serialization interface that maps rows to text (Wen et al., 5 Feb 2025). Empirically, it demonstrates that retrieval can make LLM-based TabICL scale with data size, achieve strong performance on many datasets, and provide complementary predictions in ensembles, while also exposing unresolved limitations: no universally optimal retrieval policy, lower aggregate performance than the best tuned numeric tabular models, and sensitivity to feature interactions and distributional quirks.