Target-Adaptive Text-Tabular Prediction

• Target-adaptive text-tabular prediction is a modeling paradigm that integrates structured and unstructured data by conditioning predictions on specific target semantics.
• It employs text serialization and target-aware representations to recast tabular tasks as generative sequence problems using in-context examples and evidence selection.
• Empirical results highlight improved low-resource performance and flexible adaptation, while challenges include numeric precision, prompt overflow, and context management.

Searching arXiv for papers on target-adaptive text-tabular prediction and closely related tabular foundation models. Target-adaptive text-tabular prediction denotes a family of methods in which prediction is performed over mixed structured and textual row representations while the prediction target is not hard-coded into a fixed dataset-specific head. Instead, the target is introduced through instructions, metadata, target tokens, target-aware encodings, in-context examples, or task-specific decoders. In this formulation, tabular classification and regression are recast as generative sequence modeling, target imputation, in-context learning, or multimodal representation learning over serialized rows, metadata, dialogue, and other unstructured fields. The paradigm appears in universal tabular modeling with instruction-following LLMs, retrieval-based long-context predictors, target-aware representation benchmarks, and adaptation-heavy settings such as predicting the decisions of previously unseen AI agents from a few prior games (Wang et al., 2023, Schindler et al., 5 Nov 2025, Arazi et al., 11 May 2026, Shapira et al., 12 May 2026).

1. Formal problem formulations

A central starting point is the contrast with standard tabular supervision. In traditional tabular modeling, each dataset has its own predictor $f_n : S_n \to T_n$, so a new target column or a new dataset requires a new model. UniPredict formulates a universal alternative with a single model $f_{\text{univ}} : S \to T$ such that $f_{\text{univ}}(S_n; M_n)=T_n$, where $S_n$ is a tabular sample and $M_n$ is dataset metadata. In that setting, the target column becomes part of the conditioning information rather than a fixed classifier head, and few-shot learning is defined as adapting a model trained on many source datasets to a new dataset $D_{n+1}$ with only a small amount of labeled data (Wang et al., 2023).

A related formalization appears in tabular in-context learning. TabSieve defines a table dataset as

$\mathcal{D}=\{(x_i,y_i)\}_{i=1}^{N},$

with query row $x^{(q)}$, labeled context

$\mathcal{C}=\{(x^{(k)},y^{(k)})\}_{k=1}^{K},$

and predictor

$$\hat{y}^{(q)} = f_{\theta'}\!\big(x^{(q)}, \mathcal{C}\big).$$

The target-adaptive element is that the prediction for the query is conditioned on a task-specific support set rather than on a fixed learned mapping alone. The same logic is specialized in controlled bargaining and negotiation games, where each decision point is converted into a table row that combines structured game state, offer history, and dialogue, and adaptation is performed with $f_{\text{univ}} : S \to T$0 complete prior games from the same unseen target agent (Wang et al., 12 Feb 2026, Shapira et al., 12 May 2026).

These formulations distinguish two kinds of adaptation that are often conflated. One is adaptation across changing target columns or tasks within tabular datasets; another is adaptation to a specific target entity, such as an unseen agent, via a small labeled history. A plausible implication is that “target-adaptive” in this literature refers less to domain adaptation in the classical covariate-shift sense than to conditioning prediction on explicit target semantics or target-specific evidence.

2. Serialization and target-aware representations

The most direct implementation path is text serialization. UniPredict constructs prompts as

$f_{\text{univ}} : S \to T$1

where $f_{\text{univ}} : S \to T$2 is metadata, $f_{\text{univ}} : S \to T$3 is the serialized row, and $f_{\text{univ}} : S \to T$4 is an instruction describing the prediction task. Its heavy prompt variant uses re-formatted metadata such as dataset purpose, target definition, and column semantics, while the light variant uses only the object or features. The heavy template is summarized by the instruction: “Below is the description of a dataset, an object profile from the dataset and a target description. Predict the target by the given information of the object.” This makes the target column instruction-addressable rather than structurally fixed (Wang et al., 2023).

The broader serialization literature treats this mapping explicitly. One formulation defines $f_{\text{univ}} : S \to T$5, where a tabular dataset $f_{\text{univ}} : S \to T$6 is converted into serialized texts $f_{\text{univ}} : S \to T$7 by template filling. Rows become sentence-like descriptions, and performance depends on feature selection, feature scaling, and missing-data handling before serialization. The same study reports that feature selection implemented with SHAP values from XGBoost or an ANOVA F-test is the only preprocessing step that consistently helps, whereas feature scaling has nuanced or marginal effects and filler-sentence imputation for missing values can substantially perturb logits (Ono et al., 2024).

Other methods make the target a first-class representational object rather than a prompt string. TabSTAR verbalizes each feature as text and also verbalizes the target: in classification, every possible class label becomes its own target token; in regression, a single target token is verbalized from the target name. Numerical features are represented both as clipped standardized values,

$f_{\text{univ}} : S \to T$8

and as semantic verbalizations using 10 quantile bins. The target tokens then interact with feature tokens inside a 6-layer Transformer encoder, producing semantically target-aware representations rather than target-agnostic text embeddings (Arazi et al., 23 May 2025).

The same emphasis on task-conditioned representation appears in benchmark construction. MulTaBench defines Target-Aware Representations (TAR) as embeddings tuned to the prediction target rather than kept frozen. Its text-tabular curation accepts a dataset only when joint multimodal utility and awareness utility both exceed threshold criteria: $f_{\text{univ}} : S \to T$9

$f_{\text{univ}}(S_n; M_n)=T_n$0

and

$f_{\text{univ}}(S_n; M_n)=T_n$1

with $f_{\text{univ}}(S_n; M_n)=T_n$2 and $f_{\text{univ}}(S_n; M_n)=T_n$3. This makes target-adaptation a benchmark desideratum rather than only an architectural choice (Arazi et al., 11 May 2026).

A more explicitly supervised encoding strategy appears in SG-XDEAT, which constructs a raw stream $f_{\text{univ}}(S_n; M_n)=T_n$4 and a target-aware stream $f_{\text{univ}}(S_n; M_n)=T_n$5, then concatenates them as

$f_{\text{univ}}(S_n; M_n)=T_n$6

Categorical features in the target-aware stream are encoded by a DecisionTreeEncoder, numerical features by PLE-T, and the two views interact through Cross-Encoding and Cross-Dimension self-attention with Adaptive Sparse Self-Attention. This is target conditioning by supervised tokenization rather than by prompt engineering or target tokens (Cheng et al., 14 Oct 2025).

3. Learning objectives and adaptation mechanisms

UniPredict departs from ordinary label generation by introducing target augmentation. It defines

$f_{\text{univ}}(S_n; M_n)=T_n$7

where $f_{\text{univ}}(S_n; M_n)=T_n$8 are class categories and $f_{\text{univ}}(S_n; M_n)=T_n$9 are calibrated probabilities. For classification, labels are one-hot encoded and then smoothed into probability distributions; for regression, continuous targets are discretized into quantile bins. The external probability estimator is an isotonic-calibrated XGBoost predictor with $S_n$0, and the augmented target is serialized as text such as “class $S_n$1; class $S_n$2; \ldots”. Training minimizes

$S_n$3

and test-time output is mapped back to a class by regex extraction and

$S_n$4

This objective makes supervision confidence-aware while preserving a generative interface (Wang et al., 2023).

TabGemma uses a different adaptation mechanism: target imputation. During each training step, it samples a table, uniformly samples 256 rows, chooses one column as the prediction target, includes all selected rows with target values present in the text, and computes loss only on tokens belonging to the target column. In abstract notation, if $S_n$5 denotes the target-column token sequence and $S_n$6 the feature tokens, the training objective maximizes

$S_n$7

Because the model is autoregressive and uses causal attention, every target cell acts as a supervised sample, and there is no need for a fixed context/query split during training. The method is schema-agnostic because the same row serialization and target-imputation behavior are applied across tables with different numbers of columns, feature types, and target columns (Schindler et al., 5 Nov 2025).

TabSieve adds an explicit intermediate action: selecting evidence rows before predicting the target. Its supervised cold-start corpus, TabSieve-SFT-40K, is synthesized from 331 real tables and contains 41,083 select-then-predict trajectories after strict filtering. The subsequent TAB-GRPO stage optimizes format validity, evidence quality, and answer correctness jointly, with dynamic task-advantage balancing to stabilize mixed classification and regression training. In this setting, target adaptation is inseparable from evidence choice: the system is trained not merely to predict from context, but to decide which context should count as evidence (Wang et al., 12 Feb 2026).

The AI-agent prediction setting introduces yet another mechanism: separation of representation and adaptation. A small frozen LLM acts as an Observer, reading public state and dialogue; its answer is discarded and its hidden state becomes a decision-oriented feature, while the downstream tabular foundation model performs the actual target-specific adaptation using source-population rows and $S_n$8 target-agent examples. The paper explicitly contrasts this with direct LLM-as-Predictor prompting, arguing that direct prompting compresses current state and adaptation examples into a single generated output and does not naturally integrate labeled source-population structure (Shapira et al., 12 May 2026).

4. Context, retrieval, and evidence selection

Context management is a defining technical issue in target-adaptive text-tabular prediction because serialized rows are token-inefficient and because additional examples can help or harm depending on their relevance. TabGemma addresses this with a compact n-gram-based retrieval system. Each cell is represented by a bag of character n-grams hashed into a fixed 256-dimensional vector per cell; per-cell vectors are concatenated into a row embedding, L2-normalized, and indexed with FAISS. At inference, the default setting retrieves $S_n$9 context rows for a 128k-token window, and the appendix reports that performance improves monotonically with more retrieved rows, especially on classification. The same analysis shows that retrieval quality and long-context handling degrade as tables become larger and wider because the candidate pool grows and the 128k window becomes a bottleneck (Schindler et al., 5 Nov 2025).

TabSieve treats context more selectively. Rather than concatenating all candidate rows into a prompt and hoping the model will attend to the right ones, it first selects a small set of informative in-table rows as evidence and then predicts conditioned on the selected subset. Its noisy-context analysis distinguishes No-Select, Evidence-$M_n$0, and Noise-$M_n$1 settings, showing that evidential rows help, noise rows hurt, and the gap widens as $M_n$2 grows. In one attention study on Qwen3-8B, mean attention on evidential rows increases from 0.210 to 0.254 and combined attention on target plus evidential rows rises from 0.466 to 0.499 once explicit selection is introduced (Wang et al., 12 Feb 2026).

In counterpart modeling for AI agents, the adaptation context is defined in terms of whole games. For each unseen target agent, $M_n$3 previously observed games are supplied, and all round-by-round decisions from those games enter the adaptation pool. The model also uses an agent-identity indicator, a one-hot marker over the source agents plus the target agent, so that target-specific rows can serve as an anchor during prediction on held-out games. This setup makes target adaptation concrete: the same predictive machinery must specialize to a new hidden policy from limited interaction history rather than merely average over a population (Shapira et al., 12 May 2026).

These approaches share a common premise: few-shot context is not uniformly useful. A plausible implication is that modern text-tabular systems increasingly treat context selection, retrieval, or compression as part of the prediction problem itself rather than as a peripheral prompt-engineering step.

5. Empirical performance across benchmarks and applications

UniPredict evaluates universal modeling by training a single GPT-2-based model on 169 Kaggle tabular datasets, capped at 7,500 samples per dataset for a total of 366,786 training samples, and comparing it with per-dataset baselines including XGBoost, MLP, TabNet, FT-Transformer, and TabLLM. On this 169-dataset benchmark, UniPredict-heavy reports a 2.2% absolute accuracy gain over XGBoost, a 5.4% relative improvement over the best tree-boosting baseline, and a 13.4% relative improvement over the best neural-network baseline, FT-Transformer; its average accuracy is about 0.81. On 62 unseen few-shot datasets with fewer than 100 samples each, training fractions from 10% to 90%, and 30 adaptation epochs, UniPredict-heavy shows especially strong low-resource behavior, including an over 118% advantage over XGBoost at the 10% training split (Wang et al., 2023).

TabGemma evaluates a schema-agnostic in-context learner over 123 tasks from CARTE, TextTab, and TabArena-lite. On the full benchmark aggregate it achieves 83.6 accuracy and 60.7 soft-clipped $M_n$4, while the paper emphasizes its stronger behavior on semantically rich benchmarks: 79.3 accuracy and 70.3 $M_n$5 on CARTE, and 84.1 accuracy and 31.6 $M_n$6 on TextTab. The reported conclusion is that LLM-based in-context learning can match or surpass the state of the art in classification on CARTE and TextTab, but conventional approaches remain stronger overall on regression and on numerics-heavy TabArena (Schindler et al., 5 Nov 2025).

Benchmarking work isolates the importance of target-aware representations at the dataset level. MulTaBench begins from 56 unique text-tabular candidates, retains 20 datasets after joint-signal and task-awareness filtering, and reports that Joint TAR consistently outperforms Joint Frozen for all learners during curation. On the final text-tabular subset, mean score rises from 0.774 with Frozen embeddings to 0.792 with Contextualized / TAR embeddings; the largest individual gain is on Jigsaw Toxicity, where the score increases from 0.806 to 0.926, a gain of +0.119 (Arazi et al., 11 May 2026).

Application-specific evidence comes from predicting the decisions of unfamiliar AI agents. Training on 13 frontier-LLM agents and testing on 91 held-out scaffolded agents, the full Observer-based tabular model outperforms direct LLM-as-Predictor prompting and game+text baselines. At $M_n$7, the best bargaining response predictor reaches AUC .831 versus .791 for game+text features and .770 for LLM-as-Predictor, while in negotiation it reaches AUC .852 versus .803 and .785. For bargaining proposal prediction, the Observer reduces nominal \$M_n$8552 to \$473, a 14% reduction (Shapira et al., 12 May 2026).

TabSTAR provides a further classification-oriented result for semantically rich tabular data with text features. On its 10K condition, TabSTAR reports 0.809 versus 0.783 for TabPFN-v2, 0.756 for CatBoost-Tuned, and 0.744 for XGBoost-Tuned; in the unlimited-data condition, TabSTAR-Unlimit reaches 0.874 while second-best CatBoost-Tuned-Unlimit reaches 0.734. Its pretraining study also shows monotonic improvement with more pretraining datasets, from 0.352 to 0.786 on normalized classification score and from 0.338 to 0.811 on normalized regression score across 0, 16, 64, and 256 pretraining datasets (Arazi et al., 23 May 2025).

6. Limitations, controversies, and architectural directions

A major controversy concerns whether text serialization plus language-model fine-tuning should be regarded as a replacement for conventional tabular machine learning. A dedicated comparison concludes that pretrained LMs should not replace conventional tabular methods and that gradient-boosted models remain stronger or more robust overall on typical tabular tasks. The same study finds that MTEB rankings do not transfer cleanly to serialized tabular prediction, that LLMs are slower and more resource-intensive than boosting methods, and that adaptation is fragile because performance depends strongly on serialization choices, irrelevant features, and imputation details (Ono et al., 2024).

Failure analyses in target-adaptive systems are correspondingly concrete. UniPredict attributes poor performance to too many columns causing prompt overflow, poorly represented feature values, inadequate or ambiguous metadata, and other dataset-specific issues; its heavy version benefits from informative metadata, whereas the light version avoids bad metadata but can struggle when raw feature values are hard to interpret (Wang et al., 2023). TabGemma identifies unstable numeric tokenization and limited context size as the main practical hurdles, proposes signed scientific notation such as

$M_n$9

to reduce variability, and still reports that regression remains weaker than conventional methods as data grows and that larger, wider tables degrade retrieval precision and long-context effectiveness (Schindler et al., 5 Nov 2025).

A related limitation appears in TabPFN-based text integration. The TabPFN Text Adapter removes the PCA bottleneck by freezing both sentence encoder and TabPFN and training only a lightweight projector from 384-dimensional sentence embeddings into a short sequence of 192-dimensional TabPFN-space tokens. On TextTabBench it is the best among TabPFN-based methods for regression, but in classification it remains behind PCA-30 and behind ConTextTab; the paper explicitly notes that compressing text into a small TabPFN token space likely loses information and suggests future conditioning on the full text column, other columns, or even the target (Tajjar et al., 3 Jun 2026).

The field also exhibits a design tension between target-aware and target-agnostic representation learning. TabSTAR, SG-XDEAT, and MulTaBench foreground target-conditioned embeddings, target-aware streams, or task-aware tuning. FlexTab argues for the opposite decomposition: a single shared encoder should learn target-agnostic row embeddings, while task-specific decoders perform the final specialization for classification, regression, anomaly detection, clustering, entity matching, and relational entity classification. FlexTab reports that many earlier in-context learners entangle feature representation with a specific prediction target, whereas its shared encoder and task-specific decoders achieve state-of-the-art performance on classification, regression, anomaly detection, and entity matching while remaining competitive on relational entity classification (Polewczyk et al., 29 Jun 2026). This suggests an unresolved architectural question: whether adaptation should be injected early into the representation itself or deferred to a decoder operating over reusable row embeddings.

Taken together, the literature supports a narrow but important conclusion. Target-adaptive text-tabular prediction is not a single architecture but a design space organized around how the target enters the system: by instruction, by target token, by target-aware preprocessing, by evidence selection, by retrieval, or by decoder specialization. Its strongest results occur when text carries genuine semantic signal, when few-shot transfer matters, or when target identity itself is part of the inference problem. Its recurrent weaknesses are numeric precision, context bottlenecks, serialization sensitivity, and the continued strength of conventional boosting methods on many standard tabular benchmarks.