--- title: Tabular In-Context Learning Models url: https://www.emergentmind.com/topics/tabular-in-context-learning-icl-models type: topic --- # Tabular In-Context Learning Models Tabular in-context learning (ICL) models are a class of neural architectures that achieve accurate predictions on structured tabular data by conditioning directly on labeled training examples, presented as context, without updating model parameters. Transformer-based tabular foundation models, trained on massive synthetic tabular datasets, underlie this approach and demonstrate highly competitive performance versus gradient-boosted trees, notably requiring no downstream fine-tuning. As ICL for tabular data moves toward deployment in fairness-sensitive domains such as finance, health, and policy, understanding and mitigating algorithmic bias in tabular foundation models is increasingly critical. "Towards Fair In-Context Learning with Tabular Foundation Models" [2505.09503] provides a rigorous characterization of the architecture, workflows, fairness metrics, and tested pre-processing strategies for bias mitigation in tabular ICL. ## 1. Transformer Architectures and Pre-training Tabular ICL relies on transformer backbones adapted for structured data. Categorical features are one-hot encoded or embedded via learned lookup tables; numerical features undergo quantile normalization before projection into a shared embedding space. Column and row embeddings encode feature slot and position, respectively. Pre-training uses millions of synthetic tabular datasets drawn from random causal graphs, with each task formulated as tabular classification (≤500 features, ≤10,000 samples) in a single forward pass. Representative variants include TabPFN (handling up to 10,000 examples/500 features) and TabICL (scaling context capacity to 500,000 examples). ICL prediction proceeds by constructing a single sequence of K labeled records and one query, formatted as  [CLS] embed(x₁), embed(y₁), ..., embed(x_K), embed(y_K), embed(x_{K+1}), [PT] The final-layer output at [PT] passes through a classification head to yield $$p(y_{K+1} \mid x_{K+1}, \text{context})$$. ## 2. Demonstration Selection and Context Construction Bias in tabular ICL arises through the choice of in-context demonstration records. The context set is selected from a labeled training pool $$\mathcal{D}=\{(x_i, y_i, s_i)\}_{i=1}^N$$, where $$s_i$$ denotes a binary sensitive attribute (e.g. gender, age group). Test-time predictions are conditioned on selected context records, with no model updates. The strategies analyzed include: - **Correlation Remover**: Implements Feldman et al. (2015) by projecting each non-sensitive feature $$z^j$$ orthogonally to $$S$$, yielding decorrelated features:  $$w_j^* = \arg\min_{w\in\mathbb{R}}\,\|\mathbf{z}^j - S w\|_2^2,\quad z^{j*} = z^j - S w_j^*$$  Composite feature $$Z' = (1-\alpha)Z + \alpha Z^*$$ for $$\alpha\in[0,1]$$; the sensitive attribute is dropped prior to demonstration selection. - **Group-Balanced Selection**: Ensures equal representation in context by sampling $$K/2$$ examples from each group (defined by $$s_i$$), and one additional from the minority group for odd $$K$$. - **Uncertainty-Based Selection**: For each candidate, trains a sensitive-attribute classifier $$h(x)\approx \mathbb{P}(s=1|x)$$. Uncertainty score can be entropy  $$H_i = -[h(x_i) \log h(x_i) + (1-h(x_i)) \log (1-h(x_i))]$$  or conformal prediction set size; selects K examples with highest uncertainty. In experiments, conformal prediction (MAPIE) is preferred, with examples having an α-level prediction set of cardinality two. ## 3. Fairness Metrics for In-Context Predictions Fairness is evaluated using established group metrics. For predictions $$f(x)\in\{0,1\}$$, - $$\pi_g = \mathbb{P}(f(x)=1\mid s=g)$$    (demographic parity per group) - $$\text{TPR}_g = \mathbb{P}(f(x)=1\mid s=g, y=1)$$ (sensitive group True Positive Rate) - $$\text{FPR}_g = \mathbb{P}(f(x)=1\mid s=g, y=0)$$ (sensitive group False Positive Rate) Group-fairness gaps are defined: - Demographic parity gap:    $$\Delta \text{DP} = |\pi_0 - \pi_1|$$ - Equal opportunity gap:      $$\Delta \text{EOP} = |\text{TPR}_0 - \text{TPR}_1|$$ - Equalized odds gap:       $$\Delta \text{EOD} = |\text{FPR}_0 - \text{FPR}_1| + |\text{TPR}_0 - \text{TPR}_1|$$ ## 4. Experimental Protocol and Representative Results Experiments span eight real tabular datasets: several Folktables tasks, Diabetes (BRFSS), German Credit (age ≤25 vs >25), and CelebA (gender vs attractiveness). The workflow applies an 80/20 train/test split, with the held-out 20% dedicated to uncertainty-model fitting. On the 80% remainder, 5-fold cross-validation (×10 seeds) yields mean ± std metrics. Context size K is maximized per model capacity (≤10,000 for TabPFN). Baselines include: - Vanilla ICL (random context) - Balanced subsampling - Correlation remover ($$\alpha$$ swept in [0,1]) - Uncertain_lr (logistic regression entropy/conformal uncertainty) - Uncertain_tabpfn (foundation model-based uncertainty) Key findings: - Group-balanced demonstration selection yields only marginal fairness improvement relative to vanilla ICL. - Correlation remover frequently increases $$\Delta \text{DP}$$, $$\Delta \text{EOP}$$, and $$\Delta \text{EOD}$$: at test time, the transformer re-infers $$s$$ via subtle reconstructed feature patterns. - Uncertainty-based selection, especially with foundation-model-derived uncertainty ("uncert_tabpfn"), consistently provides the lowest group fairness gaps across datasets, with only ≈1–2 percentage point reduction in predictive accuracy. - Pareto-front analysis over conformal coverage parameter $$\epsilon$$ shows uncertainty selection dominates other pre-processing strategies in the fairness-accuracy trade-off. TabPFN slightly outperforms TabICL for combined fairness–accuracy, but both benefit substantially from uncertainty-based selection. ## 5. Implications for Fairness Mitigation in Tabular ICL Bias mitigation in tabular ICL fundamentally hinges on demonstration selection rather than model retraining. Correlation-based preprocessing, a mainstay in classic fair ML, may prove counterproductive: the transformer leverages intricate feature dependencies to restore dropped sensitive attributes post-decorrelation. Simple group-balanced contexts are insufficient for robust fairness. In contrast, selecting demonstration records for maximal uncertainty under a sensitive-attribute model minimizes the risk of context-driven bias propagation. This approach operates solely at the data presentation layer and does not require updating the pretrained tabular foundation model. ## 6. Future Directions and Open Challenges The most effective fairness regimen found—uncertainty-based context selection—suggests further study of model-driven demonstration retrieval algorithms, potentially integrating advanced conformal prediction methods or active sampling protocols. Extending these techniques to multi-class sensitive attributes, structured outputs, and regression remains open. Likewise, formal proofs of group-fairness guarantees under context uncertainty selection are needed. As tabular ICL models propagate into privacy-regulated domains, intersectional analyses (fairness by multiple overlapping sensitive groups) and compatibility with privacy-preserving data frameworks (such as DP-TabICL [2403.05681]) will be increasingly important. ## 7. Summary Table: Fairness Strategies and Effects | Selection Method | Fairness Impact | Accuracy Cost | |----------------------|---------------------|----------------| | Balanced Sampling | Marginal reduction | None | | Correlation Remover | Often increases gap | None/small | | Uncertainty-Based | Consistent reduction| ~1–2pp | Uncertainty-based demonstration selection using conformal prediction outperforms both balanced and correlation-removal approaches for reducing group fairness gaps ($$\Delta \text{DP}, \Delta \text{EOP}, \Delta \text{EOD}$$) in tabular in-context learning, with minimal accuracy compromise [2505.09503]. This elevates demonstration retrieval methods to the central tool for fair in-context learning over tabular data. Source: https://www.emergentmind.com/topics/tabular-in-context-learning-icl-models