Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

Abstract: Tabular Foundation Models (TFMs) have demonstrated strong empirical performance as black-box inference engines through in-context learning. However, their use in transfer learning is limited by two obstacles: strict context-size constraints and sensitivity to distribution shifts between source and target tasks. Directly pooling heterogeneous source data can therefore lead to negative transfer. To address these challenges, we propose Context-Constrained Transfer Learning via ANchoring and DIstillation (TL-ANDI), a posterior-aware distillation framework for TFMs. TL-ANDI constructs a compact source context by solving a budget-constrained optimal transport problem whose cost jointly measures target covariate coverage and posterior compatibility. The selected anchor samples are then equipped with locally distilled labels and combined with a residual calibration step using target data.

• The paper introduces TL-ANDI, a transfer learning framework that employs budget-constrained optimal transport to select source anchors and denoise labels.
• It demonstrates robust performance across homogeneous and heterogeneous tabular data, significantly reducing mean squared and misclassification errors.
• Empirical evaluations confirm that TL-ANDI avoids negative transfer, outperforming traditional methods in both regression and classification tasks.

Context-Constrained Transfer Learning for Tabular Foundation Models via Data Distillation

Introduction and Motivation

Tabular Foundation Models (TFMs), notably exemplified by TabPFN and Limix, have demonstrated empirical dominance in in-context learning (ICL) for tabular domains. These models, however, face fundamental bottlenecks when adapting for transfer learning (TL): (i) architectural restrictions on context size, limiting their capacity to ingest large-scale or multi-domain datasets; and (ii) acute sensitivity to distribution shift, with naive inclusion of heterogeneous source samples often resulting in negative transfer. Addressing these dual challenges is pivotal for leveraging TFMs as general-purpose inference engines across domains.

TL-ANDI: Posterior-Aware Optimal Transport Distillation Framework

The paper introduces TL-ANDI, a principled pipeline for transfer learning with TFMs under context constraints, formalized as a budget-constrained optimal transport (OT) distillation framework. TL-ANDI strategically selects source anchors and distills informative pseudo-labels, allowing the TFM to ingest a compact and highly target-compatible training context.

The method proceeds in three stages:

1. OT Anchoring: Source samples are selected by minimizing a transport cost that jointly incorporates target covariate coverage and posterior compatibility. The cost function includes a Euclidean covariate distance and a posterior discrepancy penalty, estimated using local kernel smoothing and pilot target-only predictions.
2. Local Label Distillation: The noisy, original source labels are replaced by kernel-smoothed pseudo-labels, reducing variance and denoising the anchors.
3. Residual Calibration and Validation: The transferred predictors are validated using independent folds, including the target-only baseline in the candidate set, guaranteeing no negative transfer under empirical risk minimization.

Theoretical Guarantees

The framework is supported by two formal results:

• Oracle Inequality for Source Context Quality: The distilled context achieves target-compatibility over the test covariate region whenever a transferable source subset exists, with explicit decomposition into transferability, distillation, target-pilot, and optimization error components.
• Validation-Based No-Negative-Transfer Guarantee: The final predictor selected by validation is statistically safeguarded against negative transfer, always performing at least as well as the target-only baseline up to a validation error term.

Empirical Evaluations

Extensive simulation studies and real-world experiments are conducted. All evaluations employ TabPFN-2.5 as the base TFM and benchmark TL-ANDI against Target-only, TL-RAND (random source sampling), and in classification, TabPFN-kNN (test-specific covariate-based retrieval).

Regression Simulations: Homogeneous and Heterogeneous Source Settings

Under homogeneous sources, TL-ANDI marginally outperforms TL-RAND due to posterior-aware selection and distillation, with both outperforming Target-only. For heterogeneous sources, TL-RAND suffers pronounced negative transfer; TL-ANDI robustly isolates informative source components and achieves the lowest MSE across all settings.

Figure 1: MSE distribution for Target-only and Vanilla-TL under homogeneous linear posterior shift, showing impact of target sample size.

Figure 2: MSE distribution for Target-only and Vanilla-TL under homogeneous nonlinear posterior shift, highlighting robustness across varying sample sizes.

Figure 3: MSE among Target-only, TL-RAND, TL-ANDI under heterogeneous source settings with variable source-component overlap, demonstrating TL-ANDI's robustness.

Figure 4: MSE for Target-only, TL-RAND, TL-ANDI under heterogeneous sources across target sample sizes, showing consistent performance gains for TL-ANDI.

Ablation Studies

Ablation experiments confirm that in homogeneous cases, variance reduction from distillation is beneficial when context budget is limited, while in heterogeneous settings, OT anchoring is the critical driver of negative transfer mitigation. The integrated approach further reduces MSE by combining targeted selection and denoising.

Classification Simulations

For binary classification, TL-ANDI surpasses Target-only, TL-RAND, and TabPFN-kNN. The latter fails to account for posterior heterogeneity despite covariate proximity, reinforcing the necessity of posterior-aware anchor selection.

Figure 5: Mis-classification error for Target-only, TabPFN-kNN, TL-RAND, TL-ANDI across demographic target domains, showing TL-ANDI's superiority.

Real Data Experiments

California Housing Dataset

The leave-one-out ocean proximity experiment validates TL-ANDI's robustness; transfer learning is beneficial, but only TL-ANDI reliably improves prediction error by targeting covariate and posterior-compatible source samples.

Figure 6: Mean Prediction Error (MPE) for Target-only, TL-RAND, TL-ANDI across four ocean proximity domains, illustrating TL-ANDI's consistent advantage.

Diabetes Health Indicators Dataset

TL-ANDI consistently outperforms baseline methods across all demographic splits, particularly in settings with pronounced distribution shifts. The methodology demonstrates both positive transfer and resilience against negative transfer due to careful anchor selection.

Figure 5: Mis-classification error for all methods across six demographic domains, evidencing TL-ANDI's stability and effectiveness.

Practical and Theoretical Implications

TL-ANDI provides a scalable solution for transfer learning with TFMs, circumventing architectural limitations and minimizing transfer risk. The OT anchoring introduces facility-location-type combinatorial optimization into TFM context construction, directly tying source selection to minimization of transfer risk. The validation-based safeguard is critical for practical deployment in heterogeneous real-world datasets, ensuring performance is never degraded below target-only inference.

Methodological integration between OT-based selection, kernel-based distillation, and validation-driven aggregation positions TL-ANDI as a robust framework for context-constrained transfer learning. Future work should target scalability for high-dimensional data (addressing kernel bandwidth limitations) and explore adaptation to TFMs with dynamic context architectures. Incorporating sparse or feature-selective approaches into the OT pipeline may further enhance the method's effectiveness in complex, high-dimensional environments.

Conclusion

TL-ANDI establishes a theoretically justified and empirically validated approach for transfer learning with Tabular Foundation Models under context constraints. Posterior-aware OT anchoring combined with local distillation and validation-driven aggregation achieves reliable negative transfer mitigation and improved performance in both regression and classification tasks, even under severe domain heterogeneity and sample size restrictions. The framework has immediate practical relevance for deploying TFMs in multi-domain tabular data applications and constitutes a foundation for further research on scalable, high-dimensional transfer learning methodologies (2607.04809).