Transfer of GIST under distribution shift

Investigate the transfer of Generator-state-Informed Supervision (GIST) to settings involving distribution shift beyond the evaluated TabArena, TALENT, and BeyondArena benchmarks.

Background

The paper introduces GIST, a framework for tabular foundation model pretraining that uses information retained during synthetic task generation to construct conditional query-label targets. By averaging query losses over conditional label distributions, GIST preserves the population objective and can reduce gradient covariance relative to sampled hard-label supervision. Experiments across TabICLv2, TabSwift, and TabPFN-2.5 report improvements on IID TabArena and TALENT benchmarks and selected temporal and grouped OOD tasks in BeyondArena.

Despite these results, the paper does not establish how well GIST transfers under broader or more demanding distribution shifts. The authors explicitly identify transfer under distribution shift as an unresolved direction for future investigation.