Selecting and composing test-time compute recipes for unseen datasets

Determine how to choose and compose an appropriate test-time compute recipe for a new tabular dataset, including how model–data interactions determine whether adaptation, aggregation, or context-construction strategies will improve predictions on that dataset.

Background

The paper evaluates three families of test-time computation for tabular foundation models: adaptation of pretrained parameters, aggregation of predictions from multiple configurations, and construction or expansion of the conditioning context. Although adaptation and selective aggregation produce benchmark-level gains, the improvements are uneven across datasets, and context-selection methods can either improve or degrade performance depending on the task.

The authors report that aggregate benchmark gains do not reliably indicate whether a particular recipe will benefit an unseen dataset. Their tested dataset-level predictors achieve limited accuracy in forecasting the magnitude of gains, motivating an unresolved problem of identifying informative model–data signals and using them to guide the selection and composition of test-time compute strategies.