Addressing Conditional Shift for Practical Effect Generalization

Determine practical and statistically valid methods to address conditional shift—the change in the conditional distribution of the outcome Y given observed covariates X—when generalizing effect estimates from a source population to a target population, including well-controlled multi-site replication settings where adjusting for observed covariate shift does not explain the distributional differences between populations.

Background

The paper highlights that many generalization methods rely on the covariate shift assumption, under which adjusting for differences in the distribution of observed covariates across populations is sufficient because the conditional distribution of outcomes given covariates remains invariant. Recent empirical evaluations, however, show that covariate shift often explains only a small fraction of distributional differences, implying non-negligible conditional shift—i.e., changes in P(Y|X) across settings.

This observation raises a practical challenge: in real-world generalization tasks, especially in controlled multi-site replication studies where outcome data in the target population are unavailable, it is necessary to develop valid procedures to account for conditional shift. The authors explicitly note that it is unclear how to address this conditional shift to achieve credible external validity, motivating their proposed predictive role of covariate shift as a way forward.