A Review of Generalizability and Transportability (2102.11904v1)

Abstract: When assessing causal effects, determining the target population to which the results are intended to generalize is a critical decision. Randomized and observational studies each have strengths and limitations for estimating causal effects in a target population. Estimates from randomized data may have internal validity but are often not representative of the target population. Observational data may better reflect the target population, and hence be more likely to have external validity, but are subject to potential bias due to unmeasured confounding. While much of the causal inference literature has focused on addressing internal validity bias, both internal and external validity are necessary for unbiased estimates in a target population. This paper presents a framework for addressing external validity bias, including a synthesis of approaches for generalizability and transportability, the assumptions they require, as well as tests for the heterogeneity of treatment effects and differences between study and target populations.

Generalizability and Transportability: A Framework for Causal Inference

The paper entitled "A Review of Generalizability and Transportability" by Irina Degtiar and Sherri Rose provides an extensive examination of the challenges and methodologies involved in extending causal inferences from paper samples to broader target populations. The authors address the critical need to assess and address external validity bias in studies estimating causal effects, a component often overlooked despite its pivotal role in ensuring the applicability of results derived from both randomized controlled trials (RCTs) and observational studies to real-world settings.

Framework for Addressing External Validity Bias

The paper presents a comprehensive framework designed to guide researchers in assessing and addressing external validity bias post data collection. This framework includes considerations around defining estimands, evaluating assumptions necessary for generalizability and transportability, and methods for evaluating the similarity of paper and target populations as well as treatment effect heterogeneity. The authors emphasize the necessity of both internal and external validity for unbiased causal estimates, instead of focusing solely on internal validity, which is traditionally prioritized.

Estimands and Assumptions

In causal inference, the choice of estimand—typically the Population Average Treatment Effect (PATE)—defines the target population for inference. The authors carefully distinguish between internal validity (precision within the paper sample) and external validity (generalizability or transportability to a broader population), delineating the assumptions required for each. These include conditional exchangeability for treatment and paper selection, positivity of treatment and selection, and the stable unit treatment value assumption (SUTVA).

Evaluating Generalizability and Treatment Effect Heterogeneity

Degtiar and Rose explore methodologies for assessing dissimilarity between paper and target populations, which include propensity score-based methods and machine learning approaches such as Bayesian Additive Regression Trees (BART). The paper explores tests for treatment effect heterogeneity—ranging from prespecified to data-driven subgroup analyses—highlighting the importance of identifying effect modifiers that may differ across populations. This evaluation is pivotal for determining whether paper results can be extrapolated to a target population with different characteristics.

Analytical Methods for Generalizability and Transportability

The synthesis includes detailed explorations of analytical methods for generalizing and transporting paper results to a target population, categorized into weighting and matching methods, outcome regression models, and combined propensity score and outcome regression techniques. Part of this taxonomy includes inverse probability weighting, regression modeling, and doubly robust approaches, each tailored for addressing specific data availability and paper design challenges. The review supports the integration of advanced statistical methods like machine learning to enhance these analytical processes, ensuring flexibility and robustness against assumption violations.

Practical Implications and Future Considerations

Beyond these technical evaluations, the paper provides critical guidance for methodological innovation and practical application within research settings. It urges researchers to explicitly define target populations, incorporate generalizability considerations in paper designs, and maintain transparency regarding assumptions. For methodological researchers, it emphasizes the need to make computation tools available to increase adoption of such methods in applied settings.

In conclusion, Degtiar and Rose's paper reflects an essential advancement in understanding and operationalizing generalizability and transportability within causal inference. It bridges the gap between theoretical approaches and practical application, offering a definitive resource for statisticians, epidemiologists, and health policy researchers committed to deriving more broadly applicable and reliable causal estimates. This synthesis provides a nuanced narrative supported by a robust framework to navigate the complex terrain of extending causality beyond paper samples to relevant populations in diverse fields.