Comparative Evaluation of Difference in Differences Methods for Staggered Adoption Interventions

Abstract: Staggered adoption is a common approach for implementing healthcare interventions, where different units adopt the program at different times. Difference-in-differences (DiD) methods are frequently used to evaluate the effects of such interventions. Nonetheless, recent research has shown that classical DiD approaches designed for a single treatment start date can produce biased estimates in staggered adoption settings, particularly due to treatment effect heterogeneity across adoption and calendar time. Several alternative methods have been developed to address these limitations. However, these methods have not been fully systematically compared, and their practical utility remains unclear. Motivated by a payment program implemented by a healthcare provider in Hawaii, we provide a comprehensive review of the staggered adoption setting and a selection of DiD methods suitable for this context. We begin with a theoretical overview of these methods, followed by a simulation study designed to resemble the characteristics of our application, where the intervention is implemented at the cluster level. Our results show that the current methods tend to under-perform when the number of clusters is small, but improve as the number of clusters increases. We then apply the methods to evaluate the real-world payment program intervention and offer practical recommendations for researchers implementing DiD methods for staggered adoption settings. Finally, we translate our findings into practical guidance for applied researchers choosing among DiD methods for staggered adoption settings.