Estimating treatment-effect heterogeneity across sites, in multi-site randomized experiments with few units per site (2405.17254v3)
Abstract: In multi-site randomized trials with many sites and few randomization units per site, an Empirical-Bayes estimator can be used to estimate the variance of the treatment effect across sites. When this estimator indicates that treatment effects do vary, we propose estimators of the coefficients from regressions of site-level effects on site-level characteristics that are unobserved but can be unbiasedly estimated, such as sites' average outcome without treatment, or site-specific treatment effects on mediator variables. In experiments with imperfect compliance, we show that the sign of the correlation between local average treatment effects (LATEs) and site-level characteristics is identified, and we propose a partly testable assumption under which the variance of LATEs is identified. We use our results to revisit Behaghel et al (2014), who study the effect of counseling programs on job seekers' job-finding rate, in 200 job placement agencies in France. We find considerable treatment-effect heterogeneity, both for intention to treat and LATE effects, and the treatment effect is negatively correlated with sites' job-finding rate without treatment.
- Sampling-based versus design-based uncertainty in regression analysis. Econometrica 88(1), 265–296.
- Heterogeneity and endogenous compliance: Implications for scaling class size interventions. Technical report, National Bureau of Economic Research.
- Allcott, H. (2015). Site selection bias in program evaluation. The Quarterly journal of economics 130(3), 1117–1165.
- Identification of causal effects using instrumental variables. Journal of the American statistical Association 91(434), 444–455.
- Sharp bounds on the variance in randomized experiments.
- Private and public provision of counseling to job seekers: Evidence from a large controlled experiment. American economic journal: applied economics 6(4), 142–174.
- Assessing external validity. Research in Economics 75(3), 274–285.
- From local to global: External validity in a fertility natural experiment. Journal of Business & Economic Statistics (just-accepted), 1–48.
- Eicker, F. et al. (1963). Asymptotic normality and consistency of the least squares estimators for families of linear regressions. The Annals of Mathematical Statistics 34(2), 447–456.
- Probability and random processes. Oxford university press.
- Predicting the efficacy of future training programs using past experiences at other locations. Journal of econometrics 125(1-2), 241–270.
- Huber, P. J. et al. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, Volume 1, pp. 221–233. University of California Press.
- Imbens, G. W. and J. D. Angrist (1994). Identification and estimation of local average treatment effects. Econometrica 62(2), pp. 467–475.
- Imbens, G. W. and C. F. Manski (2004). Confidence intervals for partially identified parameters. Econometrica 72(6), 1845–1857.
- General forms of finite population central limit theorems with applications to causal inference. Journal of the American Statistical Association 112(520), 1759–1769.
- Liang, K.-Y. and S. L. Zeger (1986). Longitudinal data analysis using generalized linear models. Biometrika 73(1), 13–22.
- Liu, R. Y. et al. (1988). Bootstrap procedures under some non-iid models. The Annals of Statistics 16(4), 1696–1708.
- Morris, C. N. (1983). Parametric empirical bayes inference: theory and applications. Journal of the American statistical Association 78(381), 47–55.
- Experimentation at scale. Journal of Economic Perspectives 31(4), 103–124.
- Neyman, J. (1923). On the application of probability theory to agricultural experiments. essay on principles. section 9. translated in Statistical Science 5(4), 465-472, 1990.
- Raudenbush, S. W. and H. S. Bloom (2015). Learning about and from a distribution of program impacts using multisite trials. American Journal of Evaluation 36(4), 475–499.
- Randomized experiments in education, with implications for multilevel causal inference. Annual review of statistics and its application 7, 177–208.
- Robins, J. M. (1988). Confidence intervals for causal parameters. Statistics in medicine 7(7), 773–785.
- External Validity in a Stochastic World: Evidence from Low-Income Countries. The Review of Economic Studies.
- Walters, C. R. (2015). Inputs in the production of early childhood human capital: Evidence from head start. American Economic Journal: Applied Economics 7(4), 76–102.
- White, H. et al. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. econometrica 48(4), 817–838.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
