Neymanian inference in randomized experiments

Abstract: In his seminal 1923 work, Neyman studied the variance estimation problem for the difference-in-means estimator of the average treatment effect in completely randomized experiments. He proposed a variance estimator that is conservative in general and unbiased under homogeneous treatment effects. While widely used under complete randomization, there is no unique or natural way to extend this estimator to more complex designs. To this end, we show that Neyman's estimator can be alternatively derived in two ways, leading to two novel variance estimation approaches: the imputation approach and the contrast approach. While both approaches recover Neyman's estimator under complete randomization, they yield fundamentally different variance estimators for more general designs. In the imputation approach, the variance is expressed in terms of observed and missing potential outcomes and then estimated by imputing the missing potential outcomes, akin to Fisherian inference. In the contrast approach, the variance is expressed in terms of unobservable contrasts of potential outcomes and then estimated by exchanging each unobservable contrast with an observable contrast. We examine the properties of both approaches, showing that for a large class of designs, each produces non-negative, conservative variance estimators that are unbiased in finite samples or asymptotically under homogeneous treatment effects.