High confidence inference on the probability an individual benefits from treatment using experimental or observational data with known propensity scores (2205.09094v3)

Abstract: We seek to understand the probability an individual benefits from treatment (PIBT), an inestimable quantity that must be bounded in practice. Given the innate uncertainty in the population-level bounds on PIBT, we seek to better understand the margin of error for their estimation in order to discern whether the estimated bounds on PIBT are tight or wide due to random chance or not. Toward this goal, we present guarantees to the estimation of bounds on marginal PIBT, with any threshold of interest, for a randomized experiment setting or an observational setting where propensity scores are known. We also derive results that permit us to understand heterogeneity in PIBT across learnable sub-groups delineated by pre-treatment features. These results can be used to help with formal statistical power analyses and frequentist confidence statements for settings where we are interested in assumption-lean inference on PIBT through the target bounds with minimal computational complexity compared to a bootstrap approach. Through a real data example from a large randomized experiment, we also demonstrate how our results for PIBT can allow us to understand the practical implication and goodness of fit of an estimate for the conditional average treatment effect (CATE), a function of an individual's baseline covariates.