Likelihood-ratio inference on differences in quantiles

Abstract: Quantiles can represent key operational and business metrics, but the computational challenges associated with inference has hampered their adoption in online experimentation. One-sample confidence intervals are trivial to construct; however, two-sample inference has traditionally required bootstrapping or a density estimator. This paper presents a new two-sample difference-in-quantile hypothesis test and confidence interval based on a likelihood-ratio test statistic. A conservative version of the test does not involve a density estimator; a second version of the test, which uses a density estimator, yields confidence intervals very close to the nominal coverage level. It can be computed using only four order statistics from each sample.