Dealing With Ratio Metrics in A/B Testing at the Presence of Intra-User Correlation and Segments

Abstract: We study ratio metrics in A/B testing at the presence of correlation among observations coming from the same user and provides practical guidance especially when two metrics contradict each other. We propose new estimating methods to quantitatively measure the intra-user correlation (within segments). With the accurately estimated correlation, a uniformly minimum-variance unbiased estimator of the population mean, called correlation-adjusted mean, is proposed to account for such correlation structure. It is proved theoretically and numerically better than the other two unbiased estimators, naive mean and normalized mean (averaging within users first and then across users). The correlation-adjusted mean method is unbiased and has reduced variance so it gains additional power. Several simulation studies are designed to show the estimation accuracy of the correlation structure, effectiveness in reducing variance, and capability of obtaining more power. An application to the eBay data is conducted to conclude this paper.