---
title: 'CUPED: Variance Reduction in A/B Testing'
url: https://www.emergentmind.com/topics/controlled-experiments-using-pre-experiment-data-cuped
type: topic
---

# CUPED: Variance Reduction in A/B Testing

Controlled Experiments Using Pre-experiment Data (CUPED) is a variance-reduction technique for randomized experiments, especially large-scale online A/B testing, that uses pre-treatment covariates to remove predictable variation from post-experiment outcomes while preserving unbiasedness of the average treatment effect. In its classical form, CUPED adjusts the outcome with a control-variate term built from a pre-experiment metric, often cutting confidence interval widths in half or more, and subsequent work has reframed it as a broader augmentation framework that extends beyond simple mean metrics and beyond purely pre-experiment adjustment [2311.17858; 2312.02935].

## 1. Classical estimator and core variance-reduction mechanism

In the standard two-arm setup, treatment and control are compared on a post-experiment outcome \(Y\). The unadjusted estimator is the difference in means,
\[
\hat{\tau}_{\text{diff}}=\bar Y_1-\bar Y_0.
\]
CUPED introduces a pre-experiment covariate \(X\), often the same metric measured in a pre-period, and forms an adjusted outcome
\[
Y^{\text{CUPED}} = Y - \theta (X - \mathbb{E}[X]).
\]
The treatment effect is then estimated as
\[
\hat{\tau}_{\text{CUPED}}=\bar Y^{\text{CUPED}}_1-\bar Y^{\text{CUPED}}_0.
\]
Under randomization, this adjustment is unbiased for any fixed \(\theta\), because the pre-experiment covariate is independent of treatment assignment [2606.04110].

The variance-minimizing coefficient in the classical single-covariate formulation is
\[
\theta^\star=\frac{\operatorname{Cov}(Y,X)}{\operatorname{Var}(X)}.
\]
With this choice, the variance of the adjusted estimator is reduced by approximately \(1-\rho^2\), where \(\rho\) is the correlation between \(X\) and \(Y\). In the basic case where \(X\) is the pre-period value of the goal metric and \(Y\) is the post-period value, the relative variance is
\[
\frac{\operatorname{Var}(\hat\delta_{\text{basic r.a.}})}{\operatorname{Var}(\hat\delta_{\text{original}})}=1-\rho^2,
\]
so strong pre/post correlation directly translates into narrower confidence intervals [2311.17858].

The same adjustment can be written as regression. For mean metrics, CUPED is equivalent to estimating
\[
Y_i = \alpha + \delta T_i + \beta X_i + \varepsilon_i,
\]
or, with treatment–covariate interaction in richer specifications, interpreting the coefficient on \(T_i\) as the adjusted ATE. This equivalence explains why CUPED is often described as regression adjustment, even though later work argues that this description is narrower than the underlying estimator logic [2606.18750].

## 2. CUPED as mean-preserving augmentation

A central reinterpretation of CUPED presents it not primarily as unit-level regression residualization, but as a mean-preserving augmentation of an estimator. In this view, one starts with any estimator \(\widehat{\delta}\) of a target \(\delta\) and adds an augmentation term \(\widehat{\delta}_0\) satisfying
\[
\mathbb{E}[\widehat{\delta}_0]=0.
\]
The augmented estimator is
\[
\widehat{\delta}^*=\widehat{\delta}+\theta \widehat{\delta}_0,
\]
with variance-minimizing coefficient
\[
\theta^*=-\frac{\operatorname{Cov}[\widehat{\delta}_0,\widehat{\delta}]}{\operatorname{Var}[\widehat{\delta}_0]}.
\]
Its minimized variance is
\[
\operatorname{Var}[\widehat{\delta}^*(\theta^*)]
=
\operatorname{Var}[\widehat{\delta}]
\bigl(1-\operatorname{Corr}[\widehat{\delta},\widehat{\delta}_0]^2\bigr).
\]
This is closer to the control-variate interpretation of the 2013 CUPED paper and does not require a parametric model for \(Y\mid X\) [2312.02935].

That augmentation perspective immediately extends CUPED beyond simple averages. A pre-experiment difference in any metric \(M\),
\[
\Delta_{\text{pre}}(M)=M(\mathbf X_{\text{pre}})_t-M(\mathbf X_{\text{pre}})_c,
\]
is mean-zero under randomization and can serve as an augmentation term. The same logic therefore applies to ratio metrics, percentile metrics, and other estimators for which a pre-experiment analogue exists. The same paper further pushes the framework beyond pre-experiment data by introducing approximate null augmentation (ANA), in which an in-experiment component with effect believed to be close to zero is used as a variance-reducing augmentation. In Airbnb ranking experiments, this decomposition roughly doubled the number of statistically significant detections relative to the raw metric, from \(4/25\) to \(8/25\) [2312.02935].

This reformulation clarifies a common misconception. Regression adjustment is a special case of CUPED for mean metrics, but CUPED itself is broader: it is an estimator-level construction built around mean-zero augmentations.

## 3. Theoretical limits of pre-experiment-only adjustment

Because classical CUPED can already produce large reductions in confidence-interval width, a natural question is whether sufficiently elaborate feature engineering from pre-experiment data could drive the variance of the treatment-effect estimator arbitrarily close to zero. The answer given in “On the Limits of Regression Adjustment” is negative [2311.17858].

That result studies a basic adjustment using only the pre-experiment value of the goal metric and compares it to the best possible regression adjustment constructed from richer pre-treatment features. Let \(\rho=\operatorname{cor}(Y_{\text{pre}},Y_{\text{post}})\), let \(\sigma=\operatorname{cor}(X,Y_{\text{pre}})\), and let \(\tau=\operatorname{cor}(X,Y_{\text{post}})\) for an engineered covariate \(X\). Under the temporal predictability assumption
\[
\tau \le \sigma,
\]
the paper proves
\[
\frac{1-\tau^2}{1-\rho^2} \ge \frac{1}{1+\rho}.
\]
Equivalently,
\[
\operatorname{Var}(\hat\delta_{\text{advanced r.a.}})
\ge
\frac{1}{1+\rho}
\operatorname{Var}(\hat\delta_{\text{basic r.a.}}).
\]

The interpretation is sharp. When the basic adjustment is already very effective, so that \(\rho\approx 1\), the best possible advanced pre-treatment-only regression adjustment can reduce the remaining variance by at most \(50\%\). In confidence-interval terms, the maximum additional reduction in interval width is about \(29\%\). If \(\rho=0\), the bound implies no additional improvement at all. The paper’s conclusion is that a simple form of regression adjustment using just the pre-experiment value of the goal metric captures most of the attainable benefit in many online experimentation settings [2311.17858].

The paper also identifies cases where the bound need not hold. Strong seasonality and cases where the post-treatment aggregation window is much longer than the pre-period can make the post-period easier to predict than the pre-period. This suggests that the theorem is a typical-case limit for standard online experiments rather than a universal impossibility result.

## 4. Generalizations to richer predictors, richer metrics, and in-experiment augmentation

A major modern direction is to replace the single pre-period covariate with flexible predictors built from many pre-treatment variables. “Towards Optimal Variance Reduction in Online Controlled Experiments” develops debiased, cross-fitted estimators for count and ratio metrics and proves semiparametric efficiency under mild convergence conditions. In LinkedIn experiments, the proposed optimal procedure for ratio metrics reduced up to \(80\%\) of variance compared to the standard difference-in-mean estimator and also further reduced up to \(30\%\) of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates [2110.13406].

Another extension combines pre-experiment and in-experiment covariates. The key idea is to use pre-experiment data \(X\) for a CUPAC-style first-stage prediction and then use in-experiment covariates \(Z\) that are not affected by treatment for an additional linear residualization step. Under the paper’s assumptions, this yields asymptotic normality with a consistent variance estimator and achieves greater variance reduction than CUPED and CUPAC without introducing additional computation complexity. Applied to 29 Etsy experiments, the method reached substantial variance reduction over CUPAC with the inclusion of only a few in-experiment covariates [2410.09027].

One-sided triggering creates a different extension point. In experiments where only treatment users can trigger exposure, “Zero to Hero” constructs a CUPED-style mean-zero augmentation term from the trigger-complement group, using weighting schemes built from pre-experiment or in-experiment observations. The resulting ITT estimator remains unbiased when the augmentation term has expectation zero, can incorporate both pre-experiment and in-experiment observations, and in simulations and a real-world experiment achieved precision improvements as large as if triggering status were fully observed; in some cases it even outperformed trigger-dilute analysis [2112.13299].

Taken together, these developments preserve the central CUPED principle—subtract predictable, mean-zero variation from the estimator—while loosening three classical restrictions: linearity, mean metrics only, and pre-experiment-only covariates.

## 5. Heavy-tailed metrics, robustness, and altered estimands

CUPED’s standard formulations are especially attractive when outcomes are well-behaved. Heavy-tailed monetization metrics expose their limits. In ShareChat ranking and recommendation experiments, user-level GMV was extremely heavy-tailed, with the top \(\sim 0.01\%\) of users dominating both mean and variance. In that regime, CUPED alone reduced variance by \(47.62\%\), but the test remained conservative and under-powered. A combined post-stratification-plus-CUPED estimator, built from strata defined on pre-period GMV and population weights, reduced variance by \(99.3\%\) to \(99.7\%\) relative to raw GMV and achieved equivalent statistical confidence with \(\sim 45\%\) less traffic than standard metrics [2606.04110].

That framework comes with a caveat. Because the post-stratified estimand deliberately down-weights whales, it is not recommended when the primary product goal is to affect whales. This is a general pattern in robust CUPED variants: large efficiency gains often come with a shift in weighting or in the estimand itself.

A second robust route explicitly changes the estimand. “Improving Sensitivity in A/B Tests: Integrating CUPED with Trimmed Mean Techniques” combines CUPED with Yuen’s t-test by trimming first and then applying CUPED at the level of trimmed means. The resulting estimator is unbiased for the trimmed treatment effect rather than the full mean effect. In simulations with lognormal and zero-inflated lognormal outcomes, the combined method produced much higher power than Welch’s test with or without CUPED while keeping false positive rates near nominal [2510.03468].

A third route keeps the mean-based ATE target but changes the residual model. “STATE” replaces Gaussian residual assumptions with a Student’s \(t\)-distribution and uses a variational EM algorithm for robust regression adjustment. On Meituan data, STATE achieved over \(50\%\) variance reduction relative to CUPAC/MLRATE and extended the same strategy to ratio metrics through a linear transformation that preserves unbiased estimation [2407.16337].

These results show that CUPED is not a single method but a family of variance-reduction strategies. For heavy-tailed outcomes, the relevant choice is no longer only how predictive the covariates are, but also whether one is willing to alter weighting, alter the estimand, or alter the residual model.

## 6. Moving pre-experiment data into experimental design

Several papers relocate the CUPED idea from analysis to design. “Optimal Data Collection for Randomized Control Trials” uses pre-experimental data to choose both sample size and the subset of covariates to collect under a budget constraint, minimizing the mean squared error of the regression-adjusted ATE estimator. In two empirical applications, the procedure produced gains of up to \(58\%\), measured either as reductions in data collection costs or as improvements in precision [1603.03675].

“Synthetic Design” pushes further by using pre-treatment panel outcomes to choose treated units, control units, and weights jointly. The estimator becomes a difference between weighted average outcomes, a form that nests both difference-in-means and synthetic-control estimators. In simulations based on U.S. state-level data, the optimization-based designs improved mean squared error and statistical power relative to randomized trials [2112.00278].

A more directly randomized alternative is Covariate Ordered Systematic Sampling (COSS). COSS sorts units by a pre-experiment covariate and assigns them alternately to treatment and control. The estimator remains a difference in means, but the design locally matches units on the covariate. Theory in the paper shows variance behavior analogous to a \(1-r^2\) reduction under monotonic relationships, and empirical marketing experiments found better sensitivity gains than commonly used variance reduction techniques like CUPED, especially for nonlinear settings and metrics such as GMV [2404.10381].

These design-stage methods do not replace CUPED’s analysis-stage role in ordinary user-randomized A/B testing. They show, rather, that pre-experiment data can be used earlier: to choose which covariates to collect, which units to treat, or how to assign treatment, instead of only how to analyze outcomes after randomization.

## 7. Nonstandard designs, variance estimation, and production deployment

In nonstandard experimental designs, CUPED’s familiar formulas interact with clustering, switching, multiple arms, and sampling stages. “Design-Aware Variance Reduction for Switchback Experiments” studies CUPED, CUPAC, and doubly robust estimators under clustered time-dependent randomization. There, CUPED’s gains depend strongly on macro-level predictability: with low temporal autocorrelation its standard-error ratio versus the raw estimator is close to \(0.97\), while at \(\rho=0.9\) it reaches about \(0.62\). The paper emphasizes that cluster-robust standard errors are essential and that finite-cluster effects and interference can limit improvements [2606.27662].

“Ensuring Trustworthy Online A/B Testing” systematizes several practical lessons from ByteDance’s platform. It shows that full-sample CUPED can occasionally increase variance, that split-sample CUPED is preferable for mean metrics, that regression-based CUPED should use treatment–covariate interactions together with robust sandwich variance, and that naive reliance on standard variance estimators in multi-arm experiments and two-stage sampling designs can lead to severely misleading inferences [2606.18750].

At the platform level, this has a clear implication. CUPED is not only a point-estimation technique; it is also an inference problem. Once experiments depart from the simple two-arm i.i.d. setting, the choice of adjustment specification and the choice of variance estimator become inseparable. The mature form of CUPED therefore includes not just a control-variate formula, but a design-aware specification of slopes, centering, and standard errors.

Controlled Experiments Using Pre-experiment Data began as a simple industry method for subtracting pre-period noise from post-period outcomes. It is now better understood as a general mean-preserving augmentation framework whose classical \(1-\rho^2\) efficiency gain remains a useful baseline, whose pre-experiment-only limits are now theoretically characterized, and whose modern extensions reach ratio metrics, percentiles, triggering designs, heavy-tailed outcomes, switchbacks, multi-arm experiments, and design-stage optimization. Across these settings, the common principle is unchanged: use information that is independent of treatment assignment but predictive of outcomes to reduce variance without changing the causal target.

Source: https://www.emergentmind.com/topics/controlled-experiments-using-pre-experiment-data-cuped