Valuing Winners: When and How to Correct for Selection Bias in Randomized Experiments

Abstract: Decision-makers often deploy the best-performing treatment from a randomized experiment, creating a winner's curse: selection favors treatments whose observed outcomes are high partly because of statistical noise, so the naïve estimate of the winner is upward biased. We distinguish two forms of winner's curse, bias relative to the true best treatment (global) and bias relative to the selected treatment's true mean (selective), and link them to regret from deploying a suboptimal treatment. This framework defines seven decision-relevant evaluation targets: mean bias, mean squared error, and confidence interval coverage for the global and selective winner's curse, and mean regret. We then show that methods that perform well on one target can perform poorly on others, so corrections should be matched to the manager's objective. Across simulations with varying effect sizes, multiple-arm settings, and data calibrated to an online A/B testing platform, no method dominates uniformly: the plug-in estimator performs best when treatment differences are large, cross-fitting performs best when treatments are similar, and resampling methods often achieve low mean squared error for moderate differences. We also introduce an adaptive empirical likelihood procedure that delivers asymptotically valid confidence intervals across settings without the tuning sensitivity of resampling-based methods.

• The paper distinguishes between global and selective winner’s curse, clarifying that the bias difference equals post-selection regret and highlighting the need for context-driven corrections.
• The paper evaluates multiple correction methods—plug-in, sample splitting, resampling, conditional inference, and adaptive empirical likelihood—demonstrating trade-offs in bias, MSE, and regret.
• The paper introduces an adaptive empirical likelihood approach that achieves robust, computationally efficient confidence intervals across various effect size regimes, including multi-arm settings.

Valuing Winners: Selection Bias and Post-Selection Inference in Randomized Experiments

Introduction and Motivation

Randomized experiments, including A/B tests and field RCTs, underpin empirical decision-making across social science and industry. The operational approach is typically to deploy the treatment arm with the best observed performance. However, the statistical underpinning of this “pick-the-winner” rule introduces non-trivial selection bias—colloquially, the winner’s curse—because the winner is selected conditional on achieving the highest observed outcome, which can overstate its true effect due to noise. This paper distinguishes between global and selective winner’s curse (bias relative to the best true treatment vs. bias relative to the true mean of the selected arm), connects these definitions to ex-post regret, and provides a unified framework to analyze statistical corrections and inference targets.

Definitions and Objectives

The paper formally characterizes three distinct objectives for decision-makers post-selection:

• Selective Winner’s Curse (WC_select): Bias in the estimated outcome of the deployed treatment relative to its own true mean.
• Global Winner’s Curse (WC_global): Bias relative to the true best treatment’s mean.
• Regret: Expected loss from selecting a non-optimal arm, strictly a function of selection accuracy, not bias in outcome estimates.

Through simulation and analytical derivations, the authors show these concepts are interlinked by the identity $WC_{select} - WC_{global} = regret$. This decomposition clarifies that no correction strategy can universally optimize bias, MSE, and coverage for all targets simultaneously; optimal choice is context-dependent.

Review and Categorization of Winner’s Curse Corrections

The analysis encompasses five principal methodological classes:

• Plug-in Estimator: Simply selects the empirical best arm and uses its sample mean as the estimate. It is consistent but exhibits positive bias for inferential targets due to selection.
• Sample Splitting / Cross-Fitting: Reduces selection bias by decoupling winner choice from performance estimation via data partitioning and averaging split-specific estimates. The approach is unbiased but incurs higher variance and increased regret due to reduced power for winner identification.
• Resampling (Bootstrap, m-out-of-n Bootstrap, Hadamard Derivative, Shrinkage): These approaches correct bias and support inference via resampled distributions, but are vulnerable to miscalibration at “kink” points where group means are tied, and are sensitive to tuning parameters.
• Conditional (Selective) Inference: Conditions inference on selection events to provide valid coverage for the deployed arm, employing truncated normal theory. Extensions include hybrid approaches that ensure bounded (non-infinite) confidence intervals even near ties.
• Empirical Likelihood (EL): The paper’s key contribution is a pre-test adaptive EL method which switches between standard $\chi^2$ and chi-bar-squared critical values for coverage, depending on whether the setting is at a “kink.” This delivers robust, computationally efficient confidence intervals across all regimes and is less susceptible to tuning parameter pathology.

Simulation Study: Numerical Results

A comprehensive Monte Carlo simulation across effect sizes, sample sizes, and numbers of treatments benchmarks all major approaches on bias, MSE, and empirical confidence interval coverage for both inferential targets, as well as regret.

Notable findings include:

• For small effect sizes (Cohen’s $d \sim 0$): Selection bias is pronounced; cross-fitting achieves near-zero bias at the cost of higher variance and regret.
• For moderate effect sizes ($0.2 \leq d \leq 0.5$): Resampling methods (esp. m-out-of-n bootstrap) and empirical Bayes approaches trade off bias and MSE efficiently, although performance is tuning parameter sensitive.
• For large effect sizes ($d \geq 0.8$): Plug-in estimators achieve minimal bias, MSE, and regret—correction is unnecessary.
• Empirical Likelihood (EL): Consistently provides nominal coverage for both targets, unaffected by underlying noise distribution (Gaussian/Bernoulli), and gracefully scales to multiclass settings.
• Regret: Plug-in estimators are asymptotically minimax-optimal for regret across all regimes.

Theoretical Implications

The non-differentiability of the maximum operator at ties renders standard bootstrap, Delta-method, and Wald inference inconsistent—a critical insight for practitioners relying on off-the-shelf estimators. This nonregularity propagates to post-selection inference, explaining much of the instability and tuning-dependence in certain resampling correction pipelines.

The adaptive EL approach, leveraging chi-bar-squared asymptotics, provides a principled and computationally efficient path to valid inference at boundaries—a generalizable result in post-selection analysis.

Practical Recommendations

The appropriate debiasing/correction treatment depends on the experimenter’s operational goal and the anticipated underlying treatment effect regime. The paper provides scenario-based rules-of-thumb for A/B test calibration:

Effect Size Regime	Bias (Select)	Bias (Global)	Regret	Coverage (Select/Global)
Null ($d=0$)	Cross-fitting	Cross-fitting	N/A	Sample Splitting / EL
Small ($d = 0.2$)	Cross-fitting	Bootstrap	Plug-in	Empirical Likelihood
Moderate ($d=0.5$)	CF/Bootstrap	CF/Bootstrap	Plug-in	Empirical Likelihood
Large ($d=0.8$)	Plug-in	Plug-in	Plug-in	Plug-in

No method dominates in all settings. For business-scale A/B testing (small effects, many arms), empirical likelihood and empirical Bayes intervals offer robust coverage and efficient implementation, even when sample sizes per arm fall.

Extensions to Multi-Arm and Real-World Platforms

The framework and methods generalize to multi-arm settings, with the EL method’s adaptive critical value generalizing via multinomial chi-bar-squared distribution. Application to Optimizely data confirms that real-world A/B testing platforms not only encounter non-Gaussian, sparse effects, but also require robust correction in the presence of many null arms—settings where standard resampling procedures may be unreliable.

Conclusion

This paper establishes that post-selection evaluation targets in randomized experiments are multidimensional and require goal-aware correction: optimal bias, MSE, and coverage choices are inherently regime-dependent and no “one-size-fits-all” post-selection inference or bias correction exists. The theoretical analysis demystifies the underpinnings of the winner’s curse, clarifies trade-offs among correction algorithms, and contributes an adaptive empirical likelihood test—offering both theoretical soundness and practical advantages for high-throughput, many-arm A/B environments. These insights are central for future automated experimentation, high-dimensional optimal policy selection, and methodological research on post-selection inference.