AI-Assisted Variance Reduction in Randomized Experiments

Abstract: Generative AI and LLMs can produce realistic predictions of human behavior from rich, unstructured inputs with little to no task-specific training data. Recent work uses these digital twin'' predictions to supplement human responses in surveys and experiments. We study the special case of using AI-generated predictions to reduce variance in randomized experiments. We argue that doing so requires no new estimators and that researchers can simply include AI predictions as covariates in standard regression adjustment, analogous to adjusting for a prognostic score. A benefit of this approach is ado no harm'' property whereby the adjusted estimator reverts to the unadjusted difference in means when predictions are uninformative. Other methods, such as variants of prediction-powered inference, do not have this guarantee. We provide implementation guidance, including how to obtain continuous scores from discrete LLM outputs and how to use LLMs to featurize unstructured inputs as auxiliary covariates. We demonstrate these ideas in simulations and three empirical applications: a survey mega-study, an email marketing A/B test, and a large-scale technology platform experiment. Overall, efficiency gains are real if modest, with greater benefits in studies that contain substantial text and other unstructured data. We also confirm the do no harm property empirically. Given these gains and limited costs, we recommend adjusting for AI-generated predictions as a regular empirical practice.

• The paper demonstrates that integrating AI-generated predictions as auxiliary covariates in regression adjustment reliably reduces variance in randomized experiments.
• It compares model-assisted, regression-adjusted, and digital twin estimators, highlighting that calibrated regression adjustment consistently overcomes systematic prediction bias.
• Key simulation and empirical evaluations confirm that while AI predictions yield modest efficiency gains, traditional methods perform robustly when baseline covariates strongly capture outcome signals.

AI-Assisted Covariate Adjustment for Variance Reduction in Randomized Experiments

Motivation and Context

Recent advances in generative AI, particularly LLMs, have enabled robust prediction of human behavior based on rich, unstructured inputs, facilitating the "digital twin" paradigm in empirical research. However, replacing human respondents with pure synthetic samples introduced systematic biases and reduced reliability. The methodological response has focused on hybrid estimators that combine AI-based predictions with observed experimental data for improved inference. This paper rigorously studies the specific use of AI-generated outcome predictions for variance reduction in randomized experiments, offering a unified perspective anchored in classical regression adjustment.

Framework and Estimator Comparisons

The paper considers the standard finite-population experiment setting: observed outcomes $Y_i^{obs}$, treatment assignments $Z_i$, and baseline covariates $X_i$. AI-generated predictions $_i(z)$, potentially derived from LLMs or tabular ML, are available for each unit under both treatment conditions. These predictions are treated as auxiliary covariates to increase estimator precision, free from any assumptions of unbiasedness or calibration.

Three principal estimator categories are analyzed:

• Model-assisted estimators: Adjust the synthetic mean using empirical differences between AI predictions and observed outcomes, yielding unbiasedness but potentially inflating variance if prediction quality is low. Variance is reduced only if $\rho(z,z) > 0.5$, where $\rho(z,z)$ is the within-arm correlation between the AI prediction and outcome.
• Regression-adjusted (calibrated) estimators: Linear regression of outcome on the treatment indicator and AI predictions (potentially interacted) ensures approximate unbiasedness and weakly reduces variance, regardless of predictor informativeness. The design effect is $1 - \rho(z, z)^2$, guaranteeing the "do no harm" property: variance is never increased, and adjustment defaults to the difference-in-means when predictions are uninformative.
• Digital twin estimators: Impute missing potential outcomes using only AI predictions, leading to substantial bias unless predictions are perfectly calibrated, rendering these unsafe for ATE estimation.

The theoretical analysis is augmented by explicit bias and variance derivations—proving that regression calibration absorbs systematic prediction bias, while uncalibrated approaches are highly susceptible to variance inflation and estimator failure.

Simulation Results

Controlled simulations sweep prediction quality ($R^2_{}$) and outcome predictability ($R^2_m$), quantifying gains from AI assistance under realistic conditions. Results show that regression calibration maintains low variance, even with poor AI predictions, contrasting sharply with uncalibrated methods.

Figure 1: Variance reduction across prediction quality and outcome predictability for continuous and binary outcomes.

Simulations for binary and count outcomes demonstrate that continuous probability predictions from LLMs (e.g., log-probabilities or averaged completions) offer noticeably greater variance reduction than hard binarizations, although precision gains are generally modest when outcome entropy is high.

Robustness to Prediction Bias

Estimation robustness to systematic prediction bias $b$ is evaluated. The uncalibrated model-assisted estimator incurs rapidly increasing variance with $Z_i$0, while regression calibration is completely insensitive, as the bias is absorbed in intercepts.

Figure 2: Design effect as a function of systematic AI prediction bias $Z_i$1, highlighting the robustness of calibrated regression adjustment.

Additional simulations compare regression adjustment to AIPW-style estimators and digital twins under symmetric and asymmetric prediction bias. Regression calibration uniquely preserves reliability and coverage; AIPW can overcover (deflate power), and digital twins fail catastrophically under asymmetric bias regimes.

Empirical Evaluation

Digital Twin Survey Mega-Study

In large-scale survey experiments, regression adjustment using LLM predictions yields moderate variance reduction (up to 16.4%), but the model-assisted and digital twin estimators produce either increased variance or substantial bias. LLM-extracted features, used in a ridge regression, match or marginally exceed the performance of direct LLM predictions, indicating the value of hybrid feature engineering. The correlation between predictions and outcomes is a strong determinant of achievable variance reduction.

Email Marketing A/B Test

On structured tabular data (Hillstrom dataset), Ridge and XGBoost regressions using the directly measured baseline features outperform LLM-based predictions by a factor of two, emphasizing the context-dependence of AI prediction utility. When covariates strongly capture outcome signal, classic ML methods remain optimal.

Online Platform Experiment

LLM predictions of conversion likelihood (from rich user activity narratives) contribute minimal efficiency gains (~1%) beyond pre-treatment outcomes, which deliver standard CUPED-like variance reduction (~15%). This underscores that, despite their flexibility for unstructured data, LLMs may not improve upon strong baseline signals in highly structured, industry environments.

Practical Guidance and Implementation Issues

• Regression adjustment with AI predictions is plug-and-play: simply include the predictions as covariates in an OLS model.
• Predictions should be constructed from strictly pre-treatment information to avoid data leakage and maintain design validity.
• Continuous scores (log-probabilities, averaged draws) are preferable for binary/count outcomes; direct probability elicitation from LLMs is sometimes necessary to overcome calibration issues.
• When rich unstructured context exists, LLMs that featurize text can be powerful auxiliary covariates. Combining LLM-based features and predictions with structured data often maximizes variance reduction.

Implications and Future Directions

Theoretical guarantees and empirical validation strongly support the routine inclusion of AI-generated predictions in regression adjustment for randomized experiments, especially where unstructured data dominates or traditional features are weak. However, efficiency gains are context-sensitive and often modest. The approach's robustness addresses both practical and statistical challenges associated with AI integration.

Potential extensions include subgroup analysis using unit-level predicted treatment effects, deployment of AI predictions as surrogate endpoints for early stopping, ML-assisted randomization tests, and formal transportability to mixed subject populations. The methodology's design-based foundation can be further generalized to adaptive or sequential experimental settings.

Conclusion

Regression calibration with AI-generated predictions provides a principled, reliable, and practical route for variance reduction in randomized experiments. It seamlessly integrates modern generative models with classical causal inference machinery, offering a simple, transparent, and statistically sound adjustment mechanism. Empirical practitioners should adopt this adjustment as standard practice, but remain vigilant about prediction quality, context dependence, and pipeline integrity.