Modeling Covariate Transition for Efficient Estimation of Longitudinal Treatment Effects in Randomized Experiments

Abstract: We present a regression-adjustment framework designed for the estimation of longitudinal treatment effects in randomized experiments under static regimes. While regression-adjustment methods are useful for variance reduction in randomized experiments by using pre-treatment covariates, they usually focus only on average effects, from which we cannot obtain valuable insights into when the effects appear and how long they continue. To address this issue, we consider intermediate outcomes and evolving post-treatment covariates over time, and we represent such dynamic trajectories using transition kernels. Furthermore, we establish the asymptotic normality and the semiparametric efficiency bound for our estimator, enabling more powerful statistical inference. Simulation studies and empirical analysis using A/B test data from a streaming platform in Japan show the practical advantages of our method.

• The paper presents a dynamic regression-adjusted estimator that uses covariate transition modeling to reduce variance and achieve semiparametric efficiency.
• It integrates Neyman-orthogonal scores and cross-fitting with high-dimensional machine learning to ensure robust and accurate inference.
• Empirical evaluations on synthetic and real A/B test data demonstrate reduced RMSE and tighter confidence intervals compared to traditional methods.

Modeling Covariate Transition for Efficient Estimation of Longitudinal Treatment Effects in Randomized Experiments

Introduction

The estimation of longitudinal treatment effects in randomized experiments has traditionally been challenged by the inability to efficiently leverage post-treatment covariate dynamics without introducing bias. While regression adjustment using pre-treatment covariates is standard for variance reduction, such approaches miss the temporal structure inherent in evolving covariate processes and their causal relationships with outcomes. This paper addresses these deficiencies by introducing a dynamic regression-adjusted estimation framework predicated on transition kernel modeling of covariate dynamics, together with Neyman-orthogonal moment functions and cross-fitting. The contributions include theoretical guarantees (asymptotic normality and semiparametric efficiency), as well as strong empirical results on both synthetic and real A/B test data.

Methodological Framework

The central estimation target is the sequence of expected potential outcomes, $\mu_{\bar{w}}(t)$, for outcome $Y$ and time-indexed treatment assignments $\bar{w}$, leading to the definition of trajectory-specific longitudinal average treatment effects. Standard empirical estimators, while unbiased and consistent, can suffer from high variance, particularly when longitudinal treatment histories are sparse in the data.

The framework models the joint evolution of covariates and outcomes over time through transition kernels $p^{(\tau)}$, which represent the conditional law of future covariate histories given the past. Dynamic regression adjustment is achieved through forward integration along these kernels, resulting in estimators that capture the influence of both pre- and post-treatment covariates without distorting the marginal estimand, thus avoiding the classical post-treatment bias problem.

The estimator is characterized by an augmented inverse-propensity-weighting structure, where three terms are combined: doubly-robust outcome adjustment, trajectory-wide marginalization via recursive integration, and an auxiliary correction isolating stochastic innovations from realized covariate paths. Learning of mean and transition functions can leverage high-dimensional or nonlinear machine learning models, with cross-fitting assuring robustness to overfitting and convergence rate conditions.

The moment condition problem is articulated via Neyman-orthogonal scores that ensure first-order insensitivity to nuisance estimation errors. This structure facilitates valid inference in semi-/non-parametric settings with complex or high-dimensional nuisance components.

Theoretical Results

A rigorous analysis establishes that the proposed dynamic regression-adjusted estimator achieves root-$n$ consistency and asymptotic normality, contingent on mild regularity conditions and sufficiently accurate nuisance estimation (see below for finite-sample performance). Crucially, the influence function associated with the estimator coincides with the semiparametric efficiency bound for longitudinal average treatment effects under randomized static regimes. This implies that, asymptotically, no regular estimator can improve upon the variance attained by this method.

Experimental Evaluation

Synthetic Data Analysis

Simulation studies were conducted using coupled non-linear dynamical systems where covariates evolve under a modified Lorenz-96 model influenced by past outcomes and treatment. The results consistently demonstrated lower root mean squared error (RMSE) and more compact confidence intervals for the proposed dynamic regression-adjusted estimators compared to both unadjusted empirical estimators and static (non-dynamic) regression adjustments.

Figure 2: Statistical properties of different estimators on synthetic data ($n=1000$): dynamic adjustment leads to reduced RMSE and confidence interval width vs. empirical estimators, while maintaining nominal coverage.

Further, the improvement in variance reduction is robust across sample sizes, and the efficiency gains amplify with larger samples.

Figure 1: RMSE reduction (%) of adjusted estimators vs. empirical estimator across different $n$; largest gains achieved by RF-MLP models as $n$ increases.

Empirical Analysis on Streaming Platform A/B Test

The methodology was applied to a proprietary large-scale A/B test, where the outcome was user viewing time. By exploiting daily longitudinal measurements, dynamic regression adjustment resulted in notably narrower confidence intervals and meaningful decreases in estimated standard errors compared to empirical estimators.

Figure 3: (a) Estimated longitudinal treatment effects with 95% CIs on daily viewing time (content-based vs. interaction-based recommendations); (b) Standard error reduction of 0.4–20.4% across the experimental period from dynamic adjustment.

Implications

From a practical standpoint, this framework addresses a persistent inefficiency in large-scale online experiments, providing a scalable approach to exploit evolving covariate information for tighter confidence intervals and thus increased statistical power—without introducing structural bias. Theoretically, the result situates itself at the intersection of causal inference, semiparametric statistics, and dynamic systems, providing the first efficient estimator for full longitudinal trajectories under static randomized regimes with measurable guarantees.

By leveraging flexible machine learning for nuisance estimation in a Neyman-orthogonal setup, the methodology is model-agnostic and adapts to the complexity of real-world covariate and outcome processes. The demonstrated empirical efficiency over classical g-formula and AIPW baselines accentuates the practical utility.

Future Directions

Potential avenues for extension include adaptation to observational settings with time-varying confounding, dynamic/adaptive experimental regimes (e.g., bandits, reinforcement learning), and inference for distributional treatment effects or more granular heterogeneity analyses. Dealing with irregular sampling, missing data, and complex compliance structures via continuous-time transition modeling represents another salient research direction.

Further, scalable and computationally efficient implementations—including variance reduction for high-dimensional or long-horizon settings—could broaden applicability. Finally, integration with uncertainty quantification techniques for machine-learned nuisance components can further improve inference reliability.

Conclusion

This work establishes a comprehensive framework for efficient estimation and inference of longitudinal treatment effects in randomized experiments with dynamic covariate evolution. By correctly modeling covariate transitions and combining modern machine learning for nuisance estimation with robust semiparametric theory, the proposed methodology achieves both theoretical efficiency bounds and empirical gains in estimation precision. The implications span both methodological theory and practical experimentation in modern online platforms.