Bandits for Efficient Experimentation: Adapting to Control Group, Preferences, and Context Drifts

Abstract: We consider a variant of the linear contextual stochastic multi-armed bandits, where the learner must provide recommendations to a group of users, each having its personalized preference vector, and in the presence of context distributions that are drifting over time. Under practitioner-friendly assumptions, we reduce this setting to linear bandit with stationary mean but heteroskedastic and non-stationary noise. We further study the case when the learner must ensure the mean reward of each decision must exceed that of a baseline strategy $\boldsymbolπ_0$ at each decision step. We introduce Dri-MED, an algorithm inspired from the linear version of the MED strategy, and carefully adapted to handle the non-stationary heteroskedastic noise. We show that the instance-dependent regret scales as $\tilde{\mathcal O}\left(\fracκ{\tildeΔ}d^2(\log(T)\right)$, where $\tildeΔ$ is the constraint-aware sub-optimality gap subject to policy $π_0$, with variance-aware multiplicative term $κ$ that we carefully handle using heteroskedastic regression. We further show Dri-MED enjoys $\tilde{\mathcal{O}}(d)$ expected constraint violations. Our numerical results suggest that Dri-MED significantly outperforms conservative baselines that ignores the drift and preference structure.

• The paper demonstrates a drift-adaptive bandit approach (Dri-MED) that integrates safety constraints, evolving context distributions, and personalized preference modeling.
• It introduces a Lagrangian-augmented MED-type algorithm with instance-dependent regret bounds and constraint violations in non-stationary, heteroskedastic settings.
• Empirical results show that Dri-MED achieves substantially lower cumulative regret and near-zero constraint violations compared to traditional stationary LinCB baselines.

Bandits for Efficient Experimentation under Control Group, Preferences, and Context Drifts

Introduction and Problem Formulation

This work introduces and analyzes a variant of the linear contextual stochastic multi-armed bandit (LinCB) framework where the learner provides sequential recommendations to a population of users (or experimental units), each with personalized preferences, under context distributions that drift over time. The problem is further complicated by the necessity of enforcing a per-decision safety constraint—the expected reward for each user must exceed that of a prescribed baseline (control group) policy, even as contexts evolve. This setting closely models practical experimental design in domains such as agriculture, clinical trials, and recommender systems, where both ethical constraints and contextual non-stationarity are critical.

The formalism assumes a finite arm set $\mathcal{A}$, user population $H$, and $L$ episodes; at each episode and for each user, contexts are drawn from evolving distributions, rewards are observed, and personalized preference vectors $p_h\in\mathbb{R}^M$ induce user-dependent payoffs. The core challenge is learning an allocation strategy that maximizes cumulative reward and minimizes both regret and constraint violation, robust to non-stationary, heteroskedastic noise and personalized preferences.

Methodology: Drift-Adaptive MED-Type Algorithms

The paper introduces Dri-MED (Drift-Adaptive Minimum Empirical Divergence), building upon the MED (Minimum Empirical Divergence) philosophy, but generalizing classical linear bandit methods to incorporate:

1. Context drift and heteroskedasticity: The reward noise variance can shift both between users and across episodes, reflecting context drift and personalized heterogeneity.
2. Baseline control group constraint: A Lagrangian dual-augmented objective assures that policy recommendations are safe with respect to an unknown, stochastic baseline estimate.
3. Preference-based allocation: Arm selection is personalized by each user’s preference direction and context-dependent variance, ensuring that allocations remain preference-divergent and constraint-aware.

The sampling distribution for each user and episode is determined by a Lagrangian-augmented design, balancing empirical performance, sufficient exploration, and constraint satisfaction. The proposed algorithm leverages discounted, episode-adaptive regression to absorb context drift, constrains arm pulls based on a pessimistic (inflated confidence) baseline estimate, and employs randomized probability matching across arms to facilitate both exploration and policy evaluation via inverse-propensity weights.

Theoretical Guarantees

Regret and Constraint Violation Bounds

A primary contribution is the derivation of instance-dependent upper bounds for cumulative regret and expected constraint violations. Notably, the regret bound scales as

$$\tilde{\mathcal{O}}\left(\frac{\kappa d^2}{\tilde{\Delta} \log^2(LH)}\right),$$

where $d$ is the (feature) dimension, $\kappa$ is the variance condition number capturing the ratio between largest and smallest per-arm/user/episode noise, and $\tilde\Delta$ incorporates both optimality gap and constraint gap enforced by the baseline policy (see the main text for precise definitions).

Crucially, the bound quantifies the interplay between context drift (via $\kappa$), the safety constraint (via $\tilde\Delta$), and problem complexity (via $H$0). The expected cumulative number of constraint violations is shown to be $H$1, independent of time horizon, under sub-Gaussian noise.

These bounds are non-asymptotic and instance-dependent, matching or improving on minimax rates for unconstrained or stationary variants, and decisively characterizing the impact of safety constraints and context heterogeneity, which prior works on LinCB do not capture.

Numerical Results

Empirical evaluation on synthetic environments simulating various context drift regimes (no drift, gradual, periodic, abrupt) is presented. Dri-MED and its deterministic variant Dri-IMED are compared to state-of-the-art stationary LinCB baselines—OFUL, LinMED, LinIMED, and LinTS—which lack support for context drift, personalized preferences, and baseline constraints.

Figure 1: Cumulative regret (left), estimated violation rate (center), and true violation rate (right) over time for gradual and periodic drift regimes.

Dri-MED consistently achieves substantially lower cumulative regret and near-zero constraint violations after initial exploration, robust across all drift regimes. The performance gap with respect to the best unconstrained baseline (OFUL) is persistent and grows with episode count, signifying the necessity of drift adaptation and baseline-aware design.

Ablation results (see supplementary figures) examine sensitivity to baseline quantile and satisficing tolerance, demonstrating the method’s robustness and supporting the theoretical analysis.

Figure-Based Insights

Figure 2: Cumulative regret, estimated violation, and true constraint violation for all drift varieties, confirming robustness of Dri-MED and Dri-IMED across synthetic environments.

Figure 3: Ablation study on abrupt drift; illustrates regret sensitivity to the baseline policy quantile and constraint tolerance.

Figure 4: Per-user arm allocation fractions; Dri-MED increasingly concentrates pulls on the oracle arm, while the unconstrained baseline exhibits significant suboptimal arm exploration.

Pull-mass analyses show that Dri-MED homes almost exclusively on the optimal arm per user, but only when safe to do so, a behavior not realized by baseline algorithms.

Implications, Limitations, and Future Directions

This work forges the first synthesis of context-drift adaptation, personalized preference modeling, and baseline-aware (safe) LinCB in a unified, practical framework. The implications are immediate for practical sequential experiment design in domains with noisy, drifting, and personalized feedback (e.g., adaptive clinical studies, industrial A/B testing, and personalized recommenders).

The regret bounds provide actionable guidance for practitioners: heteroskedasticity and non-stationarity can be efficiently absorbed via drift-adaptive regression, but safety constraints fundamentally increase effective problem hardness (quantified by $H$2 and $H$3). The empirical effectiveness demonstrates that traditional stationary LinCB methodologies—even with adaptive exploration—fail under non-stationarity or preference heterogeneity.

A significant open direction is relaxing the assumption of stationary mean rewards. Allowing mean reward drift would broaden practical applications (e.g., engagement decay in recommenders, non-stationary treatment effects in medicine). Robustness to heavy-tailed (non-sub-Gaussian) noise is also unfinished—the present analysis leverages sub-Gaussianity at critical steps. Designing MED-variant algorithms tolerant to heavy-tailed or misspecified noise processes remains a crucial open area. Moreover, extensions to combinatorial action spaces or finite memory/adaptive batch bandit variants raise intricate theoretical questions.

Conclusion

This manuscript provides a rigorous, general-purpose framework for sequential experimental design under context drift, personalized preferences, and operationally crucial safety constraints, as formalized in the context-drifted, baseline-constrained LinCB model. Dri-MED’s theoretical and empirical performance establishes it as a practical mechanism for reliable and efficient sequential experimentation under complex, realistic noise and preference structures (2606.09802).