---
title: 'Dri-MED: Drift-Adaptive Conservative Bandit'
url: https://www.emergentmind.com/topics/dri-med
type: topic
---

# Dri-MED: Drift-Adaptive Conservative Bandit

Dri-MED, short for **Drift-adaptive Minimum Empirical Divergence**, is an algorithm for conservative sequential experimentation in a **linear contextual stochastic multi-armed bandit** setting with **multiple users**, **personalized preference vectors**, and **drifting context distributions**. In the formulation introduced in “Bandits for Efficient Experimentation: Adapting to Control Group, Preferences, and Context Drifts” [2606.09802], the learner must make recommendations over repeated episodes while remaining competitive with a known baseline policy $\pi_0$. The method combines a reduction to a stationary-mean linear bandit with **heteroskedastic, non-stationary noise**, a discounted weighted estimator, MED-style randomized allocation, and a Lagrangian safety mechanism. Its stated applications include agriculture, clinical trials, recommendation, and other forms of sequential experimentation in which heterogeneity, drift, and control-group conservativeness must be handled jointly [2606.09802].

## 1. Problem formulation

Dri-MED is defined over episodes $\ell \in [L]$, with $H$ users in each episode. For each user $h$ in episode $\ell$, the learner selects an arm $A_{h,\ell}\in\mathcal A$, observes an $M$-dimensional feedback vector $Y_{h,\ell}$, and obtains scalar reward through the user-specific preference vector $p_h \in \mathbb R^M$:
$$
r_{h,\ell}(a) \defn Y_{h,\ell}^\top p_h.
$$
Contexts are sampled from an episode-specific distribution $C_\ell$, so drift is modeled through changing context distributions rather than fixed covariates. The paper emphasizes that users have heterogeneous scalarizations, and that practitioners often require recommendations not to fall substantially below a control-group or baseline policy $\pi_0$ [2606.09802].

The conservative requirement is encoded through a per-round comparison with the baseline mean reward. Violations are counted by
$$
\text{Violation}(\pi_0) = \mathbb{E}\!\left[ \sum_{\ell=1}^L \sum_{h=1}^H  1\!\Big\{{\bf m}_{h,\ell}(A_{h,\ell})<(1-\epsilon)\mathbb{E}_{a\sim \pi_0}[{\bf m}_{h,\ell}(a)]\Big\}\right],
$$
where $\epsilon\in[0,1]$ is the conservativeness tolerance. Regret is the cumulative gap to the optimal arm for each user and episode:
$$
Reg = \mathbb{E}\!\left[ \sum_{\ell=1}^L \sum_{h=1}^H \Delta_{A_{h,\ell},h,\ell} \right].
$$
This makes Dri-MED a baseline-constrained experimentation algorithm rather than a purely reward-maximizing bandit policy.

## 2. Reduction to a heteroskedastic linear bandit

The paper’s key modeling step is a reduction under two assumptions. First, conditional on arm and context,
$$
Y_{h,\ell} \mid (A_{h,\ell}=a, C_{h,\ell}=c)\sim N(m_a,\Sigma_{a,c}),
$$
with context affecting the covariance but not the mean:
$$
\mathbb E[Y_{h,\ell}\mid A_{h,\ell}=a,C_{h,\ell}=c]=m_a.
$$
Second, the mean is linear:
$$
m_a=\Phi(a)\theta,
$$
where $\theta\in\mathbb R^d$ is unknown and $\Phi(a)\in\mathbb R^{M\times d}$ is known [2606.09802].

This implies that the scalar reward mean for user $h$ is
$$
\mathbb E[r_{h,\ell}(a)\mid A_{h,\ell}=a,C_{h,\ell}=c]
= \theta^\top \Phi(a)^\top p_h.
$$
The effective scalar variance is
$$
\sigma^2_{a,h,\ell}=p_h^\top \Sigma_{a,C_{h,\ell}}p_h,
$$
and Dri-MED defines normalized features
$$
\phi_{a,h,\ell}=\frac{1}{\sigma_{a,h,\ell}}\Phi(a)^\top p_h.
$$
After rescaling, the learner faces
$$
\tilde r_{h,\ell}(a)=\frac{r_{h,\ell}(a)}{\sigma_{a,h,\ell}}
\sim N(\theta^\top \phi_{a,h,\ell},1).
$$

The important implication is that the mean parameter $\theta$ remains stationary, while drift enters through **time-varying heteroskedasticity** and user-specific effective features. This is one of the central cautionary points in the paper: Dri-MED does **not** address arbitrary mean drift. Its theoretical treatment depends on the context-independent mean assumption, so the non-stationarity it adapts to is variance and normalization drift induced by changing context distributions [2606.09802].

## 3. Algorithmic design

Dri-MED is explicitly described as being inspired by the linear version of the MED strategy, but adapted to heteroskedastic non-stationary noise. Its first stage constructs user- and episode-specific normalized features and then estimates $\theta$ with discounted heteroskedastic ridge regression:
$$
\hat{\theta}_{\ell-1}
=
\arg\min_\theta
\sum_{s=1}^{\ell-1}\gamma_{\rm decay}^{\ell-1-s}\sum_{h=1}^H
\left(\theta^\top \phi_{A_{h,s},h,s}-\tilde r_{h,s}\right)^2
+\frac{\lambda}{2}\|\theta\|_2^2,
$$
with discounted Gram matrix
$$
V_{\ell-1}^{\gamma_{\text{decay}}}
=
\sum_{s=1}^{\ell-1}\gamma_{\rm decay}^{\ell-1-s}
\sum_{h=1}^H
\phi_{A_{h,s},h,s}\phi_{A_{h,s},h,s}^\top
+\lambda \gamma_{\rm decay}^{\ell-1}I.
$$
Discounting is the mechanism by which old data are downweighted when context distributions drift [2606.09802].

The second stage makes the baseline conservative through optimism. Dri-MED estimates the current baseline value and inflates it to an optimistic upper confidence estimate:
$$
\hat{\mu}_{0,\ell}^+
=
\hat{\mu}_{0,\ell}
+
\sqrt{\beta_{\ell}(\delta_\ell)}
\left\|
\sum\nolimits_a \pi_0(a)\bar{\phi}_{a,\ell}
\right\|_{(V_{\ell-1}^{\gamma_{\text{decay}}})^{-1}},
$$
then defines the threshold
$$
\tau_\ell=(1-\epsilon)\hat\mu_{0,\ell}^+.
$$
Because the baseline is estimated optimistically, the resulting safety threshold is stringent, making the algorithm cautious about declaring an arm acceptable.

Within each episode and user, Dri-MED computes an empirical best arm, empirical gaps, and lower confidence bounds:
$$
\mathrm{LCB}(a,h,\ell)
=
\langle \hat\theta_{h,\ell},\phi_{a,h,\ell}\rangle
-
\sqrt{\beta_{h,\ell}(\alpha_{h,\ell})}
\|\phi_{a,h,\ell}\|_{(V_{h,\ell}^{\gamma_{\rm decay}})^{-1}}.
$$
Its MED-style score is then exponentially penalized when the arm’s lower confidence bound falls below the conservative threshold:
$$
\tilde f_h(a)
=
f_h(a)\cdot
\exp\!\Bigl(-\nu_{h,\ell}\cdot \max\{0,\tau_\ell-\mathrm{LCB}(a,h,\ell)\}\Bigr).
$$
This Lagrangian factor is the main safety modification. Dri-MED then combines approximate $G$-optimal design, a point mass on the empirical best arm, and uniform exploration:
$$
q_h(a)
=
\alpha_{\rm opt}\,q_h^{\rm opt}(a)
+
\alpha_{\rm emp}\,\mathbf 1[a=\hat a_{h,\ell}]
+
(1-\alpha_{\rm opt}-\alpha_{\rm emp})\frac1{|\mathcal A|}.
$$
A saturation test forces additional exploration for geometrically under-sampled arms, and a dual variable is updated after each decision:
$$
\nu_{h,\ell}
\leftarrow
\max\left\{ 0,\,
\nu_{h,\ell} + \eta_{h,\ell}\left(\xi_{h,\ell}-\frac{\epsilon}{1-\epsilon}\right)
\right\},
$$
with $\eta_{h,\ell}=(\ell h)^{-1/2}$ [2606.09802].

## 4. Regret, safety, and theoretical guarantees

The analysis decomposes regret into safe and violating components,
$$
Reg = Reg^{\rm safe} + Reg^{\rm viol},
$$
and introduces a constraint-aware gap through
$$
\frac{1}{\tilde{\Delta}}
\defn
\max\left\{
\frac{1}{\Delta_{\min}},
\frac{B-\Delta_0}{(\Delta_0+\epsilon\mu_0)^2}
\right\}.
$$
The paper’s main theorem states that, under Assumptions 1–3,
$$
Reg
=
\tilde{\mathcal O}\left(
\kappa\,
\frac{ d^2\left(\log^2(LH)+\log(\Sigma_{LH})\right) }{ \tilde{\Delta} }
\right),
$$
where $\kappa$ is a variance-aware multiplicative term derived from heteroskedasticity, and $\Sigma_{LH}$ is a discounted cumulative variance term. The abstract summarizes this more compactly as instance-dependent regret of order
$$
\tilde{\mathcal O}\!\left(\frac{\kappa}{\tilde\Delta}d^2\log(T)\right),
$$
with $T=LH$ [2606.09802].

The constraint guarantee is separate. Dri-MED is shown to satisfy
$$
\mathrm{Violation}(\pi_0)=\tilde{\mathcal O}(d).
$$
This is not a zero-violation guarantee. A common misunderstanding would be to read Dri-MED as exact safe control; the formal result is instead that expected violations scale only logarithmically-hidden linearly in the feature dimension. The safety argument depends on confidence ellipsoids for discounted regression, MED-style saturation control, and lower bounds on violating-round gaps relative to the baseline.

## 5. Empirical evaluation

The reported experiments are synthetic. The main setup uses $H=10$ users, $A=5$ arms, $L=1000$ episodes, $d=4$, $M=4$, and 128 random seeds. User preferences are drawn from a Dirichlet distribution and fixed across episodes. The observation model is
$$
Y_{h,\ell}\sim
\mathcal N\!\left(
\Phi(A_{h,\ell})\theta^\star,\;
\sigma_r(1+\|C_{h,\ell}\|)\Sigma_{A_{h,\ell}}
\right),
$$
so the mean is fixed while the noise scale changes with context. The baseline policy is chosen as the median-performance arm per user. The study evaluates no drift, gradual drift, periodic drift, and abrupt drift, with Dri-MED hyperparameters including $\gamma_{\rm decay}=0.99$, $\epsilon=0.1$, $\nu_0=1.0$, $\alpha_{\rm opt}=0.99$, and $\alpha_{\rm emp}=0.005$ [2606.09802].

The empirical findings are consistent across drift regimes. Dri-MED and Dri-IMED achieve substantially lower cumulative regret than OFUL, LinMED, LinIMED, and LinTS. OFUL is reported as the strongest baseline, but still about an order of magnitude worse. The paper further states that Dri-MED rapidly drives the true violation rate close to zero after initial exploration, whereas the baselines maintain persistent violations. In arm-allocation plots, Dri-MED concentrates nearly all mass on the oracle-optimal arm for each user, while LinMED spreads mass across suboptimal arms. The reported interpretation is that ignoring drift, heteroskedasticity, or personalized preferences leads to materially worse behavior under the same experimentation budget [2606.09802].

## 6. Scope, interpretation, and limitations

Dri-MED is best understood as a **baseline-conservative experimentation algorithm** rather than a general non-stationary contextual bandit in the strongest possible sense. Its theory relies crucially on the mean-stationarity assumption
$$
\mathbb E[Y_{h,\ell}\mid A_{h,\ell}=a,C_{h,\ell}=c]=m_a,
$$
so if contexts alter the mean as well as the covariance, the paper’s reduction no longer applies. This suggests that Dri-MED is most appropriate when drift manifests as changes in user mix, context distribution, or noise scale rather than structural reward-function drift [2606.09802].

Several practical constraints follow directly from the formulation. First, one needs access to, or a reasonable estimate of, the heteroskedastic variances $\sigma_{a,h,\ell}^2$, because both features and rewards are normalized by them. Second, Dri-MED is more computationally involved than optimism-based baselines such as OFUL because it requires confidence computations and an ApproxDesign call. Third, its conservative mechanism is relative to a known baseline policy $\pi_0$, which is appropriate in settings such as clinical trials or recommendation systems with existing control policies, but less natural when no baseline is available. The paper also notes an implementation-level virtue of randomized allocation: explicit design probabilities are useful for downstream offline evaluation via inverse propensity scoring.

A further interpretive point concerns the name. In the supplied arXiv literature, Dri-MED refers explicitly to the bandit algorithm above, not to a medical diagnosis system or medication recommender. Its connection to clinical domains arises through experimentation with control groups and personalized outcomes, not through medical reasoning architecture. That distinction matters because the algorithm’s contribution is methodological: it unifies personalized scalarization, context-distribution drift, and conservative baseline constraints inside a MED-style linear bandit framework [2606.09802].

Source: https://www.emergentmind.com/topics/dri-med