---
title: Doubly-Robust UCB Methods
url: https://www.emergentmind.com/topics/doubly-robust-upper-confidence-bound-dr-ucb
type: topic
---

# Doubly-Robust UCB Methods

Searching arXiv for papers on Doubly-Robust UCB and related contextual bandit methods.
Doubly-Robust Upper Confidence Bound (DR-UCB) denotes a class of optimistic bandit algorithms that integrate doubly-robust estimation with upper-confidence-based action selection. In the supplied literature, the term has two closely related but distinct instantiations. In "Sequential Decision Problems with Missing Feedback" [2507.19596], DR-UCB is a fully nonparametric algorithm for stochastic multi-armed bandits with missing rewards, where missingness may depend on observable covariates and, indirectly, on rewards. In "Doubly-Robust Lasso Bandit" [1907.11362], the same perspective appears in a sparse high-dimensional contextual bandit algorithm whose confidence mechanism is implicit rather than expressed as an explicit per-arm bonus. In both cases, the common principle is to construct pseudo-outcomes or reward estimators that combine inverse-propensity correction with an outcome model, then use high-probability confidence control to support optimistic action choice.

## 1. Conceptual basis

The doubly-robust construction originates in missing data and causal inference, where an estimator combines a propensity model and an outcome model. The defining property is that consistency can be retained if either the propensity model or the outcome model is correctly specified. The bandit setting naturally admits this viewpoint because rewards for unobserved actions, or sometimes even for chosen actions, are missing by design or by the data-generating process.

In the missing-feedback formulation of [2507.19596], at each round the learner chooses an arm \(A_t\), but the realized reward may or may not be observed. For each arm \(a\), Nature draws \((R_{a,t}, C_{a,t}, X_{a,t})\), where \(R_{a,t}\) is reward, \(C_{a,t}\in\{0,1\}\) indicates whether the reward is observed, and \(X_{a,t}\) is an observable covariate vector. The target is the unconditional mean reward \(\theta_a := \mathbb{E}_\nu[R_a]\), and the difficulty is that naive empirical averages of observed rewards can converge to \(\widetilde{\theta}_a := \mathbb{E}[R_a \mid C_a=1]\) rather than \(\theta_a\) when missingness depends on reward through covariates [2507.19596].

In the sparse contextual-bandit formulation of [1907.11362], only the reward of the selected arm is observed, while the contexts of all arms are available. The algorithm defines a doubly-robust pseudo-reward for the average context \(\bar b(t)=\frac{1}{N}\sum_{i=1}^N b_i(t)\), thereby recasting adaptive bandit feedback as a missing-data estimation problem. The central role of the doubly-robust device is not merely debiasing but variance control under adaptive sampling, which enables high-dimensional Lasso estimation with logarithmic dependence on ambient dimension [1907.11362].

These two usages are methodologically aligned. Both use a doubly-robust score to restore valid estimation under nontrivial observation mechanisms, and both attach confidence control to that estimator in order to drive exploration. This suggests a broader interpretation of DR-UCB as an optimistic bandit design pattern rather than a single fixed algorithm.

## 2. Missing-feedback formulation

The explicit DR-UCB algorithm in [2507.19596] studies stochastic multi-armed bandits over horizon \(T\) with a finite arm set \(A=\{1,\dots,A\}\). Across arms, \((R_a,C_a,X_a)\) are independent; over time, \((R_{a,t},C_{a,t},X_{a,t})\) are i.i.d. draws from \((R_a,C_a,X_a)\). Rewards satisfy sub-Gaussian conditions: \(R_a \sim \mathsf{sG}(\sigma_a)\), and, when covariates are used, \(R_a\mid X_a \sim \mathsf{sG}(\sigma_a)\) [2507.19596].

The paper distinguishes two missingness regimes. Under reward-independent missingness, \(C_a \perp R_a\), standard UCB can be adapted and retains nearly optimal regret. Under the more general regime, the paper assumes a covariate-based conditional ignorability condition. For each arm \(a\), either model-based CIA holds,
\[
\mathbb{E}_\nu[R_a \mid X_a, C_a] = \mathbb{E}_\nu[R_a \mid X_a] =: \theta_a(X_a) \quad \text{a.s.},
\]
or design-based CIA holds,
\[
\mathbb{E}_\nu[C_a \mid X_a, R_a] = \mathbb{E}_\nu[C_a \mid X_a] =: q_a(X_a)\in[\underline{q},1] \quad \text{a.s.}
\]
In addition, \(|\theta_a(X_a)| \le \overline{K}_\theta\) uniformly [2507.19596].

Within this formulation, DR-UCB estimates the unconditional arm mean via a doubly-robust average over rounds in which arm \(a\) was pulled:
\[
\widehat{R}^{\text{DR}}_a(t)
:=
\frac{1}{P_a(t)}
\sum_{\ell=1}^{t-1}
\mathbf{1}[A_\ell=a]
\left(
\frac{C_{a,\ell}\big(R_{a,\ell}-\hat{\theta}_a(X_{a,\ell})\big)}{\hat{q}_a(X_{a,\ell})}
+
\hat{\theta}_a(X_{a,\ell})
\right),
\]
where \(P_a(t)\) is the number of times arm \(a\) has been pulled up to time \(t-1\), \(\hat{\theta}_a(\cdot)\) estimates the conditional mean reward, and \(\hat{q}_a(\cdot)\) estimates the conditional observation probability [2507.19596].

The optimistic index takes the form
\[
\widetilde{R}^{\text{DR}}_a(t,\delta)
=
\widehat{R}^{\text{DR}}_a(t)
+
b^{\text{DR}}_{a,t}(\delta),
\]
with
\[
b^{\text{DR}}_{a,t}(\delta)
=
\overline{\sigma}\sqrt{\frac{2\ln(2AT/\delta)}{P_a(t)}}
+
b^{\text{res}}_{a,t}(\delta),
\]
where \(b^{\text{res}}_{a,t}(\delta)\) captures nuisance-estimation error and is assumed to be of smaller order than \(P_a(t)^{-1/2}\) [2507.19596]. Action selection is then classical UCB:
\[
A_t = \arg\max_{a\in A}\widetilde{R}^{\text{DR}}_a(t,\delta).
\]

The algorithm is described as fully nonparametric because \(\hat{\theta}_a\) and \(\hat{q}_a\) may be estimated by arbitrary machine-learning methods, including forests or neural nets, subject to prediction-error conditions [2507.19596].

## 3. Doubly-robust estimation and confidence construction

The doubly-robust structure is the defining technical ingredient. In [2507.19596], the two nuisance functions are
\[
\theta_a(X_a) := \mathbb{E}_\nu[R_a \mid X_a], \qquad
q_a(X_a) := \mathbb{E}_\nu[C_a \mid X_a].
\]
The estimator combines a regression term \(\hat{\theta}_a(X)\) with an inverse-propensity residual correction. If \(\hat{q}_a\) is accurate, the inverse-weighted residual corrects selection bias. If \(\hat{\theta}_a\) is accurate, the regression term is itself consistent. The paper formalizes double robustness by requiring that for each arm there exist \(\widetilde{q}_a,\widetilde{\theta}_a\) such that either \(\widetilde{q}_a=q_a\) or \(\widetilde{\theta}_a=\theta_a\), together with \(L^2\) convergence conditions on the nuisance estimators [2507.19596].

The rate conditions are expressed through empirical errors:
\[
\text{Err}_t(\hat{q}_a) \lesssim P_a(t)^{-\alpha_q}, \qquad
\text{Err}_t(\hat{\theta}_a) \lesssim P_a(t)^{-\alpha_\theta}, \qquad
\text{Err}_t(\hat{q}_a)\,\text{Err}_t(\hat{\theta}_a) \lesssim P_a(t)^{-\alpha},
\]
with \(\alpha>1/2\) [2507.19596]. The product condition ensures that the bias from nuisance estimation is asymptotically dominated by the stochastic term of order \(P_a(t)^{-1/2}\).

A central result is a high-probability concentration inequality for the DR estimator under adaptive sampling. Defining the failure event
\[
F^{\text{DR}}(\delta)
:=
\left\{
\exists a\in A,\; \exists t\le T :
\big|\widehat{R}^{\text{DR}}_a(t)-\theta_a\big|
\ge b^{\text{DR}}_{a,t}(\delta)
\right\},
\]
the paper proves a tail bound using martingale difference techniques à la Freedman (1975) [2507.19596]. The adaptive bandit trajectory is not i.i.d., but the DR score is conditionally mean-zero once nuisances are treated as fixed, so concentration remains available through martingale methods.

In [1907.11362], the doubly-robust estimator appears as a pseudo-reward rather than a direct arm-wise mean estimator:
\[
\hat{r}(t)
=
\bar{b}(t)^\top \hat{\beta}(t-1)
+
\frac{1}{N}\frac{r_{a(t)}(t)-b_{a(t)}(t)^\top \hat{\beta}(t-1)}{\pi_{a(t)}(t)}.
\]
Here \(\pi_{a(t)}(t)\) is the algorithm-controlled propensity of the chosen arm, and \(\hat{r}(t)\) is unbiased for \(\bar b(t)^\top\beta\) whenever all propensities are positive [1907.11362]. The crucial point is that the conditional variance of this pseudo-reward can remain \(O(1)\) even when propensities decay with time, provided they satisfy a suitable lower bound and the regression estimate is sufficiently accurate. This variance control underwrites a high-probability Lasso error bound, which functions as an implicit confidence radius over the parameter \(\beta\).

## 4. UCB structure and algorithmic variants

The explicit UCB structure is clearest in [2507.19596]. The algorithm initializes by pulling each arm once, then iteratively updates counts \(P_a(t)\) and \(N_a(t)\), nuisance estimates \(\hat q_a,\hat\theta_a\), the doubly-robust reward estimate \(\widehat{R}^{\text{DR}}_a(t)\), and the optimistic index \(\widetilde{R}^{\text{DR}}_a(t,\delta)\). The choice rule is the standard optimistic rule
\[
A_t := \arg\max_{a\in A}\widetilde{R}^{\text{DR}}_a(t,\delta)
\]
[2507.19596].

By contrast, [1907.11362] does not present an explicit per-arm analytical bonus. Instead, it uses a greedy choice with structured randomized exploration. After an initial pure exploration phase of length \(z_T\), the algorithm sets
\[
\lambda_{1t} := \lambda_1\sqrt{\frac{\log t + \log d}{t}},
\]
draws \(m_t \sim \mathrm{Ber}(\lambda_{1t})\), and either explores uniformly if \(m_t=1\) or exploits greedily:
\[
a(t)=\arg\max_{1\le i\le N} b_i(t)^\top \hat{\beta}(t-1).
\]
The parameter estimate is updated through Lasso on the dataset of average contexts and doubly-robust pseudo-rewards:
\[
\hat{\beta}(t)
=
\arg\min_{\beta\in\mathbb{R}^d}
\left\{
\frac{1}{t}\sum_{(\bar b,\hat r)\in\mathbb{S}}
(\hat r-\bar b^\top\beta)^2
+
\lambda_{2t}\|\beta\|_1
\right\},
\quad
\lambda_{2t}:=\lambda_2\sqrt{\frac{\log t + \log d}{t}}.
\]
The paper interprets this as UCB-style because the analysis yields a high-probability bound on \(\|\hat{\beta}(t)-\beta\|_1\), and instantaneous regret is controlled by that parameter error [1907.11362].

A concise comparison is given below.

| Variant | Feedback structure | Confidence mechanism |
|---|---|---|
| DR-UCB in [2507.19596] | Chosen arm’s reward may be missing | Explicit index \(\widehat{R}^{\text{DR}}_a + b^{\text{DR}}_{a,t}\) |
| DR-Lasso Bandit as DR-UCB view in [1907.11362] | Only chosen arm reward observed; all arm contexts observed | Implicit parameter confidence via Lasso error bound |

This suggests that “DR-UCB” can denote either a concrete optimistic index algorithm for missing feedback or a broader methodological template in which doubly-robust estimation supports high-probability optimism.

## 5. Regret guarantees

In [2507.19596], the pseudo-regret is
\[
\text{Regret}(\pi;\nu)
=
T\overline{\theta}
-
\sum_{t=1}^T \theta_{A_t},
\qquad
\overline{\theta}:=\max_a \theta_a.
\]
Under Assumptions 2 and 3, Theorem 2 states that for any \(\delta\in(0,1)\),
\[
\text{Regret}(\pi^{\text{DR-UCB}})
\;\lesssim\;
\frac{4 \overline{\sigma}\sqrt{A T\ln(2AT/\delta)}}{\underline{q}}
\quad \text{with probability } \ge 1-\delta-\delta_{\mathfrak{c}},
\]
where \(\delta_{\mathfrak{c}}\) reflects the failure probability of nuisance-error bounds [2507.19596]. The paper also gives a minimax lower bound
\[
\text{Regret}^\star(\mathcal{C}_j)
\ge
\frac{\sqrt{T}(A-1)}{16e},
\qquad j\in\{1,2\},
\]
showing that the achieved \(\widetilde{O}(\sqrt{T})\) rate is nearly minimax optimal [2507.19596].

In [1907.11362], the main regret theorem for the Doubly-Robust Lasso Bandit states that under the sparse linear contextual model and Assumptions A1–A4, with probability at least \(1-2\delta\),
\[
R(T)=O\big(s_0 \sqrt{T}\log(dT)\big).
\]
The bound has two especially notable properties: it does not depend on the number of arms \(N\), and it scales logarithmically in the ambient dimension \(d\) rather than polynomially [1907.11362]. The proof partitions regret into initial uniform exploration, later randomized exploration, and greedy exploitation, with the greedy term controlled by the high-probability Lasso error rate
\[
\|\hat{\beta}(t)-\beta\|_1
=
O\!\left(
s_0\sqrt{\frac{\log d + \log t}{t}}
\right)
\]
[1907.11362].

The two regret theories operate in different model classes and therefore are not directly numerically comparable. Nonetheless, both emphasize that doubly-robust estimation preserves a \(\sqrt{T}\)-type regret order under observation mechanisms that would break naive estimators.

## 6. Relation to standard UCB and other bandit methods

The most direct comparison in [2507.19596] is with standard UCB under missing rewards. When missingness is reward-independent, standard UCB remains valid. The paper defines a regularized empirical mean based on observed rewards,
\[
\widehat{R}^{\text{UCB}}_a(t)
:=
\frac{1}{N_a(t)+\lambda}
\sum_{\ell=1}^{t-1}
\mathbf{1}[A_\ell=a]\, C_{a,\ell}R_{a,\ell},
\]
and an optimistic index with bonus \(b^{\text{UCB}}_{a,t}(\delta)\). Under Assumption 1 and mild regularization, Theorem 1 shows
\[
\text{Regret}(\pi^{\text{UCB}})
\;\lesssim\;
\frac{4 \overline{\sigma}\sqrt{2AT\ln(2AT/\delta_1)}}{\underline{q}_\lambda}
\]
with high probability [2507.19596].

The difficulty arises when missingness is reward-dependent. The paper provides a two-arm example with
\[
(\theta_1,\theta_2)=(0.5,1), \quad
(\widetilde{\theta}_1,\widetilde{\theta}_2)=(1.16,1.08), \quad
(q_1,q_2)=(0.25,0.9).
\]
Here the truly optimal arm is arm 2, but the arm with the highest observed-data mean is arm 1, so standard UCB can converge to the wrong arm and incur linear regret [2507.19596]. DR-UCB is designed precisely to avoid this failure mode.

The same paper contrasts DR-UCB with IPW-only and regression-only variants. IPW-UCB uses
\[
\widehat{R}^{\text{IPW}}_a(t)
=
\frac{1}{P_a(t)}
\sum_{\ell=1}^{t-1}
\mathbf{1}[A_\ell=a]\frac{C_{a,\ell}R_{a,\ell}}{\hat q_a(X_{a,\ell})},
\]
while regression-UCB uses
\[
\widehat{R}^{\text{reg}}_a(t)
=
\frac{1}{P_a(t)}
\sum_{\ell=1}^{t-1}
\mathbf{1}[A_\ell=a]\hat\theta_a(X_{a,\ell}).
\]
Each is single-robust; the first requires correct \(\hat q_a\), the second correct \(\hat\theta_a\). DR-UCB combines both and preserves correctness if at least one is well specified [2507.19596].

In the high-dimensional contextual setting, [1907.11362] compares the Doubly-Robust Lasso Bandit with classical linear bandit methods and sparse bandit approaches. Standard LinUCB-type methods are described as low-dimensional and typically polynomial in \(d\). Sparse linear bandit methods such as Abbasi-Yadkori et al. (2012), Carpentier and Munos (2012), and Gilton and Willett (2017) are discussed as either scaling with \(\sqrt d\), requiring restricted arm geometries, or relying on prior support information. The paper also compares against Lasso Bandit and MCP Bandit in the contextual bandit with linear rewards setting, highlighting that those methods use forced sampling and yield regret linear in \(N\), whereas the proposed method shares information across arms and avoids explicit forced-sampling schedules [1907.11362].

## 7. Practical considerations, simulations, and scope

In [2507.19596], practical implementation requires maintaining arm pull counts \(P_a(t)\), observed-reward counts \(N_a(t)\), nuisance models \(\hat{\theta}_a\) and \(\hat{q}_a\), and the DR estimator with its confidence radius. The paper highlights two nuisance-estimation strategies. The first is a separate batch approach, denoted M1, which uses an external dataset independent of the online run. The second is a leave-one-out online approach, denoted M2, in which nuisance models used at time \(t\) are fit on rounds \(1,\dots,t-2\), excluding the most recent observation, to approximate the conditional independence logic of cross-fitting [2507.19596]. Truncation of the propensity estimate,
\[
\hat q_a(x)\in[\underline q,1],
\]
is required to prevent instability from near-zero denominators [2507.19596].

Its simulation study uses two arms, horizon \(T=5000\), one-dimensional Gaussian covariates, and three scenarios: no missing data, reward-independent missingness, and reward-dependent missingness. In the biased missingness scenario, the naive observed-data means become \((1.16,1.08)\) although the true means remain \((0.5,1)\). Standard UCB then frequently selects the wrong arm, whereas DR-UCB gradually learns the correct best arm and exhibits sub-linear regret consistent with the theoretical \(\widetilde{O}(\sqrt{T})\) prediction [2507.19596].

In [1907.11362], the simulation study compares DR Lasso Bandit with Lasso Bandit on synthetic data with \(d=100\), \(s_0=5\), \(N\in\{10,20,50,100\}\), and correlated arm contexts. The reported findings are that for small \(N=10\), Lasso Bandit converges faster asymptotically, but DR Lasso Bandit has lower early-stage regret due to the exploration burden of forced sampling in the comparator. As \(N\) increases, DR Lasso Bandit significantly outperforms Lasso Bandit; its regret is empirically almost unaffected by \(N\). Performance also improves as arm-context correlation increases, which the paper attributes to the use of average contexts and shared information across arms [1907.11362].

A common misconception is to equate double robustness with immunity to all misspecification. The papers do not make that claim. In [2507.19596], the guarantees require covariate-based ignorability and nuisance conditions, including the product-rate condition with \(\alpha>1/2\). If both nuisance models are poor, the guarantees degrade. In [1907.11362], the method depends on the sparse linear model, boundedness, the i.i.d. context assumption over time, compatibility, and sub-Gaussian noise. Thus, double robustness addresses a specific bias-variance problem created by partial observation; it does not eliminate the need for structural assumptions.

Taken together, the two papers position DR-UCB as an approach for optimistic sequential decision-making under partial or selective feedback. In the missing-feedback multi-armed bandit setting, it is an explicit UCB algorithm with doubly-robust arm-wise mean estimation and nearly-optimal \(\widetilde{O}(\sqrt{T})\) worst-case regret [2507.19596]. In the sparse high-dimensional contextual setting, it appears as an implicit UCB methodology in which doubly-robust pseudo-rewards enable high-probability parameter confidence and regret \(O(s_0\sqrt{T}\log(dT))\) independent of the number of arms [1907.11362]. These formulations differ in model class and estimator architecture, but both exemplify the same central idea: optimism can remain statistically valid under missing or selectively observed feedback when confidence bounds are built on doubly-robust estimators rather than naive empirical means.

Source: https://www.emergentmind.com/topics/doubly-robust-upper-confidence-bound-dr-ucb