DRTS: Double Reverse Thompson Sampling

• Double Reverse Thompson Sampling (DRTS) is a contextual bandit algorithm that integrates doubly robust estimation with Bayesian resampling to utilize both observed and imputed arm rewards.
• It leverages pseudo-rewards and advanced statistical concentrations to achieve tighter regret bounds, scaling as O(d√T) and outperforming classical methods like LinTS.
• Empirical evaluations demonstrate DRTS's advantages in stable early-stage parameter estimation and lower cumulative regret, all without the need for forced exploration.

Double Reverse Thompson Sampling (DRTS), formally known as Doubly Robust Thompson Sampling, is a contextual bandit algorithm for linear payoffs that augments standard Thompson Sampling with a doubly-robust estimation mechanism, drawing on missing data techniques to construct pseudo-rewards for all arms at each round. The core innovation is the combination of doubly-robust estimators with Bayesian resampling, yielding improved regret bounds over classical algorithms such as LinTS and providing a more efficient exploitation of observed and unobserved contextual information (Kim et al., 2021).

1. Formal Model and Doubly-Robust Estimation

Consider a multi-armed contextual bandit setting with $N$ arms. At each round $t$, each arm $i\in\{1,\,\dots,N\}$ is characterized by a context vector $X_i(t)\in\mathbb{R}^d$. The stochastic reward $Y_i(t)$ is generated according to a linear model: $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$ where $\beta \in \mathbb{R}^d$, $\|\beta\|_2\le 1$, and $\eta_i(t)$ is a conditionally zero-mean, $\sigma$-sub-Gaussian noise term. Only the reward associated with the chosen arm, $t$0, is observed. The challenge is to leverage the information in all available contexts (including those for unchosen arms) despite the missing rewards.

To address the missing data problem, a doubly-robust (DR) pseudo-reward is constructed for every arm at every round. With $t$1 denoting the probability of selecting arm $t$2 given the history $t$3, and $t$4 denoting an imputation parameter (typically a ridge regression fit on observed data), the DR pseudo-reward for arm $t$5 is

$t$6

This construction ensures $t$7 regardless of the correctness of $t$8. These DR pseudo-rewards enable each round’s data to contribute all $t$9 contexts into the ridge regression update for $i\in\{1,\,\dots,N\}$0: $i\in\{1,\,\dots,N\}$1 where $i\in\{1,\,\dots,N\}$2 is a regularization parameter.

2. Algorithmic Description

The DRTS procedure comprises the following sequence at each round:

1. Observation: The learner observes all arms’ contexts $i\in\{1,\,\dots,N\}$3.
2. Sampling: For each arm $i\in\{1,\,\dots,N\}$4, sample

$i\in\{1,\,\dots,N\}$5

and compute $i\in\{1,\,\dots,N\}$6.

1. Selection and Resampling: Select $i\in\{1,\,\dots,N\}$7. Estimate (or approximate) $i\in\{1,\,\dots,N\}$8. If $i\in\{1,\,\dots,N\}$9, play $X_i(t)\in\mathbb{R}^d$0; otherwise, repeat sampling (up to $X_i(t)\in\mathbb{R}^d$1 tries) to ensure a minimum probability threshold $X_i(t)\in\mathbb{R}^d$2 for the chosen arm.
2. Feedback and Update: After observing $X_i(t)\in\mathbb{R}^d$3, compute $X_i(t)\in\mathbb{R}^d$4 for all arms. Update

$X_i(t)\in\mathbb{R}^d$5

then $X_i(t)\in\mathbb{R}^d$6 and $X_i(t)\in\mathbb{R}^d$7.

1. Imputation parameter (optional): Recompute $X_i(t)\in\mathbb{R}^d$8 using ridge regression on selected arms.

This procedure guarantees all-arm context usage at every step, bypassing the statistical inefficiency of using only the chosen arm's context as in classical LinTS.

3. Additive Regret Decomposition

The regret at time $X_i(t)\in\mathbb{R}^d$9 is defined by the difference between the expected reward of the optimal arm and the selected arm: $Y_i(t)$0 where $Y_i(t)$1. The analysis introduces the "super-unsaturated arms" set $Y_i(t)$2 and shows, with high probability, that the selected arm $Y_i(t)$3. The instantaneous regret decomposes additively: $Y_i(t)$4 This leads to two distinct regret components: estimation error and exploration bonus, facilitating a cleaner analysis than the saturated/unsaturated regime-splitting in LinTS.

4. Regret Bounds and Dimension Dependence

Assuming the minimum eigenvalue $Y_i(t)$5 of the average context-covariance matrix satisfies

$Y_i(t)$6

and under standard sub-Gaussian/martingale assumptions, DRTS admits a dimension-free estimation error

$Y_i(t)$7

Plugging this and an exploration term bounded by $Y_i(t)$8 into the cumulative regret sum over $Y_i(t)$9 gives the main bound: $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$0 In common scenarios—e.g., isotropic context distributions—$Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$1, so $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$2. This surpasses the $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$3 bound of LinTS by a factor $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$4 owing to the more efficient exploitation of the all-arms context matrix and dimension-free concentration bounds.

5. Comparative Advantages Over LinTS

DRTS achieves its improved regret for several reasons:

• Universal Context Utilization: The doubly-robust estimator enables every round’s regression matrix to incorporate all $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$5 contexts, in contrast to LinTS and LinUCB, which restrict updates to the chosen arm's context. This enhances statistical efficiency.
• Dimension-Free Martingale Bounds: The theory leverages novel vector- and matrix-martingale concentration inequalities that avoid the classical $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$6 penalty in estimation error.
• Clean Regret Decomposition: The algorithm removes the need to distinguish between "high-variance" and "low-variance" arms (the unsaturated/saturated dichotomy of prior analyses), resulting in a single additive decomposition for regret and further reducing complexity.

Collectively these innovations yield regret bounds scaling as $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$7 rather than $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$8, aligning DRTS performance with theoretically optimal rates up to logarithmic factors in many regimes (Kim et al., 2021).

6. Empirical Evaluation

Simulations with $Y_i(t) = X_i(t)^\top \beta + \eta_i(t),$9 and $\beta \in \mathbb{R}^d$0 arms compared DRTS to LinTS (Agrawal & Goyal ’13) and BLTS (IPW-based LinTS, Dimakopoulou et al. ’19). The findings include:

• Lower cumulative regret across all tested dimensions and arm counts, typically a 10–20% reduction.
• Improved early-stage estimation stability for $\beta \in \mathbb{R}^d$1, attributable to comprehensive context usage at each time step and adaptively growing regularization $\beta \in \mathbb{R}^d$2.
• Absence of forced exploration or explicit warm-up phases, a byproduct of the resampling device and thresholding on arm selection probabilities.

A summary table of empirical comparisons:

Algorithm	Cumulative Regret	Early $\beta \in \mathbb{R}^d$3 Estimation	Forced Exploration
DRTS	Lowest (10–20%↓)	Most stable	None
LinTS	Higher	Less stable	Required
BLTS	Intermediate	Intermediate	Required

These empirical advantages persist across both moderate and higher dimensions.

7. Broader Context and Related Methodologies

DRTS is positioned at the intersection of contextual bandit research, missing data estimation, and advanced martingale analysis. The method extends prior works exploiting importance weighting and doubly robust estimators for off-policy evaluation to the online bandit context. It distinguishes itself from prior art—such as “BLTS” (IPW-based LinTS, Dimakopoulou et al. 2019) and “Balanced Linear Thompson Sampling” (Kim et al. 2019)—by furnishing the first regret bound for LinTS that exploits the minimum context-covariance eigenvalue without explicit dimension dependence (Kim et al., 2021).

Further implications include the circumvention of complicated arm saturation definitions and technical machinery required in LinTS’s regret proofs, potentially generalizing to other settings where all-context information is available but observations are sparse due to selection.

DRTS thus represents an overview of doubly robust imputation, dimension-free probabilistic analysis, and Bayesian exploration, yielding both theoretical and practical advancements for contextual bandit problems with linear structure.