Bandit Mirror Descent Overview

• Bandit mirror descent is a framework for online optimization that uses Bregman divergence and partial-information loss estimators to adapt to different feedback scenarios.
• It underpins methods for adversarial bandits, delayed feedback, non-stationary convex optimization, and distributed zeroth-order optimization in practical applications.
• Its analysis links mirror-descent stability terms with information-ratio arguments to yield near-optimal regret bounds across a variety of online learning settings.

Bandit mirror descent is the application of mirror descent to online learning with partial, bandit, or zeroth-order feedback, where the learner updates a decision distribution or point using an estimated loss or gradient and a Bregman divergence rather than a Euclidean metric. In the literature it appears as online stochastic mirror descent for adversarial finite-armed bandits, as two-point or multi-point bandit mirror descent for bandit convex optimization and games, and as a modular wrapper for delayed, distributed, and inference-aware settings. Taken together, these works suggest that bandit mirror descent is best understood as a geometry-sensitive framework for partial-information online optimization rather than a single canonical algorithm (Lattimore et al., 2020, Huang et al., 2023, He et al., 6 Aug 2025).

1. Foundations and geometric interpretation

Mirror descent is specified by a strictly convex potential and its associated Bregman divergence. A standard form is

$$\theta_{t+1} = \arg\min_{\theta \in \Theta} \left\{ \langle \theta, \nabla f_t(\theta_t) \rangle + \frac{1}{\alpha_t} D_G(\theta, \theta_t) \right\},$$

with

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$

Raskutti and Mukherjee prove that mirror descent induced by Bregman divergences is equivalent to natural gradient descent on the dual Riemannian manifold induced by the convex conjugate, and conclude that mirror descent is the steepest descent direction on that dual manifold (Raskutti et al., 2013).

This geometric viewpoint matters directly in bandit settings because the mirror map determines both feasible exploration and the scale of stability terms in the regret analysis. The literature uses negative entropy for simplex-based bandits, Tsallis entropy for adversarial finite-armed bandits, log-barrier regularization when strict positivity and stability are required, Euclidean geometry in standard convex domains, entropy on the simplex, and $\|x\|_p^2/2$ on the cross-polytope (Zimmert et al., 2019, Halder et al., 10 Mar 2026, He et al., 6 Aug 2025). A common simplification is to identify bandit mirror descent with EXP3 alone; the literature does not support that identification, since EXP3 is only one stochastic mirror descent instance within a broader class (Halder et al., 10 Mar 2026).

2. Partial-information updates and estimator design

The canonical bandit mirror descent pattern replaces the true gradient or loss vector by an estimator constructed from the observed feedback. In the adversarial bandit setting, one form of the update is

$$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$

where $\hat\ell_t$ is an importance-weighted estimate derived from the played action and observed signal. The 2020 analysis of “exploration by optimisation” emphasizes that, to attain optimal rates, the exploration distribution and the loss estimator must be tightly coupled and are chosen by solving a convex optimization problem at each step (Lattimore et al., 2020).

For finite-armed bandits, importance weighting is the basic mechanism. In delayed multi-armed bandits, for example, the estimator may take the familiar form

$$\widehat{l}_{t,a} = \frac{\mathbb{I}[A_t=a]\,l_{t,a}}{x_{t,a}},$$

after which the mirror update is applied when feedback becomes available rather than necessarily when the action is played (Huang et al., 2023). In the Tsallis-entropy analysis of adversarial $k$-armed bandits, shifted estimators are used to reduce variance, and the worst-case variance is reported to decrease by a factor of $4$ (Zimmert et al., 2019).

In bandit convex optimization and distributed zeroth-order optimization, the estimator is built from function values rather than arm-wise losses. For two-point bandit feedback in distributed composite optimization, the estimator is

$$g_{i,t} = \frac{d}{2\delta}\left( \ell_{i,t}(x_{i,t} + \delta u_{i,t}) - \ell_{i,t}(x_{i,t} - \delta u_{i,t}) \right)u_{i,t},$$

which is unbiased for the gradient of a smoothed version of the loss (Yuan et al., 2020). In non-stationary bandit convex optimization, the proposed algorithm samples $s_t \sim \mathrm{Unif}(\partial \mathbb{B}_1^d)$, queries $$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$0, and uses

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$1

with the stated advantage that the variance depends only logarithmically on the dimension when using the $$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$2-sphere (He et al., 6 Aug 2025).

Game-theoretic variants use multi-point pseudo-gradient estimates. In merely coherent games, each player performs an optimistic mirror descent update and then forms a centered multi-point estimator

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$3

whose bias is $$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$4 and variance is $$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$5 (Huang et al., 2023).

3. Regret analysis, stability, and the information ratio

A central development in the modern theory is the formal connection between mirror-descent stability terms and information-theoretic information-ratio arguments. One form of the online stochastic mirror descent regret bound is

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$6

which yields

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$7

after optimizing the learning rate (Zimmert et al., 2019). The corresponding 2020 paper shows that bounds on the information ratio imply bounds on the mirror-descent stability term, thereby translating Bayesian-information arguments into explicit adversarial algorithms (Lattimore et al., 2020).

This connection produces sharp finite-armed guarantees. With a suitable Tsallis entropy mirror map and a shifted loss estimator, OSMD achieves

$$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$8

for $$D_G(\theta, \theta') = G(\theta) - G(\theta') - \langle \nabla G(\theta'), \theta - \theta' \rangle.$$9-armed adversarial bandits (Zimmert et al., 2019). The “exploration by optimisation” construction goes further and gives an efficient algorithm for adversarial bandits with

$\|x\|_p^2/2$0

matching exactly the best known information-theoretic upper bound in the finite-armed case (Lattimore et al., 2020). The same line of work also improves bounds for bandits with graph feedback and for online linear optimization on $\|x\|_p^2/2$1-balls (Zimmert et al., 2019).

A useful distinction follows from these analyses. In regret theory, “stability” refers to the per-round control of the mirror-descent variance term or its information-ratio analogue. In later inference-oriented work, the same word is used in a different sense, namely stability of the sampling frequencies under adaptive data collection. The two notions are related by regularization and averaging, but they are not identical (Zimmert et al., 2019, Halder et al., 10 Mar 2026).

4. Major regimes and representative guarantees

Bandit mirror descent now spans finite-armed adversarial learning, delayed feedback, distributed zeroth-order optimization, and non-stationary bandit convex optimization. The main guarantees reported in the supplied literature are summarized below.

Setting	Representative construction	Reported guarantee
$\|x\|_p^2/2$2-armed adversarial bandits	Tsallis-entropy OSMD; exploration by optimisation	$\|x\|_p^2/2$3; $\|x\|_p^2/2$4
Delayed adversarial MAB and linear bandits	Banker-OMD	$\|x\|_p^2/2$5; $\|x\|_p^2/2$6
Distributed composite optimization with bandit feedback	$\|x\|_p^2/2$7	$\|x\|_p^2/2$8 in the Euclidean case; $\|x\|_p^2/2$9 in $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$0 settings
Non-stationary two-point bandit convex optimization	BMD and PBMD	$$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$1 in Euclidean space; $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$2 on the simplex

These guarantees show that the framework is not confined to static regret on the simplex. In distributed online composite optimization, bandit mirror descent achieves the same average regularized regret order as the full-information algorithm while using only two function values per node per round (Yuan et al., 2020). In non-stationary bandit convex optimization, the parameter-free PBMD meta-algorithm runs an ensemble of BMD base learners over a geometric grid of step sizes and adapts to unknown path variation without tuning to $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$3 (He et al., 6 Aug 2025). This suggests that the main source of variation across applications is not the mirror-descent principle itself, but the estimator, geometry, and comparator class (Yuan et al., 2020, He et al., 6 Aug 2025, Zimmert et al., 2019, Lattimore et al., 2020, Huang et al., 2023).

5. Delays, regularization, and statistical inference

Delayed feedback historically disrupted the telescoping argument underlying ordinary mirror-descent analyses. Banker Online Mirror Descent addresses this by almost completely decoupling delay handling from task-specific OMD design. Its central device is a “banking” mechanism that tracks pending updates corresponding to actions whose feedback has not yet arrived; when feedback is revealed, the accumulated “debt” is applied to the underlying OMD update. In the detailed exposition, the new action is formed as a convex combination in the dual space of previously unlocked savings, optionally supplemented by an overdraft from a default distribution when savings are insufficient (Huang et al., 2023).

The resulting guarantees have the characteristic delayed form

$$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$4

where $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$5 is the horizon and $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$6 is the total feedback delay. The framework is reported to yield $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$7 regret for delayed scale-free adversarial MAB and the first delayed adversarial linear bandit algorithm with $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$8 regret; the detailed exposition states the linear-bandit bound as $$Q_{t+1} = \operatorname*{argmin}_{q \in \mathcal{A}} \left\{ \langle q, \hat\ell_t \rangle + \frac{1}{\eta} D(q, Q_t) \right\},$$9 (Huang et al., 2023). A related presentation emphasizes delayed adversarial MAB, delayed adversarial linear bandits, and a delayed best-of-both-worlds MAB setting, with nearly-optimal performance in all three (Huang et al., 2021).

Regularization has also been used to make bandit mirror descent statistically inferentially tractable under adaptive sampling. The 2026 paper on regularized stochastic mirror descent establishes a general criterion: if the average iterates converge in ratio to a non-random probability vector,

$\hat\ell_t$0

then the induced bandit algorithm is stable in the Lai–Wei sense (Halder et al., 10 Mar 2026). Using a log-barrier regularizer and Tsallis-family mirror maps, the authors prove that regularized-EXP3 satisfies this criterion, that Wald-type confidence intervals for linear functionals of the mean parameter achieve nominal coverage, and that the same algorithms attain minimax-optimal regret guarantees up to logarithmic factors (Halder et al., 10 Mar 2026). They further show robustness to corruption: a modified variant preserves asymptotic normality of empirical arm means even under $\hat\ell_t$1 adversarial corruptions, in contrast to UCB, which is said to suffer linear regret even under logarithmic levels of corruption (Halder et al., 10 Mar 2026).

6. Games, actual-play convergence, and broader roles

Bandit mirror descent is also used beyond single-agent regret minimization. In merely coherent games, optimistic mirror descent combined with multi-point pseudo-gradient estimates produces almost sure convergence of both the iterate sequence and the actual sequence of play to a critical point, provided the query radius $\hat\ell_t$2 and sample size $\hat\ell_t$3 are chosen so that $\hat\ell_t$4 and $\hat\ell_t$5. The paper explicitly stresses that this is achieved without extra Tikhonov regularization terms or additional norm conditions (Huang et al., 2023).

In atomic congestion games under partial information, players observe only the cost of the path they actually played. The proposed bandit mirror-descent family therefore uses episodes with a fixed mixed strategy inside each episode, empirical path-cost estimates,

$\hat\ell_t$6

and a mirror-descent update

$\hat\ell_t$7

The reported guarantee is that, after sufficiently many episodes and with high probability, the potential satisfies $\hat\ell_t$8, which yields an approximate Nash equilibrium and approximate social-cost guarantees (Chen et al., 2016).

A broader implication is that mirror descent with bandit feedback can also operate at the meta-algorithmic level. The Corral master algorithm combines multiple bandit algorithms using OMD with the log-barrier mirror map

$\hat\ell_t$9

together with importance weighting and base-wise adaptive learning rates. The stated purpose is to prevent starvation of base algorithms that initially look poor but later outperform the rest, and the regret bounds are designed to track the best base algorithm up to the master’s exploration cost (Agarwal et al., 2016). These results suggest that bandit mirror descent is not merely a direct optimization routine; it is also a general control mechanism for exploration, coordination, and robustness across heterogeneous partial-information learners.