Unified Framework of Distributional Regret in Multi-Armed Bandits and Reinforcement Learning

Abstract: We study the distribution of regret in stochastic multi-armed bandits and episodic reinforcement learning through a unified framework. We formalize a distributional regret bound as a probabilistic guarantee that holds uniformly over all confidence levels $δ\in (0,1]$, thereby characterizing the regret distribution across the full range of $δ$. We present a simple UCBVI-style algorithm with exploration bonus $\min{c_{1,k}/N, c_{2,k}/\sqrt{N}}$, where $N$ denotes the visit count and $(c_{1,k},c_{2,k})$ are user-specified parameters. For arbitrary parameter sequences, we derive general gap-independent and gap-dependent distributional regret bounds, yielding a principled characterization of how the parameters control the trade-off between expected performance, tail risk, and instance-dependent behavior. In particular, our bounds achieve optimal trade-offs between expected and distributional regret in both minimax and instance-dependent regimes. As a special case, for multi-armed bandits with $A$ arms and horizon $T$, we obtain a distributional regret bound of order $\mathcal{O}(\sqrt{AT}\log(1/δ))$, confirming the conjecture of Lattimore & Szepesvári (2020, Section 17.1) for the first time.

• The paper introduces a unified framework that defines distributional regret as a deterministic upper bound parameterized by confidence levels, matching minimax lower bounds.
• It proposes a UCBVI-style algorithm with flexible exploration bonuses that generalize both gap-independent and gap-dependent regimes, sharpening instance-dependent bounds.
• The framework integrates theoretical advances with practical implications, providing principled tools for risk-sensitive online learning in both tabular bandits and episodic RL.

Unified Distributional Regret Framework in MAB and RL: An Expert Summary

Overview

The paper "Unified Framework of Distributional Regret in Multi-Armed Bandits and Reinforcement Learning" (2605.05102) introduces a rigorous framework for analyzing the distributional properties of regret in both stochastic multi-armed bandit (MAB) and episodic reinforcement learning (RL) models. It formalizes the notion of distributional regret as a deterministic upper bound parameterized by confidence level $\delta$, and proposes a UCBVI-style algorithm with parameterized exploration bonuses. The theoretical guarantees provided match minimax lower bounds for worst-case expected and distributional regret in MAB, resolving a longstanding conjecture in the literature, and further generalize to episodic RL.

Technical Contributions

Distributional Regret Bounds

The paper defines the distributional regret bound $\mathcal{R}(K, \delta)$ as the smallest deterministic value such that regret exceeds it with probability at most $\delta$, for all $\delta \in (0, 1]$. This distributional perspective subsumes expected and fixed-confidence bounds, allowing for a principled characterization of the entire risk profile, including tail behavior.

Unified Algorithmic Framework

The algorithm ("EQO+") generalizes UCBVI-type methods to allow for a flexible exploration bonus $$\min\left\{\frac{c_{1,k}}{N}, \frac{c_{2,k}}{\sqrt{N}}\right\}$$, with $N$ denoting visit counts and $c_{1,k}, c_{2,k}$ user-specified. The bonus design captures both gap-independent (minimax) and gap-dependent (instance-dependent) regimes, supporting arbitrary sequences of exploration parameters. The special case where $c_{1,k}$ alone is used recovers prior 'EQO' methods [lee2025minimax], while gap-dependent behavior is enabled via the square-root bonus term.

Generalized Regularity Assumption

A regularity condition is introduced, bounding the sub-exponential norm of reward-plus-optimal-value random variables. This simultaneously subsumes sub-Gaussian noise (bandit literature) and bounded reward (RL literature), establishing a unified technical foundation for regret analysis in both settings.

Optimality in Multi-Armed Bandits

For multi-armed bandits with $A$ arms and horizon $T$:

• The distributional regret bound is shown to be $\mathcal{R}(K, \delta)$0.
• The expected regret is $\mathcal{R}(K, \delta)$1, matching minimax lower bounds up to constants. This confirms a major conjecture on the tightness of regret bounds for stochastic bandits [lattimore2020bandit, Section~17.1].

General Distributional Regret Bounds in RL

For tabular episodic MDPs:

• Both gap-independent and gap-dependent bounds are derived for arbitrary parameter choices.
• The trade-offs between expected regret, tail risk, and instance-dependent performance are precisely quantified.
• The transformation from high-probability bounds to expectation (via integration over $\mathcal{R}(K, \delta)$2) avoids unnecessary logarithmic factors, achieving sharp constants.

Comparative Analysis

This framework strictly generalizes prior UCB-type distributional approaches:

• The bonus parameters are fully tunable, unlike prior works where only specific choices were analyzed [simchi2023regret, simchi2025simple].
• The distributional regret bounds in MAB improve over existing results by removing extraneous logarithmic factors and sharpening gap-dependent terms.
• The RL results provide the first unified analysis spanning both bandit and episodic RL, with bounds matching current lower bounds in both regimes up to log factors.

Strong Numerical Results and Contradictory Claims

• For MAB, $\mathcal{R}(K, \delta)$3 distributional regret and $\mathcal{R}(K, \delta)$4 expectation rates are established as optimal, confirming a previously unresolved conjecture.
• The paper asserts that previous algorithms fail to achieve these tight rates due to additional logarithmic or instance-dependent terms.
• Instance-dependent regret bounds in RL are improved, notably reducing previous $\mathcal{R}(K, \delta)$5 scaling to $\mathcal{R}(K, \delta)$6, a substantial improvement in the gap-dependent regime.

Practical and Theoretical Implications

Practical

• The tunable exploration bonus allows practitioners to balance expected and tail risk regret for specific applications, with principled control over statistical risk aversion.
• The integration technique for expectation enables practitioners to derive sharp expected regret bounds from distributional guarantees, removing log-overhead from overly conservative union bounds.

Theoretical

• The unified framework reconciles bandit and RL analyses, providing a blueprint for extensions to non-tabular settings and function approximation.
• The generalized regularity assumption opens the door for analysis under broader noise classes (sub-exponential, Poisson, etc.), potentially extending to semi-parametric and nonparametric RL.

Future Developments

• Extension of distributional regret analysis to RL with function approximation (linear, kernel, deep) is a natural future direction.
• Joint analysis of exploration-exploitation trade-offs under non-standard noise models in large-scale MDPs could leverage the unified framework.
• Lower bounds for distributional regret in RL beyond tabular models remain open, representing opportunities for further sharpness.

Conclusion

This paper establishes a unified and rigorous framework for distributional regret analysis in both MAB and RL, with algorithmic and theoretical results matching (or improving upon) previous lower bounds and conjectures. The approach provides practitioners and theorists with principled tools to analyze and tune regret distributions, with implications for advanced online learning, statistical risk management, and large-scale RL. The extension to more general settings and further sharpness in the tabular RL regime are clear directions for future work.