A Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits

Abstract: We adapt the analysis of policy gradient for continuous time $k$-armed stochastic bandits by Lattimore (2026) to the standard discrete time setup. As in continuous time, we prove that with learning rate $η= O(Δ{\min}^2/(Δ{\max} \log(n)))$ the regret is $O(k \log(k) \log(n) / η)$ where $n$ is the horizon and $Δ{\min}$ and $Δ{\max}$ are the minimum and maximum gaps.

• The paper introduces a Lyapunov-based method to derive non-asymptotic, high-probability regret bounds for softmax policy gradient in stochastic bandits.
• It employs discrete-time martingale techniques and explicit logit potential functions to ensure robust policy concentration on optimal actions.
• The study underscores the critical role of learning rate selection and gap dependence, paving the way for adaptive algorithms to reduce regret.

Lyapunov Analysis of Softmax Policy Gradient for Stochastic Bandits

Problem Setting and Algorithmic Framework

The paper provides a technical analysis of the softmax policy gradient (PG) algorithm applied to stochastic $k$-armed bandits in discrete time. The agent operates over a finite horizon $n \geq k$, with reward vector $\mu \in [0,1]^k$. At each round $t$, the agent samples an action $A_t$ according to a softmax policy parameterized by logits $\theta_t$, observes a reward $Y_t$ with expected value $\mu_{A_t}$, and updates using the standard bandit-PG rule:

$$\theta_{t+1,a} = \theta_{t,a} + \eta (\mathbb{I}\{A_t = a\} - \pi_{t,a}) Y_t$$

where $\pi_{t,a}\propto \exp(\theta_{t,a})$ is the action probability under the current policy, and $n \geq k$0 is a fixed learning rate. The regret is defined as

$n \geq k$1

under the assumption (w.l.o.g.) that $n \geq k$2 with at least one suboptimal action, and $n \geq k$3.

Main Contribution: Lyapunov Regret Bound

The core result is a non-asymptotic high-probability regret bound for discrete-time policy gradient. Specifically, the analysis establishes that for sufficiently small learning rate $n \geq k$4, the expected regret satisfies

$n \geq k$5

where $n \geq k$6 denotes the relevant suboptimality gap(s). The proof adapts continuous-time Lyapunov techniques to the discrete case, constructing an explicit Lyapunov potential over the logit vector to control the evolution of the policy distribution and ensure concentration mass around the optimal arms.

Key Technical Lemmas

• Logit Conservation and Lower Bounds: The sum of logits is conserved; each logit is lower bounded with high probability, established via a martingale/supermartingale argument.
• Optimal vs. Suboptimal Logit Gap Stability: The logit gap $n \geq k$7 between optimal and suboptimal actions is shown to rarely fall below $n \geq k$8, providing robustness in policy concentration.
• Inverse Mass Control: On the “good” event where all logits are well-behaved, the Lyapunov potential tightly upper bounds $n \geq k$9, decreasing as the sum of optimal logits increases.

The Lyapunov argument uses a carefully designed function whose first and second derivatives admit tight control, making Taylor approximation and maximal inequalities effective for handling the discrete increments.

Discussion of Sharpness, Dependence, and Comparisons

A central claim is that the dependence on $\mu \in [0,1]^k$0 is unavoidable, up to logarithmic factors, unless the discrete setting is drastically more benign than continuous time, referenced via a lower bound established in the continuous analysis [L26cpg]. The quadratic gap dependence dominates; the additional logarithmic factor in $\mu \in [0,1]^k$1 or $\mu \in [0,1]^k$2 is likely an artifact of proof technique and may be removable with more refined control of the logit process.

Comparative discussion with prior works clarifies the novelty and limitations:

• The result strengthens prior claims by [mei2023stochastic] by clarifying rates and correcting previous oversights.
• In the special case of equal gaps, [baudry2025does] produces a slightly better bound for certain regimes, indicating that some remaining slack exists in the general upper bound.
• The main theorem contradicts claims in [baudry2025does] that asymptotically large $\mu \in [0,1]^k$3 with moderate learning rate must yield polynomial regret, resolving the tension by distinguishing between non-asymptotic guarantees and asymptotic lower bounds.

Implications and Prospects for Stochastic Bandit Policy Gradient

The analysis provides a rigorous foundation for fixed learning rate softmax PG in the stochastic bandit regime, clarifying both achievable regret and limitations. The Lyapunov potential constructed translates naturally to the discrete setting—suggesting that continuous-time techniques can be ported with careful discretization and concentration control.

However, the proof highlights sensitivity to the learning rate. Further progress may hinge on analyzing adaptive or epoch-wise increasing learning rates, as the main technical barrier is localized to worst-case "bad" events in early rounds. Understanding the typical behavior of the logit vector and policy distribution under high-probability events could facilitate more aggressive learning schedules, potentially lowering regret bounds and relaxing dependence on unknown gaps.

From an algorithmic perspective, the result underscores the robustness but also the conservatism of vanilla PG for exploration in simple stochastic bandits relative to UCB-type or gap-based exploration algorithms. Sharp gap dependence remains the key limitation to optimality.

Conclusion

This paper develops a discrete-time Lyapunov analysis for softmax policy gradient in stochastic bandits, yielding an explicit regret upper bound with concrete dependence on the problem parameters and the learning rate (2603.26547). The proof technique demonstrates that the regret and the necessary learning rate are controlled under a generic gap regime, affirming theoretical guarantees previously available only in continuous-time or under restrictive assumptions. Open questions concern removal of logarithmic factors, characterization of optimal adaptive schedules, and extension to more general settings such as contextual bandits or MDPs.