Softmax gradient policy for variance minimization and risk-averse multi armed bandits

Abstract: Algorithms for the Multi-Armed Bandit (MAB) problem play a central role in sequential decision-making and have been extensively explored both theoretically and numerically. While most classical approaches aim to identify the arm with the highest expected reward, we focus on a risk-aware setting where the goal is to select the arm with the lowest variance, favoring stability over potentially high but uncertain returns. To model the decision process, we consider a softmax parameterization of the policy; we propose a new algorithm to select the minimal variance (or minimal risk) arm and prove its convergence under natural conditions. The algorithm constructs an unbiased estimate of the objective by using two independent draws from the current's arm distribution. We provide numerical experiments that illustrate the practical behavior of these algorithms and offer guidance on implementation choices. The setting also covers general risk-aware problems where there is a trade-off between maximizing the average reward and minimizing its variance.

• The paper proposes a softmax gradient policy that minimizes reward variance for risk-averse decision-making in multi-armed bandit problems.
• It leverages paired sampling to construct an unbiased variance estimator and employs a variant of the REINFORCE algorithm for effective optimization.
• Empirical evaluations demonstrate rapid regret minimization and reliable optimal arm identification across various synthetic and challenging risk scenarios.

Softmax Gradient Policy for Variance Minimization and Risk-Averse Multi-Armed Bandits

Introduction and Motivation

The study addresses the design and theoretical analysis of algorithms for Multi-Armed Bandit (MAB) problems under a risk-aware paradigm. Classical MAB algorithms target maximal expected reward, yet many domains—including finance, medicine, and safety-critical systems—require that decision-makers control not only expected outcomes but also variability. The work departs from mean-centric bandits and investigates policies that aim to minimize reward variance, thus ensuring outcome stability. This shift aligns with broader trends in distributional reinforcement learning, where the full distributional properties of returns are considered.

Risk-Aware Bandit Framework and Algorithmic Innovations

The approach is built on a softmax parameterization of the MAB policy, following the tradition of policy gradient methods. The policy distribution $\Pi_H$ is parameterized via a preference vector $H$, and the learning dynamics are governed by a variant of the REINFORCE gradient estimator customized for the variance-minimization objective.

Critically, the variance estimate used for gradient computation is constructed via paired sampling: for a selected arm at each timestep, two independent rewards are obtained. The composite reward is then defined as $\frac{1}{2}(R_t - R_t')^2$, providing an unbiased estimator of variance without separately estimating means. This approach diverges from standard UCB or LCB bandits, which typically maintain distinct estimates of means and variances over time. The resulting policy gradient update is: $H_{t+1}(a) = H_t(a) - \rho_t g_t(a)$ where

$$g_t(a) = [R_t - \bar{R}_t] (1_{a=A_t} - \Pi_{H_t}(a))$$

and $\bar{R}_t$ is a moving baseline which can be updated incrementally.

This method extends to more general risk-aware objectives by introducing linear weights for mean and variance, resulting in a target function: $$J_r(\Pi) = \lambda_\sigma V_{A \sim \Pi}[R(A)] + \lambda_\mu \mathbb{E}_{A \sim \Pi}[R(A)]$$ with corresponding adjustments to the composite reward in the algorithm.

Theoretical Analysis

The convergence of the proposed algorithms is established using stochastic approximation theory and recent results on softmax policy gradients. The update rule produces an unbiased estimate of the gradient of the risk function under standard filtration arguments. Under mild regularity assumptions—namely, distinct objective values across arms and bounded rewards—the policy is proven to almost surely converge to a Dirac mass on the optimal (minimum variance or optimal trade-off) arm, even with constant learning rates.

This strong almost-sure convergence is notable given the non-convex nature of the optimization landscape induced by the softmax parameterization. The proof leverages the global convergence properties of stochastic policy gradient methods as developed in [mei2023stochastic].

Numerical Evaluation

Empirical results on synthetic bandit problems—both well separated and nearly degenerate scenarios—demonstrate the practical efficiency and robustness of the approach. For a two-arm toy problem with variances 1 and 2, convergence of regret to zero and identification of the optimal arm at near-100% frequency are observed.

Figure 1: The toy example for $k=2$ arms, illustrating average regret (left) and frequency of optimal arm selection (right), plus 95% confidence intervals over 1000 runs.

In a ten-arm extension, where nine arms have identical variances and the last is optimal, the method maintains rapid regret minimization and high optimal arm selection rates.

Figure 2: The toy example for $k=10$ arms, showing average regret (left) and optimal arm frequency (right) with similar convergence properties.

For more challenging regimes with random means and narrowly separated variances, the algorithm continues to minimize regret effectively even when optimal arm frequencies cannot realistically reach 100% due to statistical indistinguishability.

Figure 3: Average regret in a difficult $k=10$-arm scenario with randomly assigned means and variances over 2000 steps and 1000 trials.

Figure 4: Average frequency of optimal arm selections in the same $H$0-arm regime, reflecting practical identification limits imposed by near-equal variances.

Implications and Prospects

The policy gradient framework promoted here offers several appealing theoretical and practical features:

• No need for explicit mean/variance tracking: Estimation is performed online via paired samples, reducing estimator bias and potential complications from simultaneous mean-variance tracking.
• Provably convergent algorithms: Under reasonable regularity and practical boundedness, the algorithm is shown to converge to deterministic, optimal policies.
• Extensible to general risk objectives: The methodology supports composite risk functionals, making it broadly applicable to other notions of risk such as mean-variance tradeoffs, CVaR, or custom quantile objectives.

In practice, this approach has immediate utility for applications where risk minimization is essential—for example, in recommendation systems prioritizing consistency, portfolio optimization under volatility constraints, or healthcare interventions where outcome robustness is paramount.

Future developments may extend these methods to contextual bandits, non-stationary settings, or general reinforcement learning scenarios with temporal dependencies. There is also scope to incorporate advanced variance reduction techniques for sample-efficient gradient estimation and to theoretically analyze convergence under heavy-tailed reward distributions.

Conclusion

This work advances the analysis and practical deployment of risk-aware MABs by introducing a gradient-based, softmax-parameterized framework for variance minimization. The approach distinguishes itself by unbiased on-the-fly variance estimation, global convergence guarantees, and demonstrable empirical efficacy across a spectrum of regimes. It establishes a solid foundation for further theoretical exploration and for practical algorithms in risk-sensitive sequential decision-making contexts.

---

Reference: "Softmax gradient policy for variance minimization and risk-averse multi armed bandits" (2604.00241)