Mellowmax: A Smooth, Stable RL Operator

• Mellowmax is a parametric, differentiable, non-expansive operator that aggregates action-value vectors to balance between the hard max and averaged values in RL.
• It ensures smoothness (C² differentiability) and non-expansion, enabling stable and theoretically robust Bellman backups in both single-agent and multi-agent settings.
• Extensions like Soft Mellowmax further reduce overestimation bias and enhance convergence through techniques such as Anderson mixing and regularization.

Mellowmax is a parametric, differentiable, non-expansive operator for aggregating action-value vectors, developed to address stability, smoothness, and overestimation issues in value-based reinforcement learning. As an alternative to the non-differentiable hard maximum and the potentially expansive Boltzmann softmax, Mellowmax and its extensions, such as Soft Mellowmax and maximum-entropy variants, have become integral components in modern deep and multi-agent reinforcement learning frameworks.

1. Mathematical Definition and Formal Properties

Given an action-value vector $Q(s, \cdot) \in \mathbb{R}^{|\mathcal{A}|}$ at state $s$ and inverse-temperature parameter $\omega > 0$, the Mellowmax operator is defined by

$$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$

This operator interpolates between the hard maximum and the average:

• As $\omega \to \infty$, $\mathrm{mm}_\omega(Q) \to \max_{a} Q(s,a)$
• As $\omega \to 0$, $$\mathrm{mm}_\omega(Q) \to \frac{1}{|\mathcal{A}|} \sum_{a} Q(s,a)$$

A commonly used variant is Soft Mellowmax (SM2) which introduces a secondary parameter $\alpha$:

$$\pi_{\alpha}(a|s) = \frac{\exp(\alpha Q(s,a))}{\sum_{b} \exp(\alpha Q(s,b))}$$

$s$0

Key mathematical properties of Mellowmax:

• Twice continuously differentiable ($s$1), with bounded derivatives on bounded domains, enabling gradient-based optimization and stability analysis (Sun et al., 2021).
• Non-expansive in the $s$2 norm: for any $s$3,

$s$4

This property ensures that the corresponding Bellman operator is a $s$5-contraction as required for standard value-iteration convergence (Gan et al., 2020).

• The Bellman-Mellowmax operator is given by

$s$6

2. Theoretical Motivation and Analysis

The hard $s$7 operator is non-expansive but non-differentiable, which can preclude the use of smooth fixed-point theorems and create instability in deep RL with function approximation (Sun et al., 2021). The Boltzmann softmax is differentiable but not always non-expansive, potentially introducing contraction violations and bias (Gan et al., 2020).

Mellowmax is designed to combine the strengths of both: it is smooth (enabling theoretical analyses such as Taylor expansions for convergence rates) and non-expansive (preserving contraction properties essential for fixed-point and stability guarantees under stochastic approximation with function approximation) (Gan et al., 2020, Sun et al., 2021). Theoretical analyses show that for policy-iteration schemes augmented with Anderson mixing, substituting the Bellman optimality operator with Mellowmax yields enhanced contraction radii and more robust convergence bounds (Sun et al., 2021).

For Soft Mellowmax, additional results include an explicit performance-gap bound between the fixed point of the SM2 operator and the true optimal $s$8, and a strictly smaller overestimation bias compared to the classical max operator (Gan et al., 2020).

3. Integration in Value-Based Algorithms and Multi-Agent Settings

Single-Agent Value-Based Methods

The Bellman optimality backup

$s$9

can be replaced by

$\omega > 0$0

or by the Soft Mellowmax backup,

$\omega > 0$1

The temporal-difference loss is minimized with respect to current parameters, using the differentiability of the backup (Gan et al., 2020, Zhang et al., 2022).

Anderson Mixing and Acceleration

In policy iteration and fixed-point value approximation, Mellowmax is used in the target operator $\omega > 0$2 within damped Anderson mixing updates. This preserves both the contraction property and smoothness necessary for rigorous convergence acceleration results. In the stabilized Anderson acceleration scheme, experimental results show consistent improvements in learning speed and final performance, especially when Mellowmax is used together with regularization (Sun et al., 2021).

Multi-Agent RL (MARL)

In value-decomposition frameworks such as QMIX, Mellowmax or Soft Mellowmax is applied to local action spaces for each agent, avoiding exponential computational cost. The hybrid TD-$\omega > 0$3 update in MAST-QMIX uses agent-wise Soft Mellowmax for target formation, ensuring stable value propagation in dynamic sparse training regimes without requiring a max over the joint action space (Hu et al., 2024).

Representative pseudocode pattern (Editor’s term):

$\omega \to \infty$4 (Hu et al., 2024)

4. Overestimation Bias, Stability, and Performance Bounds

The standard max-operator in the Bellman backup is known to induce positive bias in $\omega > 0$4-learning due to noisy maximization over finite samples. Mellowmax, being a soft maximum, reduces this bias: its output is always within $\omega > 0$5, strictly capping overestimation. For Soft Mellowmax, the bias is further reduced, with explicit bounds proven under i.i.d.\ error assumptions in the $\omega > 0$6-values (Gan et al., 2020, Zhang et al., 2022).

The contraction and smoothness of Mellowmax and SM2 enable robust convergence of empirical RL algorithms, even under function approximation, and with recursive least-squares or Anderson acceleration schemes (Sun et al., 2021, Zhang et al., 2022). In multi-agent contexts, applying agent-local SM2 instead of a joint max prevents bias accumulation as the number of agents grows, which would otherwise scale linearly in vanilla QMIX (Gan et al., 2020).

5. Practical Considerations and Empirical Findings

Mellowmax and its variants require selection of the inverse-temperature parameter $\omega > 0$7. Typical practical choices, as reported in the literature, are:

• For classic control / gridworlds: $\omega > 0$8.
• For Atari: $\omega > 0$9, with $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$0 to $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$1 being effective trade-offs between bias and sharpness.

SM2 introduces an additional $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$2 parameter, commonly set such that $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$3 is not too large relative to $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$4, balancing weighting concentration and smoothness (Gan et al., 2020, Hu et al., 2024).

Empirical studies across DQN, DDQN, and episodic control variants on Atari, PLE, MinAtar, and multi-agent StarCraft benchmarks demonstrate that Mellowmax-based and SM2-based backups yield:

• Stronger stability of learning
• Faster convergence (especially when combined with Anderson or regularization)
• Consistently higher or more reliable final performance versus hard max and softmax baselines
• Reduction in overestimation bias and more conservative, accurate $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$5-estimates
• Robustness against hyperparameter sensitivity and instability in sparse or high-dimensional action spaces (Sun et al., 2021, Gan et al., 2020, Hu et al., 2024, Sarrico et al., 2019)

6. Limitations and Extensions

While Mellowmax provides a principled trade-off and stability benefits, several limitations and proposed solutions have been documented:

• The fixed-point performance bound (distance to $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$6) under Mellowmax contraction is generally unknown; only in the SM2 extension is an explicit bound derived (Gan et al., 2020).
• The $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$7 parameter must generally be tuned per domain; large $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$8 recovers greedy max (risking overestimation), small $$\mathrm{mm}_\omega(Q(s,\cdot)) = \frac{1}{\omega} \log\left(\frac{1}{|\mathcal{A}|} \sum_{a \in \mathcal{A}} e^{\omega Q(s,a)}\right)$$9 oversmooths (diluting high-value actions).
• Uniform weighting in Mellowmax can depress backups in large action spaces or sparse-reward problems. SM2 and maximum-entropy variants address this by reweighting via a soft-policy (Gan et al., 2020, Hu et al., 2024, Sarrico et al., 2019).
• In episodic control, root-finding for a state-dependent temperature in MEMEC may be computationally significant, mitigated by warm-starting or limiting solver steps (Sarrico et al., 2019).

Enhanced operators such as SM2 or hybrid dynamic-target techniques in sparse MARL (MAST-QMIX) further address these issues, achieving rigorous bias control and enabling robust, sample-efficient learning in high-dimensional or distributed RL settings (Gan et al., 2020, Hu et al., 2024).

7. Table: Summary of Core Properties by Operator

Operator	Smoothness	Non-expansive	Performance Bound
max	No	Yes	No
softmax	Yes	No (Lipschitz $\omega \to \infty$0)	No
Mellowmax ($\omega \to \infty$1)	Yes ($\omega \to \infty$2)	Yes	No
Soft Mellowmax (SM2)	Yes ($\omega \to \infty$3)	Yes	Yes (explicit bound)

Both Mellowmax and Soft Mellowmax can be directly plugged into single-agent and multi-agent value-based RL, deep Q-learning, and episodic control for improved stability, sample efficiency, and theoretical guarantees on learning dynamics (Sun et al., 2021, Gan et al., 2020, Hu et al., 2024, Sarrico et al., 2019, Zhang et al., 2022).