Smoothed Bellman Operator

• Smoothed Bellman operator is a family of modified Bellman mappings that replace the hard max with smooth approximations (e.g., softmax, log-sum-exp) to improve differentiability and stability.
• They reduce overestimation bias and gradient variance while sometimes preserving contraction, thereby benefiting learning dynamics in reinforcement learning and control applications.
• These operators underpin advanced methods such as soft residual minimization, SBEED, and smoothed policy iteration, balancing bias with improved optimization behavior.

Searching arXiv for relevant papers on smoothed Bellman operators and closely related formulations. The smoothed Bellman operator denotes a family of Bellman-type mappings in which the hard maximization or other nonsmooth components of dynamic programming are replaced, regularized, averaged, or otherwise softened to improve differentiability, stability, or analytical tractability. In reinforcement learning and control, the term covers several distinct constructions: softmax-weighted Bellman backups for $Q$-iteration, entropy-regularized log-sum-exp operators, smoothed Bellman residual objectives under weighted $L_p$ geometries, consistency operators used in SBEED, temporally averaged smoothed policy-improvement operators, annealed interpolations between optimality and policy-evaluation backups, and semigroup-smoothed Bellman mappings for infinite-dimensional Hamilton–Jacobi–Bellman equations (Song et al., 2018, Yang et al., 8 Apr 2026, Touati et al., 2020, Nanda et al., 11 May 2026, Omura et al., 6 Jun 2025, Bolli et al., 29 Oct 2025). Across these settings, smoothing modifies the operator’s geometry and bias properties, typically trading exact Bellman optimality for regularity, reduced overestimation, improved optimization behavior, or stronger existence theory.

1. Canonical operator forms

In a finite discounted Markov decision process with state space $\mathcal{S}$, action space $\mathcal{A}$, reward $R$, transition kernel $P$, and discount $\gamma\in(0,1)$, the standard Bellman optimality operator on action values is

$$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$

This operator is the unsmoothed reference point in several later constructions (Song et al., 2018).

A widely studied smoothed variant is the softmax-weighted Bellman operator

$$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$

where $\beta$ is an inverse temperature parameter. As $L_p$0, the inner softmax-weighted average approaches the max operator (Song et al., 2018). The same work also contrasts this operator with mellowmax,

$L_p$1

with $L_p$2, whose Bellman backup replaces the inner max by $L_p$3 (Song et al., 2018).

A second canonical smoothing replaces max by log-sum-exp directly. In the $L_p$4-space formulation of soft Bellman residual minimization,

$L_p$5

with temperature $L_p$6 (Yang et al., 8 Apr 2026). In value space, the analogous entropy-smoothed operator is

$L_p$7

which also appears in the analysis of entropy-regularized control and SBEED (Touati et al., 2020).

Smoothing can also appear in a policy-conditioned consistency form rather than an optimality form. SBEED employs

$L_p$8

with $L_p$9 (Touati et al., 2020). Here the smoothing is induced by entropy regularization through the log-policy term rather than by a direct replacement of $\mathcal{S}$0.

A different generalization is the smoothed Bellman operator in doubly smoothed policy iteration: $\mathcal{S}$1 where $\mathcal{S}$2 is a bounded, non-negative, concave statewise regularizer and $\mathcal{S}$3 is the smoothing level (Nanda et al., 11 May 2026). This encompasses Shannon entropy, Tsallis entropy, shifted negative squared norm, and the zero function.

In continuous-action online reinforcement learning, smoothing may be time dependent. One construction interpolates between the Bellman optimality operator and the Bellman operator through

$\mathcal{S}$4

with $\mathcal{S}$5 annealed during training (Omura et al., 6 Jun 2025). The continuous-action implementation uses expectile regression rather than an explicit max.

Outside finite-dimensional RL, the phrase also describes a semigroup-smoothed Bellman mapping for stationary HJB equations: $\mathcal{S}$6 where $\mathcal{S}$7 is the Ornstein–Uhlenbeck semigroup (Bolli et al., 29 Oct 2025). Here smoothing is induced by the semigroup rather than by entropy or softmax.

2. Approximation to Bellman optimality and induced bias

For softmax-weighted backups, the central question is how closely the smoothed operator tracks the Bellman optimality operator. Under bounded rewards and finite action spaces, the difference between $\mathcal{S}$8 and the softmax-weighted scalar

$\mathcal{S}$9

admits explicit upper and lower bounds controlled by the action-gap envelope $\mathcal{A}$0 (Song et al., 2018). The same work proves that the iterates $\mathcal{A}$1 converge to the standard Bellman iterates $\mathcal{A}$2 exponentially fast in $\mathcal{A}$3, with the rate determined by positive action gaps (Song et al., 2018).

The asymptotic deviation from optimal action values is also bounded. Specifically, if $\mathcal{A}$4 denotes the optimal $\mathcal{A}$5-function, then the softmax iterates remain within a band below $\mathcal{A}$6 whose width shrinks with increasing $\mathcal{A}$7 (Song et al., 2018). The paper emphasizes that this is a suboptimality envelope rather than a fixed-point distance because $\mathcal{A}$8 is not generally a contraction and may not have a unique fixed point (Song et al., 2018).

Entropy-smoothed log-sum-exp operators admit a different limiting interpretation. In the soft residual minimization framework, $\mathcal{A}$9 as $R$0 (Yang et al., 8 Apr 2026). In SBEED, the entropy-smoothed operator satisfies the pointwise relation

$R$1

which yields the smoothing bias bound

$R$2

and a performance bias scaling as $R$3 relative to unregularized optimal control (Touati et al., 2020).

These bias statements show that smoothing typically introduces a controlled downward or regularization-induced deviation from exact Bellman optimality. A plausible implication is that the practical value of smoothing often depends less on recovering the exact optimality equation and more on how the altered backup interacts with approximation error, optimization, and statistical noise.

3. Contraction, monotonicity, and operator geometry

A major source of divergence between smoothed Bellman operators is whether smoothing preserves contraction. The softmax-weighted operator of the form used in deep $R$4-learning is in general not a contraction; the cited counterexample traces to Littman, 1996 (Song et al., 2018). By contrast, the mellowmax operator is a contraction, and the log-sum-exp operator $R$5 is a $R$6-contraction in $R$7 with unique fixed point $R$8 (Song et al., 2018, Yang et al., 8 Apr 2026).

The distinction between weighted-average softmax and log-sum-exp is therefore structural rather than terminological. The former modifies the target by replacing $R$9 with a softmax-weighted expectation of $P$0 values; the latter replaces it by a convex-conjugate log-sum-exp value. Both are “soft,” but only the second inherits the familiar contraction argument in the sup norm in the cited analyses (Song et al., 2018, Yang et al., 8 Apr 2026).

The weighted $P$1 analysis of soft Bellman residual minimization extends the geometric viewpoint. For weights $P$2 summing to one, the weighted norm

$P$3

induces an effective Lipschitz factor

$P$4

Then

$P$5

Whenever $P$6, which holds for sufficiently large $P$7, the soft operator becomes a contraction in $P$8 (Yang et al., 8 Apr 2026). As $P$9, $\gamma\in(0,1)$0, aligning the optimization geometry with the Bellman operator’s native $\gamma\in(0,1)$1 contraction (Yang et al., 8 Apr 2026).

Monotonicity is preserved more broadly. For the softmax-weighted scalar $\gamma\in(0,1)$2, the derivative with respect to $\gamma\in(0,1)$3 is nonnegative, so the softmax-weighted average increases monotonically with inverse temperature (Song et al., 2018). The log-sum-exp operator $\gamma\in(0,1)$4 is monotone in the sense that $\gamma\in(0,1)$5 implies $\gamma\in(0,1)$6 (Yang et al., 8 Apr 2026). The smoothed operators $\gamma\in(0,1)$7 and $\gamma\in(0,1)$8 in doubly smoothed policy iteration are likewise monotone and $\gamma\in(0,1)$9-contractive with factor $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$0 (Nanda et al., 11 May 2026).

This contrast gives rise to a common misconception: that all smoothed Bellman operators sacrifice contraction. The literature summarized here shows that this is false. Some smoothings break contraction, some preserve it exactly, and others recover it only in a suitable geometry or under sufficiently large $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$1 (Song et al., 2018, Yang et al., 8 Apr 2026, Nanda et al., 11 May 2026).

4. Overestimation control and stabilization in value-based learning

The most prominent practical motivation for smoothing in value-based RL is overestimation mitigation. Under the noise model used in the softmax Bellman analysis, where $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$2 and all actions have equal true value at state $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$3, the expected max noise is nonnegative: $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$4 For the softmax aggregation,

$$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$5

so the softmax backup induces less or equal overestimation than the max backup for any $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$6 (Song et al., 2018). The reduction is bounded using the same action-gap quantities that govern approximation quality (Song et al., 2018).

This analysis helps explain an empirical paradox reported in deep $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$7-learning: the softmax Bellman operator leads to sub-optimal $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$8-functions in the tabular sense, yet when combined with DQN it can produce superior policies and even outperform double $$T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').$$9-learning on the reported Atari suite (Song et al., 2018). The paper argues that the reduced overestimation error and reduced gradient norm and variance are plausible drivers of this phenomenon (Song et al., 2018).

The continuous-control annealing work presents a related but temporally structured argument. If $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$0 is perturbed by zero-mean noise $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$1, then Jensen’s inequality yields

$$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$2

formalizing the upward bias of max-like backups (Omura et al., 6 Jun 2025). The proposed remedy is not to fix a single smoothing level, but to begin training with a more max-like critic and gradually anneal toward policy-evaluation-style backups. In practice this is implemented by expectile loss

$$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$3

where $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$4 induces a more upper-tail-focused, max-like critic and $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$5 recovers standard $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$6 regression (Omura et al., 6 Jun 2025).

The empirical findings are consistent with a two-phase interpretation: early overestimation can accelerate learning and correlate with higher policy entropy, whereas later reduction toward expectation backups lowers bias and improves stability (Omura et al., 6 Jun 2025). This suggests that “smoothed Bellman operator” may denote not only a static surrogate for $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$7, but also a training-time schedule through a family of operators.

5. Residual minimization, SBEED, and optimization-oriented smoothing

A separate lineage uses smoothing to make Bellman residual objectives differentiable and compatible with function approximation. In contraction-aligned soft Bellman residual minimization, linear approximations of the form $$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$8 are fitted by minimizing

$$T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),$$9

under a weighted $\beta$0 norm (Yang et al., 8 Apr 2026). For sufficiently large $\beta$1 such that $\beta$2, the paper proves sandwich bounds connecting the residual objective to the distance from the soft fixed point: $\beta$3 This yields quasi-optimality of the minimizer $\beta$4 relative to the best approximation to $\beta$5, with multiplicative gap

$\beta$6

which decreases monotonically with $\beta$7 and converges to $\beta$8 as $\beta$9 (Yang et al., 8 Apr 2026).

The same framework derives explicit gradients. With even integer $L_p$00,

$L_p$01

and because the soft operator is differentiable, its Jacobian can be expressed using the Boltzmann policy induced by $L_p$02: $L_p$03 so

$L_p$04

(Yang et al., 8 Apr 2026). Large-$L_p$05 instability is handled by residual normalization, which rescales the gradient without changing stationary points (Yang et al., 8 Apr 2026).

SBEED uses smoothing in a different way. Rather than minimizing a direct Bellman residual, it introduces a saddle-point objective to avoid double-sampling bias in the empirical smoothed Bellman error (Touati et al., 2020). With i.i.d. batch data and helper class $L_p$06, the objective is

$L_p$07

where

$L_p$08

and

$L_p$09

The helper function $L_p$10 estimates the consistency target and cancels the variance term caused by the inner expectation (Touati et al., 2020).

The sharp analysis of SBEED proves a suboptimality decomposition

$L_p$11

where $L_p$12 is a tight $L_p$13-type concentrability coefficient (Touati et al., 2020). The resulting guarantee has linear dependence on the horizon factor $L_p$14 and a fast realizable statistical rate $L_p$15, improving earlier SBEED analysis that yielded slower rates (Touati et al., 2020).

Taken together, these works position smoothing not merely as a surrogate for $L_p$16, but as a means of making Bellman objectives differentiable, variance-aware, and better matched to contraction geometry under approximation.

6. Smoothed policy iteration and natural policy gradient

In doubly smoothed policy iteration, smoothing acts both across policies and within the greedy step. The algorithm maintains an averaged action-value sequence

$L_p$17

and defines the next policy through the smoothed greedification condition

$L_p$18

or explicitly,

$L_p$19

(Nanda et al., 11 May 2026).

The smoothing is therefore double: temporal averaging in $L_p$20 and regularized greedy improvement via $L_p$21 (Nanda et al., 11 May 2026). The analysis relies only on monotonicity and contraction of the smoothed Bellman operators. It proves monotonic policy improvement,

$L_p$22

and a one-step recursion

$L_p$23

(Nanda et al., 11 May 2026). Under constant $L_p$24, the resulting convergence is geometric, with iteration complexity $L_p$25 for computing an $L_p$26-optimal policy (Nanda et al., 11 May 2026).

A key contribution of this framework is the exact equivalence between natural policy gradient and a specific doubly smoothed policy iteration instance. With $L_p$27 equal to Shannon entropy, $L_p$28, and

$L_p$29

the policy sequence generated by DSPI coincides with the sequence generated by tabular dual-averaged natural policy gradient (Nanda et al., 11 May 2026). This yields a Bellman-operator interpretation of natural policy gradient in which the smoothing is intrinsic to the mirror map and temporal averaging, not added as an extraneous regularizer.

The unregularized case $L_p$30 recovers dual-averaged policy iteration, for which the paper proves finite termination under deterministic greedy updates and fixed tie breaking (Nanda et al., 11 May 2026). This places smoothed Bellman operators on a continuum between classical policy iteration, mirror-descent-style methods, and natural policy gradient.

7. Infinite-dimensional HJB formulations and broader interpretation

In infinite-dimensional control, smoothing addresses a different obstruction: lack of regularity in the HJB equation rather than nondifferentiability of $L_p$31. For the stationary HJB

$L_p$32

classical mild solution theory may fail because of lack of strong Feller smoothing, unbounded control operators, and state-dependent costs (Bolli et al., 29 Oct 2025). The cited work addresses this by lifting the problem to a trajectory space and exploiting partial smoothing of the Ornstein–Uhlenbeck semigroup.

The Bellman nonlinearity is

$L_p$33

and the mild solution is represented as

$L_p$34

This integral resolvent form is the smoothed Bellman operator in that setting (Bolli et al., 29 Oct 2025). The semigroup $L_p$35 regularizes only along appropriate lifted and $L_p$36-directional coordinates, quantified through operators $L_p$37 satisfying

$L_p$38

(Bolli et al., 29 Oct 2025). With sufficiently large discount $L_p$39, the corresponding fixed-point map becomes a contraction on the lifted function class, yielding existence and uniqueness of a regular mild solution, together with verification and optimal feedback synthesis (Bolli et al., 29 Oct 2025).

This infinite-dimensional usage extends the notion of smoothing beyond regularized maximization. The operator is “smoothed” because the Bellman nonlinearity is precomposed with a semigroup and integrated against the discount kernel, thereby gaining differentiability properties otherwise unavailable (Bolli et al., 29 Oct 2025). A plausible implication is that the unifying idea behind smoothed Bellman operators is not a particular formula such as log-sum-exp, but the replacement of a raw Bellman map by one with improved analytical regularity.

8. Comparative perspective and recurring trade-offs

The literature distinguishes several recurrent trade-offs.

Construction	Main benefit	Main limitation
Softmax-weighted backup	Reduces overestimation and gradient variance; directly defines a policy	Not Bellman-optimal; not generally a contraction (Song et al., 2018)
Log-sum-exp / entropy smoothing	Differentiable; $L_p$40-contraction in $L_p$41; unique soft fixed point	Introduces regularization bias relative to hard optimality (Yang et al., 8 Apr 2026, Touati et al., 2020)
Mellowmax	Contraction; reduces overestimation	Does not directly define a policy without extra steps (Song et al., 2018)
Weighted $L_p$42 soft residual minimization	Aligns residual objective with contraction geometry as $L_p$43 increases	Requires tuning $L_p$44, weights, and temperature; large $L_p$45 may cause large gradients (Yang et al., 8 Apr 2026)
SBEED consistency operator	Avoids double-sampling bias; sharp batch guarantees	Analysis depends on finite classes and concentrability (Touati et al., 2020)
DSPI smoothed operators	Distribution-free geometric convergence; exact NPG equivalence	Model-based evaluation in the presented analysis (Nanda et al., 11 May 2026)
Annealed optimality-to-evaluation smoothing	Early acceleration with later bias reduction in continuous control	Performance depends on annealing schedule (Omura et al., 6 Jun 2025)
Semigroup-smoothed HJB mapping	Existence, uniqueness, and feedback synthesis in infinite dimensions	Requires lifting machinery and partial smoothing hypotheses (Bolli et al., 29 Oct 2025)

Several misconceptions recur in discussions of the topic. One is that smoothing always means entropy regularization. The surveyed works show instead that smoothing may arise from softmax weighting, log-sum-exp conjugacy, regularized greedification, residual geometry, expectile annealing, or semigroup resolvents (Song et al., 2018, Yang et al., 8 Apr 2026, Nanda et al., 11 May 2026, Omura et al., 6 Jun 2025, Bolli et al., 29 Oct 2025). Another is that smoothing is synonymous with improved contraction; this is true for some operators but false for the softmax-weighted Bellman operator (Song et al., 2018). A third is that smoothing necessarily degrades performance because it induces suboptimal value functions. The Atari and continuous-control results indicate that, in the presence of function approximation and optimization noise, a biased operator may nonetheless improve learned policies by controlling overestimation and stabilizing updates (Song et al., 2018, Omura et al., 6 Jun 2025).

Open questions remain. The annealed continuous-control framework notes that convergence properties of time-varying smoothed operators under function approximation remain open (Omura et al., 6 Jun 2025). The contraction-aligned residual analysis points toward broader connections between function approximation, residual geometry, and Bellman contraction beyond linear settings (Yang et al., 8 Apr 2026). The softmax Bellman work identifies cooling schedules and deeper bias–variance theory as natural future directions (Song et al., 2018). In this sense, the smoothed Bellman operator is less a single operator than a technical program: modifying Bellman backups or Bellman equations so that approximation, optimization, and control-theoretic regularity can be handled more effectively than under the hard optimality map alone.