---
title: Smoothed Bellman Operator
url: https://www.emergentmind.com/topics/smoothed-bellman-operator
type: topic
---

# Smoothed Bellman Operator

Searching arXiv for relevant papers on smoothed Bellman operators and closely related formulations.
The smoothed Bellman operator denotes a family of Bellman-type mappings in which the hard maximization or other nonsmooth components of dynamic programming are replaced, regularized, averaged, or otherwise softened to improve differentiability, stability, or analytical tractability. In reinforcement learning and control, the term covers several distinct constructions: softmax-weighted Bellman backups for $Q$-iteration, entropy-regularized log-sum-exp operators, smoothed Bellman residual objectives under weighted $L_p$ geometries, consistency operators used in SBEED, temporally averaged smoothed policy-improvement operators, annealed interpolations between optimality and policy-evaluation backups, and semigroup-smoothed Bellman mappings for infinite-dimensional Hamilton–Jacobi–Bellman equations [1812.00456], [2604.06837], [2007.03749], [2605.10671], [2506.05968], [2510.25894]. Across these settings, smoothing modifies the operator’s geometry and bias properties, typically trading exact Bellman optimality for regularity, reduced overestimation, improved optimization behavior, or stronger existence theory.

## 1. Canonical operator forms

In a finite discounted Markov decision process with state space $\mathcal{S}$, action space $\mathcal{A}$, reward $R$, transition kernel $P$, and discount $\gamma\in(0,1)$, the standard Bellman optimality operator on action values is
\[
T(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\max_{a'}Q(s',a').
\]
This operator is the unsmoothed reference point in several later constructions [1812.00456].

A widely studied smoothed variant is the softmax-weighted Bellman operator
\[
T_\beta(Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\sum_{a'}\frac{\exp(\beta Q(s',a'))}{\sum_{\bar a}\exp(\beta Q(s',\bar a))}Q(s',a'),
\]
where $\beta$ is an inverse temperature parameter. As $\beta\to\infty$, the inner softmax-weighted average approaches the max operator [1812.00456]. The same work also contrasts this operator with mellowmax,
\[
\mathrm{mm}_\beta(Q)(s)=\frac{1}{\beta}\log\Big(\frac{1}{m}\sum_{a'}\exp[\beta Q(s,a')]\Big),
\]
with $m=|\mathcal{A}|$, whose Bellman backup replaces the inner max by $\mathrm{mm}_\beta(Q(s',\cdot))$ [1812.00456].

A second canonical smoothing replaces max by log-sum-exp directly. In the $Q$-space formulation of soft Bellman residual minimization,
\[
(F_\lambda Q)(s,a)=R(s,a)+\gamma\sum_{s'}P(s'\mid s,a)\,\lambda\ln\Big(\sum_{u\in\mathcal{A}}\exp(Q(s',u)/\lambda)\Big),
\]
with temperature $\lambda>0$ [2604.06837]. In value space, the analogous entropy-smoothed operator is
\[
(T_\lambda V)(s)=\lambda\ln\sum_a \exp\Big(\frac{r(s,a)+\gamma\mathbb{E}_{s'\mid s,a}V(s')}{\lambda}\Big),
\]
which also appears in the analysis of entropy-regularized control and SBEED [2007.03749].

Smoothing can also appear in a policy-conditioned consistency form rather than an optimality form. SBEED employs
\[
(C_\lambda^\pi V)(s,a)=R(s,a)+\gamma(PV)(s,a)-\lambda\ln\pi(a\mid s),
\]
with $(PV)(s,a)=\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[V(s')]$ [2007.03749]. Here the smoothing is induced by entropy regularization through the log-policy term rather than by a direct replacement of $\max$.

A different generalization is the smoothed Bellman operator in doubly smoothed policy iteration:
\[
[\mathcal{H}_{\eta}(Q)](s,a)=\mathcal{R}(s,a)+\gamma\sum_{s'}p(s'\mid s,a)\max_{\mu\in\Delta(\mathcal{A})}\big(\mu^\top Q(s')+\eta \nu(\mu)\big),
\]
where $\nu$ is a bounded, non-negative, concave statewise regularizer and $\eta\ge 0$ is the smoothing level [2605.10671]. This encompasses Shannon entropy, Tsallis entropy, shifted negative squared norm, and the zero function.

In continuous-action online reinforcement learning, smoothing may be time dependent. One construction interpolates between the Bellman optimality operator and the Bellman operator through
\[
\mathcal{T}_{\beta(t)}Q(s,a)=r(s,a)+\gamma\mathbb{E}_{s'|s,a}\Big[\beta(t)\max_{a'}Q(s',a')+(1-\beta(t))\mathbb{E}_{a'\sim\pi(\cdot|s')}Q(s',a')\Big],
\]
with $\beta(t)$ annealed during training [2506.05968]. The continuous-action implementation uses expectile regression rather than an explicit max.

Outside finite-dimensional RL, the phrase also describes a semigroup-smoothed Bellman mapping for stationary HJB equations:
\[
\widetilde{\mathcal{B}}[v](x)=R_\lambda\big[\ell_0(\cdot)+H_{\min}(\nabla^B v(\cdot))\big](x),
\quad
R_\lambda[\phi](x)=\int_0^\infty e^{-\lambda t}P_t[\phi](x)\,dt,
\]
where $P_t$ is the Ornstein–Uhlenbeck semigroup [2510.25894]. Here smoothing is induced by the semigroup rather than by entropy or softmax.

## 2. Approximation to Bellman optimality and induced bias

For softmax-weighted backups, the central question is how closely the smoothed operator tracks the Bellman optimality operator. Under bounded rewards and finite action spaces, the difference between $\max_a Q(s,a)$ and the softmax-weighted scalar
\[
g_{Q(s,\cdot)}(\beta)=\sum_{a'}\mathrm{softmax}_\beta(Q(s,\cdot))_{a'}\,Q(s,a')
\]
admits explicit upper and lower bounds controlled by the action-gap envelope $\widehat{\delta}(s)$ [1812.00456]. The same work proves that the iterates $T_\beta^kQ_0$ converge to the standard Bellman iterates $T^kQ_0$ exponentially fast in $\beta$, with the rate determined by positive action gaps [1812.00456].

The asymptotic deviation from optimal action values is also bounded. Specifically, if $Q^*$ denotes the optimal $Q$-function, then the softmax iterates remain within a band below $Q^*$ whose width shrinks with increasing $\beta$ [1812.00456]. The paper emphasizes that this is a suboptimality envelope rather than a fixed-point distance because $T_\beta$ is not generally a contraction and may not have a unique fixed point [1812.00456].

Entropy-smoothed log-sum-exp operators admit a different limiting interpretation. In the soft residual minimization framework, $F_\lambda Q\to\mathcal{T}^*Q$ as $\lambda\to 0$ [2604.06837]. In SBEED, the entropy-smoothed operator satisfies the pointwise relation
\[
\max_a q_a \ge \lambda\ln\sum_a e^{q_a/\lambda}\ge \max_a q_a-\lambda\ln|\mathcal{A}|,
\]
which yields the smoothing bias bound
\[
|(T_\lambda V)(s)-(TV)(s)|\le \lambda\ln|\mathcal{A}|
\]
and a performance bias scaling as $\frac{\lambda\ln|\mathcal{A}|}{1-\gamma}$ relative to unregularized optimal control [2007.03749].

These bias statements show that smoothing typically introduces a controlled downward or regularization-induced deviation from exact Bellman optimality. A plausible implication is that the practical value of smoothing often depends less on recovering the exact optimality equation and more on how the altered backup interacts with approximation error, optimization, and statistical noise.

## 3. Contraction, monotonicity, and operator geometry

A major source of divergence between smoothed Bellman operators is whether smoothing preserves contraction. The softmax-weighted operator of the form used in deep $Q$-learning is in general not a contraction; the cited counterexample traces to Littman, 1996 [1812.00456]. By contrast, the mellowmax operator is a contraction, and the log-sum-exp operator $F_\lambda$ is a $\gamma$-contraction in $\|\cdot\|_\infty$ with unique fixed point $Q_\lambda^*$ [1812.00456], [2604.06837].

The distinction between weighted-average softmax and log-sum-exp is therefore structural rather than terminological. The former modifies the target by replacing $\max$ with a softmax-weighted expectation of $Q$ values; the latter replaces it by a convex-conjugate log-sum-exp value. Both are “soft,” but only the second inherits the familiar contraction argument in the sup norm in the cited analyses [1812.00456], [2604.06837].

The weighted $L_p$ analysis of soft Bellman residual minimization extends the geometric viewpoint. For weights $w_i>0$ summing to one, the weighted norm
\[
\|x\|_{p,w}=\Big(\sum_{i=1}^n w_i |x_i|^p\Big)^{1/p}
\]
induces an effective Lipschitz factor
\[
\gamma_{p,w}=\gamma\cdot n^{1/p}\cdot\Big(\frac{w_{\max}}{w_{\min}}\Big)^{1/p}.
\]
Then
\[
\|F_\lambda Q-F_\lambda Q'\|_{p,w}\le \gamma_{p,w}\|Q-Q'\|_{p,w}.
\]
Whenever $\gamma_{p,w}<1$, which holds for sufficiently large $p$, the soft operator becomes a contraction in $\|\cdot\|_{p,w}$ [2604.06837]. As $p\to\infty$, $\gamma_{p,w}\to\gamma$, aligning the optimization geometry with the Bellman operator’s native $\|\cdot\|_\infty$ contraction [2604.06837].

Monotonicity is preserved more broadly. For the softmax-weighted scalar $g_x(\beta)$, the derivative with respect to $\beta$ is nonnegative, so the softmax-weighted average increases monotonically with inverse temperature [1812.00456]. The log-sum-exp operator $F_\lambda$ is monotone in the sense that $Q\le Q'$ implies $F_\lambda Q\le F_\lambda Q'$ [2604.06837]. The smoothed operators $\mathcal{H}_\eta$ and $\mathcal{H}_\eta^\pi$ in doubly smoothed policy iteration are likewise monotone and $\ell_\infty$-contractive with factor $\gamma$ [2605.10671].

This contrast gives rise to a common misconception: that all smoothed Bellman operators sacrifice contraction. The literature summarized here shows that this is false. Some smoothings break contraction, some preserve it exactly, and others recover it only in a suitable geometry or under sufficiently large $p$ [1812.00456], [2604.06837], [2605.10671].

## 4. Overestimation control and stabilization in value-based learning

The most prominent practical motivation for smoothing in value-based RL is overestimation mitigation. Under the noise model used in the softmax Bellman analysis, where $Q_t(s,a)=V^*(s)+\varepsilon_a$ and all actions have equal true value at state $s$, the expected max noise is nonnegative:
\[
\mathbb{E}[\max_a \varepsilon_a]\ge 0.
\]
For the softmax aggregation,
\[
\mathbb{E}\Big[\sum_a \frac{\exp(\beta\varepsilon_a)}{\sum_{\bar a}\exp(\beta\varepsilon_{\bar a})}\,\varepsilon_a\Big]
\le
\mathbb{E}[\max_a\varepsilon_a],
\]
so the softmax backup induces less or equal overestimation than the max backup for any $\beta\ge 0$ [1812.00456]. The reduction is bounded using the same action-gap quantities that govern approximation quality [1812.00456].

This analysis helps explain an empirical paradox reported in deep $Q$-learning: the softmax Bellman operator leads to sub-optimal $Q$-functions in the tabular sense, yet when combined with DQN it can produce superior policies and even outperform double $Q$-learning on the reported Atari suite [1812.00456]. The paper argues that the reduced overestimation error and reduced gradient norm and variance are plausible drivers of this phenomenon [1812.00456].

The continuous-control annealing work presents a related but temporally structured argument. If $Q(s',a')$ is perturbed by zero-mean noise $\epsilon_{s',a'}$, then Jensen’s inequality yields
\[
\mathbb{E}\big[\mathcal{T}^*(Q+\epsilon)(s,a)\big]\ge \mathcal{T}^*Q(s,a),
\]
formalizing the upward bias of max-like backups [2506.05968]. The proposed remedy is not to fix a single smoothing level, but to begin training with a more max-like critic and gradually anneal toward policy-evaluation-style backups. In practice this is implemented by expectile loss
\[
L_2^\tau(u)=|\tau-\mathbbm{1}(u<0)|u^2,
\]
where $\tau\to 1$ induces a more upper-tail-focused, max-like critic and $\tau=0.5$ recovers standard $L_2$ regression [2506.05968].

The empirical findings are consistent with a two-phase interpretation: early overestimation can accelerate learning and correlate with higher policy entropy, whereas later reduction toward expectation backups lowers bias and improves stability [2506.05968]. This suggests that “smoothed Bellman operator” may denote not only a static surrogate for $\max$, but also a training-time schedule through a family of operators.

## 5. Residual minimization, SBEED, and optimization-oriented smoothing

A separate lineage uses smoothing to make Bellman residual objectives differentiable and compatible with function approximation. In contraction-aligned soft Bellman residual minimization, linear approximations of the form $Q_\theta=\Phi\theta$ are fitted by minimizing
\[
f_p(\theta)=\frac{1}{p}\|F_\lambda(\Phi\theta)-\Phi\theta\|_{p,w}^p
=
\frac{1}{p}\|\delta_\theta\|_{p,w}^p,
\quad
\delta_\theta=F_\lambda(\Phi\theta)-\Phi\theta
\]
under a weighted $L_p$ norm [2604.06837]. For sufficiently large $p$ such that $\gamma_{p,w}<1$, the paper proves sandwich bounds connecting the residual objective to the distance from the soft fixed point:
\[
\frac{(1-\gamma_{p,w})^p}{p}\|Q_\theta-Q_\lambda^*\|_{p,w}^p
\le
f_p(\theta)
\le
\frac{(1+\gamma_{p,w})^p}{p}\|Q_\theta-Q_\lambda^*\|_{p,w}^p.
\]
This yields quasi-optimality of the minimizer $\theta_p^*$ relative to the best approximation to $Q_\lambda^*$, with multiplicative gap
\[
C(p)=\frac{1+\gamma_{p,w}}{1-\gamma_{p,w}},
\]
which decreases monotonically with $p$ and converges to $\frac{1+\gamma}{1-\gamma}$ as $p\to\infty$ [2604.06837].

The same framework derives explicit gradients. With even integer $p\ge 2$,
\[
\nabla_\theta f_p(\theta)=\sum_{i=1}^n w_i (e_i^\top \delta_\theta)^{p-1}\nabla_\theta(e_i^\top \delta_\theta),
\]
and because the soft operator is differentiable, its Jacobian can be expressed using the Boltzmann policy induced by $Q_\theta$:
\[
\pi_\theta(a\mid s)=\frac{\exp(Q_\theta(s,a)/\lambda)}{\sum_{u}\exp(Q_\theta(s,u)/\lambda)},\qquad
\nabla_Q F_\lambda(Q_\theta)=\gamma P\Pi^{\pi_\theta},
\]
so
\[
\nabla_\theta\delta_\theta=(\gamma P\Pi^{\pi_\theta}-I)\Phi
\]
[2604.06837]. Large-$p$ instability is handled by residual normalization, which rescales the gradient without changing stationary points [2604.06837].

SBEED uses smoothing in a different way. Rather than minimizing a direct Bellman residual, it introduces a saddle-point objective to avoid double-sampling bias in the empirical smoothed Bellman error [2007.03749]. With i.i.d. batch data and helper class $G$, the objective is
\[
\min_{V\in V,\;\pi\in P}\max_{g\in G}\Big\{L_D(V;V,\pi)-R_D(g;V,\pi)\Big\},
\]
where
\[
L_D(V;V,\pi)=\frac{1}{n}\sum_{i=1}^n\big(V(s_i)-r_i-\gamma V(s_i')+\lambda\ln\pi(a_i\mid s_i)\big)^2
\]
and
\[
R_D(g;V,\pi)=\frac{1}{n}\sum_{i=1}^n\big(g(s_i,a_i)-r_i-\gamma V(s_i')+\lambda\ln\pi(a_i\mid s_i)\big)^2.
\]
The helper function $g$ estimates the consistency target and cancels the variance term caused by the inner expectation [2007.03749].

The sharp analysis of SBEED proves a suboptimality decomposition
\[
J(\pi^\star)-J(\hat\pi)
\le
\frac{\lambda\ln|\mathcal{A}|}{1-\gamma}
+
\frac{2\sqrt{C_2}}{1-\gamma}\|\hat V-C_\lambda^{\hat\pi}\hat V\|_{2,\mu},
\]
where $C_2$ is a tight $L_2$-type concentrability coefficient [2007.03749]. The resulting guarantee has linear dependence on the horizon factor $(1-\gamma)^{-1}$ and a fast realizable statistical rate $1/\sqrt{n}$, improving earlier SBEED analysis that yielded slower rates [2007.03749].

Taken together, these works position smoothing not merely as a surrogate for $\max$, but as a means of making Bellman objectives differentiable, variance-aware, and better matched to contraction geometry under approximation.

## 6. Smoothed policy iteration and natural policy gradient

In doubly smoothed policy iteration, smoothing acts both across policies and within the greedy step. The algorithm maintains an averaged action-value sequence
\[
\bar Q_{k+1}=(1-\beta_k)\bar Q_k+\beta_k Q^{\pi_k},
\]
and defines the next policy through the smoothed greedification condition
\[
\mathcal{H}_{\eta_k}^{\pi_{k+1}}(\bar Q_{k+1})=\mathcal{H}_{\eta_k}(\bar Q_{k+1}),
\qquad
\eta_k=\tau\prod_{j=1}^k(1-\beta_j),
\]
or explicitly,
\[
\pi_{k+1}(s)\in \argmax_{\mu\in\Delta(\mathcal{A})}
\Big\{\mu^\top \bar Q_{k+1}(s)+\tau\prod_{j=1}^k(1-\beta_j)\nu(\mu)\Big\}
\]
[2605.10671].

The smoothing is therefore double: temporal averaging in $\bar Q_{k+1}$ and regularized greedy improvement via $\nu$ [2605.10671]. The analysis relies only on monotonicity and contraction of the smoothed Bellman operators. It proves monotonic policy improvement,
\[
Q^{\pi_k}\le Q^{\pi_{k+1}},
\]
and a one-step recursion
\[
\|Q^*-\bar Q_{k+1}\|_\infty
\le
(1-(1-\gamma)\beta_k)\|Q^*-\bar Q_k\|_\infty
+
\gamma\beta_k\eta_{k-1}\nu_{\max}
\]
[2605.10671]. Under constant $\beta$, the resulting convergence is geometric, with iteration complexity $\mathcal{O}((1-\gamma)^{-1}\log((1-\gamma)^{-1}\epsilon^{-1}))$ for computing an $\epsilon$-optimal policy [2605.10671].

A key contribution of this framework is the exact equivalence between natural policy gradient and a specific doubly smoothed policy iteration instance. With $\nu=h$ equal to Shannon entropy, $\tau=1/\alpha_0$, and
\[
\beta_k=\frac{\alpha_k}{\sum_{i=0}^k \alpha_i},
\]
the policy sequence generated by DSPI coincides with the sequence generated by tabular dual-averaged natural policy gradient [2605.10671]. This yields a Bellman-operator interpretation of natural policy gradient in which the smoothing is intrinsic to the mirror map and temporal averaging, not added as an extraneous regularizer.

The unregularized case $\nu\equiv 0$ recovers dual-averaged policy iteration, for which the paper proves finite termination under deterministic greedy updates and fixed tie breaking [2605.10671]. This places smoothed Bellman operators on a continuum between classical policy iteration, mirror-descent-style methods, and natural policy gradient.

## 7. Infinite-dimensional HJB formulations and broader interpretation

In infinite-dimensional control, smoothing addresses a different obstruction: lack of regularity in the HJB equation rather than nondifferentiability of $\max$. For the stationary HJB
\[
\lambda v(x)=\mathcal{A}[v](x)+\ell_0(x)+H_{\min}(\nabla^B v(x)),
\]
classical mild solution theory may fail because of lack of strong Feller smoothing, unbounded control operators, and state-dependent costs [2510.25894]. The cited work addresses this by lifting the problem to a trajectory space and exploiting partial smoothing of the Ornstein–Uhlenbeck semigroup.

The Bellman nonlinearity is
\[
\mathcal{B}[v](x)=\ell_0(x)+H_{\min}(\nabla^B v(x)),
\]
and the mild solution is represented as
\[
v(x)=\int_0^\infty e^{-\lambda t}(P_t[\mathcal{B}[v]])(x)\,dt.
\]
This integral resolvent form is the smoothed Bellman operator in that setting [2510.25894]. The semigroup $P_t$ regularizes only along appropriate lifted and $B$-directional coordinates, quantified through operators $\widehat\Lambda^{P,B}(t)$ satisfying
\[
\|\widehat\Lambda^{P,B}(t)\|_{L(K,L^2_\rho)}\le \kappa_0(t^{-\gamma}\vee 1),
\qquad \gamma\in(0,1)
\]
[2510.25894]. With sufficiently large discount $\lambda$, the corresponding fixed-point map becomes a contraction on the lifted function class, yielding existence and uniqueness of a regular mild solution, together with verification and optimal feedback synthesis [2510.25894].

This infinite-dimensional usage extends the notion of smoothing beyond regularized maximization. The operator is “smoothed” because the Bellman nonlinearity is precomposed with a semigroup and integrated against the discount kernel, thereby gaining differentiability properties otherwise unavailable [2510.25894]. A plausible implication is that the unifying idea behind smoothed Bellman operators is not a particular formula such as log-sum-exp, but the replacement of a raw Bellman map by one with improved analytical regularity.

## 8. Comparative perspective and recurring trade-offs

The literature distinguishes several recurrent trade-offs.

| Construction | Main benefit | Main limitation |
|---|---|---|
| Softmax-weighted backup | Reduces overestimation and gradient variance; directly defines a policy | Not Bellman-optimal; not generally a contraction [1812.00456] |
| Log-sum-exp / entropy smoothing | Differentiable; $\gamma$-contraction in $\|\cdot\|_\infty$; unique soft fixed point | Introduces regularization bias relative to hard optimality [2604.06837], [2007.03749] |
| Mellowmax | Contraction; reduces overestimation | Does not directly define a policy without extra steps [1812.00456] |
| Weighted $L_p$ soft residual minimization | Aligns residual objective with contraction geometry as $p$ increases | Requires tuning $p$, weights, and temperature; large $p$ may cause large gradients [2604.06837] |
| SBEED consistency operator | Avoids double-sampling bias; sharp batch guarantees | Analysis depends on finite classes and concentrability [2007.03749] |
| DSPI smoothed operators | Distribution-free geometric convergence; exact NPG equivalence | Model-based evaluation in the presented analysis [2605.10671] |
| Annealed optimality-to-evaluation smoothing | Early acceleration with later bias reduction in continuous control | Performance depends on annealing schedule [2506.05968] |
| Semigroup-smoothed HJB mapping | Existence, uniqueness, and feedback synthesis in infinite dimensions | Requires lifting machinery and partial smoothing hypotheses [2510.25894] |

Several misconceptions recur in discussions of the topic. One is that smoothing always means entropy regularization. The surveyed works show instead that smoothing may arise from softmax weighting, log-sum-exp conjugacy, regularized greedification, residual geometry, expectile annealing, or semigroup resolvents [1812.00456], [2604.06837], [2605.10671], [2506.05968], [2510.25894]. Another is that smoothing is synonymous with improved contraction; this is true for some operators but false for the softmax-weighted Bellman operator [1812.00456]. A third is that smoothing necessarily degrades performance because it induces suboptimal value functions. The Atari and continuous-control results indicate that, in the presence of function approximation and optimization noise, a biased operator may nonetheless improve learned policies by controlling overestimation and stabilizing updates [1812.00456], [2506.05968].

Open questions remain. The annealed continuous-control framework notes that convergence properties of time-varying smoothed operators under function approximation remain open [2506.05968]. The contraction-aligned residual analysis points toward broader connections between function approximation, residual geometry, and Bellman contraction beyond linear settings [2604.06837]. The softmax Bellman work identifies cooling schedules and deeper bias–variance theory as natural future directions [1812.00456]. In this sense, the smoothed Bellman operator is less a single operator than a technical program: modifying Bellman backups or Bellman equations so that approximation, optimization, and control-theoretic regularity can be handled more effectively than under the hard optimality map alone.

Source: https://www.emergentmind.com/topics/smoothed-bellman-operator