---
title: Doubly Smoothed Policy Iteration (DSPI)
url: https://www.emergentmind.com/topics/doubly-smoothed-policy-iteration-dspi
type: topic
---

# Doubly Smoothed Policy Iteration (DSPI)

Searching arXiv for the specified paper and closely related context.
arXiv search query: 2605.10671
Doubly Smoothed Policy Iteration (DSPI) is a Bellman-operator framework for reinforcement learning in which each policy is obtained by applying a regularized greedy step to a weighted average of past $Q$-functions. In the formulation introduced in "Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework," DSPI gives an exact policy-iteration interpretation of natural policy gradient, while also subsuming classical policy iteration, dual-averaged policy iteration, and more general policy dual averaging methods [2605.10671]. Its defining feature is the simultaneous use of two forms of smoothing: averaging in $Q$-space and regularization in the policy-improvement step.

## 1. Problem setting and Bellman-operator background

DSPI is formulated for a finite discounted Markov decision process
$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$, where $\mathcal S$ has size $n$, $\mathcal A$ has size $m$, $\mathcal R(s,a)\in[0,1]$, and $\gamma\in(0,1)$. A stationary policy $\pi$ maps each state $s$ to a distribution $\pi(\cdot\mid s)$ over actions. Its state-action value function is
\[
Q^\pi(s,a)=\EE_\pi\Bigl[\sum_{t=0}^\infty\gamma^t\,\mathcal R(S_t,A_t)\,\big|\,S_0=s,A_0=a\Bigr],
\]
and its value function is
\[
V^\pi(s)=\sum_a\pi(a\mid s)\,Q^\pi(s,a).
\]

The optimal $Q$-function is characterized by the Bellman optimality equation
\[
Q^*=\mathcal H(Q^*),\qquad
[\mathcal H(Q)](s,a)=\mathcal R(s,a)+\gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').
\]
For a fixed policy $\pi$, the evaluation operator is
\[
\mathcal H^\pi(Q)=\mathcal R+\gamma\,P^\pi\,Q,
\]
where
\[
[P^\pi Q](s,a)=\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a').
\]

Both $\mathcal H$ and each $\mathcal H^\pi$ are $\gamma$-contractions in $\|\cdot\|_\infty$ and are monotone in the sense that $Q_1\le Q_2\implies \mathcal H(Q_1)\le \mathcal H(Q_2)$, with the analogous property for $\mathcal H^\pi$. These two properties—monotonicity and contraction—form the analytic backbone of DSPI. The framework is notable because its convergence analysis uses only these properties of smoothed Bellman operators, rather than distribution-dependent arguments or adaptive, trajectory-dependent stepsize constructions [2605.10671].

## 2. Smoothed Bellman operators

The first structural ingredient of DSPI is Bellman smoothing through regularization. Fix a concave, nonnegative regularizer $\nu:\Delta(\mathcal A)\to\mathbb R_{\ge 0}$, such as Shannon entropy. For $\eta\ge 0$, define the smoothed optimality operator
\[
[\mathcal H_\eta(Q)](s,a)
=\mathcal R(s,a)
+\gamma\sum_{s'}p(s'\!\mid s,a)\;
\max_{\mu\in\Delta(\mathcal A)}
\bigl\{\mu^\top Q(s')+\eta\,\nu(\mu)\bigr\},
\]
and, for any policy $\pi$, the smoothed evaluation operator
\[
[\mathcal H_\eta^\pi(Q)](s,a)
=\mathcal R(s,a)
+\gamma\sum_{s'}p(s'\!\mid s,a)\bigl\{\pi(s')^\top Q(s')+\eta\,\nu(\pi(s'))\bigr\}.
\]

For each $\eta$, the operators $\mathcal H_\eta$ and $\mathcal H_\eta^\pi$ remain $\gamma$-contractions in $\|\cdot\|_\infty$ and remain monotone in $Q$. This preservation is central: the regularized problem stays within the same Bellman-operator template as ordinary dynamic programming.

The smoothing parameter $\eta$ controls the strength of regularization. Large $\eta$ corresponds to stronger smoothing, while $\eta=0$ recovers the unregularized Bellman operator. The regularized greedy step therefore interpolates between exact maximization and a soft-greedy choice induced by $\nu$.

## 3. DSPI update rule

DSPI is described as “doubly smoothed” because it introduces two distinct modifications to classical policy iteration. First, it replaces the most recent $Q$-function by a running average $\bar Q_k$. Second, it replaces the greedy policy-improvement step by a regularized greedy step.

The iteration is:
1. Initialize $\pi_0(s)\in \operatorname{Argmax}_\mu \nu(\mu)$ for each state $s$.
2. Set $\bar Q_0\leftarrow 0$.
3. For $k=0,1,2,\dots$:
   \[
   \bar Q_{k+1}\leftarrow (1-\beta_k)\bar Q_k+\beta_k Q^{\pi_k},\qquad 0<\beta_k\le 1,
   \]
   \[
   \eta_k\leftarrow \tau \cdot \prod_{j=1}^k(1-\beta_j),
   \]
   and choose $\pi_{k+1}$ so that
   \[
   \mathcal H_{\eta_k}^{\pi_{k+1}}(\bar Q_{k+1})=\mathcal H_{\eta_k}(\bar Q_{k+1}).
   \]

Equivalently, for each state $s$,
\[
\pi_{k+1}(s)\in\operatorname{Argmax}_{\mu\in\Delta(\mathcal A)}
\Bigl\{\mu^\top \bar Q_{k+1}(s)
+\tau\!\!\prod_{j=1}^k(1-\beta_j)\,\nu(\mu)\Bigr\}.
\]

The parameter $\beta_k$ trades off freshness versus stability of the running average $\bar Q_k$, and $\eta_k$ controls the strength of regularization. As $\beta_k\to 1$, $\eta_k\to 0$ and DSPI reduces to classical greedy policy iteration. This construction makes explicit that DSPI smooths both the value information used for improvement and the improvement map itself [2605.10671].

## 4. Specializations and the relation to natural policy gradient

A defining feature of DSPI is that several apparently different policy-optimization algorithms appear as exact special cases.

**Classical policy iteration.** Setting $\beta_k\equiv 1$ removes averaging, and $\eta_k=0$. The policy-improvement step becomes
\[
\pi_{k+1}\in\operatorname{Argmax}_\mu \mu^\top Q^{\pi_k},
\]
which is the exact greedy update.

**Dual-averaged policy iteration, unregularized form.** Setting $\nu\equiv 0$ removes entropy or other regularization while keeping $\beta_k\in(0,1]$. Averaging then occurs only in $Q$-space, and the policy update is exact greedy with respect to $\bar Q$.

**Natural policy gradient.** If one chooses a strongly-concave mirror map $\omega$, takes Shannon entropy $h$ for $\nu$, and sets
\[
\beta_k=\frac{\alpha_k}{\sum_{i=0}^k\alpha_i},\qquad
\tau=\frac{1}{\alpha_0},\qquad
\nu(\mu)=h(\mu),\quad h(\mu)=-\sum_a \mu(a)\log \mu(a),
\]
then
\[
\bar Q_{k+1}=\frac{\sum_{i=0}^k \alpha_i Q^{\pi_i}}{\sum_{i=0}^k \alpha_i},
\qquad
\eta_k=\frac{1}{\sum_{i=0}^k \alpha_i},
\]
and the soft-greedy step becomes exactly the natural policy gradient update
\[
\theta_{k+1}=\theta_k+\alpha_k\,Q^{\pi_k},\qquad
\pi_{k+1}\in\operatorname{Argmax}_\pi\{\langle \theta_{k+1},\pi\rangle+h(\pi)\}.
\]

The same identification extends to the full policy dual-averaging class by replacing the entropy-based mirror map with any convex divergence-generator $\omega$. The conceptual significance is that natural policy gradient is not merely analogous to a smoothed policy-improvement method; it admits an exact formulation as doubly smoothed policy iteration [2605.10671]. This reframes NPG within dynamic programming rather than treating it solely as a policy-space first-order method.

## 5. Convergence theory and finite termination

The DSPI analysis proceeds through Bellman-operator arguments that parallel classical policy iteration but operate in the doubly smoothed setting. One key lemma is monotonic improvement:
\[
Q^{\pi_k}\le Q^{\pi_{k+1}}\qquad \text{for all }k.
\]
The proof combines the Bellman equation for $Q^{\pi_k}$ with the DSPI soft-greedy condition and then applies monotonicity and translation invariance.

A second lemma gives a one-step contraction for the averaged iterate. If
\[
\Delta_k=\|Q^*-\bar Q_k\|_\infty,
\]
then for $k\ge 1$,
\[
\Delta_{k+1}\le (1-(1-\gamma)\beta_k)\,\Delta_k+\gamma\,\beta_k\,\eta_{k-1}\,\nu_{\max},
\]
where $\nu_{\max}=\max_\mu \nu(\mu)$.

From this recursion, together with the relation between $Q$-function error and value-function error, one obtains a global geometric convergence theorem. If $\beta_0=1$, $\beta_k\equiv \beta\in(0,1)$, and $\tau\nu_{\max}\le 1$, then for all $k\ge 1$,
\[
\|V^*-V^{\pi_k}\|_\infty
\le
(1-(1-\gamma)\beta)^{k-1}
\bigl[\gamma\|V^*-V^{\pi_0}\|_\infty+1\bigr].
\]
Consequently, to reach $\varepsilon$-accuracy it suffices to take
\[
k=O\Bigl(\frac{1}{1-\gamma}\log\Bigl(\frac{1}{(1-\gamma)\varepsilon}\Bigr)\Bigr).
\]

The abstract emphasizes that this yields distribution-free global geometric convergence of DSPI, and therefore the same iteration complexity for standard natural policy gradient and policy dual averaging, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes [2605.10671].

In the unregularized case $\nu\equiv 0$, corresponding to dual-averaged policy iteration, DSPI admits a stronger conclusion: finite termination. With deterministic tie-breaking, $\beta_0=1$, and $\beta_k\equiv \beta$, every
\[
k^*=\Bigl\lceil\frac{1}{\beta(1-\gamma)}\ln\frac{2}{1-\gamma}\Bigr\rceil
\]
iterations eliminate at least one suboptimal action. Since there are at most $n(m-1)$ suboptimal actions, the method terminates in
\[
n(m-1)\;\Bigl\lceil\frac{1}{\beta(1-\gamma)}\ln\frac{2}{1-\gamma}\Bigr\rceil
=
O\bigl(nm\,(1-\gamma)^{-1}\log((1-\gamma)^{-1})\bigr)
\]
steps with an optimal policy.

A common misconception is that geometric convergence guarantees for NPG require either auxiliary regularization or state-distribution assumptions. The DSPI formulation shows that, within this Bellman-operator framework, standard NPG can be analyzed globally and distribution-free.

## 6. Extensions to function approximation and stochastic shortest path

The same framework extends beyond tabular discounted MDPs. One extension uses linear function approximation with log-linear policies
\[
\pi_\theta(a\mid s)\propto \exp\{\phi_{s,a}^\top \theta\},
\]
where features lie in $\mathbb R^d$. At iteration $k$, one solves the least-squares TD fit
\[
\min_w\,\EE_{s,a}\bigl[Q^{\pi_k}(s,a)-\phi_{s,a}^\top w\bigr]^2
\]
to obtain
\[
W_k^*(s,a)=\phi_{s,a}^\top w_k^*\approx Q^{\pi_k}(s,a).
\]
DSPI is then run with
\[
\bar W_{k+1}=(1-\beta_k)\bar W_k+\beta_k W_k^*,
\]
followed by the same soft-greedy step.

Under the uniform sup-norm approximation condition
\[
\|W_k^*-Q^{\pi_k}\|_\infty\le \varepsilon,
\]
the method retains geometric contraction up to an additive $O\bigl(\varepsilon/(1-\gamma)^2\bigr)$ term. In particular, natural policy gradient with log-linear policies converges in
\[
O\bigl((1-\gamma)^{-1}\log((1-\gamma)^{-1}\varepsilon^{-1})\bigr)
\]
iterations to within $O(\varepsilon/(1-\gamma)^2)$ of optimal [2605.10671].

A second extension treats undiscounted stochastic shortest path (SSP) problems. In the SSP setting there is a terminal state and proper policies, meaning the terminal state is reached with probability $1$. Under the standard assumption that all stationary policies are proper, the Bellman operator is a contraction in a weighted $\infty$-norm $\|\cdot\|_\xi$. Smoothed operators $\widetilde H_\eta$ and $\widetilde H_\eta^\pi$ are defined in the same way as in the discounted case, but without the discount factor in front of future $Q$.

Running the corresponding DSPI\_SSP iteration with constant $\beta$ yields
\[
\|Q^*-\bar Q_k\|_\xi
\le
(1-(1-\kappa)\beta)^{k-1}\|Q^*-\bar Q_1\|_\xi+O(\eta),
\]
where $\kappa<1$ is the weighted contraction factor. Using equivalence of norms, this implies
\[
\|V^*-V^{\pi_k}\|_\infty
=
O\bigl((1-(1-\kappa)\beta)^k\bigr),
\]
and hence
\[
k=O\bigl((1-\kappa)^{-1}\log((1-\kappa)^{-1}\varepsilon^{-1})\bigr).
\]
The same specialization that identifies NPG in the discounted setting also recovers NPG for SSP with matching iteration complexity.

These extensions suggest that the DSPI perspective is not restricted to a narrowly tabular interpretation of policy iteration. A plausible implication is that Bellman-operator smoothing provides a common analytic language for policy optimization across discounted, approximate, and proper undiscounted regimes, so long as monotonicity and contraction survive in an appropriate norm.

Source: https://www.emergentmind.com/topics/doubly-smoothed-policy-iteration-dspi