Doubly Smoothed Policy Iteration (DSPI)

• Doubly Smoothed Policy Iteration (DSPI) is a reinforcement learning framework that smooths both Q-value estimates and policy updates using regularization and running averages.
• It unifies classical policy iteration, natural policy gradient, and dual-averaged methods under a Bellman-operator framework with enhanced convergence properties.
• The approach guarantees global geometric convergence and finite termination under specific conditions, and it extends naturally to function approximation and stochastic shortest path problems.

Searching arXiv for the specified paper and closely related context. arXiv search query: (Nanda et al., 11 May 2026) Doubly Smoothed Policy Iteration (DSPI) is a Bellman-operator framework for reinforcement learning in which each policy is obtained by applying a regularized greedy step to a weighted average of past $Q$-functions. In the formulation introduced in "Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework," DSPI gives an exact policy-iteration interpretation of natural policy gradient, while also subsuming classical policy iteration, dual-averaged policy iteration, and more general policy dual averaging methods (Nanda et al., 11 May 2026). Its defining feature is the simultaneous use of two forms of smoothing: averaging in $Q$-space and regularization in the policy-improvement step.

1. Problem setting and Bellman-operator background

DSPI is formulated for a finite discounted Markov decision process $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$, where $\mathcal S$ has size $n$, $\mathcal A$ has size $m$, $\mathcal R(s,a)\in[0,1]$, and $\gamma\in(0,1)$. A stationary policy $\pi$ maps each state $Q$0 to a distribution $Q$1 over actions. Its state-action value function is

$Q$2

and its value function is

$Q$3

The optimal $Q$4-function is characterized by the Bellman optimality equation

$Q$5

For a fixed policy $Q$6, the evaluation operator is

$Q$7

where

$Q$8

Both $Q$9 and each $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$0 are $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$1-contractions in $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$2 and are monotone in the sense that $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$3, with the analogous property for $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$4. These two properties—monotonicity and contraction—form the analytic backbone of DSPI. The framework is notable because its convergence analysis uses only these properties of smoothed Bellman operators, rather than distribution-dependent arguments or adaptive, trajectory-dependent stepsize constructions (Nanda et al., 11 May 2026).

2. Smoothed Bellman operators

The first structural ingredient of DSPI is Bellman smoothing through regularization. Fix a concave, nonnegative regularizer $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$5, such as Shannon entropy. For $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$6, define the smoothed optimality operator

$$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$7

and, for any policy $$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$8, the smoothed evaluation operator

$$\mathcal M=(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$$9

For each $\mathcal S$0, the operators $\mathcal S$1 and $\mathcal S$2 remain $\mathcal S$3-contractions in $\mathcal S$4 and remain monotone in $\mathcal S$5. This preservation is central: the regularized problem stays within the same Bellman-operator template as ordinary dynamic programming.

The smoothing parameter $\mathcal S$6 controls the strength of regularization. Large $\mathcal S$7 corresponds to stronger smoothing, while $\mathcal S$8 recovers the unregularized Bellman operator. The regularized greedy step therefore interpolates between exact maximization and a soft-greedy choice induced by $\mathcal S$9.

3. DSPI update rule

DSPI is described as “doubly smoothed” because it introduces two distinct modifications to classical policy iteration. First, it replaces the most recent $n$0-function by a running average $n$1. Second, it replaces the greedy policy-improvement step by a regularized greedy step.

The iteration is:

1. Initialize $n$2 for each state $n$3.
2. Set $n$4.
3. For $n$5:

$n$6

$n$7

and choose $n$8 so that

$n$9

Equivalently, for each state $\mathcal A$0,

$\mathcal A$1

The parameter $\mathcal A$2 trades off freshness versus stability of the running average $\mathcal A$3, and $\mathcal A$4 controls the strength of regularization. As $\mathcal A$5, $\mathcal A$6 and DSPI reduces to classical greedy policy iteration. This construction makes explicit that DSPI smooths both the value information used for improvement and the improvement map itself (Nanda et al., 11 May 2026).

4. Specializations and the relation to natural policy gradient

A defining feature of DSPI is that several apparently different policy-optimization algorithms appear as exact special cases.

Classical policy iteration. Setting $\mathcal A$7 removes averaging, and $\mathcal A$8. The policy-improvement step becomes

$\mathcal A$9

which is the exact greedy update.

Dual-averaged policy iteration, unregularized form. Setting $m$0 removes entropy or other regularization while keeping $m$1. Averaging then occurs only in $m$2-space, and the policy update is exact greedy with respect to $m$3.

Natural policy gradient. If one chooses a strongly-concave mirror map $m$4, takes Shannon entropy $m$5 for $m$6, and sets

$m$7

then

$m$8

and the soft-greedy step becomes exactly the natural policy gradient update

$m$9

The same identification extends to the full policy dual-averaging class by replacing the entropy-based mirror map with any convex divergence-generator $\mathcal R(s,a)\in[0,1]$0. The conceptual significance is that natural policy gradient is not merely analogous to a smoothed policy-improvement method; it admits an exact formulation as doubly smoothed policy iteration (Nanda et al., 11 May 2026). This reframes NPG within dynamic programming rather than treating it solely as a policy-space first-order method.

5. Convergence theory and finite termination

The DSPI analysis proceeds through Bellman-operator arguments that parallel classical policy iteration but operate in the doubly smoothed setting. One key lemma is monotonic improvement: $\mathcal R(s,a)\in[0,1]$1 The proof combines the Bellman equation for $\mathcal R(s,a)\in[0,1]$2 with the DSPI soft-greedy condition and then applies monotonicity and translation invariance.

A second lemma gives a one-step contraction for the averaged iterate. If

$\mathcal R(s,a)\in[0,1]$3

then for $\mathcal R(s,a)\in[0,1]$4,

$\mathcal R(s,a)\in[0,1]$5

where $\mathcal R(s,a)\in[0,1]$6.

From this recursion, together with the relation between $\mathcal R(s,a)\in[0,1]$7-function error and value-function error, one obtains a global geometric convergence theorem. If $\mathcal R(s,a)\in[0,1]$8, $\mathcal R(s,a)\in[0,1]$9, and $\gamma\in(0,1)$0, then for all $\gamma\in(0,1)$1,

$\gamma\in(0,1)$2

Consequently, to reach $\gamma\in(0,1)$3-accuracy it suffices to take

$\gamma\in(0,1)$4

The abstract emphasizes that this yields distribution-free global geometric convergence of DSPI, and therefore the same iteration complexity for standard natural policy gradient and policy dual averaging, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes (Nanda et al., 11 May 2026).

In the unregularized case $\gamma\in(0,1)$5, corresponding to dual-averaged policy iteration, DSPI admits a stronger conclusion: finite termination. With deterministic tie-breaking, $\gamma\in(0,1)$6, and $\gamma\in(0,1)$7, every

$\gamma\in(0,1)$8

iterations eliminate at least one suboptimal action. Since there are at most $\gamma\in(0,1)$9 suboptimal actions, the method terminates in

$\pi$0

steps with an optimal policy.

A common misconception is that geometric convergence guarantees for NPG require either auxiliary regularization or state-distribution assumptions. The DSPI formulation shows that, within this Bellman-operator framework, standard NPG can be analyzed globally and distribution-free.

6. Extensions to function approximation and stochastic shortest path

The same framework extends beyond tabular discounted MDPs. One extension uses linear function approximation with log-linear policies

$\pi$1

where features lie in $\pi$2. At iteration $\pi$3, one solves the least-squares TD fit

$\pi$4

to obtain

$\pi$5

DSPI is then run with

$\pi$6

followed by the same soft-greedy step.

Under the uniform sup-norm approximation condition

$\pi$7

the method retains geometric contraction up to an additive $\pi$8 term. In particular, natural policy gradient with log-linear policies converges in

$\pi$9

iterations to within $Q$00 of optimal (Nanda et al., 11 May 2026).

A second extension treats undiscounted stochastic shortest path (SSP) problems. In the SSP setting there is a terminal state and proper policies, meaning the terminal state is reached with probability $Q$01. Under the standard assumption that all stationary policies are proper, the Bellman operator is a contraction in a weighted $Q$02-norm $Q$03. Smoothed operators $Q$04 and $Q$05 are defined in the same way as in the discounted case, but without the discount factor in front of future $Q$06.

Running the corresponding DSPI_SSP iteration with constant $Q$07 yields

$Q$08

where $Q$09 is the weighted contraction factor. Using equivalence of norms, this implies

$Q$10

and hence

$Q$11

The same specialization that identifies NPG in the discounted setting also recovers NPG for SSP with matching iteration complexity.

These extensions suggest that the DSPI perspective is not restricted to a narrowly tabular interpretation of policy iteration. A plausible implication is that Bellman-operator smoothing provides a common analytic language for policy optimization across discounted, approximate, and proper undiscounted regimes, so long as monotonicity and contraction survive in an appropriate norm.