---
title: Dual-Averaged Policy Iteration
url: https://www.emergentmind.com/topics/dual-averaged-policy-iteration
type: topic
---

# Dual-Averaged Policy Iteration

Searching arXiv for the cited papers to ground the article in current sources.
{"query":"id:2605.10671 OR \"Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework\"","max_results":5}
{"query":"2605.10671","max_results":5,"source":"arxiv"}
to=arxiv_search  大发游戏 ＿天天json code
{"query":"2605.10671","max_results":5}
to=arxiv_search  彩神争霸大发json code
{"query":"2603.10199","max_results":5}
Dual-Averaged Policy Iteration is the unregularized, dual-averaged instantiation of Doubly-Smoothed Policy Iteration (DSPI) in a finite discounted Markov decision process. In this formulation, each policy is obtained by taking a greedy step with respect to a running average of past $Q$-functions rather than with respect to only the most recent evaluation. In the Bellman-operator framework of "Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework" [2605.10671], this zero-regularizer case exactly recovers a policy-iteration-like procedure that admits monotonic policy improvement, Bellman-residual contraction, and a finite-termination theorem.

## 1. Bellman-operator form

The relevant setting is a finite discounted MDP $(\mathcal S,\mathcal A,p,\mathcal R,\gamma)$. The policy-evaluation Bellman operator is
\[
[\mathcal H^\pi Q](s,a)
=
\mathcal R(s,a)
+
\gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),
\]
and the optimality operator is
\[
[\mathcal H Q](s,a)
=
\mathcal R(s,a)
+
\gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').
\]

DSPI maintains a running average
\[
\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},
\]
and then selects $\pi_{k+1}$ through a regularized greedy step. The framework uses the smoothed optimality operator
\[
[\mathcal H_\eta Q](s,a)
=
\mathcal R(s,a)
+
\gamma\sum_{s'}p(s'\mid s,a)
\max_{\mu\in\Delta(\mathcal A)}
\{\mu^\top Q(s')+\eta\,\nu(\mu)\}.
\]

Dual-Averaged Policy Iteration arises by turning off the regularization, i.e. by choosing $\nu\equiv 0$. In that case, $\mathcal H_\eta\equiv\mathcal H$ and $\mathcal H^\pi_\eta\equiv\mathcal H^\pi$, and DSPI becomes the update pair
\[
\bar Q_{k+1}=(1-\beta_k)\bar Q_k+\beta_kQ^{\pi_k},
\]
followed by a greedy policy selection with respect to $\bar Q_{k+1}$:
\[
\pi_{k+1}(s)\in\arg\max_{\mu\in\Delta(\mathcal A)}\mu^\top\bar Q_{k+1}(s).
\]
Equivalently, $\pi_{k+1}$ is any policy satisfying
\[
\mathcal H^{\pi_{k+1}}(\bar Q_{k+1})=\mathcal H(\bar Q_{k+1}).
\]

Within this operator view, Dual-Averaged Policy Iteration is not an approximation to policy iteration. It is the exact zero-smoothing member of the DSPI family [2605.10671].

## 2. Dual-averaged mirror-descent interpretation

The same algorithm can be written through a dual variable
\[
\theta_{k+1}=\theta_k+\alpha_k\,Q^{\pi_k},
\qquad \alpha_k>0,
\]
with primal recovery
\[
\pi_{k+1}(s)=\arg\max_{\mu\in\Delta(\mathcal A)}\mu^\top\theta_{k+1}(s).
\]
For the zero regularizer, this is the mirror-descent primal map.

The equivalence between the Bellman-operator form and the dual-averaged mirror-descent form is obtained by induction:
\[
\bar Q_{k+1}
=
\frac{\sum_{i=0}^k\alpha_i\,Q^{\pi_i}}{\sum_{i=0}^k\alpha_i}.
\]
If the averaging weights are normalized so that
\[
\beta_k=\frac{\alpha_k}{\sum_{i=0}^k\alpha_i},
\]
then the primal policy recovered from $\theta_{k+1}$ is exactly the greedy policy with respect to $\bar Q_{k+1}$.

This establishes that the algorithm can be understood either as Bellman-operator iteration on a weighted average of past value information or as a dual-averaging procedure over $Q$-functions. The two descriptions coincide exactly in the unregularized case [2605.10671].

## 3. Monotonic improvement and Bellman-residual contraction

The analysis rests on two structural properties. The first is monotonic policy improvement. The paper shows by induction that for every $k$,
\[
Q^{\pi_k}\le \mathcal H^{\pi_{k+1}}\bigl(Q^{\pi_k}\bigr)
\quad\Longrightarrow\quad
Q^{\pi_k}\le Q^{\pi_{k+1}}.
\]
Thus, although the greedy step is taken with respect to an averaged $Q$-function, the sequence of policies remains monotonically improving in the standard ordering on action-value functions.

The second property is contraction of the Bellman residual. Defining
\[
r_k:=Q^*-\mathcal H^{\pi_k}(Q^*)\ge 0,
\]
the analysis proves
\[
\|r_{k+1}\|_\infty
\le
\bigl(1-(1-\gamma)\beta_k\bigr)\,\|r_k\|_\infty.
\]
When $\beta_k\equiv\beta$, the residual therefore decreases geometrically.

These two facts are the basis for both asymptotic convergence and finite termination. The paper explicitly emphasizes that Dual-Averaged Policy Iteration shares the same two pillars as Howard’s PI—monotonic policy improvement and a contraction in the Bellman residual—even though it uses a smoothed average of past $Q$-functions instead of the most recent one [2605.10671].

## 4. Finite termination

For the unregularized greedy case, the main finite-termination statement is formulated as follows. Suppose $\nu\equiv 0$, $\beta_0=1$, and $\beta_k\equiv\beta\in(0,1)$. If a fixed tie-breaking rule is used so that each $\pi_k$ is deterministic, then after at most
\[
n\,(m-1)\,\Bigl\lceil\tfrac1{\beta(1-\gamma)}\ln\bigl(2(1-\gamma)^{-1}\bigr)\Bigr\rceil
\]
iterations Dual-Averaged Policy Iteration has reached an optimal policy.

The proof sketch has three components. First, the residual-contraction bound implies that for
\[
k^*:=\Bigl\lceil\frac1{\beta(1-\gamma)}\ln\bigl(2(1-\gamma)^{-1}\bigr)\Bigr\rceil
\]
one has $\|r_k\|_\infty<\|r_0\|_\infty$ for $k\ge k^*$, equivalently that there exists some state-action pair whose Bellman error strictly decreased. Second, a certificate lemma shows that whenever the Bellman error at a state-action pair decreases, one of the originally greedy actions in some state must have been eliminated from the support of $\pi_k$, and that action will never reappear. Third, since there are at most $n(m-1)$ suboptimal actions to eliminate and at least one is eliminated every $k^*$ steps, the stated bound follows.

A common concern is that averaging past $Q$-functions might destroy the combinatorial argument used in classical policy iteration. The paper directly rejects that conclusion: the averaging does not destroy the elimination-of-suboptimal-actions argument, because eventually the averaged $Q$-values separate the best action in each state and they remain distinct thereafter. In this sense, Dual-Averaged Policy Iteration is described as a lazy, or inertial, version of classical PI that nevertheless retains its finite-termination property [2605.10671].

## 5. Position within the DSPI framework

DSPI is introduced as a Bellman-operator framework that includes policy iteration, Dual-Averaged Policy Iteration, natural policy gradient, and more general policy dual averaging methods as special cases. The unregularized greedy limit gives Dual-Averaged Policy Iteration; strictly positive smoothing gives the regularized variants.

Using only monotonicity and contraction of smoothed Bellman operators, the DSPI analysis proves distribution-free global geometric convergence. A consequence is that standard natural policy gradient and policy dual averaging achieve an iteration complexity of
\[
\mathcal O\bigl((1-\gamma)^{-1}\log((1-\gamma)^{-1}\epsilon^{-1})\bigr)
\]
for computing an $\epsilon$-optimal policy, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes.

The contrast with Dual-Averaged Policy Iteration is explicit. Natural policy gradient with entropy regularization approximates PI and DPI, but does not terminate in finitely many steps because the entropy smoothing remains strictly positive at every iteration. The unregularized case, by contrast, exactly recovers a finite-termination PI-like procedure mediated through a running average of $Q$-functions. The same Bellman-operator framework is also stated to extend to discounted MDPs with linear function approximation and stochastic shortest path problems [2605.10671].

## 6. Continuous-action policy dual averaging and actor acceleration

A closely related line of work studies Policy Dual Averaging in continuous state and action spaces. In "Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces" [2603.10199], the setting is an infinite-horizon discounted MDP $(\mathcal S,\mathcal U,P,c,\gamma)$ with continuous state space $\mathcal S\subset\mathbb R^{n_s}$ and action space $\mathcal U\subset\mathbb R^{n_u}$, closed and convex. There, Policy Dual Averaging is presented as a Policy Mirror Descent framework that replaces the prox-to-current-policy regularizer by a fixed prox-center $\pi_0$ and accumulates all past advantages in a dual variable.

The cumulative regularized objective at state $s$ is
\[
\Psi_k(s,a):=\sum_{t=0}^k \beta_t\,\psi^{\pi_t}(s,a)+\lambda_k\,D\bigl(a,\pi_0(s)\bigr),
\]
and the exact update is
\[
\pi_{k+1}(s)=\arg\min_{a\in\mathcal U}\Psi_k(s,a).
\]
With
\[
G_k(s,a)=\sum_{t=0}^k\beta_t\,\psi^{\pi_t}(s,a),
\]
the primal update may be written as
\[
\pi_{k+1}(s)
=
\arg\min_{a\in\mathcal U}\{G_k(s,a)+\lambda_k D(a,\pi_0(s))\}
=
\nabla\omega^*\Bigl(\nabla\omega(\pi_0(s))-\tfrac1{\lambda_k}\,\partial_a G_k(s,\cdot)\Bigr),
\]
where $\omega^*$ is the convex conjugate of the 1-strongly convex generator $\omega$. Two cases are singled out: the Euclidean norm, with a projected update, and the KL-regularizer, with
\[
\pi_{k+1}(a\mid s)\propto \pi_0(a\mid s)\,\exp\!\bigl(-\tfrac1{\lambda_k}G_k(s,a)\bigr).
\]

The same report develops actor-accelerated PDA for the case where solving the exact statewise optimization is computationally expensive. A learned policy network approximates the solution of the optimization sub-problems, and a sum-advantage network $\psi^\Sigma(s,a;\theta_\psi)$ approximates the scaled dual $G_k(s,a)/\bar\beta_k$. Under the stated assumptions, the convex-case theorem yields $O(1/k)$ convergence up to $\varsigma$ and $\epsilon$ when $\beta_t=t+1$ and $\lambda_t=\tilde\mu_d$, while the nonconvex-case theorem states that a near-stationary point in the advantage landscape is found in $O(1/\epsilon)$ iterations. The actor approximation appears as an additional optimality-gap term $\epsilon_{opt,k}(s)\le \epsilon$, so convergence degrades gracefully in $\epsilon$.

This continuous-action literature is not identical to Dual-Averaged Policy Iteration in the finite unregularized Bellman-operator sense. However, it clarifies the broader policy dual averaging family to which DPI belongs. A plausible implication is that the zero-regularizer finite-termination result isolates a distinctly discrete, greedy limit within a larger mirror-descent and dual-averaging landscape [2603.10199].

Source: https://www.emergentmind.com/topics/dual-averaged-policy-iteration