Dual-Averaged Policy Iteration

• Dual-Averaged Policy Iteration is an unregularized DSPI variant that uses a running average of Q-functions for greedy policy updates, ensuring consistent improvement across iterations.
• It provides a dual-averaged mirror descent perspective that links Bellman operator iterations with dual variable updates to recover the exact greedy policy.
• The approach guarantees finite termination and geometric contraction of the Bellman residual, paralleling classical policy iteration in achieving optimal policies.

Searching arXiv for the cited papers to ground the article in current sources. {"^^^^^^^^2query2 OR \2^^^^^^^^"Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework\"","max_results":5} {"^^^^^^^^2query2 to=arxiv_search 大发游戏 ＿天天json code {"^^^^^^^^2query2 to=arxiv_search 彩神争霸大发json code {"^^^^^^^^2query2 Dual-Averaged Policy Iteration is the unregularized, dual-averaged instantiation of Doubly-Smoothed Policy Iteration (DSPI) in a finite discounted Markov decision process. In this formulation, each policy is obtained by taking a greedy step with respect to a running average of past PRESERVED_PLACEHOLDER_^^^^2query2^^^^-functions rather than with respect to only the most recent evaluation. In the Bellman-operator framework of "Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework" (&&&^^^^2query2^^^^&&&), this zero-regularizer case exactly recovers a policy-iteration-like procedure that admits monotonic policy improvement, Bellman-residual contraction, and a finite-termination theorem.

^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^. Bellman-operator form

The relevant setting is a finite discounted MDP PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^. The policy-evaluation Bellman operator is

$$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$

and the optimality operator is

$$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$

DSPI maintains a running average

$$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$

and then selects $\pi_{k+1}$ through a regularized greedy step. The framework uses the smoothed optimality operator

$$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$

Dual-Averaged Policy Iteration arises by turning off the regularization, i.e. by choosing $\nu\equiv 0$. In that case, $\mathcal H_\eta\equiv\mathcal H$ and $\mathcal H^\pi_\eta\equiv\mathcal H^\pi$, and DSPI becomes the update pair

PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2query2^^^^

followed by a greedy policy selection with respect to PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2id:(Nanda et al., 11 May 2026) OR \2^^^^: PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \22^^^^ Equivalently, PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^3 is any policy satisfying

PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^4

Within this operator view, Dual-Averaged Policy Iteration is not an approximation to policy iteration. It is the exact zero-smoothing member of the DSPI family (&&&^^^^2query2^^^^&&&).

2. Dual-averaged mirror-descent interpretation

The same algorithm can be written through a dual variable

PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^5

with primal recovery

PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^6

For the zero regularizer, this is the mirror-descent primal map.

The equivalence between the Bellman-operator form and the dual-averaged mirror-descent form is obtained by induction: PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^7 If the averaging weights are normalized so that

PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^8

then the primal policy recovered from PRESERVED_PLACEHOLDER_^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^9 is exactly the greedy policy with respect to $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$^^^^2query2^^^^.

This establishes that the algorithm can be understood either as Bellman-operator iteration on a weighted average of past value information or as a dual-averaging procedure over $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^-functions. The two descriptions coincide exactly in the unregularized case (&&&^^^^2query2^^^^&&&).

3. Monotonic improvement and Bellman-residual contraction

The analysis rests on two structural properties. The first is monotonic policy improvement. The paper shows by induction that for every $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$2,

$$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$3

Thus, although the greedy step is taken with respect to an averaged $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$4-function, the sequence of policies remains monotonically improving in the standard ordering on action-value functions.

The second property is contraction of the Bellman residual. Defining

$$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$5

the analysis proves

$$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$6

When $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$7, the residual therefore decreases geometrically.

These two facts are the basis for both asymptotic convergence and finite termination. The paper explicitly emphasizes that Dual-Averaged Policy Iteration shares the same two pillars as Howard’s PI—monotonic policy improvement and a contraction in the Bellman residual—even though it uses a smoothed average of past $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$8-functions instead of the most recent one (&&&^^^^2query2^^^^&&&).

4. Finite termination

For the unregularized greedy case, the main finite-termination statement is formulated as follows. Suppose $$[\mathcal H^\pi Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\sum_{a'}\pi(a'\mid s')\,Q(s',a'),$$9, $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$^^^^2query2^^^^, and $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^. If a fixed tie-breaking rule is used so that each $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$2 is deterministic, then after at most

$$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$3

iterations Dual-Averaged Policy Iteration has reached an optimal policy.

The proof sketch has three components. First, the residual-contraction bound implies that for

$$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$4

one has $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$5 for $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$6, equivalently that there exists some state-action pair whose Bellman error strictly decreased. Second, a certificate lemma shows that whenever the Bellman error at a state-action pair decreases, one of the originally greedy actions in some state must have been eliminated from the support of $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$7, and that action will never reappear. Third, since there are at most $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$8 suboptimal actions to eliminate and at least one is eliminated every $$[\mathcal H Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a)\max_{a'}Q(s',a').$$9 steps, the stated bound follows.

A common concern is that averaging past $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$^^^^2query2^^^^-functions might destroy the combinatorial argument used in classical policy iteration. The paper directly rejects that conclusion: the averaging does not destroy the elimination-of-suboptimal-actions argument, because eventually the averaged $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^-values separate the best action in each state and they remain distinct thereafter. In this sense, Dual-Averaged Policy Iteration is described as a lazy, or inertial, version of classical PI that nevertheless retains its finite-termination property (&&&^^^^2query2^^^^&&&).

5. Position within the DSPI framework

DSPI is introduced as a Bellman-operator framework that includes policy iteration, Dual-Averaged Policy Iteration, natural policy gradient, and more general policy dual averaging methods as special cases. The unregularized greedy limit gives Dual-Averaged Policy Iteration; strictly positive smoothing gives the regularized variants.

Using only monotonicity and contraction of smoothed Bellman operators, the DSPI analysis proves distribution-free global geometric convergence. A consequence is that standard natural policy gradient and policy dual averaging achieve an iteration complexity of

$$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$2

for computing an $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$3-optimal policy, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes.

The contrast with Dual-Averaged Policy Iteration is explicit. Natural policy gradient with entropy regularization approximates PI and DPI, but does not terminate in finitely many steps because the entropy smoothing remains strictly positive at every iteration. The unregularized case, by contrast, exactly recovers a finite-termination PI-like procedure mediated through a running average of $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$4-functions. The same Bellman-operator framework is also stated to extend to discounted MDPs with linear function approximation and stochastic shortest path problems (&&&^^^^2query2^^^^&&&).

6. Continuous-action policy dual averaging and actor acceleration

A closely related line of work studies Policy Dual Averaging in continuous state and action spaces. In "Actor-Accelerated Policy Dual Averaging for Reinforcement Learning in Continuous Action Spaces" (Gao et al., 10 Mar 2026), the setting is an infinite-horizon discounted MDP $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$5 with continuous state space $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$6 and action space $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$7, closed and convex. There, Policy Dual Averaging is presented as a Policy Mirror Descent framework that replaces the prox-to-current-policy regularizer by a fixed prox-center $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$8 and accumulates all past advantages in a dual variable.

The cumulative regularized objective at state $$\bar Q_{k+1}=(1-\beta_k)\,\bar Q_k+\beta_k\,Q^{\pi_k},$$9 is

$\pi_{k+1}$^^^^2query2^^^^

and the exact update is

$\pi_{k+1}$^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^

With

$\pi_{k+1}$2

the primal update may be written as

$\pi_{k+1}$3

where $\pi_{k+1}$4 is the convex conjugate of the ^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^-strongly convex generator $\pi_{k+1}$5. Two cases are singled out: the Euclidean norm, with a projected update, and the KL-regularizer, with

$\pi_{k+1}$6

The same report develops actor-accelerated PDA for the case where solving the exact statewise optimization is computationally expensive. A learned policy network approximates the solution of the optimization sub-problems, and a sum-advantage network $\pi_{k+1}$7 approximates the scaled dual $\pi_{k+1}$8. Under the stated assumptions, the convex-case theorem yields $\pi_{k+1}$9 convergence up to $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$^^^^2query2^^^^ and $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$^^^^2id:(Nanda et al., 11 May 2026) OR \2^^^^ when $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$2 and $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$3, while the nonconvex-case theorem states that a near-stationary point in the advantage landscape is found in $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$4 iterations. The actor approximation appears as an additional optimality-gap term $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$5, so convergence degrades gracefully in $$[\mathcal H_\eta Q](s,a) = \mathcal R(s,a) + \gamma\sum_{s'}p(s'\mid s,a) \max_{\mu\in\Delta(\mathcal A)} \{\mu^\top Q(s')+\eta\,\nu(\mu)\}.$$6.

This continuous-action literature is not identical to Dual-Averaged Policy Iteration in the finite unregularized Bellman-operator sense. However, it clarifies the broader policy dual averaging family to which DPI belongs. A plausible implication is that the zero-regularizer finite-termination result isolates a distinctly discrete, greedy limit within a larger mirror-descent and dual-averaging landscape (Gao et al., 10 Mar 2026).