Policy Dual Averaging: Overview & Applications

• Policy dual averaging is a method that updates policies by aggregating cumulative dual signals with regularization, ensuring stable and globally informed decisions.
• It applies to both discrete and continuous action spaces using mechanisms like entropy-regularized softmax, hierarchical refinement, and actor-accelerated optimization.
• The approach unifies classical dual averaging with modern reinforcement learning and adversarial methods, offering regret bounds and implicit proximal regularization benefits.

Searching arXiv for recent and foundational papers on policy dual averaging and dual averaging. Policy dual averaging denotes a family of update rules in which the optimized object is a policy or policy-defining control variable, past first-order or bandit feedback is accumulated in a dual representation, and the next policy is obtained by a regularized primal mapping rather than by a purely local step. In the supplied literature, the term ranges from exact dual averaging on the simplex underlying Hedge (Baes et al., 2011), to hierarchical policy updates over distributions in adversarial zeroth-order learning (Héliou et al., 2021), to reinforcement-learning policy improvement via cumulative advantage minimization in continuous action spaces (Gao et al., 10 Mar 2026). Closely related work on dual averaging in deep learning and nonconvex stochastic optimization clarifies the core mechanism: accumulated gradients induce an implicit, time-varying regularization centered at a reference point, which provides a useful lens for interpreting policy updates even when the underlying papers are not themselves about reinforcement learning (Jelassi et al., 2020, Liu et al., 27 May 2025).

1. Definition and conceptual scope

Policy dual averaging is best understood as the policy-space analogue of classical dual averaging. Classical dual averaging maintains an accumulated dual variable and computes the next primal iterate by solving a linearized objective regularized by a prox term. In the Euclidean unconstrained case discussed in modern deep-learning work, the update is

$s_k = s_{k-1} + \lambda_k g_k,$

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$

with closed form

$x_{k+1} = x_0 - \frac{s_k}{\beta_k},$

so the iterate is determined by the entire accumulated dual signal rather than only by the latest gradient (Jelassi et al., 2020).

In policy settings, the same pattern appears with the decision variable interpreted as a stochastic policy, an action distribution, or a policy-defining dual control vector. On the simplex, the policy is a probability vector $x_t\in\Delta_n$, the dual variable is the cumulative weighted loss vector, and the update is an entropy-regularized best response (Baes et al., 2011). In adversarial zeroth-order optimization over a continuous action space, the updated object is a mixed strategy $q_t$, and the policy update has the explicit dual-averaging form

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$

where $Q$ is the regularized choice map on the policy class (Héliou et al., 2021). In continuous-action reinforcement learning, policy dual averaging updates the policy statewise by minimizing a cumulative regularized advantage surrogate,

$$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$

$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$

so the new policy is defined by accumulated policy-improvement signals relative to a fixed prox-center $\hat\pi_0=\pi_0$ (Gao et al., 10 Mar 2026).

A common conceptual distinction throughout this literature is the contrast with mirror descent or local policy-gradient steps. Mirror-descent-style methods regularize relative to the current iterate; dual averaging regularizes relative to a fixed reference while aggregating all previous feedback. This difference is explicit in the comparison between Policy Mirror Descent and Policy Dual Averaging in continuous-action reinforcement learning, where PDA regularizes with respect to the distance between the new policy and the initial policy, whereas PMD regularizes with respect to the distance between the new policy and the current policy (Gao et al., 10 Mar 2026).

2. Classical dual averaging foundations for policy updates

The most direct finite-dimensional policy interpretation appears in the identification of Hedge as a dual averaging method on the simplex (Baes et al., 2011). The online decision set is

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$0

so $$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$1 is simultaneously a portfolio vector, an expert-mixture weight vector, and a stochastic policy over $$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$2 discrete actions. Dual averaging maintains

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$3

and maps this dual average to a policy through

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$4

With the entropy-type prox

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$5

the mirror operator becomes the softmax map

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$6

so the policy is an exponential-weights distribution over cumulative scores (Baes et al., 2011).

This derivation yields an exact bridge between multiplicative-weights policy updates and dual averaging. If losses satisfy $$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$7 and are transformed by

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$8

then the unnormalized Hedge weights evolve as

$$x_{k+1} = \arg\min_{x\in\mathbb{R}^n} \left\{ \langle s_k, x\rangle + \frac{\beta_k}{2}\|x-x_0\|_2^2 \right\},$$9

Under the specialization

$x_{k+1} = x_0 - \frac{s_k}{\beta_k},$0

this becomes

$x_{k+1} = x_0 - \frac{s_k}{\beta_k},$1

which is exactly Hedge (Baes et al., 2011).

This simplex case is foundational for policy dual averaging because it makes precise the generic structure: maintain cumulative action scores in dual space; recover a policy by a regularized projection; use entropy when the policy class is a simplex; and tune the dual weights $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$2 and projection scale $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$3 to trade off stability and responsiveness. The same paper also shows that increasing $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$4 yields recency-weighted regret guarantees, which suggests a route to nonstationary policy updates when recent feedback should matter more than stale feedback (Baes et al., 2011).

A broader optimization-theoretic perspective comes from unified analyses of mirror descent and dual averaging. In that framework, dual averaging is the “lazy” endpoint of a family in which the primal point is always recovered from a dual variable via

$x_{k+1} = x_0 - \frac{s_k}{\beta_k},$5

but the dual state can either be a cumulative sum, as in DA, or a more local representation, as in MD (Juditsky et al., 2019). This suggests that policy dual averaging and policy mirror descent are not disjoint constructions but endpoints of a larger design family.

3. Policy dual averaging on the simplex and in continuous policy spaces

In discrete action spaces, the simplex formulation already yields a complete policy dual averaging recipe. The policy is $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$6, the loss of the policy is $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$7, and entropy regularization produces a softmax over cumulative weighted losses (Baes et al., 2011). This is policy dual averaging in its most literal form.

A more elaborate policy-space construction appears in hierarchical dual averaging for adversarial zeroth-order non-convex optimization (Héliou et al., 2021). There, the policy is not a point in $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$8 but a mixed strategy $x_{k+1} = x_0 - \frac{s_k}{\beta_k},$9 over a continuous action space $x_t\in\Delta_n$0. The learner samples $x_t\in\Delta_n$1, observes only the scalar payoff $x_t\in\Delta_n$2, and updates the policy distribution rather than a deterministic action. For a fixed measurable partition $x_t\in\Delta_n$3, the feasible policy class is

$x_t\in\Delta_n$4

and the dual-averaging update is

$x_t\in\Delta_n$5

Because the policy is a distribution, not a direct point decision, randomization is intrinsic rather than auxiliary (Héliou et al., 2021).

The hierarchical construction replaces a fixed discretization by a refining sequence of covers $x_t\in\Delta_n$6. If $x_t\in\Delta_n$7 is the number of cells in the current cover, then the paper uses a logarithmic schedule

$x_t\in\Delta_n$8

which implies

$x_t\in\Delta_n$9

This creates an evolving bias-variance compromise: coarse covers at early times reduce estimator variance, while refinement later reduces approximation error. The resulting update is

$q_t$0

with partition refinement at scheduled split times (Héliou et al., 2021).

A different continuous policy-space instantiation is the reinforcement-learning formulation in which policies are deterministic maps $q_t$1 over continuous states and actions (Gao et al., 10 Mar 2026). The optimization target is

$q_t$2

and the improvement signal is the approximate advantage-like quantity

$q_t$3

Policy dual averaging then updates the policy by minimizing the cumulative surrogate $q_t$4 defined above. This is a direct policy-space realization of dual averaging: cumulative policy-improvement information is aggregated in dual form, and the next policy is the regularized minimizer of that aggregate (Gao et al., 10 Mar 2026).

4. Geometry, regularization, and implicit stabilization

A central theme in policy dual averaging is that the accumulated dual signal must be tempered by a regularizer. On the simplex, negative entropy gives the softmax policy map and induces exploration-smoothing geometry appropriate to probability distributions (Baes et al., 2011). In hierarchical dual averaging, the regularizer is analyzed relative to the Fisher information metric rather than a fixed ambient norm. The primal Fisher norm is

$q_t$5

and the corresponding dual norm is

$q_t$6

This geometry is state-dependent and is explicitly motivated by the variance structure of importance-weighted zeroth-order estimators (Héliou et al., 2021).

The Fisher geometry differs materially from Euclidean regularization. Rarely sampled cells are penalized sharply in the primal metric and weakly in the dual metric, which is precisely aligned with the behavior of importance-weighted payoff estimators. This is why the paper can control second moments in the Fisher dual norm even when no comparable global norm bound is available (Héliou et al., 2021). A plausible implication is that policy dual averaging on stochastic-policy spaces is often best formulated in an information-geometric metric rather than in Euclidean coordinates.

In Euclidean deep-learning dual averaging, the regularization role is made explicit through an exact equivalence with stochastic gradient descent on a sequence of regularized objectives (Jelassi et al., 2020). If

$q_t$7

then the $q_t$8-th dual-averaging update on $q_t$9 is equivalent to SGD on $$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$0 provided

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$1

Under the Modernized Dual Averaging parameterization

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$2

this yields a decaying uncentered $$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$3-regularization centered at initialization (Jelassi et al., 2020). For policy dual averaging, this does not constitute an RL theorem, but it clarifies a transferable mechanism: dual averaging can be read as optimization with a time-varying proximal penalty around a reference policy parameter vector.

The same perspective is sharpened in recent nonconvex theory. In Euclidean stochastic dual averaging,

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$4

is exactly SGD on the surrogate sequence

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$5

so accumulated gradients induce an implicit time-varying quadratic regularization (Liu et al., 27 May 2025). This suggests that policy dual averaging should often be interpreted not merely as gradient accumulation but as repeated optimization of regularized policy surrogates.

5. Reinforcement-learning formulations

The most explicit reinforcement-learning realization in the supplied material is actor-accelerated Policy Dual Averaging for continuous action spaces (Gao et al., 10 Mar 2026). The paper defines the performance-difference quantity

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$6

and uses the performance difference lemma

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$7

The exact PDA update is the statewise minimization of the cumulative regularized surrogate $$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$8, with regularizer

$$S_{t+1} = S_t + \hat u_t,\qquad q_{t+1}=Q(\eta_{t+1}S_{t+1}),$$9

In the implementation, the Euclidean choice

$Q$0

is used (Gao et al., 10 Mar 2026).

The paper’s principal motivation is computational. In continuous action spaces, exact PDA requires solving

$Q$1

for each encountered state, which is prohibitively slow when done online. Actor acceleration replaces repeated exact minimization by a learned actor $Q$2 satisfying

$Q$3

The approximation is formalized by the optimality-gap condition

$Q$4

with bounded error $Q$5 (Gao et al., 10 Mar 2026).

The practical algorithm uses three learned components: a value network $Q$6, a cumulative advantage network $Q$7, and an actor network $Q$8. To stabilize the cumulative objective numerically, the paper introduces the scaled running average

$Q$9

and uses the schedule

$$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$0

The actor is then trained by minimizing

$$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$1

evaluated at $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$2 (Gao et al., 10 Mar 2026).

The theory distinguishes favorable convex or weakly convex action-subproblem structure from the genuinely nonconvex case. In the convex regime, the paper gives weighted performance bounds with additive error terms reflecting critic approximation $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$3 and actor approximation $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$4. In the $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$5 case, with $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$6 and $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$7, the asymptotic error floor is on the order of $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$8, while actor approximation contributes vanishing $$\pi_{k+1}(s) =\arg\min_{a \in \mathbb{U}} \tilde{\Psi}_k(s, a),$$9 and $$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$0 terms (Gao et al., 10 Mar 2026). In the nonconvex case, the guarantee becomes a negative-advantage bound rather than a full global-convergence statement.

The empirical scope is broad. The method is evaluated on MuJoCo-v4/v5, Box2D, and several OR-Gym environments, and actor-accelerated PDA is reported to outperform PPO on many continuous-control tasks while remaining runtime-competitive (Gao et al., 10 Mar 2026). Those findings concern this particular PDA instantiation rather than policy dual averaging in general.

6. Variants, guarantees, and limitations

The supplied literature shows that “policy dual averaging” is not a single algorithm but a structural template with several distinct regimes.

Regime	Updated object	Core update
Hedge / simplex DA	$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$1	softmax of cumulative weighted losses
Hierarchical DA	mixed strategy $$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$2	$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$3 on refining covers
RL PDA	policy $$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$4	statewise minimization of cumulative regularized advantages

For discrete-action online learning, the theory is regret-based. The dual-averaging reinterpretation of Hedge yields standard and improved regret guarantees, including the bound

$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$5

for the paper’s “Optimal Hedge” tuning (Baes et al., 2011). For adversarial zeroth-order continuous control, hierarchical exponential weights achieves

$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$6

for static regret and

$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$7

under the specified tuning, with the static rate identified as minimax optimal (Héliou et al., 2021). For RL PDA, the guarantees are finite-time performance bounds under Lipschitzness, weak convexity, bounded deterministic critic approximation error, and bounded actor optimality gap (Gao et al., 10 Mar 2026).

There are also relevant non-policy results that matter for the conceptual foundation. Modernized Dual Averaging in supervised deep learning shows that a non-adaptive, momentum-augmented dual-averaging method can match SGD+momentum on vision tasks and Adam on several transformer/NLP tasks while being interpretable as SGD with decaying uncentered $$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$8-regularization (Jelassi et al., 2020). More recent theory proves that stochastic dual averaging converges for smooth nonconvex stochastic optimization at rate

$$\tilde{\Psi}_k(s, a) := \sum_{t=0}^k \beta_t \tilde{\psi}(s, a; \theta_t) + \lambda_k D(\hat\pi_0(s), a),$$9

establishing that dual averaging is not confined to convex analysis (Liu et al., 27 May 2025). These papers do not establish RL-specific policy claims, but they materially strengthen the general case for dual averaging as a modern nonconvex optimization primitive.

The limitations are correspondingly regime-specific. The RL paper requires assumptions on Lipschitz continuity, weak convexity of the approximate advantage surrogate, bounded deterministic critic approximation error, and bounded actor optimality gap, and the nonconvex case remains incomplete in the sense that the paper proves a negative-advantage guarantee rather than full global convergence (Gao et al., 10 Mar 2026). The hierarchical zeroth-order theory applies to adversarial online learning with Lipschitz payoffs and randomized policies, not to actor-critic RL (Héliou et al., 2021). The deep-learning and nonconvex-SDA papers work in Euclidean parameter space and do not address KL geometry, Markovian sampling, or biased policy-gradient estimators (Jelassi et al., 2020, Liu et al., 27 May 2025).

Taken together, the literature supports a precise encyclopedia-level characterization. Policy dual averaging is a regularized cumulative-feedback methodology in which the policy is recovered from an accumulated dual signal rather than moved by a purely local step. On the simplex it coincides with entropy-regularized exponential weights; in adversarial continuous domains it becomes distributional dual averaging with hierarchical refinement and Fisher geometry; in continuous-action reinforcement learning it becomes cumulative advantage minimization around a fixed reference policy, with actor acceleration used to amortize the policy-improvement subproblem [(Baes et al., 2011); (Héliou et al., 2021); (Gao et al., 10 Mar 2026)]. The broader dual-averaging literature suggests that its practical effect is often best understood as implicit proximal regularization around a reference point, an interpretation that plausibly unifies these seemingly different policy-update rules (Jelassi et al., 2020, Liu et al., 27 May 2025).