Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework

Abstract: In this work, we show that natural policy gradient, a core algorithm in reinforcement learning, admits an exact formulation as a smoothed and averaged form of policy iteration. Specifically, we introduce doubly smoothed policy iteration (DSPI), a Bellman-operator framework in which each policy is obtained by applying a regularized greedy step to a weighted average of past $Q$-functions. DSPI includes policy iteration, dual-averaged policy iteration, natural policy gradient, and more general policy dual averaging methods as special cases. Using only monotonicity and contraction of smoothed Bellman operators, we prove distribution-free global geometric convergence of DSPI. Consequently, standard natural policy gradient and policy dual averaging achieve an iteration complexity of $\mathcal{O}((1-γ)^{-1}\log((1-γ)^{-1}ε^{-1}))$ for computing an $ε$-optimal policy, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes. For the unregularized greedy case, corresponding to dual-averaged policy iteration, we also prove finite termination. The same Bellman-operator framework further extends to discounted MDPs with linear function approximation and stochastic shortest path problems.

• The paper establishes DSPI as a unifying framework that connects natural policy gradient and dual-averaging with classical policy iteration through Bellman operators.
• The paper demonstrates global geometric convergence of DSPI, showing that NPG and PDA attain optimal iteration complexity without adaptive stepsizes or extraneous regularization.
• The paper’s framework extends to MDPs with linear function approximation and stochastic shortest path problems, indicating broad practical applications.

Natural Policy Gradient as Doubly Smoothed Policy Iteration: A Bellman-Operator Framework

Context and Motivation

The natural policy gradient (NPG) algorithm has emerged as a central method for reinforcement learning (RL) within Markov Decision Processes (MDPs), particularly for scalable applications and as a foundation for advanced actor-critic methods. While prior theoretical analyses of NPG fall into two camps—treating NPG as a continuous optimization procedure, or as a non-exact policy iteration method—there has been a lack of a unified perspective that directly connects classical policy iteration (PI) and modern gradient-based methods such as NPG and policy dual averaging (PDA).

Nanda and Chen introduce doubly smoothed policy iteration (DSPI), a Bellman-operator-based framework that encompasses PI, dual-averaged PI, NPG, PDA, and more. DSPI offers a principled path towards a unified convergence analysis for a broad class of policy optimization algorithms, underpinned by the contraction and monotonicity properties central to dynamic programming.

Doubly Smoothed Policy Iteration (DSPI) and Theoretical Unification

DSPI generalizes policy optimization by two forms of smoothing: (1) policies are updated based on a weighted average of past $Q$-functions rather than a single one, and (2) the improvements are performed via regularized greedy steps, e.g., by entropy regularization, which introduces a parameterized smoothing in the Bellman operator. The central DSPI update can be instantiated with varying policy update rules and smoothing functions (e.g., entropy, Tsallis, or none), which mounts a spectrum from classical PI (no smoothing, greedy in $Q$) to NPG/PDA (with entropy or other regularizers).

A remarkable result is the equivalence between NPG and a particular instance of DSPI. That is, under suitable parameterization, NPG’s iterates correspond exactly to those of DSPI with appropriate stepsizes and entropy regularization. PDA, based on other divergence-generating functions, is similarly subsumed.

Geometric Convergence and Finite-Termination Properties

Leveraging the contraction and monotonicity of smoothed Bellman operators, a global geometric convergence theorem is established for the entire DSPI class. Under constant stepsize $\beta$, the value function gap at iteration $k$ contracts as

$$\|V^{\pi_k} - V^*\|_\infty \leq (1 - (1-\gamma)\beta)^{k-1} \left(\gamma\|V^* - V^{\pi_0}\|_\infty + \tau\nu_{\max}\right)$$

where $\nu_{\max}$ bounds the smoothing functional, $\tau$ is a smoothing coefficient, and $\gamma$ is the discount factor.

Important numerical consequences:

• For NPG and PDA, the iteration complexity to compute an $\epsilon$-optimal policy is $$\mathcal{O}\left((1-\gamma)^{-1}\log((1-\gamma)^{-1}\epsilon^{-1})\right)$$—this matches the best-known rates but, crucially, requires no adaptive stepsize, no extraneous regularization, and applies distribution-free (i.e., regardless of the initial state distribution or concentrability coefficients).
• For dual-averaged PI (no smoothing), DSPI yields a proof of finite termination: after at most $Q$0 iterations (where $Q$1 are state, action space sizes), the algorithm finds the optimal policy.

Notably, these results hold not just for tabular MDPs but, modulo an error term, extend to MDPs with linear function approximation and to stochastic shortest path problems.

Contrasts with Prior Work

The analysis here resolves several limitations in prior literature. Existing optimization-theoretic analyses have typically required problem-dependent stepsizes or additional regularizers to induce strong convexity (cf. Polyak-Lojasiewicz or strong convexity arguments). Distribution mismatches and concentrability constants often contaminate theoretical rates, impeding interpretation or application. On the other hand, prior policy iteration–based analyses have demanded trajectory-dependent stepsize adaptation to control errors. This framework avoids these pitfalls via a direct operator-theoretic approach—using only monotonicity and contraction, relying neither on value-advantage gap–dependent stepsizes nor on penalty schedules derived from the problem trajectory.

In summary, the DSPI view establishes NPG and related methods as smoothed variants of policy iteration, with precise operator contracts and strong geometric convergence, under uniform analytic machinery.

Implications and Future Directions

By connecting gradient-based policy optimization algorithms with dynamic-programming paradigms through the DSPI formalism, the work creates substantial theoretical clarity. In practical terms, this suggests that robust convergence rates are available for NPG-like policy optimization, even without "curvature engineering" or delicate stepsize tuning, so long as the updates conform to DSPI structure.

Theoretical implications:

• The analysis provides tools for extending sharp non-asymptotic convergence rates to model-free RL and generalized policy classes.
• The contraction-based proofs may streamline or supersede hybrid approaches blending optimization and dynamic programming concepts.
• Extensions to function approximation regimes and stochastic shortest path tasks are made formally accessible within the same operator view.

Open questions and future research:

The current results presume access to exact $Q$2-functions at each policy iterate (i.e., a model-based or tabular setting). An important direction is to extend sharp global convergence and sample complexity results for model-free RL, where $Q$3 is approximated via TD-learning or Monte Carlo methods. Achieving minimax-optimal sample complexity, especially in actor-critic settings, within the DSPI paradigm remains open and would represent a significant advance.

Conclusion

Doubly smoothed policy iteration creates a unified operator-theoretic bridge between classical policy iteration and modern natural policy gradient (and PDA) methods, providing strong non-asymptotic convergence guarantees that are both distribution-free and practically realizable. Through Bellman operator contraction and monotonicity, the analysis bypasses the need for adaptive step rules and regularization, and generalizes to broader function-approximation and stochastic shortest path settings. The framework opens paths to further advances in understanding and scaling policy optimization in modern RL.