Natural Policy Gradient in RL

• Natural policy gradient is a reinforcement learning method that replaces Euclidean updates with Fisher-preconditioned, geometry-aware policy updates.
• It employs a KL divergence-based metric to ensure consistent policy improvements and reparameterization invariance for reliable convergence.
• Extensions and approximations such as natural actor-critic, COPOS, and K-FAC address computational challenges and broaden its applicability in deep and multi-agent settings.

Searching arXiv for recent and foundational papers on natural policy gradient. Natural policy gradient is a policy-search method in reinforcement learning that replaces Euclidean ascent in parameter space by steepest ascent under the information geometry of policy distributions. In its standard form, a stochastic policy $\pi_\theta$ is optimized for expected return $J(\theta)$ by replacing the vanilla direction $\nabla_\theta J(\theta)$ with the Fisher-preconditioned direction

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$

where the Fisher information matrix provides the local metric on policy space rather than an ordinary flat metric on parameters. In this sense, natural policy gradient is a geometry-aware refinement of policy gradient, and it is treated in the literature as a conceptual foundation for methods such as natural actor-critic, TRPO, and PPO, while also admitting distinct formulations through mirror descent, trust-region optimization, compatible function approximation, Bellman operators, and occupancy-measure geometry (Heeswijk, 2022, Kämmerer, 2019).

1. Geometric basis and policy-space interpretation

The central motivation for natural policy gradient is that reinforcement learning depends on how the policy distribution changes, not on the Euclidean size of a parameter update. Vanilla policy gradient uses

$$\theta \leftarrow \theta+\alpha \nabla_\theta J(\theta),$$

but this update is parameterization dependent: two parameter changes with the same Euclidean norm can induce very different changes in the policy distribution. The lecture note “Natural Policy Gradients In Reinforcement Learning Explained” states that the flaw in vanilla policy gradient is not merely poor learning-rate tuning, but using the wrong geometry; it highlights overshooting, undershooting, and the fact that a bad policy update can degrade future data because data are generated by the current policy (Heeswijk, 2022).

Natural policy gradient addresses this by viewing policies as a statistical manifold and using KL divergence as the local notion of distance between policies. The local curvature of this manifold is captured by the Fisher information matrix,

$$F(\theta)=\mathbb{E}_\theta\!\left[\nabla_\theta \log \pi_\theta(x)\,\nabla_\theta \log \pi_\theta(x)^\top\right].$$

Preconditioning by $F(\theta)^{-1}$ dampens directions that cause large policy changes and amplifies directions that barely affect the policy, so the update becomes the steepest-ascent direction under the Riemannian metric induced by the Fisher matrix rather than under the Euclidean metric. The same paper explicitly links this to reparameterization invariance: a reparameterization may change the ordinary gradient numerically, but the Fisher metric changes in a compensating way, so the natural-gradient direction in policy space remains the same (Heeswijk, 2022).

A complementary exposition in “On Policy Gradients” presents natural gradient through the discounted Fisher matrix of score features,

$$F_\theta = \int_\mathbb{S} d^\pi(s)\int_\mathbb{A}\pi(a\mid s)\,\nabla_\theta \log \pi(a\mid s)\,\nabla_\theta \log \pi(a\mid s)^T\,da\,ds,$$

and states that natural gradient provides a way to iterate through policy space instead of parameter space. That paper does not develop the full KL-manifold derivation, but it makes explicit the same geometric shift from raw parameter coordinates to distribution-aware ascent (Kämmerer, 2019).

2. Equivalent formulations: trust regions, compatibility, and policy iteration

A standard derivation of natural policy gradient begins from a KL-constrained improvement problem: maximize return while preventing the new policy from drifting too far from the old one. Using a first-order Taylor expansion of the objective and a second-order expansion of the KL term yields a local quadratic problem whose optimizer is

$$\Delta\theta \propto F(\theta)^{-1}\nabla_\theta J(\theta).$$

In this formulation, natural policy gradient is the trust-region idea in embryo: step size is determined by a KL budget rather than by an arbitrary Euclidean learning rate, and the resulting update changes the policy by approximately the same magnitude independent of parameterization (Heeswijk, 2022).

A second formulation uses compatible function approximation. If the critic is parameterized as

$f_w^\pi(s,a)=\nabla_\theta \log \pi(a\mid s)^T w,$

then “On Policy Gradients” states that

$J(\theta)$0

so the natural gradient simplifies to

$J(\theta)$1

This identity is the basis of natural actor-critic: instead of explicitly estimating and inverting the Fisher matrix, one fits a compatible critic and directly updates $J(\theta)$2 in the $J(\theta)$3 direction (Kämmerer, 2019).

A stronger equivalence result appears in “Compatible Natural Gradient Policy Search.” For exponential-family policies in their natural parameterization, together with compatible value function approximation, the paper states that natural gradient and KL-constrained trust-region optimization are not merely locally related: for the log-linear parameters, the natural gradient is the exact optimizer of the trust-region problem. The same paper also states that standard natural-gradient updates may reduce entropy according to a wrong schedule, leading to premature convergence, and proposes COPOS to bound entropy loss explicitly (Pajarinen et al., 2019).

Natural policy gradient also admits a dynamic-programming interpretation. “On the Linear convergence of Natural Policy Gradient Algorithm” treats the tabular softmax update

$J(\theta)$4

as a smooth approximation of policy iteration and as a KL-mirror-descent update (Khodadadian et al., 2021). The Bellman-operator framework “Natural Policy Gradient as Doubly Smoothed Policy Iteration” goes further and states that natural policy gradient admits an exact formulation as a smoothed and averaged form of policy iteration, with ordinary policy iteration, dual-averaged policy iteration, natural policy gradient, and more general policy dual averaging appearing as special cases of doubly smoothed policy iteration (Nanda et al., 11 May 2026).

3. Algorithmic structure and computation

In its textbook sample-based form, natural policy gradient proceeds by rolling out the current policy, estimating the policy gradient, estimating the Fisher matrix from the same samples via score outer products, computing the natural direction $J(\theta)$5, scaling the step by a trust-region radius or learning-rate surrogate, and updating $J(\theta)$6. The practical obstacle is that all of these are sample estimates, the derivation is only local, and the Fisher matrix is $J(\theta)$7, so direct inversion is expensive, roughly $J(\theta)$8, and can be numerically brittle. The same exposition identifies conjugate gradients and K-FAC as ways to mitigate these difficulties (Heeswijk, 2022).

In tabular softmax settings, the geometry simplifies substantially. In the Markov potential-game analysis of independent natural policy gradient, the logit-space update is

$J(\theta)$9

and the induced policy-space update is exactly multiplicative weights / exponentiated gradient,

$\nabla_\theta J(\theta)$0

This exact equivalence is central to both the algorithm and its proof (Fox et al., 2021).

The infinite-state queueing analysis studies an exponentiated natural-gradient step with state-dependent learning-rate parameterization,

$\nabla_\theta J(\theta)$1

with $\nabla_\theta J(\theta)$2 chosen from a statewise bound on action-gap differences. This replaces the finite-state style of global stepsize selection by a state-dependent schedule tailored to unbounded value scales (Grosof et al., 2024).

Several papers focus on making natural policy gradients practical in deep reinforcement learning. “Rank-1 Approximation of Inverse Fisher for Natural Policy Gradients in Deep Reinforcement Learning” proposes a rank-1 empirical Fisher with damping and applies the Sherman–Morrison formula to obtain a closed-form inverse-Fisher–vector product in $\nabla_\theta J(\theta)$3 time and memory, integrating the result into actor-critic as SM-ActorCritic (Huo et al., 26 Jan 2026). “Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation” instead reformulates Tikhonov-regularized natural policy gradients as vanilla policy gradients with a transformed advantage, estimates that transformation via randomized block Kaczmarz on on-policy mini-batches, and thereby avoids explicit Fisher construction, explicit inversion, conjugate-gradient solves, and architecture-specific approximations such as K-FAC (Sun, 18 May 2026). A different direction, “Reusing Historical Trajectories in Natural Policy Gradient via Importance Sampling,” studies natural policy gradient with trajectory reuse, showing how reused samples can enter the gradient estimator through importance weights while the inverse Fisher is still estimated from current-policy samples in the main analysis (Lin et al., 2024).

4. Convergence theory

Theoretical analyses of natural policy gradient differ sharply by setting, parameterization, and whether the update is exact, approximate, or sample based. In discounted finite tabular MDPs, one line of work shows that constant-step-size natural policy gradient has more structure than the classical $\nabla_\theta J(\theta)$4 guarantee suggests. “On the Linear convergence of Natural Policy Gradient Algorithm” proves that vanilla natural policy gradient with constant step size has geometric (linear) asymptotic convergence, and that an adaptive-step-size variant has linear convergence from the start and, under stronger regularity assumptions, superlinear convergence, with quadratic convergence as a special case (Khodadadian et al., 2021).

A Bellman-operator analysis strengthens this policy-iteration viewpoint. “Natural Policy Gradient as Doubly Smoothed Policy Iteration” proves distribution-free global geometric convergence for doubly smoothed policy iteration and concludes that standard natural policy gradient and policy dual averaging achieve iteration complexity

$\nabla_\theta J(\theta)$5

for computing an $\nabla_\theta J(\theta)$6-optimal policy, without modifying the MDP, adding regularization beyond the mirror map inherent in the update, or using adaptive, trajectory-dependent stepsizes (Nanda et al., 11 May 2026).

For log-linear policy classes, “Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrization” extends linear guarantees beyond the tabular softmax case. In the deterministic setting, when the $\nabla_\theta J(\theta)$7-value is approximable by a linear combination of known features up to a bias error, a geometrically increasing step size yields linear convergence toward an optimal policy; in the sample-based setting, the same linear guarantees hold up to an error term depending on estimation error, bias error, and the condition number of the feature covariance matrix (Alfano et al., 2022).

A separate line of non-asymptotic analysis studies general smooth parameterizations under Fisher non-degeneracy. “An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods” proves that NPG enjoys global convergence to the globally optimal value up to inherent approximation error due to policy parametrization, with total trajectory complexity

$\nabla_\theta J(\theta)$8

improving the $\nabla_\theta J(\theta)$9-dependence relative to prior NPG analysis (Liu et al., 2022). “Global Convergence of Natural Policy Gradient with Hessian-aided Momentum Variance Reduction” then proposes NPG-HM and states that it achieves global last-iterate $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$0-optimality with sample complexity $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$1 under generic Fisher non-degenerate policy parameterizations (Feng et al., 2024).

Setting	Representative guarantee	Paper
Discounted finite MDP, tabular softmax	Geometric asymptotic convergence; adaptive variant linear from the start	(Khodadadian et al., 2021)
DSPI/Bellman-operator view	$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$2 iteration complexity	(Nanda et al., 11 May 2026)
Log-linear policy parametrization	Linear convergence up to bias/estimation error	(Alfano et al., 2022)
General smooth parameterization	$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$3 total trajectory complexity	(Liu et al., 2022)
Fisher-non-degenerate sample-based NPG-HM	Global last-iterate $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$4-optimality with $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$5 sample complexity	(Feng et al., 2024)

5. Extensions beyond the standard single-agent finite-state setting

Natural policy gradient has been extended beyond the classical single-agent discounted finite-state setting in several directions. In mixed cooperative/competitive stochastic games, “Independent Natural Policy Gradient Always Converges in Markov Potential Games” proves that when each agent independently runs natural policy gradient, the joint policy converges in the last iterate to a Nash equilibrium under a sufficiently small constant stepsize. The key structural fact is that in Markov potential games, independent agent-wise NPG is exactly natural-gradient ascent on a global potential, with block-diagonal Fisher geometry induced by the factorized policy (Fox et al., 2021).

For decentralized fully cooperative multi-agent reinforcement learning, “Decentralized Natural Policy Gradient with Variance Reduction for Collaborative Multi-Agent Reinforcement Learning” develops MDNPG, which combines local natural-gradient preconditioning, momentum-based variance reduction, and gradient tracking over an undirected communication graph. Under the stated assumptions, MDNPG reaches an $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$6-stationary point with sample complexity $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$7, which the paper interprets as linear speedup in the number of agents (Chen et al., 2022).

For countably infinite average-reward problems, “Convergence for Natural Policy Gradient on Infinite-State Queueing MDPs” proves the first convergence-rate theorem for NPG in a general class of infinite-state average-reward MDPs. With a sufficiently good initializer satisfying a quadratic lower bound on the relative value function—MaxWeight in the queueing applications—the NPG algorithm achieves

$$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$8

providing the first convergence rate bound for NPG in this infinite-state setting (Grosof et al., 2024).

Partially observable and non-Markovian control leads to recurrent versions of natural policy gradient. “Recurrent Natural Policy Gradient for POMDPs” extends the policy gradient theorem, Fisher matrix, and compatible-function characterization to recurrent policies in POMDPs, and gives finite-time and finite-width analyses in a near-initialization regime. The paper’s main qualitative conclusion is that RNN-based recurrent NPG is efficient for short-term memory problems but faces explicit difficulties in the presence of long-term dependencies (Cayci et al., 2024).

A different extension changes the geometry itself. “Fisher-Rao Gradient Flows of Linear Programs and State-Action Natural Policy Gradients” studies a natural gradient based on the Fisher information of the state-action distribution $$\tilde{\nabla}_\theta J(\theta)=F(\theta)^{-1}\nabla_\theta J(\theta),$$9 rather than of the policy rows alone. Under this view, the RL problem becomes a linear program over the state-action polytope, the natural gradient is the Fisher–Rao gradient flow of that linear program, and the resulting state-action natural policy gradient enjoys linear convergence in rich/tabular settings and sublinear convergence up to approximation error in perturbed settings (Müller et al., 2024).

6. Relation to modern algorithms, practical limits, and recurrent points of debate

Natural policy gradient is often described as a foundation for contemporary policy optimization, but the relationship is not identity. The 2022 lecture note states that TRPO operationalizes the KL-constrained trust-region idea more robustly through constrained optimization and conjugate-gradient-based second-order methods, while PPO keeps the same underlying goal—avoid overly large policy changes—but replaces the explicit constrained second-order machinery with a simpler first-order surrogate such as clipping or penalizing policy-ratio changes. In that sense, PPO is not identical to natural policy gradient because it does not explicitly compute $$\theta \leftarrow \theta+\alpha \nabla_\theta J(\theta),$$0, even though it inherits the same principle that bounded changes in policy space are preferable to naive Euclidean steps in parameter space (Heeswijk, 2022).

A recurrent criticism concerns entropy dynamics. “Compatible Natural Gradient Policy Search” argues that standard natural-gradient updates may reduce the entropy of the policy according to a wrong schedule, leading to premature convergence. Its response, COPOS, augments the KL-constrained formulation with an explicit entropy-loss bound, thereby separating reward improvement from entropy control rather than letting entropy decay be an uncontrolled side effect of the natural-gradient update (Pajarinen et al., 2019).

Another practical debate concerns how faithfully one must approximate the Fisher geometry. The rank-1 Sherman–Morrison approximation of SMAC has linear time and memory and strong empirical results on several continuous-control tasks, but the same paper emphasizes that a rank-1 empirical Fisher captures only one curvature direction and that its “faster than PG” claim is stronger than the formal theorem, whose trajectory complexity matches the order of stochastic policy gradient analyses rather than improving the asymptotic order (Huo et al., 26 Jan 2026). RAT, by contrast, avoids explicit Fisher inversion and architecture-specific curvature approximations, but it still forms the mini-batch Gram matrix $$\theta \leftarrow \theta+\alpha \nabla_\theta J(\theta),$$1 at cost $$\theta \leftarrow \theta+\alpha \nabla_\theta J(\theta),$$2, so it is not a zero-cost natural-gradient method; it is a regularized empirical approximation that trades exactness for direct backpropagation and architectural generality (Sun, 18 May 2026).

Across these variants, one theme remains stable. Natural policy gradient is attractive because it replaces raw Euclidean parameter updates by distribution-aware policy-space updates; yet its direct implementation is often difficult in large neural policies because of local approximations, numerical brittleness, and the cost of estimating or inverting the Fisher matrix. Much of the modern literature therefore consists either of proving stronger structural properties for exact NPG in specific settings, or of designing approximations that retain enough of the Fisher geometry to preserve the advantages of natural-gradient reasoning without incurring the full computational burden (Heeswijk, 2022).