Steepest Descent Policy Optimization

• Steepest descent policy optimization is a family of methods where policy improvement is cast as the steepest step under a chosen geometry, balancing convergence, stability, and sample efficiency in reinforcement learning.
• Policy Mirror Descent (PMD) uses mirror geometry with Bregman divergence—often via negative entropy—to yield updates that mimic natural gradients and align with trust-region methods like TRPO and PPO.
• Alternatively, applying SGD (with or without Nesterov momentum) directly to policy gradients in deep RL reveals trade-offs in optimizer robustness, learning rate sensitivity, and convergence speed.

Steepest descent policy optimization denotes a family of reinforcement-learning update rules in which policy improvement is cast as a steepest step under a chosen geometry or parameterization. In one usage, the term refers to Policy Mirror Descent (PMD), where mirror descent in policy space induces steepest ascent under a Bregman geometry and, with negative entropy, coincides in the tabular case with natural policy gradient and a KL trust-region interpretation related to TRPO and PPO. In a second usage, the term refers to applying vanilla stochastic gradient descent, with or without Nesterov momentum, directly to stochastic policy-gradient objectives such as A2C and PPO surrogates. These two senses are mathematically distinct but are linked by a common concern: how the choice of geometry, optimizer, and improvement operator governs stability, convergence, and sample efficiency in policy optimization (Protopapas et al., 2024, Henderson et al., 2018).

1. Terminological scope and problem setting

In the PMD formulation, the underlying control problem is an infinite horizon discounted MDP

$\mathcal{M} = (S, A, r, \gamma, P, \rho)$

with finite state space $S$, action space $A$, rewards $r: S \times A \to [0, 1]$, discount $\gamma \in (0,1)$, transition kernel $P(s'|s,a)$, and initial state distribution $\rho$. A stationary randomized policy $\pi \in \Pi$ maps each $s \in S$ to a distribution over $A$, and its value functions are

$S$0

$S$1

The optimal value $S$2 and an optimal deterministic policy $S$3 exist. The expected Bellman operator for $S$4 is $S$5, and the optimality operator is $S$6; both are $S$7-contractions in $S$8 with unique fixed points $S$9 and $A$0, respectively (Protopapas et al., 2024).

In the deep-RL optimizer study, steepest descent is defined at the parameter level. The control objective is

$A$1

with generic policy-gradient estimator

$A$2

The steepest-descent step is then

$A$3

where $A$4 is the sampled policy gradient at iteration $A$5, obtained from A2C or PPO’s surrogate, and the implementation fixes the sign convention through minimizing a loss (Henderson et al., 2018).

A common misconception is that these usages are interchangeable. They are not. PMD defines steepestness in policy space via a mirror map and Bregman divergence, whereas SGD-based policy-gradient optimization defines steepestness in the Euclidean parameterization of a neural policy. This suggests that “steepest descent policy optimization” is best understood as an umbrella phrase whose precise meaning depends on whether the geometry is imposed on distributions over actions or on policy parameters.

2. Policy Mirror Descent as steepest descent in policy space

PMD is mirror descent in policy space with a Bregman divergence $A$6 induced by a Legendre mirror map $A$7. Its per-state update is

$A$8

Using operator notation, this is equivalent to the policy-space proximal step

$A$9

The update is state-separable, and the linearized value-improvement term is regularized by the Bregman divergence to the previous policy (Protopapas et al., 2024).

The geometric interpretation is central. Mirror descent is steepest descent in the dual geometry induced by $r: S \times A \to [0, 1]$0. For the Euclidean mirror map, PMD becomes projected Q-ascent. For the negative entropy mirror map, $r: S \times A \to [0, 1]$1 is the Kullback–Leibler divergence, and tabular PMD becomes the natural policy gradient: the mirror step is equivalent to a steepest-ascent step under the Fisher–Rao metric, so the natural gradient preconditions the score function by the inverse Fisher information. In this sense, PMD instantiates “steepest descent policy optimization” through geometry rather than through a raw Euclidean gradient step (Protopapas et al., 2024).

This same construction provides the connection to trust-region methods. The KL-based Bregman divergence implements a trust-region-like regularizer tying successive policies, paralleling TRPO/PPO’s KL trust region and clipping schemes. At the opposite extreme, when $r: S \times A \to [0, 1]$2 or $r: S \times A \to [0, 1]$3, PMD reduces to policy iteration,

$r: S \times A \to [0, 1]$4

that is, a hard unregularized 1-step greedy improvement. PMD is therefore a soft policy iteration algorithm whose steepest-descent character depends on the mirror geometry rather than on Euclidean parametrization.

3. Multi-step greedy lookahead and the emergence of h-PMD

The principal extension in “Policy Mirror Descent with Lookahead” is to replace 1-step greedy improvement with multi-step greedy lookahead. The corresponding hard-improvement scheme is h-step greedy policy iteration,

$r: S \times A \to [0, 1]$5

with $r: S \times A \to [0, 1]$6 the set of $r: S \times A \to [0, 1]$7-step greedy policies with respect to $r: S \times A \to [0, 1]$8. Because $r: S \times A \to [0, 1]$9 is a $\gamma \in (0,1)$0-contraction, $\gamma \in (0,1)$1-PI contracts with factor $\gamma \in (0,1)$2 (Protopapas et al., 2024).

The mirror-descent analogue is h-PMD:

$\gamma \in (0,1)$3

Introducing the h-lookahead value

$\gamma \in (0,1)$4

and the associated action-value

$\gamma \in (0,1)$5

the per-state update becomes

$\gamma \in (0,1)$6

The algorithm therefore preserves the mirror-descent structure and changes only the improvement signal from $\gamma \in (0,1)$7 to $\gamma \in (0,1)$8 (Protopapas et al., 2024).

Two mirror maps yield closed forms. For the Euclidean mirror map,

$\gamma \in (0,1)$9

For the negative entropy mirror map,

$P(s'|s,a)$0

with

$P(s'|s,a)$1

The KL case retains the multiplicative-weights form familiar from tabular natural policy gradient, but now driven by an $P(s'|s,a)$2-step lookahead signal rather than a 1-step advantage estimate.

The exact convergence guarantee formalizes the benefit of lookahead. For step sizes satisfying

$P(s'|s,a)$3

the iterates satisfy

$P(s'|s,a)$4

This is a dimension-free $P(s'|s,a)$5-linear convergence rate. For $P(s'|s,a)$6, h-PMD reduces to PMD; for $P(s'|s,a)$7, the contraction factor improves from $P(s'|s,a)$8 to $P(s'|s,a)$9. Letting $\rho$0 recovers h-PI and eliminates the regularization term, while finite $\rho$1 preserves the proximal mirror step (Protopapas et al., 2024).

4. Inexact h-PMD, planning, and sample complexity

Exact computation of $\rho$2 is generally unavailable, so the paper introduces an inexact update:

$\rho$3

Under a generative model, $\rho$4 is obtained by recursively building a partial tree of depth $\rho$5 and bootstrapping from Monte Carlo rollouts of $\rho$6 at the leaves. At the leaves, $\rho$7 is estimated by $\rho$8-step rollouts,

$\rho$9

after which the procedure backs up maxima over actions layer-by-layer to approximate $\pi \in \Pi$0 and then forms $\pi \in \Pi$1 (Protopapas et al., 2024).

The theory is robust to bounded evaluation error. If

$\pi \in \Pi$2

and the step sizes satisfy the corresponding inexact greedification condition with $\pi \in \Pi$3, then

$\pi \in \Pi$4

Thus the $\pi \in \Pi$5-linear rate persists, and the asymptotic bias is

$\pi \in \Pi$6

which the paper states is uniformly smaller than the 1-step PMD bias bound for $\pi \in \Pi$7 (Protopapas et al., 2024).

The generative-model sample complexity makes the gain explicit. With $\pi \in \Pi$8 iterations, $\pi \in \Pi$9, and estimator parameters tuned to target accuracy $s \in S$0 and failure probability $s \in S$1, the total samples satisfy

$s \in S$2

yielding overall sample complexity

$s \in S$3

Compared to 1-step PMD, the paper states that this improves the dependence on the effective horizon by a factor $s \in S$4 for $s \in S$5 larger than $s \in S$6. This suggests that multi-step improvement can reduce total sampling burden even when each iteration becomes more expensive.

5. Function approximation, implementation, and computational trade-offs

To scale beyond tabular state spaces, the paper considers linear approximation of $s \in S$7 with feature map $s \in S$8 and matrix $s \in S$9, writing $A$0. The inexact h-PMD update becomes

$A$1

where $A$2 is estimated by least squares from targets $A$3 computed on a design set $A$4. The analysis assumes that $A$5 has full rank and that there exists $A$6 such that for all $A$7,

$A$8

Under these conditions and a Kiefer–Wolfowitz design with $A$9, the resulting performance bound depends on $S$00 rather than on $S$01 (Protopapas et al., 2024).

The stated convergence bound is

$S$02

The paper summarizes this as preserving $S$03-linear convergence up to an approximation-and-estimation bias of order

$S$04

A plausible implication is that h-PMD is not tied to tabular analysis; its lookahead mechanism can be combined with linear value surrogates without reintroducing explicit dependence on the full state-space cardinality.

From a computational perspective, planning for $S$05 can be done by Dynamic Programming in time linear in $S$06 if the model is known. Otherwise, the paper notes that model-free RL such as Q-learning or tree-search methods such as MCTS can be used. Its analyzed Monte Carlo planner samples successor states to simulate $S$07, estimates $S$08 at leaves by rollouts of length $S$09, and backs up action maxima layer-by-layer. Bandit-guided action sampling, for example UCT, is suggested as a promising efficiency improvement, although the analysis uses the exhaustive sampler for clarity. The per-state policy update remains closed-form for the Euclidean and KL mirror maps. To avoid storing $S$10 for all states, the paper proposes computing $S$11 on demand from $S$12 during planning while memoizing states actually queried by the planner (Protopapas et al., 2024).

The principal trade-off is explicit. Larger $S$13 accelerates convergence through the factor $S$14 and tightens the asymptotic error through $S$15, but it increases per-iteration planning cost. The paper reports that, in practice, the reduction in the number of iterations and total samples can outweigh the added per-iteration cost when planning is efficient, especially with a generative model or tree search.

6. Euclidean steepest descent in deep policy-gradient methods

A separate line of work studies steepest descent policy optimization in the literal SGD sense: policy and value networks are trained with standard stochastic optimizers on A2C and PPO losses. The experiments in “Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods” evaluate SGD, SGD with Nesterov momentum (SGDNM), ASGD, Adagrad, Adadelta, RMSProp, Adam, AMSGrad, Adamax, and YellowFin on MuJoCo continuous control tasks using modified PyTorch A2C/PPO code, with observations and rewards normalized, gradient clipping with max_grad_norm = 0.5, entropy regularization turned off, value loss coefficient 1.0, and training duration of 2 million frames per run (Henderson et al., 2018).

For PPO, the clipped policy loss is

$S$16

where

$S$17

and $S$18. PPO uses GAE with

$S$19

with $S$20 and $S$21. A2C uses use_gae = false and reuses TD errors as advantages over $S$22 steps. In both cases, value regression is

$S$23

with bootstrapped $S$24-step targets. The steepest-descent baseline is plain SGD,

$S$25

and the momentum variant emphasized in the study is SGDNM,

$S$26

The empirical picture is sharply different from the policy-space PMD theory. Adaptive optimizers such as Adam, RMSProp, AMSGrad, and Adamax exhibit a narrow window of effective learning rates and diverge outside it. The paper reports, for example, that in PPO HalfCheetah asymptotic return, RMSProp goes from 3674 at lr=2e−4 to −14279 at lr=1e−2, and Adam goes from 3327 (2e−4) to −10546 (1e−2). In PPO Reacher average return, RMSProp degrades from about −10 at lr=1e−3 to −559 at lr=1e−2. By contrast, SGDNM is reported as robust across a wide range of learning rates, especially in PPO; for PPO Walker2d asymptotic return, it maintains 3402 at 5e−4, 3292 at 8e−4, and 1673 at 1e−2. The paper therefore recommends SGDNM with Nesterov and $S$27 as a strong default for PPO when extensive tuning is not feasible (Henderson et al., 2018).

Momentum is beneficial only up to a point. The study reports that $S$28 often causes divergence across both A2C and PPO, and that momentum approaching $S$29 is generally unstable. For PPO Ant average performance, the value drops to −3444 at β=0.99, and A2C exhibits catastrophic collapse at β=0.99 in several tasks. Moderate momentum, around 0.9, is near-optimal or close to optimal in many cases, including PPO Walker2d. The paper attributes this to changing loss landscapes in on-policy RL: as the policy improves, local optima shift, and high momentum overweights stale directions. Environments with early terminations, such as falling locomotion agents, appear especially sensitive because small policy changes can induce abrupt changes in returns (Henderson et al., 2018).

The experiments also provide evidence for implicit momentum. By varying the worker-to-step ratio while keeping batch size fixed, the authors observe a noisy trend in which lower explicit momentum values perform better as the worker-to-step ratio increases. This is presented as consistent with the hypothesis that aspects of sampling and on-policy staleness can add implicit momentum. A plausible implication is that Euclidean steepest descent in deep RL is not governed solely by optimizer equations; it is also shaped by rollout collection, bootstrapping, and environment dynamics.

Taken together, these results establish a contrast within the broader topic. In PMD and h-PMD, steepest descent is a geometric policy-space principle that yields exact convergence and sample-complexity guarantees, with multi-step lookahead improving the contraction factor from $S$30 to $S$31. In A2C and PPO with neural policies, steepest descent is an optimizer choice in parameter space, where robustness depends on learning-rate windows, momentum, and nonstationary objectives. The shared phrase therefore names two distinct research programs: one centered on mirror geometry and greedy operators, the other on stochastic optimization behavior in deep policy-gradient training.