---
title: Steepest Descent Policy Optimization
url: https://www.emergentmind.com/topics/steepest-descent-policy-optimization
type: topic
---

# Steepest Descent Policy Optimization

Steepest descent policy optimization denotes a family of reinforcement-learning update rules in which policy improvement is cast as a steepest step under a chosen geometry or parameterization. In one usage, the term refers to Policy Mirror Descent (PMD), where mirror descent in policy space induces steepest ascent under a Bregman geometry and, with negative entropy, coincides in the tabular case with natural policy gradient and a KL trust-region interpretation related to TRPO and PPO. In a second usage, the term refers to applying vanilla stochastic gradient descent, with or without Nesterov momentum, directly to stochastic policy-gradient objectives such as A2C and PPO surrogates. These two senses are mathematically distinct but are linked by a common concern: how the choice of geometry, optimizer, and improvement operator governs stability, convergence, and sample efficiency in policy optimization [2403.14156] [1810.02525].

## 1. Terminological scope and problem setting

In the PMD formulation, the underlying control problem is an infinite horizon discounted MDP
$\mathcal{M} = (S, A, r, \gamma, P, \rho)$
with finite state space $S$, action space $A$, rewards $r: S \times A \to [0, 1]$, discount $\gamma \in (0,1)$, transition kernel $P(s'|s,a)$, and initial state distribution $\rho$. A stationary randomized policy $\pi \in \Pi$ maps each $s \in S$ to a distribution over $A$, and its value functions are
$$
V^\pi (s) = \E_{\pi} \left[\sum_{t = 0}^{\infty} \gamma^t r(s_t, a_t) \mid s_0 = s \right],
$$
$$
Q^\pi (s,a) = \E_{\pi} \left[\sum_{t = 0}^{\infty} \gamma^t r(s_t, a_t) \mid s_0 = s, a_0 = a \right].
$$
The optimal value $V^*$ and an optimal deterministic policy $\pi^\ast$ exist. The expected Bellman operator for $\pi$ is $T^{\pi}V := M^{\pi}(r + \gamma P V)$, and the optimality operator is $T V := M^{\star}(r + \gamma P V)$; both are $\gamma$-contractions in $\|\cdot\|_\infty$ with unique fixed points $V^\pi$ and $V^*$, respectively [2403.14156].

In the deep-RL optimizer study, steepest descent is defined at the parameter level. The control objective is
$$
J(\theta) = E_{\tau \sim \pi_\theta} \left[\sum_{t=0}^{T} \gamma^t r_t \right],
$$
with generic policy-gradient estimator
$$
\nabla_\theta J(\theta) = E \left[\sum_t \nabla_\theta \log \pi_\theta(a_t|s_t) A_t \right].
$$
The steepest-descent step is then
$$
\theta_{k+1} = \theta_k - \alpha g_k,
$$
where $g_k$ is the sampled policy gradient at iteration $k$, obtained from A2C or PPO’s surrogate, and the implementation fixes the sign convention through minimizing a loss [1810.02525].

A common misconception is that these usages are interchangeable. They are not. PMD defines steepestness in policy space via a mirror map and Bregman divergence, whereas SGD-based policy-gradient optimization defines steepestness in the Euclidean parameterization of a neural policy. This suggests that “steepest descent policy optimization” is best understood as an umbrella phrase whose precise meaning depends on whether the geometry is imposed on distributions over actions or on policy parameters.

## 2. Policy Mirror Descent as steepest descent in policy space

PMD is mirror descent in policy space with a Bregman divergence $D_\phi(\cdot,\cdot)$ induced by a Legendre mirror map $\phi$. Its per-state update is
$$
\pi^{k + 1}_s \in \argmax_{\pi_s \in \Delta(\mathcal{A})} \left\{\eta_k \langle Q^{}_s, \pi_s \rangle - D_{\phi}(\pi_s, \pi^k_s)\right\}\,.
$$
Using operator notation, this is equivalent to the policy-space proximal step
$$
\pi_{k + 1} \in \argmax_{\pi \in \Pi} \left\{\eta_k \, T V^{\pi_k} - D_{\phi}(\pi, \pi_k)\right\}\,.
$$
The update is state-separable, and the linearized value-improvement term is regularized by the Bregman divergence to the previous policy [2403.14156].

The geometric interpretation is central. Mirror descent is steepest descent in the dual geometry induced by $\phi$. For the Euclidean mirror map, PMD becomes projected Q-ascent. For the negative entropy mirror map, $D_\phi$ is the Kullback–Leibler divergence, and tabular PMD becomes the natural policy gradient: the mirror step is equivalent to a steepest-ascent step under the Fisher–Rao metric, so the natural gradient preconditions the score function by the inverse Fisher information. In this sense, PMD instantiates “steepest descent policy optimization” through geometry rather than through a raw Euclidean gradient step [2403.14156].

This same construction provides the connection to trust-region methods. The KL-based Bregman divergence implements a trust-region-like regularizer tying successive policies, paralleling TRPO/PPO’s KL trust region and clipping schemes. At the opposite extreme, when $\eta_k \to \infty$ or $D_\phi \equiv 0$, PMD reduces to policy iteration,
$$
\pi_{k + 1} \in \argmax_{\pi \in \Pi} \left\{ T V^{\pi_k} \right\}\,,
$$
that is, a hard unregularized 1-step greedy improvement. PMD is therefore a soft policy iteration algorithm whose steepest-descent character depends on the mirror geometry rather than on Euclidean parametrization.

## 3. Multi-step greedy lookahead and the emergence of h-PMD

The principal extension in “Policy Mirror Descent with Lookahead” is to replace 1-step greedy improvement with multi-step greedy lookahead. The corresponding hard-improvement scheme is h-step greedy policy iteration,
$$
\pi^{k+1} \in \argmax_{\pi \in \Pi}  \left\{  T^{h-1} V^{\pi_k} \right\}\,,
$$
with $G_h(V) := \{\pi \in \Pi : M^{\pi} Q^{(h)}(V) = T^h V\}$ the set of $h$-step greedy policies with respect to $V$. Because $T$ is a $\gamma$-contraction, $h$-PI contracts with factor $\gamma^h$ [2403.14156].

The mirror-descent analogue is h-PMD:
$$
\pi_{k+1} = \argmax_{\pi \in \Pi} \left\{\eta_k \, T^{h - 1} V^{\pi_k} - D_{\phi}(\pi, \pi_k) \right\}\,.
$$
Introducing the h-lookahead value
$$
V_h^{\pi} := T^{h-1} V^{\pi},
$$
and the associated action-value
$$
Q_h^{\pi}(s,a) := r(s,a) + \gamma (P V_{h}^{\pi})(s,a)\,,
$$
the per-state update becomes
$$
\pi^{k+1}_s = \argmax_{\pi_s \in \Delta(\mathcal{A})} \left\{\eta_k \langle Q_h^{\pi_k}(s,\cdot), \pi_s \rangle - D_{\phi}(\pi_s, \pi_s^{k}) \right\}\,.
$$
The algorithm therefore preserves the mirror-descent structure and changes only the improvement signal from $Q^{\pi_k}$ to $Q_h^{\pi_k}$ [2403.14156].

Two mirror maps yield closed forms. For the Euclidean mirror map,
$$
\pi_s^{k+1} = \text{Proj}_{\Delta(\mathcal{A})}(\pi_s^k + \eta_k Q_h^{\pi_k}(s,\cdot))\,.
$$
For the negative entropy mirror map,
$$
\pi_{k+1}(a|s) = \pi_k(a|s) \frac{\exp(\eta_k Q_h^{\pi_k}(s,a))}{Z_k(s)}\,,
$$
with
$$
Z_k(s) := \sum_{a\in A} \pi_k(a|s) \exp(\eta_k Q_h^{\pi_k}(s,a)).
$$
The KL case retains the multiplicative-weights form familiar from tabular natural policy gradient, but now driven by an $h$-step lookahead signal rather than a 1-step advantage estimate.

The exact convergence guarantee formalizes the benefit of lookahead. For step sizes satisfying
$$
\eta_k \geq \frac{1}{c_k} \big\| \min_{\pi \in  G_h(V^{\pi_k})} D_{\phi}(\pi, \pi_k)\big\|_\infty,
$$
the iterates satisfy
$$
\lVert V^\star - V^{\pi_k} \rVert_\infty \leq \gamma^{hk} \left( \lVert V^\star - V^{\pi_0} \rVert_\infty + \frac{1}{1-\gamma}\sum_{t = 1}^k  \frac{c_{t - 1}{ } \gamma^{ht}\right).
$$
This is a dimension-free $\gamma^h$-linear convergence rate. For $h=1$, h-PMD reduces to PMD; for $h \geq 2$, the contraction factor improves from $\gamma$ to $\gamma^h$. Letting $\eta_k \to \infty$ recovers h-PI and eliminates the regularization term, while finite $\eta_k$ preserves the proximal mirror step [2403.14156].

## 4. Inexact h-PMD, planning, and sample complexity

Exact computation of $Q_h^{\pi_k}$ is generally unavailable, so the paper introduces an inexact update:
$$
\pi_s^{k + 1} = \argmax_{\pi_s \in \Delta(A)} \left\{\eta_k \langle \hat{Q}_h^{\pi_k}(s, \cdot), \pi_s \rangle - D_{\phi}(\pi_s, \pi_s^{k}) \right\}.
$$
Under a generative model, $\hat{Q}_h^\pi(s,a)$ is obtained by recursively building a partial tree of depth $h$ and bootstrapping from Monte Carlo rollouts of $\pi$ at the leaves. At the leaves, $V^\pi$ is estimated by $H$-step rollouts,
$$
\hat{V}^{\pi}(\tilde{s}) = \frac{1}{M_0}\sum_{j=1}^{M_0} \sum_{k=0}^{H-1} \gamma^k r(s_k^j,a_k^j),
$$
after which the procedure backs up maxima over actions layer-by-layer to approximate $V_h^\pi = T^{h-1}V^\pi$ and then forms $Q_h^\pi(s,a) = r(s,a) + \gamma (P V_h^\pi)(s,a)$ [2403.14156].

The theory is robust to bounded evaluation error. If
$$
\lVert \hat{Q}_h^{\pi_k} - Q_h^{\pi_k} \rVert_\infty \leq b,
$$
and the step sizes satisfy the corresponding inexact greedification condition with $\tilde{G}_h = \argmax_{\pi \in \Pi} M^\pi \hat{Q}_h^{\pi_k}$, then
$$
\lVert V^\star - V^{\pi_k} \rVert_\infty \leq \gamma^{hk} \left( \lVert V^\star - V^{\pi_0} \rVert_\infty + \frac{1}{1 - \gamma} \sum_{t = 1}^k \frac{c_{t - 1}{ } \gamma^{ht}\right) + \frac{2b}{(1 -\gamma)(1 - \gamma^h)}\,.
$$
Thus the $\gamma^h$-linear rate persists, and the asymptotic bias is
$$
O\!\left(\frac{b}{(1-\gamma)(1-\gamma^h)}\right),
$$
which the paper states is uniformly smaller than the 1-step PMD bias bound for $h \geq 2$ [2403.14156].

The generative-model sample complexity makes the gain explicit. With $K$ iterations, $c_k := \gamma^{2 h (k+1)}$, and estimator parameters tuned to target accuracy $\epsilon$ and failure probability $\delta$, the total samples satisfy
$$
\text{total samples} \leq K M_0 H |S| + K h M |S| |A|,
$$
yielding overall sample complexity
$$
\tilde{\mathcal{O}\Big(\frac{|S|}{h \epsilon^2(1 - \gamma)^6 (1 - \gamma^h)^2} + \frac{|S||A|}{\epsilon^2(1 - \gamma)^7}\Big).
$$
Compared to 1-step PMD, the paper states that this improves the dependence on the effective horizon by a factor $\approx 1/(1-\gamma)$ for $h$ larger than $\approx 1/(|A|(1-\gamma))$. This suggests that multi-step improvement can reduce total sampling burden even when each iteration becomes more expensive.

## 5. Function approximation, implementation, and computational trade-offs

To scale beyond tabular state spaces, the paper considers linear approximation of $Q_h^\pi$ with feature map $\psi(s,a) \in \mathbb{R}^d$ and matrix $\Psi \in \mathbb{R}^{|S||A|\times d}$, writing $Q_h^\pi \approx \Psi \theta$. The inexact h-PMD update becomes
$$
\pi_s^{k + 1} \in \argmax_{\pi_s \in \Delta(S)}\left\{\eta_k \langle (\Psi \theta_k)_s, \pi_s \rangle - D_{\phi}(\pi_s, \pi^k_s)\right\},
$$
where $\theta_k$ is estimated by least squares from targets $\hat{Q}_h^{\pi_k}$ computed on a design set $C \subset S\times A$. The analysis assumes that $\Psi$ has full rank and that there exists $\epsilon > 0$ such that for all $\pi$,
$$
\inf_\theta \|Q_h^\pi - \Psi \theta\|_\infty \leq \epsilon.
$$
Under these conditions and a Kiefer–Wolfowitz design with $|C| = O(d^2)$, the resulting performance bound depends on $d$ rather than on $|S|$ [2403.14156].

The stated convergence bound is
$$
\lVert V^\star - V^{\pi_k} \rVert_\infty \leq \gamma^{hk}\left(\lVert V^\star - V^{\pi_0} \rVert_\infty + \frac{1}{1 - \gamma}\sum_{t = 1}^k \frac{c_{t - 1}{1 - \gamma}\right) + \frac{2 \max_{i \leq k}\lVert \epsilon_{\pi_i}\rVert_\infty ( 1 + \sqrt{d}) + 2\sqrt{d}\,\epsilon}{(1 - \gamma)(1 - \gamma^h)}\,.
$$
The paper summarizes this as preserving $\gamma^h$-linear convergence up to an approximation-and-estimation bias of order
$$
O\!\left(\frac{(\|\epsilon_{\pi}\|_\infty + \epsilon)\sqrt{d}}{(1-\gamma)(1-\gamma^h)}\right).
$$
A plausible implication is that h-PMD is not tied to tabular analysis; its lookahead mechanism can be combined with linear value surrogates without reintroducing explicit dependence on the full state-space cardinality.

From a computational perspective, planning for $Q_h^\pi$ can be done by Dynamic Programming in time linear in $h$ if the model is known. Otherwise, the paper notes that model-free RL such as Q-learning or tree-search methods such as MCTS can be used. Its analyzed Monte Carlo planner samples successor states to simulate $P$, estimates $V^\pi$ at leaves by rollouts of length $H$, and backs up action maxima layer-by-layer. Bandit-guided action sampling, for example UCT, is suggested as a promising efficiency improvement, although the analysis uses the exhaustive sampler for clarity. The per-state policy update remains closed-form for the Euclidean and KL mirror maps. To avoid storing $\pi$ for all states, the paper proposes computing $\pi_{k+1}(\cdot|s)$ on demand from $\Psi \theta_k$ during planning while memoizing states actually queried by the planner [2403.14156].

The principal trade-off is explicit. Larger $h$ accelerates convergence through the factor $\gamma^h$ and tightens the asymptotic error through $(1-\gamma^h)$, but it increases per-iteration planning cost. The paper reports that, in practice, the reduction in the number of iterations and total samples can outweigh the added per-iteration cost when planning is efficient, especially with a generative model or tree search.

## 6. Euclidean steepest descent in deep policy-gradient methods

A separate line of work studies steepest descent policy optimization in the literal SGD sense: policy and value networks are trained with standard stochastic optimizers on A2C and PPO losses. The experiments in “Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methods” evaluate SGD, SGD with Nesterov momentum (SGDNM), ASGD, Adagrad, Adadelta, RMSProp, Adam, AMSGrad, Adamax, and YellowFin on MuJoCo continuous control tasks using modified PyTorch A2C/PPO code, with observations and rewards normalized, gradient clipping with `max_grad_norm = 0.5`, entropy regularization turned off, value loss coefficient `1.0`, and training duration of `2 million frames per run` [1810.02525].

For PPO, the clipped policy loss is
$$
L_{\text{CLIP}}(\theta) = E [\min(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t)],
$$
where
$$
r_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\theta_{\text{old}}}(a_t|s_t)
$$
and $\epsilon = 0.2$. PPO uses GAE with
$$
\hat{A}_t = \sum_{l=0}^{\infty} (\gamma \lambda)^l \delta_{t+l},
\qquad
\delta_t = r_t + \gamma V_\phi(s_{t+1}) - V_\phi(s_t),
$$
with $\gamma = 0.99$ and $\lambda = 0.95$. A2C uses `use_gae = false` and reuses TD errors as advantages over $n=5$ steps. In both cases, value regression is
$$
L_V(\theta_V) = E [(Y_t - V_\gamma^\pi(s_t; \theta_V))^2],
$$
with bootstrapped $N$-step targets. The steepest-descent baseline is plain SGD,
$$
\theta_{t+1} = \theta_t - \alpha g_t,
$$
and the momentum variant emphasized in the study is SGDNM,
$$
v_{t+1} = \beta v_t + g_t,
\qquad
\theta_{t+1} = \theta_t - \alpha (\beta v_{t+1} + g_t).
$$

The empirical picture is sharply different from the policy-space PMD theory. Adaptive optimizers such as Adam, RMSProp, AMSGrad, and Adamax exhibit a narrow window of effective learning rates and diverge outside it. The paper reports, for example, that in PPO HalfCheetah asymptotic return, RMSProp goes from `3674 at lr=2e−4` to `−14279 at lr=1e−2`, and Adam goes from `3327 (2e−4)` to `−10546 (1e−2)`. In PPO Reacher average return, RMSProp degrades from about `−10 at lr=1e−3` to `−559 at lr=1e−2`. By contrast, SGDNM is reported as robust across a wide range of learning rates, especially in PPO; for PPO Walker2d asymptotic return, it maintains `3402 at 5e−4`, `3292 at 8e−4`, and `1673 at 1e−2`. The paper therefore recommends SGDNM with Nesterov and $\beta \approx 0.9$ as a strong default for PPO when extensive tuning is not feasible [1810.02525].

Momentum is beneficial only up to a point. The study reports that $\beta \geq 0.98$ often causes divergence across both A2C and PPO, and that momentum approaching $1$ is generally unstable. For PPO Ant average performance, the value drops to `−3444 at β=0.99`, and A2C exhibits catastrophic collapse at `β=0.99` in several tasks. Moderate momentum, around `0.9`, is near-optimal or close to optimal in many cases, including PPO Walker2d. The paper attributes this to changing loss landscapes in on-policy RL: as the policy improves, local optima shift, and high momentum overweights stale directions. Environments with early terminations, such as falling locomotion agents, appear especially sensitive because small policy changes can induce abrupt changes in returns [1810.02525].

The experiments also provide evidence for implicit momentum. By varying the worker-to-step ratio while keeping batch size fixed, the authors observe a noisy trend in which lower explicit momentum values perform better as the worker-to-step ratio increases. This is presented as consistent with the hypothesis that aspects of sampling and on-policy staleness can add implicit momentum. A plausible implication is that Euclidean steepest descent in deep RL is not governed solely by optimizer equations; it is also shaped by rollout collection, bootstrapping, and environment dynamics.

Taken together, these results establish a contrast within the broader topic. In PMD and h-PMD, steepest descent is a geometric policy-space principle that yields exact convergence and sample-complexity guarantees, with multi-step lookahead improving the contraction factor from $\gamma$ to $\gamma^h$. In A2C and PPO with neural policies, steepest descent is an optimizer choice in parameter space, where robustness depends on learning-rate windows, momentum, and nonstationary objectives. The shared phrase therefore names two distinct research programs: one centered on mirror geometry and greedy operators, the other on stochastic optimization behavior in deep policy-gradient training.

Source: https://www.emergentmind.com/topics/steepest-descent-policy-optimization