---
title: Non-Myopic Pathwise Policy Gradients
url: https://www.emergentmind.com/topics/non-myopic-pathwise-policy-gradients-nm-ppg
type: topic
---

# Non-Myopic Pathwise Policy Gradients

Searching arXiv for the cited NM-PPG-related papers to ground the article in current records.
Non-Myopic Pathwise Policy Gradients (NM-PPG) denotes a family of policy-gradient methods that perform credit assignment over multi-step trajectories rather than only through one-step action evaluations. Across several distinct formulations, the common idea is to optimize a stochastic policy by differentiating an objective defined on full paths or multi-step rollouts, so that parameter updates reflect long-range dependencies among actions, observations, and downstream rewards or costs. In the literature, the label covers at least three closely related but application-specific lines: a hierarchical model-based policy optimization framework in path space [1912.01448], a rollout-gradient method for non-myopic Bayesian optimization [2408.07812], and a pathwise-gradient approach for non-myopic active feature acquisition [2605.05511]. Related work on $k$-step policy gradients formalizes a closely aligned mechanism—coupling policy randomness across a finite horizon to escape myopic local optima in restricted policy classes [2605.10909].

## 1. Conceptual scope and defining idea

The unifying property of NM-PPG methods is non-myopic credit assignment. In the path-space formulation, every infinite trajectory $\tau=(s_0,a_0,s_1,a_1,\dots)$ is treated as a sample from the path density
\[
p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),
\]
and the objective is the expected cumulative reward
\[
J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].
\]
A second-order update in the policy parameters then coincides with the natural gradient with respect to a Fisher metric defined on entire trajectories [1912.01448]. This makes the update sensitive to the correlational structure of induced state-action paths rather than only to local action perturbations.

In Bayesian optimization, the same non-myopic principle appears in a finite-horizon MDP whose states are Gaussian-process posteriors, actions are query points, and one-step reward is improvement over the best-so-far. There, NM-PPG optimizes a stochastic sampling policy $\pi_\theta(a\mid s)$ for the $H$-step value
\[
V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],
\]
using a pathwise derivative estimator based on reparameterized rollouts [2408.07812].

In active feature acquisition, NM-PPG is formulated as a finite-horizon POMDP in which actions either acquire a feature or choose STOP. The objective combines feature-acquisition cost with prediction loss, and a continuous relaxation enables backpropagation through the full acquisition trajectory [2605.05511]. Although the state, action, and reward semantics differ across these domains, the methodological commonality is direct differentiation through a trajectory-level objective.

A plausible implication is that NM-PPG is best understood not as a single algorithmic template, but as a design pattern: define a sequential decision problem, re-express the objective on full paths or multi-step blocks, and compute gradients that preserve long-range dependence information.

## 2. Path-space natural gradients and hierarchical model-based policy optimization

In "Hierarchical model-based policy optimization: from actions to action sequences and back" [1912.01448], NM-PPG appears as a normative second-order framework in path space. If $\pi_\theta(a\mid s)$ is treated as an exponential family in natural parameters $\theta$, then a local quadratic model around $\theta^k$ yields
\[
\Delta\theta^*=\arg\max_{\Delta\theta}\Bigl\{\nabla_\theta J(\theta^k)^\top\Delta\theta-\tfrac12\Delta\theta^\top F_{\rm path}(\theta^k)\Delta\theta\Bigr\},
\]
with optimum
\[
\Delta\theta^*=F_{\rm path}(\theta^k)^{-1}\nabla_\theta J(\theta^k).
\]
Here the pathwise Fisher information is
\[
F_{\rm path}(\theta)=\mathbb E_{\tau\sim p_\theta}[s(\tau;\theta)s(\tau;\theta)^\top]
\equiv-\mathbb E_{\tau\sim p_\theta}[\nabla_\theta^2\log p_\theta(\tau)],
\]
where $s(\tau;\theta)=\nabla_\theta\log p_\theta(\tau)$ is the trajectory score function [1912.01448].

This formulation gives the update
\[
\theta\leftarrow\theta+\alpha F_{\rm path}(\theta)^{-1}\nabla_\theta J(\theta),
\]
or, in the small-step trust-region form,
\[
\theta\leftarrow\theta+F_{\rm path}(\theta)^{-1}\nabla_\theta J(\theta).
\]
Because $J(\theta)=E_\tau[R(\tau)]$ and $\nabla_\theta J=E_\tau[(\nabla_\theta\log p_\theta(\tau))R(\tau)]$, the update is expressed in terms of path-score covariances [1912.01448].

The paper’s central interpretive claim is that such updates are hierarchy-sensitive. Long-range correlational structure in stationary state-action densities affects the natural path gradient, and in simulation the resulting policy flow prioritizes local updates in a way that reflects an intuitive state-space hierarchy [1912.01448]. The paper explicitly connects this effect to “higher-order successor representations,” which provide the algebraic machinery for analytic computation of the metric and gradient.

## 3. Exact model-based computation via successor representations

The model-based NM-PPG framework assumes access to a perfect environment model $P$, allowing expectations over trajectories to be computed analytically rather than by Monte Carlo sampling [1912.01448]. The one-step transition matrix under policy $\pi$ is
\[
T_{s,s'}=\sum_{a\in\mathcal A(s)}\pi(a\mid s)P(s'\mid s,a),
\]
and the discounted successor representation is
\[
D=(I-\lambda T)^{-1},\qquad 0<\lambda\le\gamma.
\]
The state-action to state successor tensor is
\[
E_{(s,a),s'}=\sum_{s''}P(s''\mid s,a)D_{s'',s'},
\]
so that $E_{(s,a),k}$ is the expected future count of state $k$ after taking $(s,a)$ [1912.01448].

Using these objects, one constructs the second-order co-occurrence or counter-correlation
\[
C_{(s,a),(s',a')}=\mathbb E[n_{s,a}(\tau)n_{s',a'}(\tau)],
\]
where $n_{s,a}(\tau)$ is the discounted count of visits to $(s,a)$ in a path $\tau$. Closed-form expressions for $C$ follow from $D$ and $E$. The Fisher metric $F_{\rm path}$ and the policy gradient can then be written entirely in terms of $(D,E,C)$ and the known rewards $R$ [1912.01448].

The paper’s high-level pseudocode builds $T$, then $D$, then $E$, then the co-occurrence tensor $C$, followed by the Fisher matrix and gradient, before applying the natural-gradient update and reconstructing the policy with a softmax parameterization [1912.01448]. The algorithmic significance is explicit: no sampling of long trajectories is required, and every gradient and metric term is computed analytically from successor quantities. The paper characterizes this as a model-based exact-gradient property.

The same section motivates the term non-myopic. The tensor $C_{(s,a),(s',a')}$ quantifies how often, under the current policy, action $a$ in state $s$ leads after many steps to action $a'$ in state $s'$. A reward at $(s',a')$ is therefore backed up to $(s,a)$ in proportion to this co-occurrence, with normalization terms removing spurious attributions induced by policy normalization [1912.01448]. This suggests that NM-PPG can be viewed as replacing horizon truncation or bootstrap design with a direct accounting of all horizons at once.

## 4. Pathwise rollout gradients in non-myopic Bayesian optimization

In "Differentiating Policies for Non-Myopic Bayesian Optimization" [2408.07812], Bayesian optimization is cast as an MDP whose state is the current GP posterior, equivalently the dataset $D_t=\{(x^i,y_i):i=1\ldots t\}$ together with posterior mean and covariance, and whose action is the next query point $x\in\Omega$. The transition density is induced by the GP posterior predictive, and the one-step reward is
\[
R(s_t,x_t,s_{t+1})=(f_t^*-f_{t+1})^+,
\]
where $f_t^*=\min_{i\le t}y_i$ and $f_{t+1}=y_{t+1}$ [2408.07812].

Rather than optimize a single-step acquisition function, the method considers a stochastic policy $\pi_\theta(a\mid s)$ and seeks
\[
\theta^*=\arg\max_\theta V_H^\theta(s_0).
\]
A convenient policy choice is a Gibbs policy
\[
\pi_\theta(x\mid s)=\exp\{\theta^\top\phi(s,x)\}/Z_\theta(s),
\]
with a common two-dimensional feature map
\[
\phi(s,x)=[\mu^{(s)}(x),\sigma^{(s)}(x)]^\top,
\]
where $\mu^{(s)}(x)$ and $\sigma^{(s)}(x)=\sqrt{K^{(s)}(x,x)}$ are the GP-posterior mean and standard deviation [2408.07812].

The key technical device is rollout reparameterization. If $\pi_\theta(x\mid s)$ is a location-scale family, the query can be written
\[
x_t=m_\theta(s_t)+L_\theta(s_t)\epsilon_t,
\]
with $\epsilon_t\sim N(0,I)$ or $\epsilon_t\sim{\rm Uniform}[0,1]^d$ for a Sobol-scrambled QMC sequence. The GP draw is likewise parameterized as
\[
y_t=\mu^{(s_{t-1})}(x_{t-1})+\sigma^{(s_{t-1})}(x_{t-1})\eta_t,
\]
with $\eta_t\sim N(0,1)$ [2408.07812]. Because the randomness is independent of $\theta$, the gradient may be pushed inside the expectation:
\[
\nabla_\theta V_H^\theta(s)=\mathbb E_{\epsilon,\eta}\Bigl[\nabla_\theta\Bigl(\sum_{t=0}^{H-1}r(s_t(\theta),x_t(\theta))\Bigr)\Bigr].
\]

The paper states an unbiased pathwise gradient proposition: under almost-everywhere differentiability of $r$ and the GP updates, the Monte Carlo estimator
\[
\hat g_N=\frac1N\sum_{i=1}^N\nabla_\theta\Bigl[\sum_{t=0}^{H-1}r(s_t^{(i)},x_t^{(i)})\Bigr]
\]
satisfies $\mathbb E[\hat g_N]=\nabla_\theta V_H^\theta(s)$ and has variance $O(1/N)$ [2408.07812]. It further reports that quasi-Monte Carlo Sobol draws reduce empirical variance, often by one or more orders of magnitude. Computationally, the method combines QMC trajectories, backpropagation through fantasized GP updates, stochastic gradient ascent for inner policy argmax, common random numbers across successive states, a control-variate construction using analytically known EI, and rank-one Cholesky updates based on Schur complements [2408.07812].

Empirically, the paper evaluates 15 standard synthetic test functions and compares myopic EI and PI, UCB, Knowledge Gradient, fixed-h rollout policies, and NM-PPG. It reports that all non-myopic methods outperform myopic EI and PI, and that NM-PPG achieves the best mean and median gap on 10 out of 15 problems. It also reports a per-iteration overhead of approximately $2\times$ that of $h=1$ rollout, with benefit appearing in 20–30 iterations through better minima [2408.07812].

## 5. Continuous relaxation and straight-through rollouts for active feature acquisition

In "Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients" [2605.05511], NM-PPG is developed for active feature acquisition (AFA), where features are costly and the learner adaptively chooses which values to acquire and when to stop. The problem is formulated as a finite-horizon POMDP. The latent state at time $t$ is
\[
\omega_t=(S_t,x,y)\in 2^{[d]}\times\mathcal X\times\mathcal Y,
\]
where $S_t\subseteq[d]$ is the set of acquired features. The observation is the currently revealed values,
\[
o_t=(S_t,x_{S_t}),
\]
or equivalently the masked representation
\[
x(m_t)=(m_t\odot x,m_t)\in\mathbb R^{2d}.
\]
The action space is
\[
\mathcal A=\{1,\dots,d,d+1\},
\]
where $a\in[d]$ acquires a feature at cost $c(a)>0$ and $a=d+1$ is STOP [2605.05511].

The one-step cost is
\[
C((S_t,x,y),a)=
\begin{cases}
\alpha\,c(a), & a\in[d],\\[4pt]
\ell(f_\phi(x(m_t)),y), & a=d+1,
\end{cases}
\]
and the total cost under policy $\pi_\theta$ is
\[
G(x,y,\pi_\theta)=\sum_{t=0}^{t_\theta(x)-1}\alpha\,c(a_t)+\ell(f_\phi(x(m_{t_\theta(x)})),y).
\]
The objective minimizes
\[
J(\pi_\theta)=\mathbb E_{(x,y)\sim p(\mathbf x,\mathbf y),\,\pi_\theta}[G(x,y,\pi_\theta)],
\]
which the paper states exactly matches the standard AFA objective [2605.05511].

Because discrete acquisition and stopping are not differentiable, the method introduces a continuous relaxation. The discrete mask $m_t$ becomes a soft mask $\tilde m_t\in[0,1]^d$, and a survival mass $\tilde s_t\in[0,1]$ tracks the remaining probability of continuing. A neural network outputs logits
\[
z_t=z_\theta(x(\tilde m_t))\in\mathbb R^{d+1},
\]
and a differentiable relaxed action is sampled using Gumbel-Softmax. The relaxed mask and survival dynamics are
\[
\tilde m_{t+1}=\tilde m_t+(1-\tilde m_t)\odot\tilde r_t,\qquad
\tilde s_{t+1}=\tilde s_t(1-\tilde a_{t,d+1}),
\]
leading to a soft return $\tilde G(x,y,\theta,\varepsilon)$ and surrogate objective
\[
\tilde J(\theta)=\mathbb E_{x,y}\mathbb E_\varepsilon[\tilde G(x,y,\theta,\varepsilon)].
\]
The paper states that this surrogate converges to the discrete objective as $\tau_{\rm soft}\to0$ [2605.05511].

Since all randomness is reparameterized, the pathwise gradient is
\[
\nabla_\theta\tilde J(\theta)=\mathbb E_{x,y,\varepsilon}[\nabla_\theta\tilde G(x,y,\theta,\varepsilon)].
\]
The paper emphasizes that this avoids score-function terms and thereby reduces gradient variance [2605.05511]. To reduce train-test mismatch, it introduces a straight-through rollout scheme: the forward pass uses hard feature acquisitions to update the discrete mask, while the backward pass replaces hard variables with straight-through surrogates so gradients flow through the soft relaxation. The resulting ST objective $\bar J(\theta)$ yields a biased but low-variance gradient, and the paper attributes stable training to this design together with entropy regularization and staged temperature sharpening over the schedule
\[
\tau_{\rm soft}\in\{0.8,\,0.5,\,0.2,\,0.05,\,0.02\}
\]
[2605.05511].

The empirical study spans 12 benchmarks, including Cube-NM, Syn1, Syn3, Connect4, Splice, EngineFaultDB, METABRIC, NHANES Mortality, Diabetes, MNIST, and Fashion-MNIST. Compared baselines include DiFA, GDFS, DIME, AACO, SEFA, GSMRL, and OL. The reported findings are that NM-PPG outperforms all non-myopic baselines in both stability and accuracy-cost trade-off, degrades gracefully on purely myopic datasets, learns the intended context-first acquisition paths on synthetic context tasks, finds non-myopic two-step screening policies on medical benchmarks, and depends strongly on both ST rollouts and entropy regularization for stable training [2605.05511].

## 6. Relation to $k$-step policy gradients and restricted policy classes

"Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $k$-step Policy Gradients" [2605.10909] is not presented as the same algorithm as the preceding works, but it articulates a closely related mechanism and explicitly uses the phrase “Non-Myopic Pathwise Policy Gradients (NM-PPG)” in its detailed summary. The paper’s starting point is that standard one-step policy gradients can stall at suboptimal critical points when the policy class is restricted. The stated reason is that the gradient improves the policy based on the one-step $Q$-function and tests a single action at the current state before reverting to $\pi_\theta$ immediately [2605.10909].

The proposed remedy is to couple policy randomness for $k$ consecutive steps. Let $\Pi_{\rm det}$ be a finite set of deterministic policies, and let $\tilde\pi\in\Delta(\Pi_{\rm det})$ be a correlated policy. In the $k$-step evaluation, a deterministic subpolicy is sampled at times $0,k,2k,\dots$ and then held fixed for the next $k$ steps. This induces
\[
J^{\tilde\pi,k}(s)=\mathbb E_{\tau\sim\tilde\pi|_{s,k}}\Bigl[\sum_{t=0}^\infty\gamma^t g(s_t,a_t)\Bigr],
\]
and the $k$-step performance objective
\[
J_k(\theta)\equiv J^{\tilde\pi_\theta,k}(\mu).
\]
With a reparameterization $\pi_{\rm det}=\sigma_\theta(\epsilon)$, the paper gives an informal pathwise derivative formula in which the same noise variable $\epsilon$ is shared across the $k$-step block, thereby coupling $\theta$ across multiple actions [2605.10909].

The main theoretical result is a near-optimality guarantee for critical points. Under bounded cost, differentiability and smoothness of $J^{\tilde\pi,k}(\mu)$, and full-simplex parameterization over $\Pi_{\rm det}$, any stationary point or boundary local minimum $\theta^*$ satisfies
\[
\mathbb E_{\pi_{\rm det}\sim\tilde\pi_{\theta^*}}[J^{\pi_{\rm det}}(\mu)]-J^{\pi_{\rm det}^*}(\mu)
\le \frac{8g_{\max}\gamma^k}{1-\gamma}
=O(e^{-ck}),
\]
where $c=-\ln\gamma>0$ [2605.10909]. The paper also states that projected gradient descent with $\eta=1/\beta$ and mirror descent with $\eta=\lambda/\beta$ achieve an $O(1/T)$ convergence rate up to the exponentially small $k$-step bias term.

The paper explicitly contrasts this with other multi-step devices. It states that $n$-step returns or GAE use truncated advantage estimators but do not correlate randomness, so they reduce variance without changing the landscape’s critical points; by contrast, $k$-step policy gradients alter the landscape by coupling randomness over $k$ steps [2605.10909]. This provides a sharp conceptual distinction between non-myopic pathwise policy gradients and merely longer-horizon return estimators.

## 7. Comparative interpretation, misconceptions, and research significance

The literature does not present a single universal NM-PPG formulation. Instead, the term covers several trajectory-level gradient constructions with different sources of non-myopia and different computational regimes.

| Setting | Core mechanism | Reported computational strategy |
|---|---|---|
| Hierarchical model-based policy optimization | Natural gradient in path space using the pathwise Fisher metric | Exact analytic computation from $D$, $E$, and $C$ [1912.01448] |
| Non-myopic Bayesian optimization | Reparameterized rollout gradients through GP fantasies | Monte Carlo or QMC trajectories with backpropagation [2408.07812] |
| Active feature acquisition | Continuous relaxation and straight-through pathwise differentiation | Gumbel-Softmax relaxation, ST rollouts, entropy regularization [2605.05511] |
| Restricted-policy $k$-step PG | Coupling randomness over $k$-step blocks | Projected gradient descent or mirror descent on correlated policies [2605.10909] |

A common misconception is to equate non-myopic policy gradients with standard policy gradients using longer returns. The restricted-policy analysis directly argues against this equivalence: truncated-return estimators such as $n$-step returns or GAE do not correlate the policy randomness across time and therefore do not change the critical-point structure in the same way [2605.10909]. Likewise, the model-based path-space formulation does not merely extend horizon length; it changes the optimization geometry through the Fisher metric on complete trajectories [1912.01448].

Another misconception is that NM-PPG necessarily implies model-based exactness. That property belongs specifically to the hierarchical model-based formulation, which assumes access to a perfect dynamics model and computes expectations analytically using successor representations [1912.01448]. The Bayesian-optimization and active-feature-acquisition variants instead rely on reparameterized Monte Carlo trajectories, variance reduction, and differentiable relaxations [2408.07812; 2605.05511].

The significance of the NM-PPG family lies in a shared response to myopia in sequential decision-making. In one line, long-range state-action dependencies are encoded through higher-order successor representations and a pathwise Fisher geometry [1912.01448]. In another, differentiable policy rollouts make non-myopic acquisition learning practical in Bayesian optimization [2408.07812]. In a third, end-to-end differentiable relaxations allow future prediction loss and acquisition cost to influence present feature-selection decisions across an entire trajectory [2605.05511]. Related theory shows that coupling randomness across multiple steps can eliminate spurious stationary points induced by one-step optimization in restricted policy classes, with exponentially improving guarantees in $k$ [2605.10909].

Taken together, these works suggest that the defining contribution of NM-PPG is not a single update rule but a trajectory-centric view of policy optimization: gradients should be taken with respect to the full induced path distribution, or an explicit multi-step surrogate of it, whenever one-step action perturbations fail to reflect the structure of delayed consequences.

Source: https://www.emergentmind.com/topics/non-myopic-pathwise-policy-gradients-nm-ppg