Non-Myopic Pathwise Policy Gradients

• Non-Myopic Pathwise Policy Gradients (NM-PPG) are methods that optimize stochastic policies using full trajectory evaluations to capture long-range dependencies in decision processes.
• They employ analytical techniques or reparameterized rollouts to compute gradients over multi-step trajectories, enhancing credit assignment beyond one-step action evaluations.
• By coupling randomness across actions, NM-PPG overcomes myopic local optima and has been applied in hierarchical policy optimization, Bayesian optimization, and active feature acquisition.

Searching arXiv for the cited NM-PPG-related papers to ground the article in current records. Non-Myopic Pathwise Policy Gradients (NM-PPG) denotes a family of policy-gradient methods that perform credit assignment over multi-step trajectories rather than only through one-step action evaluations. Across several distinct formulations, the common idea is to optimize a stochastic policy by differentiating an objective defined on full paths or multi-step rollouts, so that parameter updates reflect long-range dependencies among actions, observations, and downstream rewards or costs. In the literature, the label covers at least three closely related but application-specific lines: a hierarchical model-based policy optimization framework in path space (McNamee, 2019), a rollout-gradient method for non-myopic Bayesian optimization (Nwankwo et al., 2024), and a pathwise-gradient approach for non-myopic active feature acquisition (Aronsson et al., 6 May 2026). Related work on $k$-step policy gradients formalizes a closely aligned mechanism—coupling policy randomness across a finite horizon to escape myopic local optima in restricted policy classes (DeWeese et al., 11 May 2026).

1. Conceptual scope and defining idea

The unifying property of NM-PPG methods is non-myopic credit assignment. In the path-space formulation, every infinite trajectory $\tau=(s_0,a_0,s_1,a_1,\dots)$ is treated as a sample from the path density

$$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$

and the objective is the expected cumulative reward

$$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$

A second-order update in the policy parameters then coincides with the natural gradient with respect to a Fisher metric defined on entire trajectories (McNamee, 2019). This makes the update sensitive to the correlational structure of induced state-action paths rather than only to local action perturbations.

In Bayesian optimization, the same non-myopic principle appears in a finite-horizon MDP whose states are Gaussian-process posteriors, actions are query points, and one-step reward is improvement over the best-so-far. There, NM-PPG optimizes a stochastic sampling policy $\pi_\theta(a\mid s)$ for the $H$-step value

$$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$

using a pathwise derivative estimator based on reparameterized rollouts (Nwankwo et al., 2024).

In active feature acquisition, NM-PPG is formulated as a finite-horizon POMDP in which actions either acquire a feature or choose STOP. The objective combines feature-acquisition cost with prediction loss, and a continuous relaxation enables backpropagation through the full acquisition trajectory (Aronsson et al., 6 May 2026). Although the state, action, and reward semantics differ across these domains, the methodological commonality is direct differentiation through a trajectory-level objective.

A plausible implication is that NM-PPG is best understood not as a single algorithmic template, but as a design pattern: define a sequential decision problem, re-express the objective on full paths or multi-step blocks, and compute gradients that preserve long-range dependence information.

2. Path-space natural gradients and hierarchical model-based policy optimization

In "Hierarchical model-based policy optimization: from actions to action sequences and back" (McNamee, 2019), NM-PPG appears as a normative second-order framework in path space. If $\pi_\theta(a\mid s)$ is treated as an exponential family in natural parameters $\theta$, then a local quadratic model around $\theta^k$ yields

$\tau=(s_0,a_0,s_1,a_1,\dots)$0

with optimum

$\tau=(s_0,a_0,s_1,a_1,\dots)$1

Here the pathwise Fisher information is

$\tau=(s_0,a_0,s_1,a_1,\dots)$2

where $\tau=(s_0,a_0,s_1,a_1,\dots)$3 is the trajectory score function (McNamee, 2019).

This formulation gives the update

$\tau=(s_0,a_0,s_1,a_1,\dots)$4

or, in the small-step trust-region form,

$\tau=(s_0,a_0,s_1,a_1,\dots)$5

Because $\tau=(s_0,a_0,s_1,a_1,\dots)$6 and $\tau=(s_0,a_0,s_1,a_1,\dots)$7, the update is expressed in terms of path-score covariances (McNamee, 2019).

The paper’s central interpretive claim is that such updates are hierarchy-sensitive. Long-range correlational structure in stationary state-action densities affects the natural path gradient, and in simulation the resulting policy flow prioritizes local updates in a way that reflects an intuitive state-space hierarchy (McNamee, 2019). The paper explicitly connects this effect to “higher-order successor representations,” which provide the algebraic machinery for analytic computation of the metric and gradient.

3. Exact model-based computation via successor representations

The model-based NM-PPG framework assumes access to a perfect environment model $\tau=(s_0,a_0,s_1,a_1,\dots)$8, allowing expectations over trajectories to be computed analytically rather than by Monte Carlo sampling (McNamee, 2019). The one-step transition matrix under policy $\tau=(s_0,a_0,s_1,a_1,\dots)$9 is

$$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$0

and the discounted successor representation is

$$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$1

The state-action to state successor tensor is

$$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$2

so that $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$3 is the expected future count of state $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$4 after taking $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$5 (McNamee, 2019).

Using these objects, one constructs the second-order co-occurrence or counter-correlation

$$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$6

where $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$7 is the discounted count of visits to $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$8 in a path $$p_\theta(\tau)=\prod_{t\ge0}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),$$9. Closed-form expressions for $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$0 follow from $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$1 and $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$2. The Fisher metric $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$3 and the policy gradient can then be written entirely in terms of $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$4 and the known rewards $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$5 (McNamee, 2019).

The paper’s high-level pseudocode builds $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$6, then $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$7, then $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$8, then the co-occurrence tensor $$J(\theta)=\mathbb E_{\tau\sim p_\theta}\Bigl[\sum_{t\ge0}R(s_t,a_t,s_{t+1})\Bigr].$$9, followed by the Fisher matrix and gradient, before applying the natural-gradient update and reconstructing the policy with a softmax parameterization (McNamee, 2019). The algorithmic significance is explicit: no sampling of long trajectories is required, and every gradient and metric term is computed analytically from successor quantities. The paper characterizes this as a model-based exact-gradient property.

The same section motivates the term non-myopic. The tensor $\pi_\theta(a\mid s)$0 quantifies how often, under the current policy, action $\pi_\theta(a\mid s)$1 in state $\pi_\theta(a\mid s)$2 leads after many steps to action $\pi_\theta(a\mid s)$3 in state $\pi_\theta(a\mid s)$4. A reward at $\pi_\theta(a\mid s)$5 is therefore backed up to $\pi_\theta(a\mid s)$6 in proportion to this co-occurrence, with normalization terms removing spurious attributions induced by policy normalization (McNamee, 2019). This suggests that NM-PPG can be viewed as replacing horizon truncation or bootstrap design with a direct accounting of all horizons at once.

4. Pathwise rollout gradients in non-myopic Bayesian optimization

In "Differentiating Policies for Non-Myopic Bayesian Optimization" (Nwankwo et al., 2024), Bayesian optimization is cast as an MDP whose state is the current GP posterior, equivalently the dataset $\pi_\theta(a\mid s)$7 together with posterior mean and covariance, and whose action is the next query point $\pi_\theta(a\mid s)$8. The transition density is induced by the GP posterior predictive, and the one-step reward is

$\pi_\theta(a\mid s)$9

where $H$0 and $H$1 (Nwankwo et al., 2024).

Rather than optimize a single-step acquisition function, the method considers a stochastic policy $H$2 and seeks

$H$3

A convenient policy choice is a Gibbs policy

$H$4

with a common two-dimensional feature map

$H$5

where $H$6 and $H$7 are the GP-posterior mean and standard deviation (Nwankwo et al., 2024).

The key technical device is rollout reparameterization. If $H$8 is a location-scale family, the query can be written

$H$9

with $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$0 or $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$1 for a Sobol-scrambled QMC sequence. The GP draw is likewise parameterized as

$$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$2

with $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$3 (Nwankwo et al., 2024). Because the randomness is independent of $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$4, the gradient may be pushed inside the expectation: $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$5

The paper states an unbiased pathwise gradient proposition: under almost-everywhere differentiability of $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$6 and the GP updates, the Monte Carlo estimator

$$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$7

satisfies $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$8 and has variance $$V_H^\theta(s_0)=\mathbb E_{\tau\sim\pi_\theta}\Bigl[\sum_{t=0}^{H-1}R(s_t,x_t,s_{t+1})\Bigr],$$9 (Nwankwo et al., 2024). It further reports that quasi-Monte Carlo Sobol draws reduce empirical variance, often by one or more orders of magnitude. Computationally, the method combines QMC trajectories, backpropagation through fantasized GP updates, stochastic gradient ascent for inner policy argmax, common random numbers across successive states, a control-variate construction using analytically known EI, and rank-one Cholesky updates based on Schur complements (Nwankwo et al., 2024).

Empirically, the paper evaluates 15 standard synthetic test functions and compares myopic EI and PI, UCB, Knowledge Gradient, fixed-h rollout policies, and NM-PPG. It reports that all non-myopic methods outperform myopic EI and PI, and that NM-PPG achieves the best mean and median gap on 10 out of 15 problems. It also reports a per-iteration overhead of approximately $\pi_\theta(a\mid s)$0 that of $\pi_\theta(a\mid s)$1 rollout, with benefit appearing in 20–30 iterations through better minima (Nwankwo et al., 2024).

5. Continuous relaxation and straight-through rollouts for active feature acquisition

In "Non-Myopic Active Feature Acquisition via Pathwise Policy Gradients" (Aronsson et al., 6 May 2026), NM-PPG is developed for active feature acquisition (AFA), where features are costly and the learner adaptively chooses which values to acquire and when to stop. The problem is formulated as a finite-horizon POMDP. The latent state at time $\pi_\theta(a\mid s)$2 is

$\pi_\theta(a\mid s)$3

where $\pi_\theta(a\mid s)$4 is the set of acquired features. The observation is the currently revealed values,

$\pi_\theta(a\mid s)$5

or equivalently the masked representation

$\pi_\theta(a\mid s)$6

The action space is

$\pi_\theta(a\mid s)$7

where $\pi_\theta(a\mid s)$8 acquires a feature at cost $\pi_\theta(a\mid s)$9 and $\theta$0 is STOP (Aronsson et al., 6 May 2026).

The one-step cost is

$\theta$1

and the total cost under policy $\theta$2 is

$\theta$3

The objective minimizes

$\theta$4

which the paper states exactly matches the standard AFA objective (Aronsson et al., 6 May 2026).

Because discrete acquisition and stopping are not differentiable, the method introduces a continuous relaxation. The discrete mask $\theta$5 becomes a soft mask $\theta$6, and a survival mass $\theta$7 tracks the remaining probability of continuing. A neural network outputs logits

$\theta$8

and a differentiable relaxed action is sampled using Gumbel-Softmax. The relaxed mask and survival dynamics are

$\theta$9

leading to a soft return $\theta^k$0 and surrogate objective

$\theta^k$1

The paper states that this surrogate converges to the discrete objective as $\theta^k$2 (Aronsson et al., 6 May 2026).

Since all randomness is reparameterized, the pathwise gradient is

$\theta^k$3

The paper emphasizes that this avoids score-function terms and thereby reduces gradient variance (Aronsson et al., 6 May 2026). To reduce train-test mismatch, it introduces a straight-through rollout scheme: the forward pass uses hard feature acquisitions to update the discrete mask, while the backward pass replaces hard variables with straight-through surrogates so gradients flow through the soft relaxation. The resulting ST objective $\theta^k$4 yields a biased but low-variance gradient, and the paper attributes stable training to this design together with entropy regularization and staged temperature sharpening over the schedule

$\theta^k$5

(Aronsson et al., 6 May 2026).

The empirical study spans 12 benchmarks, including Cube-NM, Syn1, Syn3, Connect4, Splice, EngineFaultDB, METABRIC, NHANES Mortality, Diabetes, MNIST, and Fashion-MNIST. Compared baselines include DiFA, GDFS, DIME, AACO, SEFA, GSMRL, and OL. The reported findings are that NM-PPG outperforms all non-myopic baselines in both stability and accuracy-cost trade-off, degrades gracefully on purely myopic datasets, learns the intended context-first acquisition paths on synthetic context tasks, finds non-myopic two-step screening policies on medical benchmarks, and depends strongly on both ST rollouts and entropy regularization for stable training (Aronsson et al., 6 May 2026).

6. Relation to $\theta^k$6-step policy gradients and restricted policy classes

"Revisiting Policy Gradients for Restricted Policy Classes: Escaping Myopic Local Optima with $\theta^k$7-step Policy Gradients" (DeWeese et al., 11 May 2026) is not presented as the same algorithm as the preceding works, but it articulates a closely related mechanism and explicitly uses the phrase “Non-Myopic Pathwise Policy Gradients (NM-PPG)” in its detailed summary. The paper’s starting point is that standard one-step policy gradients can stall at suboptimal critical points when the policy class is restricted. The stated reason is that the gradient improves the policy based on the one-step $\theta^k$8-function and tests a single action at the current state before reverting to $\theta^k$9 immediately (DeWeese et al., 11 May 2026).

The proposed remedy is to couple policy randomness for $\tau=(s_0,a_0,s_1,a_1,\dots)$00 consecutive steps. Let $\tau=(s_0,a_0,s_1,a_1,\dots)$01 be a finite set of deterministic policies, and let $\tau=(s_0,a_0,s_1,a_1,\dots)$02 be a correlated policy. In the $\tau=(s_0,a_0,s_1,a_1,\dots)$03-step evaluation, a deterministic subpolicy is sampled at times $\tau=(s_0,a_0,s_1,a_1,\dots)$04 and then held fixed for the next $\tau=(s_0,a_0,s_1,a_1,\dots)$05 steps. This induces

$\tau=(s_0,a_0,s_1,a_1,\dots)$06

and the $\tau=(s_0,a_0,s_1,a_1,\dots)$07-step performance objective

$\tau=(s_0,a_0,s_1,a_1,\dots)$08

With a reparameterization $\tau=(s_0,a_0,s_1,a_1,\dots)$09, the paper gives an informal pathwise derivative formula in which the same noise variable $\tau=(s_0,a_0,s_1,a_1,\dots)$10 is shared across the $\tau=(s_0,a_0,s_1,a_1,\dots)$11-step block, thereby coupling $\tau=(s_0,a_0,s_1,a_1,\dots)$12 across multiple actions (DeWeese et al., 11 May 2026).

The main theoretical result is a near-optimality guarantee for critical points. Under bounded cost, differentiability and smoothness of $\tau=(s_0,a_0,s_1,a_1,\dots)$13, and full-simplex parameterization over $\tau=(s_0,a_0,s_1,a_1,\dots)$14, any stationary point or boundary local minimum $\tau=(s_0,a_0,s_1,a_1,\dots)$15 satisfies

$\tau=(s_0,a_0,s_1,a_1,\dots)$16

where $\tau=(s_0,a_0,s_1,a_1,\dots)$17 (DeWeese et al., 11 May 2026). The paper also states that projected gradient descent with $\tau=(s_0,a_0,s_1,a_1,\dots)$18 and mirror descent with $\tau=(s_0,a_0,s_1,a_1,\dots)$19 achieve an $\tau=(s_0,a_0,s_1,a_1,\dots)$20 convergence rate up to the exponentially small $\tau=(s_0,a_0,s_1,a_1,\dots)$21-step bias term.

The paper explicitly contrasts this with other multi-step devices. It states that $\tau=(s_0,a_0,s_1,a_1,\dots)$22-step returns or GAE use truncated advantage estimators but do not correlate randomness, so they reduce variance without changing the landscape’s critical points; by contrast, $\tau=(s_0,a_0,s_1,a_1,\dots)$23-step policy gradients alter the landscape by coupling randomness over $\tau=(s_0,a_0,s_1,a_1,\dots)$24 steps (DeWeese et al., 11 May 2026). This provides a sharp conceptual distinction between non-myopic pathwise policy gradients and merely longer-horizon return estimators.

7. Comparative interpretation, misconceptions, and research significance

The literature does not present a single universal NM-PPG formulation. Instead, the term covers several trajectory-level gradient constructions with different sources of non-myopia and different computational regimes.

Setting	Core mechanism	Reported computational strategy
Hierarchical model-based policy optimization	Natural gradient in path space using the pathwise Fisher metric	Exact analytic computation from $\tau=(s_0,a_0,s_1,a_1,\dots)$25, $\tau=(s_0,a_0,s_1,a_1,\dots)$26, and $\tau=(s_0,a_0,s_1,a_1,\dots)$27 (McNamee, 2019)
Non-myopic Bayesian optimization	Reparameterized rollout gradients through GP fantasies	Monte Carlo or QMC trajectories with backpropagation (Nwankwo et al., 2024)
Active feature acquisition	Continuous relaxation and straight-through pathwise differentiation	Gumbel-Softmax relaxation, ST rollouts, entropy regularization (Aronsson et al., 6 May 2026)
Restricted-policy $\tau=(s_0,a_0,s_1,a_1,\dots)$28-step PG	Coupling randomness over $\tau=(s_0,a_0,s_1,a_1,\dots)$29-step blocks	Projected gradient descent or mirror descent on correlated policies (DeWeese et al., 11 May 2026)

A common misconception is to equate non-myopic policy gradients with standard policy gradients using longer returns. The restricted-policy analysis directly argues against this equivalence: truncated-return estimators such as $\tau=(s_0,a_0,s_1,a_1,\dots)$30-step returns or GAE do not correlate the policy randomness across time and therefore do not change the critical-point structure in the same way (DeWeese et al., 11 May 2026). Likewise, the model-based path-space formulation does not merely extend horizon length; it changes the optimization geometry through the Fisher metric on complete trajectories (McNamee, 2019).

Another misconception is that NM-PPG necessarily implies model-based exactness. That property belongs specifically to the hierarchical model-based formulation, which assumes access to a perfect dynamics model and computes expectations analytically using successor representations (McNamee, 2019). The Bayesian-optimization and active-feature-acquisition variants instead rely on reparameterized Monte Carlo trajectories, variance reduction, and differentiable relaxations (Nwankwo et al., 2024, Aronsson et al., 6 May 2026).

The significance of the NM-PPG family lies in a shared response to myopia in sequential decision-making. In one line, long-range state-action dependencies are encoded through higher-order successor representations and a pathwise Fisher geometry (McNamee, 2019). In another, differentiable policy rollouts make non-myopic acquisition learning practical in Bayesian optimization (Nwankwo et al., 2024). In a third, end-to-end differentiable relaxations allow future prediction loss and acquisition cost to influence present feature-selection decisions across an entire trajectory (Aronsson et al., 6 May 2026). Related theory shows that coupling randomness across multiple steps can eliminate spurious stationary points induced by one-step optimization in restricted policy classes, with exponentially improving guarantees in $\tau=(s_0,a_0,s_1,a_1,\dots)$31 (DeWeese et al., 11 May 2026).

Taken together, these works suggest that the defining contribution of NM-PPG is not a single update rule but a trajectory-centric view of policy optimization: gradients should be taken with respect to the full induced path distribution, or an explicit multi-step surrogate of it, whenever one-step action perturbations fail to reflect the structure of delayed consequences.