Policy Gradient Guidance (PGG)

• Policy Gradient Guidance (PGG) is a family of methods that combines traditional policy-gradient updates with an auxiliary guidance signal to steer learning.
• It employs techniques like conditional–unconditional interpolation, analytical gradients, or likelihood rewards to modulate behavior and reduce variance.
• PGG shows benefits across discrete, continuous, multi-agent, and diffusion settings, though its performance is sensitive to guidance strength and domain specifics.

Searching arXiv for the cited PGG-related papers to ground the article in current records. Policy Gradient Guidance (PGG) denotes a family of methods that augment a baseline policy-gradient or score-based update with an auxiliary guidance signal designed to steer behavior while retaining the underlying optimization framework. In the most specific reinforcement-learning usage, PGG is the extension of classifier-free guidance from diffusion models to standard on-policy methods such as PPO by interpolating conditional and unconditional policy branches with a guidance strength $\gamma$ that can be varied at test time without retraining (Qi et al., 2 Oct 2025). Other works use the same label, or explicitly position their method as an instantiation of it, for task-aware exploration via analytical gradients through differentiable dynamics (Chang et al., 28 Mar 2026), for per-agent variance-reducing guidance in cooperative multi-agent learning (Yang et al., 23 Feb 2026), and for likelihood-guided posterior sampling in diffusion inverse problems (Tang et al., 2024). This suggests that PGG functions less as a single standardized algorithm than as a recurring design pattern.

1. Conceptual scope and recurring structure

Across the literature, PGG introduces a guidance term that is blended with a pre-existing update rule rather than replacing it outright. The base mechanism may be an on-policy actor update, an exploratory rollout policy, a per-agent actor-critic update in cooperative MARL, or a diffusion reverse-time sampler. The guidance term may come from an unconditional policy branch, analytical gradients through a differentiable model, an analytical reference state, or a likelihood reward.

Setting	Guidance source	Primary effect
On-policy PPO-style RL (Qi et al., 2 Oct 2025)	Interpolation of conditional and unconditional policy branches	Test-time control via $\gamma$
On-policy robotic RL (Chang et al., 28 Mar 2026)	Analytical gradients through differentiable dynamics	Directed exploration toward high-reward regions
Cooperative MARL (Yang et al., 23 Feb 2026)	Per-agent analytical coefficients from a reference state	Variance reduction and scale-invariant learning
Diffusion inverse problems (Tang et al., 2024)	Likelihood reward in a policy-gradient estimator	Posterior-sampling guidance

A common structural motif is the use of a guidance signal that remains compatible with an established backbone. In PPO-style PGG, the backbone is the standard advantage-weighted score-function estimator. In Where-to-Learn, the backbone is PPO updated on an augmented dataset. In DG-PG, the backbone is PPO or MAPPO with modified advantages. In DPG, the backbone is a pretrained diffusion prior whose score is augmented by a policy-gradient estimate of the likelihood term. Earlier works can also be read through this lens: PGQL combines regularized policy gradients with off-policy Q-learning guidance, and PGQ adds policy-gradient correction terms to stable off-policy value learning (O'Donoghue et al., 2016).

2. Classifier-free-guidance-style PGG for on-policy reinforcement learning

The formulation introduced in "Policy Gradient Guidance Enables Test Time Control" adapts classifier-free guidance (CFG) from diffusion models to standard, on-policy reinforcement learning, particularly PPO (Qi et al., 2 Oct 2025). The starting point is the structural similarity between the diffusion guidance score and the policy-gradient score function. In diffusion, a common form is

$$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$

or equivalently

$$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$

In policy-gradient RL, the analogous score-function identity is

$$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$

PGG therefore augments the policy with an unconditional branch and interpolates the conditional and unconditional branches. The conditional branch is the standard actor $\pi_\theta(a|s)$; the unconditional branch is an action distribution independent of $s$, implemented as $\pi_\theta(a|\varnothing)$ via a learnable null embedding passed through the same policy head. The guided policy is defined as

$$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$

with log form

$$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$

The parameter $\gamma$0 is the control parameter. The reported special cases are explicit: $\gamma$1 recovers unconditional behavior, $\gamma$2 recovers the conditional model, and $\gamma$3 amplifies the conditional signal. In the reinforcement-learning setting, this makes $\gamma$4 a test-time knob that modulates behavior without retraining.

The implementation is deliberately lightweight. For discrete actions, if $\gamma$5 denotes conditional logits and $\gamma$6 unconditional logits, the guided logits are

$\gamma$7

followed by a softmax. For continuous Gaussian policies, if $\gamma$8 and $\gamma$9 are means and $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$0 is shared covariance, the guided mean is

$$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$1

and the acting distribution is $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$2. These logit-level and mean-level constructions approximate the multiplicative interpolation in log space and are used both for rollouts and optimization.

3. Guided update, normalization cancellation, and empirical controllability

A central theoretical result of the PPO-style PGG formulation is that the state-only normalization term disappears under advantage estimation (Qi et al., 2 Oct 2025). Differentiating the guided log-density gives

$$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$3

Plugging this into the policy gradient yields an additional expectation involving $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$4. Because $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$5 depends only on $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$6, the term separates, and under advantage estimation with baseline $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$7 one has $$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$8. The normalization term therefore vanishes inside the advantage-weighted gradient, leading to the clean update

$$\nabla_x \log \hat{p}(x|y) = (1-\gamma)\nabla_x \log p(x) + \gamma \nabla_x \log p(x|y),$$9

The training procedures studied in the paper separate two regimes. One follows CFG-style conditioning dropout with $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$0: a learnable null embedding supplies the unconditional branch, rollouts are collected with the guided policy, and conditioning dropout with probability $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$1 replaces the conditional input by the null embedding during updates. The other removes dropout and instead trains with modestly larger guidance, typically $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$2. The second regime is reported as a stable, effective setting for continuous control.

The empirical pattern is domain-dependent. In discrete control, conditioning dropout improves early performance and enables strong test-time controllability. On CartPole-v1 at $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$3k steps, raising $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$4 at inference from $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$5 to $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$6 improves mean return from $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$7 to $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$8, and on Acrobot-v1 at $$\hat{p}(x|y) \propto p(x)^{1-\gamma} p(x|y)^\gamma.$$9k steps, mean return improves from $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$0 to $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$1. By $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$2k steps on CartPole-v1, multiple $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$3 settings achieve $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$4.

In continuous control, the picture is more constrained. With dropout, best performance tends to occur around $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$5, and increasing $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$6 often degrades results, especially in Humanoid and Walker2d. Without dropout and with training $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$7, moderate guidance improves stability and sample efficiency across several MuJoCo v4 tasks. At $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$8k steps on Hopper-v4, $$\nabla_\theta J(\theta) = \mathbb{E}\big[\nabla_\theta \log \pi_\theta(a|s)\,A(s,a)\big].$$9 achieves $\pi_\theta(a|s)$0, surpassing PPO’s $\pi_\theta(a|s)$1; on Walker2d-v4, $\pi_\theta(a|s)$2 reaches $\pi_\theta(a|s)$3 versus PPO’s $\pi_\theta(a|s)$4. InvertedPendulum-v4 often improves monotonically with $\pi_\theta(a|s)$5 up to approximately $\pi_\theta(a|s)$6, whereas in Humanoid-v4 and HalfCheetah-v4 larger $\pi_\theta(a|s)$7, such as $\pi_\theta(a|s)$8, tends to hurt asymptotics.

Several implementation consequences follow directly from the derivation. The critic should estimate $\pi_\theta(a|s)$9 because the cancellation of $s$0 relies on advantages computed under the acting distribution. Entropy regularization should be applied to $s$1, not $s$2, since $s$3 is the actual rollout distribution. For continuous Gaussians, the reported results keep $s$4 fixed or shared during guidance. The computational cost is higher because both conditional and unconditional branches must be evaluated, increasing compute by roughly doubling actor forward passes. A common misconception is that larger guidance is uniformly beneficial; the reported results explicitly show that excessive $s$5 harms both training and inference in continuous control.

4. Analytical PGG as directed exploration in robotic reinforcement learning

In "Where-to-Learn: Analytical Policy Gradient Directed Exploration for On-Policy Robotic Reinforcement Learning," PGG denotes a task-aware, physics-guided exploration mechanism that uses analytical policy gradients computed through a differentiable dynamics model to steer on-policy robotic RL toward high-reward regions (Chang et al., 28 Mar 2026). The motivation is a contrast with entropy maximization and novelty-seeking: stochastic actions, entropy bonuses, and intrinsic rewards often behave like Brownian motion in high-dimensional state-action spaces, whereas PGG derives exploration signals from the task objective and the robot’s dynamics.

The setup assumes differentiable dynamics

$s$6

reward $s$7, and objective

$s$8

For a truncated horizon $s$9 with terminal value $\pi_\theta(a|\varnothing)$0, the exploration objective from $\pi_\theta(a|\varnothing)$1 is

$\pi_\theta(a|\varnothing)$2

Backpropagation through the differentiable model yields a short-horizon analytical gradient. The guidance direction is

$\pi_\theta(a|\varnothing)$3

and the analytical policy gradient is approximated by

$\pi_\theta(a|\varnothing)$4

Where-to-Learn integrates this guidance into on-policy learning through a temporary exploratory policy. Starting from $\pi_\theta(a|\varnothing)$5, one performs $\pi_\theta(a|\varnothing)$6 APG epochs on $\pi_\theta(a|\varnothing)$7 to obtain $\pi_\theta(a|\varnothing)$8, collects datasets $\pi_\theta(a|\varnothing)$9 and $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$0 with fractions $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$1 and $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$2, merges them into $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$3, fits the critic on $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$4 only, and updates the primary actor on $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$5 using PPO’s clipped objective with piecewise importance weights. The reported hyperparameter ranges are $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$6, $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$7, $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$8, and $$\hat{\pi}_\theta(a|s) \propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a|s)^\gamma,$$9. The differentiable model in the experiments is a differentiable physics engine, Brax/MJX.

The empirical claims are specific. On eight tasks from the dm_control suite and locomotion in MuJoCo Playground, Where-to-Learn improves sample efficiency and training stability versus PPO and PPO+RND on most tasks. In PointMass, PGG maintains stable convergence where PPO collapses. A notable exception is Go1Getup, described as a highly discontinuous contact task, where RND outperforms PGG, highlighting the limitation of analytical guidance under severe discontinuities. On a LimX TRON 6-DOF point-foot biped, PGG halves the steps to reach comparable asymptotic return in simulation. It also improves velocity tracking, with MSE linear $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$0 reported as $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$1 versus $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$2, variance linear $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$3 as $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$4 versus $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$5, MSE angular $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$6 as $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$7 versus $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$8, and variance angular $$\log \hat{\pi}_\theta(a|s)=\gamma \log \pi_\theta(a|s)+(1-\gamma)\log \pi_\theta(a)-\log Z_\theta(s).$$9 as $\gamma$00 versus $\gamma$01. The policy transfers successfully to indoor and outdoor real-world tests without special tuning.

5. Multi-agent, replay-based, and off-policy interpretations

In cooperative MARL, "Descent-Guided Policy Gradient for Scalable Cooperative Multi-Agent Learning" presents DG-PG as a concrete realization of PGG (Yang et al., 23 Feb 2026). The motivating problem is cross-agent noise: with a common reward, agent $\gamma$02’s policy-gradient estimator depends on the sampled actions of all other agents, and the variance scales as $\gamma$03. DG-PG introduces a differentiable system state $\gamma$04, an analytical reference state $\gamma$05, and the deviation functional

$\gamma$06

For agent $\gamma$07, the local influence vector is

$\gamma$08

and the guidance coefficient is

$\gamma$09

The resulting estimator combines the standard policy-gradient term with a per-agent analytical guidance term whose coefficient does not depend on other agents’ sampled actions. DG-PG proves equilibria preservation, variance reduction from $\gamma$10 to $\gamma$11, and sample complexity $\gamma$12. In a heterogeneous cloud scheduling task with up to $\gamma$13 agents, DG-PG converges within approximately $\gamma$14 episodes at all scales; in the controlled comparison at $\gamma$15, DG-PG reaches $\gamma$16, surpassing Best-Fit’s $\gamma$17, while IPPO deteriorates sharply to $\gamma$18.

A replay-based interpretation appears in "Combining policy gradient and Q-learning," which describes PGQL as a method that augments regularized policy gradient updates with off-policy Q-learning guidance inferred from the policy’s action preferences (O'Donoghue et al., 2016). The fixed-point relation of entropy-regularized policy gradients yields

$\gamma$19

suggesting the estimator

$\gamma$20

PGQL then applies Q-learning-style Bellman updates to $\gamma$21 using replay and feeds the resulting Bellman-consistency signal back into the actor. On Atari, with all A3C hyperparameters kept identical and a single Q-learning step added per actor-learner update, PGQL is reported best in $\gamma$22 games, A3C in $\gamma$23, Q-learning in $\gamma$24, with $\gamma$25 ties. The mean normalized score under random starts is $\gamma$26 for PGQL, $\gamma$27 for A3C, and $\gamma$28 for Q-learning; the corresponding medians are $\gamma$29, $\gamma$30, and $\gamma$31.

An earlier off-policy control perspective appears in "Policy Gradient Methods for Off-policy Control," where PGQ augments stable GQ/TDC-style off-policy value learning with explicit policy-gradient correction terms (Lehnert et al., 2015). The target policy is a Boltzmann policy tied to the action-value parameters,

$\gamma$32

and the update contains terms of the form $\gamma$33 multiplied by TD-error-based quantities. The purpose is to correct for the $\gamma$34-dependence of both the Bellman operator and the stationary distribution. In Baird’s counterexample, Q-learning diverges, whereas GQ and PGQ converge to zero MSPBE in both the sampled-update and trajectory-based settings. These works were not originally framed as a unified PGG literature, but they establish a clear precedent for using value-based or policy-gradient-derived auxiliary signals to guide policy improvement.

6. PGG in diffusion posterior sampling and inverse problems

In "Solving General Noisy Inverse Problem via Posterior Sampling: A Policy Gradient Viewpoint," Diffusion Policy Gradient (DPG) instantiates PGG for inverse problems by viewing the conditional distribution over clean images given a noisy latent as a policy and the measurement likelihood as a reward (Tang et al., 2024). The measurement model is

$\gamma$35

and the posterior objective is to sample from $\gamma$36 using a pretrained diffusion prior.

The intractable term in posterior sampling is

$\gamma$37

DPG rewrites this using the REINFORCE identity by defining

$\gamma$38

so that

$\gamma$39

This is the PGG term in the diffusion setting: $\gamma$40

Because exact sampling from $\gamma$41 is expensive, DPG approximates it by

$\gamma$42

where Tweedie’s formula provides

$\gamma$43

and

$\gamma$44

The Monte Carlo estimator uses $\gamma$45 samples and a leave-one-out baseline,

$\gamma$46

with

$\gamma$47

The guidance augments the prior score additively,

$\gamma$48

and is then inserted into standard DDPM or DDIM updates.

The reported application range includes inpainting, $\gamma$49 super-resolution, Gaussian deblurring, motion deblurring, and nonlinear deblurring, under both Gaussian and Poisson noise, without task-specific fine-tuning. On FFHQ with Gaussian noise $\gamma$50, the inpainting FID is $\gamma$51 for DPG versus $\gamma$52 for DPS, with LPIPS $\gamma$53 versus $\gamma$54; for Gaussian deblurring, FID is $\gamma$55 versus $\gamma$56, with LPIPS $\gamma$57 versus $\gamma$58. On ImageNet, $\gamma$59 super-resolution yields FID $\gamma$60 versus $\gamma$61, and Gaussian deblurring yields FID $\gamma$62 versus $\gamma$63. For nonlinear deblurring on ImageNet, DPG reports FID $\gamma$64 versus $\gamma$65, LPIPS $\gamma$66 versus $\gamma$67, and PSNR $\gamma$68 versus $\gamma$69. The overhead is measurable but limited: on A100 GPUs, DPS is reported at approximately $\gamma$70 s per image, DPG with DDPM at approximately $\gamma$71 s, and DPG with DDIM at approximately $\gamma$72 s.

7. Common themes, limitations, and interpretive boundaries

The major shared property across these works is that guidance is additive or interpolative with respect to an existing optimization process. PPO-style PGG modifies the acting distribution through a conditional–unconditional interpolation while preserving a standard on-policy actor-critic backbone (Qi et al., 2 Oct 2025). Where-to-Learn uses analytical gradients only to obtain a temporary exploratory policy and then retains PPO for the main actor update (Chang et al., 28 Mar 2026). DG-PG modifies per-agent learning signals but does not require new critics or mixing networks (Yang et al., 23 Feb 2026). DPG augments the prior diffusion score rather than replacing the pretrained prior (Tang et al., 2024).

A second recurring feature is that the benefit of guidance is conditional rather than universal. In the classifier-free-guidance formulation, conditioning dropout helps discrete control but destabilizes continuous control, and excessive $\gamma$73 harms both training and test-time performance. In Where-to-Learn, analytical guidance improves most tasks but is limited by stiff or discontinuous contacts, as illustrated by Go1Getup. In DG-PG, the guarantees rely on exogeneity, alignment, and differentiable analytical models; strong misalignment can reduce performance, motivating $\gamma$74 scheduling. In DPG, Monte Carlo estimation introduces variance and computational overhead, and guidance strength $\gamma$75 must be tuned per task.

A third point concerns the scope of controllability. Only the classifier-free-guidance extension to PPO explicitly provides a single test-time control knob $\gamma$76 that modulates behavior without retraining (Qi et al., 2 Oct 2025). The robotics and MARL variants use guidance primarily during training, while DPG applies guidance during the sampling process itself (Chang et al., 28 Mar 2026). This distinction matters because the phrase “test-time control” is specific to one branch of the literature and should not be generalized to all methods called PGG.

A common misconception is therefore that PGG names a single algorithm with a fixed mathematical form. The surveyed works do not support that reading. Instead, they show a family resemblance: a policy-gradient-compatible update is supplemented by a directed signal derived from structure external to the baseline estimator, such as an unconditional prior branch, a differentiable physics model, an analytical reference state, a replay-based Bellman backup, or a measurement likelihood reward. Another misconception is that guidance simply replaces exploration heuristics. In the robotics formulation, PGG complements entropy bonuses rather than eliminating them; in PPO-style PGG, entropy regularization is still applied to the guided policy; and in diffusion, the prior score remains essential.

The resulting research picture is heterogeneous but coherent. PGG, as the term is presently used, refers to mechanisms that preserve the computational idiom of policy gradients or score-based sampling while injecting structured directional information. The technical details differ sharply across domains, but the central ambition is stable guidance without abandoning the backbone algorithm.