---
title: Normal Guidance in Policy Gradient
url: https://www.emergentmind.com/topics/normal-guidance
type: topic
---

# Normal Guidance in Policy Gradient

Policy Gradient Guidance (PGG) is a reinforcement-learning method that transfers the central idea of classifier-free guidance from diffusion models into ordinary on-policy policy-gradient optimization, specifically PPO-style methods. It augments the standard state-conditioned policy with an unconditional action branch and defines a guided policy whose behavior can be modulated at test time by a scalar guidance strength $\gamma$, without retraining the policy [2510.02148]. In this formulation, the ordinary or unguided setting is recovered exactly at $\gamma=1$, while $\gamma>1$ amplifies state-conditioned behavior and exposes a post-training control knob over action selection [2510.02148].

## 1. Conceptual basis and definition

PGG is built around an analogy to classifier-free guidance in diffusion models. In diffusion, guidance combines an unconditional model $p(x)$ and a conditional model $p(x\mid y)$ into a guided distribution of the form
\[
\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.
\]
The paper adopts the same structure in policy space by treating the standard state-conditioned actor as the conditional branch and an added state-agnostic action prior as the unconditional branch [2510.02148].

The conditional policy is the usual actor,
\[
\pi_\theta(a\mid s),
\]
while the unconditional policy ignores the state,
\[
\pi_\theta(a),
\]
or equivalently $\pi_\theta(a\mid \emptyset)$ with a learnable null embedding. The guided policy is then defined as
\[
\hat{\pi}_\theta(a\mid s)\propto \pi_\theta(a)^{\,1-\gamma}\,\pi_\theta(a\mid s)^{\,\gamma}.
\]
This multiplicative interpolation in probability space is the core definition of PGG [2510.02148].

The semantics of $\gamma$ are explicit in the formulation.

| $\gamma$ | Guided behavior |
|---|---|
| $0$ | purely unconditional behavior |
| $1$ | recovers the standard policy |
| $>1$ | amplifies conditional behavior |

This parameterization is important because it fixes the meaning of the default setting. In PGG, the normal or unguided case is not an equal mixture of conditional and unconditional branches; it is exactly $\gamma=1$, where
\[
\hat{\pi}_\theta(a\mid s)=\pi_\theta(a\mid s)
\]
up to normalization [2510.02148]. A plausible implication is that PGG should be understood less as a new policy class than as a deployable guidance mechanism layered onto a conventional actor.

## 2. Mathematical formulation and policy-gradient derivation

The main technical obstacle in transferring diffusion-style guidance to reinforcement learning is that policy optimization differentiates with respect to parameters $\theta$, not actions. For action derivatives, the normalization disappears in the same way as in diffusion:
\[
\nabla_a \log \hat{\pi}_\theta(a\mid s) = \gamma \nabla_a \log \pi_\theta(a\mid s) + (1-\gamma)\nabla_a \log \pi_\theta(a).
\]
However, parameter derivatives introduce a state-dependent normalization term:
\[
\nabla_\theta \log \hat{\pi}_\theta(a\mid s) = \gamma \nabla_\theta \log \pi_\theta(a\mid s) + (1-\gamma)\nabla_\theta \log \pi_\theta(a) - \nabla_\theta \log Z_\theta(s),
\]
with
\[
\hat{\pi}_\theta(a\mid s)=\frac{\pi_\theta(a)^{1-\gamma}\pi_\theta(a\mid s)^\gamma}{Z_\theta(s)}.
\]
The paper identifies this $Z_\theta(s)$ term as the key technical obstacle [2510.02148].

Starting from the policy gradient theorem,
\[
\nabla_\theta J(\theta)=\mathbb{E}\!\left[A\,\nabla_\theta\log \hat{\pi}_\theta(a\mid s)\right],
\]
the derivation shows that the normalization term vanishes in expectation when advantages are used properly. The crucial property is
\[
\mathbb{E}_{a\sim \hat{\pi}}[A]=0,
\]
since $A=Q(s,a)-V(s)$ under the rollout policy. This cancels the normalization contribution and yields the guided update
\[
\nabla_\theta J(\theta) = \mathbb{E}\!\left[A \left(\gamma \nabla_\theta \log \pi_\theta(a\mid s) + (1-\gamma)\nabla_\theta \log \pi_\theta(a)\right)\right].
\]
This is the central policy-gradient update under PGG [2510.02148].

The update can be read as a weighted combination of the usual conditional policy gradient and an unconditional branch gradient. When $\gamma>1$, the coefficient $(1-\gamma)$ is negative. The paper interprets this regime as amplifying the conditional branch while training the unconditional branch to capture “what not to do” [2510.02148]. This suggests that PGG is not merely inference-time interpolation; the training dynamics themselves are biased by the chosen guidance regime.

## 3. Training procedure and test-time control

The training procedure is a minimal modification of PPO. The implementation starts from CleanRL PPO, adds an unconditional branch via a learnable null embedding, collects rollouts using the guided policy
\[
\hat{\pi}_\theta(a\mid s)\propto \pi_\theta(a\mid \emptyset)^{1-\gamma}\pi_\theta(a\mid s)^\gamma,
\]
computes advantages such as GAE, and updates PPO using $\hat{\pi}_\theta$ in place of the ordinary policy [2510.02148].

Two implementation choices depend on the action space. For discrete actions, the method uses a softmax over guided logits. For continuous actions, it uses a Gaussian with guided mean and shared covariance [2510.02148]. Conceptually, the construction remains defined in probability space, but operationally it becomes a logit-space or mean-space instantiation depending on whether the policy is categorical or Gaussian.

A central design goal is post-training controllability. Once both branches have been trained, behavior can be changed at evaluation time by varying $\gamma$ without retraining. This is the precise analogue of inference-time control in classifier-free guidance [2510.02148]. The paper studies two training modes.

The first is dropout-based PGG, where conditioning dropout is used with default probability
\[
p_{\text{drop}}=0.1.
\]
This is motivated by diffusion guidance, where the unconditional branch must receive enough training signal. The second is no-dropout, biased-guidance PGG, where $p_{\text{drop}}=0$ and training is performed with $\gamma=1.1$ so that both branches receive gradient signal through the guided objective [2510.02148].

The paper treats the second regime as more stable in continuous control. A plausible implication is that, unlike diffusion models, on-policy reinforcement learning is more sensitive to data-collection perturbations induced by conditioning dropout, because dropout changes the rollout policy and therefore the state distribution.

## 4. The meaning of “normal” or unguided guidance

In this work, the normal, ordinary, or default setting is unambiguous: it is the standard conditional actor recovered at $\gamma=1$. At that value,
\[
\hat{\pi}_\theta(a\mid s)\propto \pi_\theta(a)^0 \pi_\theta(a\mid s)^1=\pi_\theta(a\mid s),
\]
so there is no effective interpolation from the unconditional branch in the deployed action distribution [2510.02148].

This point matters because classifier-free guidance in diffusion is often discussed as interpolation between conditional and unconditional predictions. PGG is parameterized differently: the standard actor is recovered exactly at $\gamma=1$, not at some intermediate value. Therefore the paper’s own terminology implies the following equivalence:
- unguided policy gradient;
- standard PPO with policy $\pi_\theta(a\mid s)$;
- PGG evaluated at $\gamma=1$.

There is one nuance. A PGG-trained model evaluated at $\gamma=1$ may still differ architecturally from vanilla PPO because it includes an unconditional branch and may have been trained with dropout or biased guidance. Functionally, however, the deployed policy at $\gamma=1$ is the paper’s no-guidance case [2510.02148].

The paper also notes that in continuous-control dropout experiments, the best results often occur at $\gamma=1.0$, which it explicitly interprets as “no effective contribution from the unconditional branch” [2510.02148]. This observation is significant because it distinguishes architectural augmentation from behavioral guidance: having the branch available is not the same as using it at deployment.

## 5. Empirical results across discrete and continuous control

The empirical study covers discrete control on CartPole-v1 and Acrobot-v1, and continuous control on MuJoCo v4 tasks HalfCheetah, Hopper, Humanoid, InvertedPendulum, Pusher, and Walker2d. The protocol uses 5 random seeds per environment, 50 test episodes per trained policy, mean and standard deviation of episodic return, training for $5\times 10^5$ timesteps on discrete tasks, and $10^6$ timesteps on continuous tasks [2510.02148].

In discrete control, guidance is strongly beneficial. With conditioning dropout, performance at $\gamma=1$ is similar to PPO, but increasing test-time $\gamma$ substantially improves returns, especially early in training. At 100k steps on CartPole-v1, the appendix table reports:
- PPO: $270.6 \pm 81.3$
- PGG $\gamma=1$: $337.8 \pm 96.4$
- PGG $\gamma=10$: $444.8 \pm 85.3$
- PGG $\gamma=20$: $458.6 \pm 66.2$

At 200k steps on CartPole-v1, PGG with $\gamma\ge 2$ reaches $500.0\pm 0.0$. On Acrobot-v1 at 100k steps, the same appendix shows:
- PPO: $-111.3 \pm 7.5$
- PGG $\gamma=1$: $-111.7 \pm 7.0$
- PGG $\gamma=5$: $-87.4 \pm 3.4$
- PGG $\gamma=20$: $-82.7 \pm 4.4$ [2510.02148]

In continuous control with $10\%$ conditioning dropout, the picture changes. The best performance often occurs with mild or no guidance,
\[
\gamma\in[1.0,1.05],
\]
and larger $\gamma$ introduces instability, especially in higher-dimensional environments such as HalfCheetah, Humanoid, and Walker2d [2510.02148]. The paper concludes that conditioning dropout, although central in diffusion CFG, does not transfer cleanly to continuous on-policy RL.

The strongest continuous-control results come from no-dropout training with modestly larger guidance:
\[
\gamma=1.1,\qquad p_{\text{drop}}=0.
\]
This improves controllability and often improves returns and stability. At 1M steps, examples reported in the appendix include:
- HalfCheetah-v4: PPO $1846.9 \pm 1026.3$, PGG $\gamma=1.05$ $2257.0 \pm 1411.6$
- Humanoid-v4: PPO $660.7 \pm 96.2$, PGG $\gamma=1.05$ $677.2 \pm 29.2$
- InvertedPendulum-v4: PPO $864.7 \pm 143.8$, PGG $\gamma=1.5$ $995.0 \pm 9.8$
- Walker2d-v4: PPO $3056.0 \pm 950.5$, PGG $\gamma=1.2$ $3508.7 \pm 694.3$, but PGG $\gamma=1.3$ falls to $2819.6 \pm 1227.4$ [2510.02148]

The aggregate empirical pattern is consistent across the paper: discrete tasks benefit from stronger-than-standard test-time guidance, while continuous tasks benefit from modest guidance and degrade under excessive sharpening. This suggests that action-space dimensionality and rollout sensitivity materially affect how much extrapolation beyond the conditional branch is tolerable.

## 6. Interpretation, limitations, and relation to broader guidance methods

PGG’s main conceptual contribution is to show that guidance, previously associated mainly with diffusion or other generative policies, can be adapted to standard on-policy policy-gradient reinforcement learning [2510.02148]. Relative to ordinary policy-gradient methods, it adds an unconditional branch, a guidance parameter $\gamma$, a guided rollout policy, and test-time behavior control without retraining.

The paper’s practical interpretation is correspondingly narrow. In simple discrete tasks, large test-time $\gamma$ can substantially improve returns and sample efficiency. In continuous control, the preferred recipe is to train without dropout, use a slightly biased training guidance such as $\gamma=1.1$, and sweep test-time $\gamma$ in a modest range such as $1.05$–$1.2$ [2510.02148]. This is presented as the cleanest practical route to controllable post-training behavior.

The paper also identifies several caveats. Guidance sensitivity is environment-dependent; too large $\gamma$ destabilizes complex tasks; the derivation assumes advantage estimates are good enough that the normalization term vanishes in expectation; if the critic is biased or poorly trained, the cancellation may be imperfect; and evaluating both conditional and unconditional branches roughly doubles actor-side compute relative to vanilla PPO [2510.02148]. It also notes that the unconditional policy itself can be weaker than vanilla PPO in some environments, so gains may appear only after extrapolation with $\gamma>1$.

In broader methodological context, PGG directly mirrors classifier-free guidance in diffusion, where unconditional and conditional branches are combined at inference time. A related diffusion result argues that applying constant guidance across all denoising steps is suboptimal and that guidance is most helpful in a limited middle interval of the trajectory [2404.07724]. PGG does not introduce an analogous stage-wise guidance schedule for reinforcement learning, but this parallel suggests a possible future direction: the usefulness of guidance may depend not only on strength but also on when and where it is applied.

PGG therefore occupies a specific place in controllable RL. It is not a new objective from scratch, nor a generative-policy method. Its novelty lies in deriving a guided policy
\[
\hat{\pi}_\theta(a\mid s)\propto \pi_\theta(a)^{1-\gamma}\pi_\theta(a\mid s)^\gamma,
\]
showing that the normalization term disappears under advantage estimation, and exposing $\gamma$ as a deployment-time control knob for standard on-policy reinforcement learning [2510.02148].

Source: https://www.emergentmind.com/topics/normal-guidance