Normal Guidance in Policy Gradient

• Normal Guidance is a reinforcement learning method that augments standard policy gradients with an unconditional action branch and a scalar control parameter γ.
• The method mathematically interpolates between conditional and unconditional branches, with the guidance strength effectively modulating the policy directly through probability space.
• Empirical results show strong performance in discrete tasks and highlight careful tuning of γ as critical for stability in continuous control environments.

Policy Gradient Guidance (PGG) is a reinforcement-learning method that transfers the central idea of classifier-free guidance from diffusion models into ordinary on-policy policy-gradient optimization, specifically PPO-style methods. It augments the standard state-conditioned policy with an unconditional action branch and defines a guided policy whose behavior can be modulated at test time by a scalar guidance strength $\gamma$, without retraining the policy (Qi et al., 2 Oct 2025). In this formulation, the ordinary or unguided setting is recovered exactly at $\gamma=1$, while $\gamma>1$ amplifies state-conditioned behavior and exposes a post-training control knob over action selection (Qi et al., 2 Oct 2025).

1. Conceptual basis and definition

PGG is built around an analogy to classifier-free guidance in diffusion models. In diffusion, guidance combines an unconditional model $p(x)$ and a conditional model $p(x\mid y)$ into a guided distribution of the form

$$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$

The paper adopts the same structure in policy space by treating the standard state-conditioned actor as the conditional branch and an added state-agnostic action prior as the unconditional branch (Qi et al., 2 Oct 2025).

The conditional policy is the usual actor,

$\pi_\theta(a\mid s),$

while the unconditional policy ignores the state,

$\pi_\theta(a),$

or equivalently $\pi_\theta(a\mid \emptyset)$ with a learnable null embedding. The guided policy is then defined as

$$\hat{\pi}_\theta(a\mid s)\propto \pi_\theta(a)^{\,1-\gamma}\,\pi_\theta(a\mid s)^{\,\gamma}.$$

This multiplicative interpolation in probability space is the core definition of PGG (Qi et al., 2 Oct 2025).

The semantics of $\gamma=1$0 are explicit in the formulation.

$\gamma=1$1	Guided behavior
$\gamma=1$2	purely unconditional behavior
$\gamma=1$3	recovers the standard policy
$\gamma=1$4	amplifies conditional behavior

This parameterization is important because it fixes the meaning of the default setting. In PGG, the normal or unguided case is not an equal mixture of conditional and unconditional branches; it is exactly $\gamma=1$5, where

$\gamma=1$6

up to normalization (Qi et al., 2 Oct 2025). A plausible implication is that PGG should be understood less as a new policy class than as a deployable guidance mechanism layered onto a conventional actor.

2. Mathematical formulation and policy-gradient derivation

The main technical obstacle in transferring diffusion-style guidance to reinforcement learning is that policy optimization differentiates with respect to parameters $\gamma=1$7, not actions. For action derivatives, the normalization disappears in the same way as in diffusion: $\gamma=1$8 However, parameter derivatives introduce a state-dependent normalization term: $\gamma=1$9 with

$\gamma>1$0

The paper identifies this $\gamma>1$1 term as the key technical obstacle (Qi et al., 2 Oct 2025).

Starting from the policy gradient theorem,

$\gamma>1$2

the derivation shows that the normalization term vanishes in expectation when advantages are used properly. The crucial property is

$\gamma>1$3

since $\gamma>1$4 under the rollout policy. This cancels the normalization contribution and yields the guided update

$\gamma>1$5

This is the central policy-gradient update under PGG (Qi et al., 2 Oct 2025).

The update can be read as a weighted combination of the usual conditional policy gradient and an unconditional branch gradient. When $\gamma>1$6, the coefficient $\gamma>1$7 is negative. The paper interprets this regime as amplifying the conditional branch while training the unconditional branch to capture “what not to do” (Qi et al., 2 Oct 2025). This suggests that PGG is not merely inference-time interpolation; the training dynamics themselves are biased by the chosen guidance regime.

3. Training procedure and test-time control

The training procedure is a minimal modification of PPO. The implementation starts from CleanRL PPO, adds an unconditional branch via a learnable null embedding, collects rollouts using the guided policy

$\gamma>1$8

computes advantages such as GAE, and updates PPO using $\gamma>1$9 in place of the ordinary policy (Qi et al., 2 Oct 2025).

Two implementation choices depend on the action space. For discrete actions, the method uses a softmax over guided logits. For continuous actions, it uses a Gaussian with guided mean and shared covariance (Qi et al., 2 Oct 2025). Conceptually, the construction remains defined in probability space, but operationally it becomes a logit-space or mean-space instantiation depending on whether the policy is categorical or Gaussian.

A central design goal is post-training controllability. Once both branches have been trained, behavior can be changed at evaluation time by varying $p(x)$0 without retraining. This is the precise analogue of inference-time control in classifier-free guidance (Qi et al., 2 Oct 2025). The paper studies two training modes.

The first is dropout-based PGG, where conditioning dropout is used with default probability

$p(x)$1

This is motivated by diffusion guidance, where the unconditional branch must receive enough training signal. The second is no-dropout, biased-guidance PGG, where $p(x)$2 and training is performed with $p(x)$3 so that both branches receive gradient signal through the guided objective (Qi et al., 2 Oct 2025).

The paper treats the second regime as more stable in continuous control. A plausible implication is that, unlike diffusion models, on-policy reinforcement learning is more sensitive to data-collection perturbations induced by conditioning dropout, because dropout changes the rollout policy and therefore the state distribution.

4. The meaning of “normal” or unguided guidance

In this work, the normal, ordinary, or default setting is unambiguous: it is the standard conditional actor recovered at $p(x)$4. At that value,

$p(x)$5

so there is no effective interpolation from the unconditional branch in the deployed action distribution (Qi et al., 2 Oct 2025).

This point matters because classifier-free guidance in diffusion is often discussed as interpolation between conditional and unconditional predictions. PGG is parameterized differently: the standard actor is recovered exactly at $p(x)$6, not at some intermediate value. Therefore the paper’s own terminology implies the following equivalence:

• unguided policy gradient;
• standard PPO with policy $p(x)$7;
• PGG evaluated at $p(x)$8.

There is one nuance. A PGG-trained model evaluated at $p(x)$9 may still differ architecturally from vanilla PPO because it includes an unconditional branch and may have been trained with dropout or biased guidance. Functionally, however, the deployed policy at $p(x\mid y)$0 is the paper’s no-guidance case (Qi et al., 2 Oct 2025).

The paper also notes that in continuous-control dropout experiments, the best results often occur at $p(x\mid y)$1, which it explicitly interprets as “no effective contribution from the unconditional branch” (Qi et al., 2 Oct 2025). This observation is significant because it distinguishes architectural augmentation from behavioral guidance: having the branch available is not the same as using it at deployment.

5. Empirical results across discrete and continuous control

The empirical study covers discrete control on CartPole-v1 and Acrobot-v1, and continuous control on MuJoCo v4 tasks HalfCheetah, Hopper, Humanoid, InvertedPendulum, Pusher, and Walker2d. The protocol uses 5 random seeds per environment, 50 test episodes per trained policy, mean and standard deviation of episodic return, training for $p(x\mid y)$2 timesteps on discrete tasks, and $p(x\mid y)$3 timesteps on continuous tasks (Qi et al., 2 Oct 2025).

In discrete control, guidance is strongly beneficial. With conditioning dropout, performance at $p(x\mid y)$4 is similar to PPO, but increasing test-time $p(x\mid y)$5 substantially improves returns, especially early in training. At 100k steps on CartPole-v1, the appendix table reports:

• PPO: $p(x\mid y)$6
• PGG $p(x\mid y)$7: $p(x\mid y)$8
• PGG $p(x\mid y)$9: $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$0
• PGG $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$1: $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$2

At 200k steps on CartPole-v1, PGG with $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$3 reaches $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$4. On Acrobot-v1 at 100k steps, the same appendix shows:

• PPO: $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$5
• PGG $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$6: $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$7
• PGG $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$8: $$\hat{p}(x\mid y)\propto p(x)^{1-\gamma} p(x\mid y)^\gamma.$$9
• PGG $\pi_\theta(a\mid s),$0: $\pi_\theta(a\mid s),$1 (Qi et al., 2 Oct 2025)

In continuous control with $\pi_\theta(a\mid s),$2 conditioning dropout, the picture changes. The best performance often occurs with mild or no guidance,

$\pi_\theta(a\mid s),$3

and larger $\pi_\theta(a\mid s),$4 introduces instability, especially in higher-dimensional environments such as HalfCheetah, Humanoid, and Walker2d (Qi et al., 2 Oct 2025). The paper concludes that conditioning dropout, although central in diffusion CFG, does not transfer cleanly to continuous on-policy RL.

The strongest continuous-control results come from no-dropout training with modestly larger guidance: $\pi_\theta(a\mid s),$5 This improves controllability and often improves returns and stability. At 1M steps, examples reported in the appendix include:

• HalfCheetah-v4: PPO $\pi_\theta(a\mid s),$6, PGG $\pi_\theta(a\mid s),$7 $\pi_\theta(a\mid s),$8
• Humanoid-v4: PPO $\pi_\theta(a\mid s),$9, PGG $\pi_\theta(a),$0 $\pi_\theta(a),$1
• InvertedPendulum-v4: PPO $\pi_\theta(a),$2, PGG $\pi_\theta(a),$3 $\pi_\theta(a),$4
• Walker2d-v4: PPO $\pi_\theta(a),$5, PGG $\pi_\theta(a),$6 $\pi_\theta(a),$7, but PGG $\pi_\theta(a),$8 falls to $\pi_\theta(a),$9 (Qi et al., 2 Oct 2025)

The aggregate empirical pattern is consistent across the paper: discrete tasks benefit from stronger-than-standard test-time guidance, while continuous tasks benefit from modest guidance and degrade under excessive sharpening. This suggests that action-space dimensionality and rollout sensitivity materially affect how much extrapolation beyond the conditional branch is tolerable.

6. Interpretation, limitations, and relation to broader guidance methods

PGG’s main conceptual contribution is to show that guidance, previously associated mainly with diffusion or other generative policies, can be adapted to standard on-policy policy-gradient reinforcement learning (Qi et al., 2 Oct 2025). Relative to ordinary policy-gradient methods, it adds an unconditional branch, a guidance parameter $\pi_\theta(a\mid \emptyset)$0, a guided rollout policy, and test-time behavior control without retraining.

The paper’s practical interpretation is correspondingly narrow. In simple discrete tasks, large test-time $\pi_\theta(a\mid \emptyset)$1 can substantially improve returns and sample efficiency. In continuous control, the preferred recipe is to train without dropout, use a slightly biased training guidance such as $\pi_\theta(a\mid \emptyset)$2, and sweep test-time $\pi_\theta(a\mid \emptyset)$3 in a modest range such as $\pi_\theta(a\mid \emptyset)$4–$\pi_\theta(a\mid \emptyset)$5 (Qi et al., 2 Oct 2025). This is presented as the cleanest practical route to controllable post-training behavior.

The paper also identifies several caveats. Guidance sensitivity is environment-dependent; too large $\pi_\theta(a\mid \emptyset)$6 destabilizes complex tasks; the derivation assumes advantage estimates are good enough that the normalization term vanishes in expectation; if the critic is biased or poorly trained, the cancellation may be imperfect; and evaluating both conditional and unconditional branches roughly doubles actor-side compute relative to vanilla PPO (Qi et al., 2 Oct 2025). It also notes that the unconditional policy itself can be weaker than vanilla PPO in some environments, so gains may appear only after extrapolation with $\pi_\theta(a\mid \emptyset)$7.

In broader methodological context, PGG directly mirrors classifier-free guidance in diffusion, where unconditional and conditional branches are combined at inference time. A related diffusion result argues that applying constant guidance across all denoising steps is suboptimal and that guidance is most helpful in a limited middle interval of the trajectory (Kynkäänniemi et al., 2024). PGG does not introduce an analogous stage-wise guidance schedule for reinforcement learning, but this parallel suggests a possible future direction: the usefulness of guidance may depend not only on strength but also on when and where it is applied.

PGG therefore occupies a specific place in controllable RL. It is not a new objective from scratch, nor a generative-policy method. Its novelty lies in deriving a guided policy

$\pi_\theta(a\mid \emptyset)$8

showing that the normalization term disappears under advantage estimation, and exposing $\pi_\theta(a\mid \emptyset)$9 as a deployment-time control knob for standard on-policy reinforcement learning (Qi et al., 2 Oct 2025).