Policy Gradient Penalty (PGP)

• Policy Gradient Penalty is a constrained maximum-entropy exploration method that applies quadratic-penalty regularization on occupancy measures in discounted MDPs.
• It transforms occupancy-space functionals into pseudo-rewards via the policy gradient theorem, efficiently balancing safety, resource, or imitation constraints.
• The method guarantees global last-iterate convergence and shows robust performance in grid-world and continuous-control tasks compared to dual-update alternatives.

Policy Gradient Penalty (PGP) is a policy-gradient method for constrained maximum-entropy exploration in discounted Markov decision processes. It is formulated as a single-loop policy-space method that enforces general convex occupancy-measure constraints via quadratic-penalty regularization, constructs pseudo-rewards for the penalized objective, and then applies the classical Policy Gradient Theorem to obtain model-free gradient estimates. The method is motivated by exploration problems in which entropy maximization over the state-action occupancy measure must coexist with safety, resource, or imitation requirements, and it is accompanied by global last-iterate convergence guarantees under hidden convexity and strong duality, together with experiments on a grid-world benchmark and two continuous-control tasks (Wolf et al., 30 Apr 2026).

1. Constrained entropy maximization in occupancy space

PGP is posed on an infinite-horizon discounted MDP with a softmax policy parameterization

$$\pi_\theta(a \mid s) = \frac{\exp(\psi(s, a; \theta))}{ \sum_{a^\prime \in \mathcal A}\exp(\psi(s, a^\prime; \theta)) }.$$

For a policy $\pi_\theta$, the state-action occupancy measure is the discounted visitation frequency

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$

Exploration is formalized through the entropy of this occupancy measure,

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$

rather than through a stepwise reward bonus. The constrained problem is

$$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$

where $R$ is a smooth convex functional of the occupancy measure (Wolf et al., 30 Apr 2026).

This formulation is notable because occupancy entropy is not additively decomposable along trajectories in the usual Bellman sense. The same is true for general convex occupancy constraints. The paper therefore treats the problem as convex in occupancy-measure space but non-convex after policy parameterization, since the map $\theta \mapsto \lambda^{\pi_\theta}$ is nonlinear. The constraint class includes linear cumulative-cost constraints such as

$$R(\lambda^{\pi_\theta})=\langle c,\lambda^{\pi_\theta}\rangle - c_{\max},$$

and imitation-style constraints such as

$R(\lambda^{\pi_\theta}) = D_{\mathrm{KL}(\lambda^{\pi_\theta}\,\|\,\lambda^{\pi_{\mathrm{ref}}})-\kappa.$

The intended solution quality is an $(\varepsilon,\delta)$-optimal policy satisfying

$\pi_\theta$0

for the original constrained problem (Wolf et al., 30 Apr 2026).

2. Quadratic-penalty regularization and pseudo-rewards

PGP converts the constrained entropy-maximization problem into the penalized objective

$\pi_\theta$1

equivalently,

$\pi_\theta$2

with penalty coefficient $\pi_\theta$3 (Wolf et al., 30 Apr 2026). The use of a quadratic penalty is central because, in the smooth setting treated by the paper, $\pi_\theta$4 preserves the regularity needed for projected SGD analysis.

The key algorithmic step is to express gradients of occupancy-space functionals as ordinary policy gradients. For any reward vector $\pi_\theta$5, the paper defines

$\pi_\theta$6

and invokes the Policy Gradient Theorem in the form

$\pi_\theta$7

Hence for a smooth occupancy functional $\pi_\theta$8,

$\pi_\theta$9

This converts occupancy-space derivatives into pseudo-rewards (Wolf et al., 30 Apr 2026).

At iteration $$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$0, PGP estimates the occupancy $$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$1 and forms pseudo-rewards from the negative entropy term and the penalty term. For entropy,

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$2

For the quadratic penalty,

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$3

The method then estimates the two corresponding policy gradients and applies the projected update

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$4

The resulting algorithm is explicitly described as single-loop, primal-only, model-free, with no dual update and no separate optimization loop for Lagrange multipliers (Wolf et al., 30 Apr 2026).

3. Regularity, hidden convexity, and last-iterate guarantees

The theory of PGP relies on smooth softmax policies, smooth occupancy-space functionals, and a local hidden-convexity property of the occupancy parameterization. The paper assumes

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$5

together with Lipschitz conditions on $$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$6 and $$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$7 (Wolf et al., 30 Apr 2026). Under these assumptions it derives smoothness of the policy-space objective through the occupancy map. For a generic occupancy functional $$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$8, the parameter-space smoothness constant is

$$\lambda^{\pi_\theta}(s,a) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^t\, \mathbb P_{\mu_0,\pi_\theta}(s_t=s,a_t=a).$$9

The quadratic penalty preserves the same structure, with occupancy-space constants scaling as $$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$0. In particular, the paper gives

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$1

and then derives an explicit $$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$2 for the full penalized objective (Wolf et al., 30 Apr 2026).

Because occupancy and gradients are estimated from truncated trajectories, the stochastic gradient estimator is biased. The paper quantifies this by showing, for the penalty objective,

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$3

and

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$4

These terms arise from finite-horizon truncation and occupancy-estimation error propagation (Wolf et al., 30 Apr 2026).

The main optimization result states that with

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$5

PGP yields

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$6

after

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$7

iterations, where $$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$8 (Wolf et al., 30 Apr 2026).

Under strong duality, the penalty parameter can be chosen as

$$\mathcal H(\lambda) = -\sum_{s,a}\lambda(s,a)\log \lambda(s,a),$$9

which yields the paper’s main last-iterate guarantee: $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$0 The corresponding complexity is

$$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$1

with total sample complexity

$$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$2

trajectory rollouts (Wolf et al., 30 Apr 2026).

4. Empirical behavior

The paper evaluates PGP on a FrozenLake-style grid world, PointMass, and SafeCartpole (Wolf et al., 30 Apr 2026). In the grid-world benchmark, PGP is compared with an unconstrained maximum-entropy baseline and a primal-dual baseline. The reported behavior is that PGP learns policies whose entropy is close to the unconstrained solution while satisfying the safety constraint, whereas the primal-dual method exhibits practical oscillation around the constraint boundary and the unconstrained baseline violates safety.

A central ablation varies the penalty coefficient $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$3. The paper reports that final entropy is largely insensitive to $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$4 once feasibility is achieved, while constraint violation decreases monotonically as $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$5 increases. Additional ablations vary batch size $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$6 and step size $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$7; larger batches help more at larger step sizes, and the theoretical dependence $$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$8 appears conservative in practice (Wolf et al., 30 Apr 2026).

The paper also studies a nonlinear imitation-style grid-world constraint

$$\max_{\theta \in \Theta} F_1(\theta) = \mathcal H(\lambda^{\pi_\theta}) \qquad \mathrm{s.t.}\qquad F_2(\theta)=R(\lambda^{\pi_\theta})\le 0,$$9

As the budget $R$0 increases, the learned policy deviates more from the reference and achieves higher entropy, illustrating that PGP is not limited to linear occupancy constraints (Wolf et al., 30 Apr 2026).

In PointMass, the method is applied with a KL occupancy constraint

$R$1

The reported effect is that increasing $R$2 allows progressively broader exploration away from the reference policy. In SafeCartpole, PGP is used with a position-based safety constraint; the experiments show diverse exploratory behavior while respecting the safety boundary. These continuous-control demonstrations rely on occupancy estimation in continuous spaces and use a critic baseline in SafeCartpole to reduce variance (Wolf et al., 30 Apr 2026).

5. Relation to nearby “policy-gradient penalty” ideas

The phrase “Policy Gradient Penalty” is unusually specific: among the papers considered here, it is directly instantiated by the method above (Wolf et al., 30 Apr 2026). Nearby literature uses related mechanisms but often means something else.

A useful counterpoint is Group Policy Gradient (GPG), which is explicitly presented as a no-penalty alternative for reasoning fine-tuning. GPG uses

$R$3

with group-normalized response rewards, and the paper is explicit that GPG uses no explicit penalty term: no KL term, no clipping term, no trust-region constraint, and no entropy regularizer. Its ablation further reports that reintroducing a KL term with $R$4 degrades performance on Qwen2.5-Math-7B, which the paper interprets as evidence that a policy-gradient penalty may be unnecessary or harmful in that setting (Chu et al., 3 Apr 2025). This suggests that PGP occupies one end of a broader design space in which policy updates may be either penalized or deliberately unconstrained.

Other nearby constructions regularize policy gradients differently. PGQL begins from entropy-regularized policy gradient and connects the regularized fixed point to an implicit action-value model, adding an off-policy Q-learning update on top (O'Donoghue et al., 2016). Safe Policy Gradients derive monotonic-improvement lower bounds of the form “linear gain minus quadratic penalty,” but the penalty is a parameter-space smoothness term rather than a penalty added to the objective (Papini et al., 2019). Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation introduces a gradient penalty on the critic,

$R$5

which regularizes the actor’s gradient pathway indirectly rather than penalizing the actor objective itself (Gao et al., 2022). Policy Gradient-based Model Free Optimal LQG Control with a Probabilistic Risk Constraint uses a Lagrangian penalty

$R$6

and updates $R$7 by projected ascent, so it is closer to primal-dual constrained policy gradient than to single-loop quadratic-penalty PGP (Naha et al., 2024).

Terminological confusion is common. PPG in “Proximal Policy Gradient” denotes Proximal Policy Gradient, not Policy Gradient Penalty (Byun et al., 2020). The GPG paper also contains an appendix typo that refers to “PGP” when the context clearly indicates GPG, not Policy Gradient Penalty (Chu et al., 3 Apr 2025).

6. Limitations and open technical issues

The main theoretical guarantees for PGP are developed for finite state and action spaces and depend on several structural assumptions: smooth softmax policies, smooth occupancy-space functionals, local invertibility of the occupancy parameterization, and strong duality for the original constrained problem (Wolf et al., 30 Apr 2026). The paper notes that entropy is only locally smooth, and that smoothing may be useful if needed. It also states that continuous-control experiments go beyond the finite-tabular setting covered by theory.

The most important practical bottleneck is occupancy estimation in continuous spaces. In the continuous-control experiments, the paper uses an MLE/GMM occupancy estimator; this makes the method scalable beyond tabular theory, but also identifies occupancy estimation as the key systems-level difficulty (Wolf et al., 30 Apr 2026). The sample complexity guarantee,

$R$8

is also relatively large, and the empirical ablations suggest that the theoretical dependence on batch size is conservative.

A broader implication is that “policy-gradient penalty” is not a single mechanism but a family of design choices. PGP uses quadratic penalty regularization on occupancy constraints; GPG argues for removing penalties altogether in some reasoning settings; safe policy-gradient analyses replace explicit penalties with lower-bound-based conservatism; and primal-dual formulations encode constraints through adaptive multipliers rather than fixed penalties (Wolf et al., 30 Apr 2026). This suggests that the central technical question is less whether a penalty should exist in principle than which object is being regularized—policy change, occupancy mismatch, critic sharpness, or risk violation—and under what statistical and optimization regime that regularization is beneficial.