Policy Gradient Projection

• Policy gradient projection is a framework that employs geometric projections to map unconstrained policy updates onto admissible domains, ensuring feasibility and improved convergence.
• It leverages divergence minimization (e.g., KL or α-divergence) to reconcile unrealizable policies with parameterized classes, enhancing stability and suboptimality rates.
• Applications span tabular, robust, and safe RL, including action-space projections, Wasserstein proximal updates, and Hilbert projections, broadening formulation diversity.

Policy gradient projection denotes a family of reinforcement-learning constructions in which a policy-improvement step is followed, approximated, or interpreted as a projection onto a feasible or realizable set. Across the literature, the projected object varies substantially: a tabular policy can be projected onto the product of action simplices in direct policy space; an improved but unrealizable policy can be projected back into a parameterized class through KL- or $\alpha$-divergence minimization; exploratory or safe actions can be projected onto state-dependent feasible sets; robust-RL methods can project either uncertainty parameters or policy iterates; and recent work has formulated projection on finite response simplices, in Wasserstein space, or in Hilbert spaces (Xiao, 2022, Ghosh et al., 2020, Gros et al., 2020, Li et al., 2023, Zhu et al., 3 Mar 2026, Zixian, 24 Feb 2026). The term is therefore not a single algorithmic primitive but a geometric motif whose precise meaning depends on the policy representation, the constraint set, and the divergence or metric used to define “closeness.”

1. Direct projected policy gradient in policy space

In the most literal usage, projected policy gradient operates directly on the policy itself, with the feasible set

$\Pi=\Delta(A)^{|S|},$

the Cartesian product of statewise probability simplices. Under direct tabular parameterization, the update is

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$

or blockwise,

$$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$

Here the gradient is taken in policy space, not parameter space, and Euclidean projection is necessary because an unconstrained step in $\mathbb R^{|S||A|}$ need not preserve nonnegativity or row-sum-one constraints (Xiao, 2022).

This formulation admits a standard gradient-mapping interpretation. With

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$

$\|G_L(\pi)\|_2$ measures first-order stationarity relative to the feasible set. A central contribution of "On the Convergence Rates of Policy Gradient Methods" (Xiao, 2022) is a weak gradient-mapping dominance analysis yielding a sharper sublinear rate for projected policy gradient: $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$

when $\rho=\mu$ and

$\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.$

The paper explicitly contrasts this with earlier $\Pi=\Delta(A)^{|S|},$0 iteration complexity, emphasizing an $\Pi=\Delta(A)^{|S|},$1 improvement for objective suboptimality (Xiao, 2022).

A more recent exact-analysis perspective sharpens the distinction between direct simplex projection and softmax parameterization. Under simplex parameterization, projected policy gradient updates each state by

$\Pi=\Delta(A)^{|S|},$2

with the explicit simplex projection

$\Pi=\Delta(A)^{|S|},$3

"Elementary Analysis of Policy Gradient Methods" proves that this projected policy gradient enjoys global linear convergence for any constant stepsize, in contrast to softmax policy gradient, which is only sublinear for constant stepsizes (Liu et al., 2024). This suggests that the Euclidean projection onto the simplex is not merely a feasibility correction; it materially changes the effective optimization geometry.

2. Projection as an operator-theoretic approximation

A different tradition treats policy-gradient methods as alternating between a policy-improvement operator $\Pi=\Delta(A)^{|S|},$4 and a projection operator $\Pi=\Delta(A)^{|S|},$5. In this view, the primitive object is not a parameter gradient step but the exact update

$\Pi=\Delta(A)^{|S|},$6

where $\Pi=\Delta(A)^{|S|},$7 is an improved but generally unrealizable policy-like object and $\Pi=\Delta(A)^{|S|},$8 maps it back to the realizable policy class $\Pi=\Delta(A)^{|S|},$9 (Ghosh et al., 2020).

In trajectory space, when returns are positive, the improvement operator is

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$0

and the projection is a forward KL projection

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$1

This yields an operator interpretation of REINFORCE as approximate KL projection of an improved trajectory distribution back into the realizable class. In the state-action formulation, the paper defines the weighted statewise KL

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$2

and the projection

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$3

The paper stresses that this is not KL over the full joint state-action distribution; it is a weighted conditional KL over action distributions, state by state (Ghosh et al., 2020).

This operator view supports a global lower bound linking return improvement to reduction in projection error. One of the central expressions is

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$4

Consequently, any policy strictly closer to the improved policy than the current policy is, in the projection divergence, has higher return. The paper further argues that the direction of KL matters: REINFORCE corresponds to a forward, covering projection, whereas PPO-like methods are associated with a reverse-KL-like, mode-seeking geometry (Ghosh et al., 2020).

Later listwise work in RL with verifiable rewards pushes this logic onto a finite response simplex. For a prompt $$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$5 and sampled responses $$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$6, the induced listwise distribution is

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$7

"Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex" makes the target-projection structure explicit by first solving

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$8

with

$$\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),$$9

yielding

$$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$0

and then projecting $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$1 to $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$2 by exact divergence minimization (Qu et al., 7 May 2026). This extends the operator picture from trajectory or state-action distributions to prompt-conditioned response simplices.

3. Action-space projection, safe sets, and bias-corrected gradients

A separate use of policy gradient projection appears in constrained control, where the policy may output an unsafe action that is then projected onto a feasible set. In deterministic form, the applied safe action is

$$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$3

and in the stochastic case a sampled action $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$4 is projected by

$$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$5

The learning problem then concerns the projected policy $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$6, not the raw policy $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$7 (Gros et al., 2020).

For deterministic policy gradients, the correction enters through the chain rule. Under LICQ and strict SOSC, the projected policy sensitivity is

$$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$8

where $$\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.$$9 spans the tangent space of the active constraints. The unbiased deterministic policy gradient is therefore

$\mathbb R^{|S||A|}$0

In the stochastic case, the paper derives

$\mathbb R^{|S||A|}$1

with $\mathbb R^{|S||A|}$2. The score must be evaluated at the pre-projection sample $\mathbb R^{|S||A|}$3, while the advantage belongs to the projected closed loop (Gros et al., 2020).

Action-constrained reinforcement learning exposes a different failure mode: the zero-gradient problem caused by differentiating through the projection Jacobian. If the executed action is

$\mathbb R^{|S||A|}$4

then

$\mathbb R^{|S||A|}$5

When the projection saturates, $\mathbb R^{|S||A|}$6 can be nearly zero, annihilating the actor gradient (Lin et al., 2021). "Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization" therefore replaces end-to-end projected policy gradients with statewise Frank–Wolfe steps in action space,

$\mathbb R^{|S||A|}$7

followed by convex interpolation and, in the neural version, regression onto feasible reference actions (Lin et al., 2021). This suggests that action projection can be safe for execution yet harmful inside the backward pass unless the policy-gradient estimator is corrected or redesigned.

The same theme appears in MPC-based deterministic policy gradients. "Bias Correction in Deterministic Policy Gradient Using Robust MPC" studies a robust-MPC policy $\mathbb R^{|S||A|}$8 with centered isotropic exploration $\mathbb R^{|S||A|}$9, followed by a posterior projection

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$0

Under smoothness assumptions, the projection error satisfies

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$1

and the projected exploration remains asymptotically centered and isotropic: $$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$2 The resulting deterministic policy gradient estimator is asymptotically unbiased (Kordabad et al., 2021).

4. Robust and constrained policy learning: simplex, weight-space, and uncertainty-set projection

Projection also arises in robust reinforcement learning, but the projected object need not be the policy. In robust infinite-horizon discounted MDPs with non-rectangular uncertainty sets $$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$3, "Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets" uses projected Langevin dynamics for adversarial policy evaluation: $$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$4 This is a Euclidean projection of the adversary’s uncertainty parameter onto the convex compact set $$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$5, not a projection of the policy itself. For policy improvement, the same paper performs a projected actor step

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$6

so policy projection onto the simplex appears only in the actor, while the efficient deterministic critic is explicitly projection-free via Frank–Wolfe updates over transition kernels (Li et al., 2023).

In finite-horizon noisy LQR, projected policy gradient appears in gain space. With a convex constraint set $$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$7 of admissible feedback gains, the projection operator is

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$8

and the update is

$$T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),$$9

The associated gradient mapping is

$\|G_L(\pi)\|_2$0

"Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon" proves

$\|G_L(\pi)\|_2$1

a global sublinear convergence guarantee to stationarity for the constrained case, in contrast to global linear convergence in the unconstrained case (Hambly et al., 2020).

A more recent safety-oriented development projects in parameter space rather than policy or action space. "Constrained Policy Optimization via Sampling-Based Weight-Space Projection" poses

$\|G_L(\pi)\|_2$2

with rollout-based safety metrics $\|G_L(\pi)\|_2$3 available only by evaluation, not by gradients. The raw step

$\|G_L(\pi)\|_2$4

is projected in a sampled low-dimensional subspace $\|G_L(\pi)\|_2$5, where $\|G_L(\pi)\|_2$6 stores recent directions and $\|G_L(\pi)\|_2$7 stores corresponding safety measurements. The reduced convex program is

$\|G_L(\pi)\|_2$8

with $\|G_L(\pi)\|_2$9. The paper establishes a safe-by-induction guarantee: starting from $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$0, all accepted iterates remain safe given feasible projections (Cao et al., 15 Dec 2025). This broadens policy gradient projection from simplex constraints and safe actions to weight-space safety regions derived from sampled rollouts.

5. Projection-free alternatives and limits of “proximal” terminology

The literature also contains repeated warnings that not every “proximal” or “projected” label corresponds to an actual projection operator. "Proximal Policy Gradient: PPO with Policy Gradient" proposes PPG as a PPO-inspired method with the log-ratio

$$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$1

and clipped objective contributions

$$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$2

Its unclipped gradient equals the vanilla policy-gradient gradient because

$$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$3

but the paper contains no operator $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$4, no proximal-point subproblem, and no exact trust-region solve. The method is therefore a clipped log-probability heuristic with approximate-KL early stopping, not a formal projection in policy space, parameter space, or simplex space (Byun et al., 2020). This is a common source of terminological confusion.

Several papers deliberately avoid projection. In robust MDP evaluation with non-rectangular uncertainty, the efficient deterministic critic uses conservative policy iteration with Frank–Wolfe updates

$$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$5

remaining in the uncertainty set by convex interpolation rather than Euclidean projection (Li et al., 2023). In action-constrained RL, Frank–Wolfe Policy Optimization avoids differentiating through action projection and proves monotone improvement and $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$6 stationarity in the tabular setting (Lin et al., 2021). In sequential zero-sum LQ games, "Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games" develops projection-free natural-gradient and quasi-Newton updates that preserve stabilizability through Riccati residual inequalities rather than projection onto the nonconvex stabilizing set (Bu et al., 2019).

These results collectively indicate that projection is not always the preferred mechanism. A plausible implication is that projection is most natural when feasibility sets are convex and explicit, such as $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$7, $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$8, or closed convex sets of feedback gains, whereas projection-free methods are preferred when the feasible region is nonconvex, open, or only implicitly characterized by dynamics, stability, or coupled uncertainty structure (Xiao, 2022, Li et al., 2023, Bu et al., 2019).

6. New geometric formulations: response simplices, Wasserstein proximal maps, and Hilbert projections

Recent work expands policy gradient projection beyond Euclidean simplex geometry. On finite response sets for LLM post-training, the projection target is a listwise distribution $$V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2$$9 on the response simplex, and exact projection gradients follow from divergence minimization. For forward KL,

$\rho=\mu$0

which yields bounded, zero-sum, self-correcting coefficients on the simplex (Qu et al., 7 May 2026). This is projection in an empirical probability simplex associated with a single prompt and sampled response group.

"Wasserstein Proximal Policy Gradient" replaces KL/Bregman geometry with a Wasserstein proximal step. For each state,

$\rho=\mu$1

An operator-splitting derivation decomposes this into an optimal-transport step

$\rho=\mu$2

followed by a heat step

$\rho=\mu$3

This is a genuine proximal policy update in Wasserstein space, with a global linear convergence theorem under a $\rho=\mu$4 transportation-information inequality (Zhu et al., 3 Mar 2026).

A different geometric reparameterization appears in "Group Orthogonalized Policy Optimization: Group Policy Optimization as Orthogonal Projection in Hilbert Space" (Zixian, 24 Feb 2026). Fixing a reference policy $\rho=\mu$5, it defines the density fluctuation field

$\rho=\mu$6

in the Hilbert space

$\rho=\mu$7

Normalization becomes the orthogonality constraint

$\rho=\mu$8

so feasible updates lie in the codimension-one subspace

$\rho=\mu$9

The work-dissipation functional

$\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.$0

has optimizer given by the Hilbert projection theorem: $\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.$1 With positivity enforced, the bounded projection becomes

$\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.$2

inducing exact sparsity for sufficiently poor actions (Zixian, 24 Feb 2026).

Taken together, these works show that policy gradient projection now spans Euclidean projection on policy simplices, KL- and $\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.$3-divergence projection, action-level Euclidean projection onto safe sets, projection of uncertainty or weights in robust optimization, Wasserstein proximal maps on action distributions, and orthogonal projection in Hilbert spaces. A common structure persists: an unconstrained improvement direction is first defined in a larger linear or distributional space, and learning then proceeds by mapping that target back into the admissible policy class or safe region under a specified geometry (Ghosh et al., 2020, Qu et al., 7 May 2026, Zhu et al., 3 Mar 2026, Zixian, 24 Feb 2026).