---
title: Policy Gradient Projection
url: https://www.emergentmind.com/topics/policy-gradient-projection
type: topic
---

# Policy Gradient Projection

Policy gradient projection denotes a family of reinforcement-learning constructions in which a policy-improvement step is followed, approximated, or interpreted as a projection onto a feasible or realizable set. Across the literature, the projected object varies substantially: a tabular policy can be projected onto the product of action simplices in direct policy space; an improved but unrealizable policy can be projected back into a parameterized class through KL- or \(\alpha\)-divergence minimization; exploratory or safe actions can be projected onto state-dependent feasible sets; robust-RL methods can project either uncertainty parameters or policy iterates; and recent work has formulated projection on finite response simplices, in Wasserstein space, or in Hilbert spaces [2201.07443], [2006.11266], [2004.00915], [2305.19004], [2603.02576], [2602.21269]. The term is therefore not a single algorithmic primitive but a geometric motif whose precise meaning depends on the policy representation, the constraint set, and the divergence or metric used to define “closeness.”

## 1. Direct projected policy gradient in policy space

In the most literal usage, projected policy gradient operates directly on the policy itself, with the feasible set
\[
\Pi=\Delta(A)^{|S|},
\]
the Cartesian product of statewise probability simplices. Under direct tabular parameterization, the update is
\[
\pi^{(k+1)}=\operatorname{proj}_{\Pi}\left(\pi^{(k)}-\eta_k \nabla V_\mu(\pi^{(k)})\right),
\]
or blockwise,
\[
\pi^{(k+1)}_s = \operatorname{proj}_{\Delta(A)} \left(\pi^{(k)}_s-\eta_k \nabla_s V_\mu(\pi^{(k)})\right), \qquad s\in S.
\]
Here the gradient is taken in policy space, not parameter space, and Euclidean projection is necessary because an unconstrained step in \(\mathbb R^{|S||A|}\) need not preserve nonnegativity or row-sum-one constraints [2201.07443].

This formulation admits a standard gradient-mapping interpretation. With
\[
T_L(\pi)=\operatorname{proj}_{\Pi}\left(\pi-\frac{1}{L}\nabla V_\mu(\pi)\right),\qquad
G_L(\pi)=L\bigl(\pi-T_L(\pi)\bigr),
\]
\(\|G_L(\pi)\|_2\) measures first-order stationarity relative to the feasible set. A central contribution of "On the Convergence Rates of Policy Gradient Methods" [2201.07443] is a weak gradient-mapping dominance analysis yielding a sharper sublinear rate for projected policy gradient:
\[
V_\rho(\pi^{(k)})-V_\rho^\star \le \frac{128|S|}{k(1-\gamma)^5} \left\|\frac{d_\rho(\pi^\star)}{\rho}\right\|_\infty^2
\]
when \(\rho=\mu\) and
\[
\eta_k=\frac{(1-\gamma)^3}{2\gamma |S|}.
\]
The paper explicitly contrasts this with earlier \(O(\epsilon^{-2})\) iteration complexity, emphasizing an \(O(\epsilon^{-1})\) improvement for objective suboptimality [2201.07443].

A more recent exact-analysis perspective sharpens the distinction between direct simplex projection and softmax parameterization. Under simplex parameterization, projected policy gradient updates each state by
\[
\pi_s^{k+1} = \Proj_{\Delta(\mathcal A)}(\pi_s^k+\eta_s^k Q^k(s,\cdot)),
\qquad
\eta_s^k=\frac{\eta_k}{1-\gamma}d_\mu^k(s),
\]
with the explicit simplex projection
\[
\Proj_{\Delta(\mathcal A)}(y)=(y+\lambda \mathbf 1)_+.
\]
"Elementary Analysis of Policy Gradient Methods" proves that this projected policy gradient enjoys global linear convergence for any constant stepsize, in contrast to softmax policy gradient, which is only sublinear for constant stepsizes [2404.03372]. This suggests that the Euclidean projection onto the simplex is not merely a feasibility correction; it materially changes the effective optimization geometry.

## 2. Projection as an operator-theoretic approximation

A different tradition treats policy-gradient methods as alternating between a policy-improvement operator \(\mathcal I\) and a projection operator \(\mathcal P\). In this view, the primitive object is not a parameter gradient step but the exact update
\[
\pi_{t+1}=\mathcal P(\mathcal I\pi_t),
\]
where \(\mathcal I\pi_t\) is an improved but generally unrealizable policy-like object and \(\mathcal P\) maps it back to the realizable policy class \(\Pi=\{\pi_\theta:\theta\in\mathbb R^d\}\) [2006.11266].

In trajectory space, when returns are positive, the improvement operator is
\[
\mathcal I_\tau\pi(\tau)=R\pi(\tau),
\]
and the projection is a forward KL projection
\[
\mathcal P_\tau \mu = \arg\min_{\pi \in \Pi} KL(\mu \,\|\, \pi).
\]
This yields an operator interpretation of REINFORCE as approximate KL projection of an improved trajectory distribution back into the realizable class. In the state-action formulation, the paper defines the weighted statewise KL
\[
D_z(\mu \,\|\, \pi)=\sum_s z(s)\,KL\big(\mu(\cdot|s)\,\|\,\pi(\cdot|s)\big),
\]
and the projection
\[
\mathcal P_V\mu = \arg\min_{z\in\Pi}\sum_s \mu(s)\, KL\big(\mu(\cdot|s)\,\|\,z(\cdot|s)\big).
\]
The paper stresses that this is not KL over the full joint state-action distribution; it is a weighted conditional KL over action distributions, state by state [2006.11266].

This operator view supports a global lower bound linking return improvement to reduction in projection error. One of the central expressions is
\[
J(\pi) \geq J(\mu) + \mathbb{E}_\mu[V^\mu(s)]
\big[D_{\mu}(\mathcal I_V\mu \,\|\, \mu) - D_{\mu}(\mathcal I_V\mu \,\|\, \pi)\big].
\]
Consequently, any policy strictly closer to the improved policy than the current policy is, in the projection divergence, has higher return. The paper further argues that the direction of KL matters: REINFORCE corresponds to a forward, covering projection, whereas PPO-like methods are associated with a reverse-KL-like, mode-seeking geometry [2006.11266].

Later listwise work in RL with verifiable rewards pushes this logic onto a finite response simplex. For a prompt \(x\) and sampled responses \(y_1,\dots,y_K\), the induced listwise distribution is
\[
P_{\theta,k}=\frac{\exp(s_{\theta,k})}{\sum_{j=1}^K \exp(s_{\theta,j})},
\qquad
s_{\theta,k}=\log\frac{\pi_\theta(y_k|x)}{\pi_b(y_k|x)}.
\]
"Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex" makes the target-projection structure explicit by first solving
\[
w^\ast=\arg\max_{w \in \Delta^{K-1}} \hat J(w)
\]
with
\[
\hat J(w)=\sum_{k=1}^K w_k R_k-\tau D_{\mathrm{KL}}(w\|P_t),
\]
yielding
\[
w_k^\ast=\frac{P_{t,k}\exp(R_k/\tau)}{\sum_{j=1}^K P_{t,j}\exp(R_j/\tau)},
\]
and then projecting \(P_\theta\) to \(w^\ast\) by exact divergence minimization [2605.06139]. This extends the operator picture from trajectory or state-action distributions to prompt-conditioned response simplices.

## 3. Action-space projection, safe sets, and bias-corrected gradients

A separate use of policy gradient projection appears in constrained control, where the policy may output an unsafe action that is then projected onto a feasible set. In deterministic form, the applied safe action is
\[
\pi^\perp_\theta(x)=\arg\min_u \frac12 \|u-\pi_\theta(x)\|^2
\quad \text{s.t.}\quad s(x,u)\le 0,
\]
and in the stochastic case a sampled action \(u_s\sim \pi_\theta[\cdot|x]\) is projected by
\[
\pi^\perp_\theta(x,u_s)=\arg\min_u \frac12 \|u-u_s\|^2
\quad \text{s.t.}\quad s(x,u)\le 0.
\]
The learning problem then concerns the projected policy \(\pi^\perp_\theta\), not the raw policy \(\pi_\theta\) [2004.00915].

For deterministic policy gradients, the correction enters through the chain rule. Under LICQ and strict SOSC, the projected policy sensitivity is
\[
\nabla_{\theta}{\pi}_{\theta}^\perp(x)
=
\nabla_{\theta}{\pi}_{\theta}(x)\,M(x),
\qquad
M(x)= \mathcal N(\mathcal N^\top H\mathcal N)^{-1}\mathcal N^\top,
\]
where \(\mathcal N\) spans the tangent space of the active constraints. The unbiased deterministic policy gradient is therefore
\[
\nabla_{\theta} J\left({\pi}^\perp_{\theta}\right)
=
\mathbb E\left[\nabla_{\theta}{\pi}_{\theta}M\nabla_{u}A_{\pi^\perp_{\theta}}\right].
\]
In the stochastic case, the paper derives
\[
{\nabla_{\theta}J}\left(\pi^\perp_{\theta}\right)
=
\mathbb E\left[
\nabla_{\theta}\log \pi_{\theta}[u_s|x]\,
A_{\pi^\perp_{\theta}}(x,u)
\right],
\]
with \(u=\pi^\perp_\theta(x,u_s)\). The score must be evaluated at the pre-projection sample \(u_s\), while the advantage belongs to the projected closed loop [2004.00915].

Action-constrained reinforcement learning exposes a different failure mode: the zero-gradient problem caused by differentiating through the projection Jacobian. If the executed action is
\[
a^{\text{exec}}(s;\theta)=\Pi_{\mathcal C(s)}(\pi(s;\theta)),
\]
then
\[
\nabla_\theta \Pi_{\mathcal C(s)}(\pi(s;\theta))
=
\nabla_z \Pi_{\mathcal C(s)}(z)\rvert_{z=\pi(s;\theta)}\,\nabla_\theta \pi(s;\theta).
\]
When the projection saturates, \(\nabla_z \Pi_{\mathcal C(s)}\) can be nearly zero, annihilating the actor gradient [2102.11055]. "Escaping from Zero Gradient: Revisiting Action-Constrained Reinforcement Learning via Frank-Wolfe Policy Optimization" therefore replaces end-to-end projected policy gradients with statewise Frank–Wolfe steps in action space,
\[
c_k(s)=\argmax_{c\in\mathcal C(s)}
\left\langle c,\nabla_a Q(s,a;\pi(\cdot;\theta_k))\rvert_{a=\theta_k(s)} \right\rangle,
\]
followed by convex interpolation and, in the neural version, regression onto feasible reference actions [2102.11055]. This suggests that action projection can be safe for execution yet harmful inside the backward pass unless the policy-gradient estimator is corrected or redesigned.

The same theme appears in MPC-based deterministic policy gradients. "Bias Correction in Deterministic Policy Gradient Using Robust MPC" studies a robust-MPC policy \(\hat\pi_\theta\) with centered isotropic exploration \(\hat e\in B(0,\eta)\), followed by a posterior projection
\[
\hat a_\perp=P(s,\hat\pi_\theta+\hat e),\qquad
\epsilon=\hat a_\perp-(\hat\pi_\theta+\hat e).
\]
Under smoothness assumptions, the projection error satisfies
\[
\|\epsilon\|\le \alpha \eta^2,
\]
and the projected exploration remains asymptotically centered and isotropic:
\[
\lim_{\bar\eta\to 0}\mathbb{E}[\hat e_\perp]=0,\qquad
\lim_{\bar\eta\to 0}\mathbb{E}\left[\frac{1}{\eta^2}\hat e_\perp\hat e_\perp^\top\right]=\frac13 I.
\]
The resulting deterministic policy gradient estimator is asymptotically unbiased [2104.02413].

## 4. Robust and constrained policy learning: simplex, weight-space, and uncertainty-set projection

Projection also arises in robust reinforcement learning, but the projected object need not be the policy. In robust infinite-horizon discounted MDPs with non-rectangular uncertainty sets \(\mathcal P=\{P^\xi:\xi\in\Xi\}\), "Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Sets" uses projected Langevin dynamics for adversarial policy evaluation:
\[
\xi^{(m+1)} = \operatorname{Proj}_{\Xi}\!\left(
\xi^{(m)}+\eta \nabla_\xi V_\pi^{P^\xi}(s_0)\big|_{\xi=\xi^{(m)}}
+\sqrt{\frac{2\eta}{\beta}}\,w_{m+1}
\right).
\]
This is a Euclidean projection of the adversary’s uncertainty parameter onto the convex compact set \(\Xi\), not a projection of the policy itself. For policy improvement, the same paper performs a projected actor step
\[
\pi^{(k+1)} = \operatorname{Proj}_{\Pi}\!\left(
\pi^{(k)}+\eta \nabla_\pi V_{\pi^{(k)}}^{P^{(k)}}(s_0)
\right),
\qquad
\Pi=\Delta(\mathcal A)^S,
\]
so policy projection onto the simplex appears only in the actor, while the efficient deterministic critic is explicitly projection-free via Frank–Wolfe updates over transition kernels [2305.19004].

In finite-horizon noisy LQR, projected policy gradient appears in gain space. With a convex constraint set \(\mathcal S\) of admissible feedback gains, the projection operator is
\[
\Pi_{\mathcal S}(\pmb y)
=
\arg\min_{\pmb x\in\mathcal S}
\frac12\sum_{t=0}^{T-1}\|x_t-y_t\|_F^2,
\]
and the update is
\[
\pmb K^{n+1}
=
\Pi_{\mathcal S}\!\left(
\pmb K^n-\eta\,\widehat{\nabla C(\pmb K^n)}
\right).
\]
The associated gradient mapping is
\[
G(\pmb K)
=
\frac{\Pi_{\mathcal S}(\pmb K-\eta\nabla C(\pmb K))-\pmb K}{2\eta}.
\]
"Policy Gradient Methods for the Noisy Linear Quadratic Regulator over a Finite Horizon" proves
\[
\frac1N\sum_{n=0}^{N-1}\sum_{t=0}^{T-1}\|G_t(\pmb K^n)\|_F^2
=
\mathcal O\!\left(\frac1N\right),
\]
a global sublinear convergence guarantee to stationarity for the constrained case, in contrast to global linear convergence in the unconstrained case [2011.10300].

A more recent safety-oriented development projects in parameter space rather than policy or action space. "Constrained Policy Optimization via Sampling-Based Weight-Space Projection" poses
\[
\min_\theta \mathcal L(\theta)\quad \text{s.t.}\quad g(\theta)\le \mathbf 0,
\]
with rollout-based safety metrics \(g(\theta)\) available only by evaluation, not by gradients. The raw step
\[
\Delta \theta_{\text{raw}}=-\eta \nabla_\theta \mathcal L(\theta_t)
\]
is projected in a sampled low-dimensional subspace \(\Delta\theta=Dc\), where \(D\) stores recent directions and \(G\) stores corresponding safety measurements. The reduced convex program is
\[
\begin{aligned}
\min_c \quad &
(c-e_m)^\top S (c-e_m) \\
\text{s.t.}\quad &
(1-\mathbf 1^\top c) g(\theta_t) + Gc
+ \frac12\left(c^\top S c + |c|^\top \mathrm{diag}(S)\right)L
\le \mathbf 0,
\end{aligned}
\]
with \(S=D^\top D\). The paper establishes a safe-by-induction guarantee: starting from \(g(\theta_0)\le 0\), all accepted iterates remain safe given feasible projections [2512.13788]. This broadens policy gradient projection from simplex constraints and safe actions to weight-space safety regions derived from sampled rollouts.

## 5. Projection-free alternatives and limits of “proximal” terminology

The literature also contains repeated warnings that not every “proximal” or “projected” label corresponds to an actual projection operator. "Proximal Policy Gradient: PPO with Policy Gradient" proposes PPG as a PPO-inspired method with the log-ratio
\[
d_t(\theta)=\log \pi_\theta(a_t\mid s_t)-\log \pi_{\theta_{\mathrm{old}}}(a_t\mid s_t),
\]
and clipped objective contributions
\[
\delta_t(\theta)=
\begin{cases}
\min(d_t(\theta),u_b), & \hat A_t\ge 0,\\[4pt]
\max(d_t(\theta),l_b), & \hat A_t<0.
\end{cases}
\]
Its unclipped gradient equals the vanilla policy-gradient gradient because
\[
\nabla_\theta d_t(\theta)=\nabla_\theta \log\pi_\theta(a_t\mid s_t),
\]
but the paper contains no operator \(\Pi_{\mathcal C}\), no proximal-point subproblem, and no exact trust-region solve. The method is therefore a clipped log-probability heuristic with approximate-KL early stopping, not a formal projection in policy space, parameter space, or simplex space [2010.09933]. This is a common source of terminological confusion.

Several papers deliberately avoid projection. In robust MDP evaluation with non-rectangular uncertainty, the efficient deterministic critic uses conservative policy iteration with Frank–Wolfe updates
\[
P^{(m+1)}=(1-\alpha_m)P^{(m)}+\alpha_m P_\epsilon,
\]
remaining in the uncertainty set by convex interpolation rather than Euclidean projection [2305.19004]. In action-constrained RL, Frank–Wolfe Policy Optimization avoids differentiating through action projection and proves monotone improvement and \(O(T^{-1/2})\) stationarity in the tabular setting [2102.11055]. In sequential zero-sum LQ games, "Global Convergence of Policy Gradient for Sequential Zero-Sum Linear Quadratic Dynamic Games" develops projection-free natural-gradient and quasi-Newton updates that preserve stabilizability through Riccati residual inequalities rather than projection onto the nonconvex stabilizing set [1911.04672].

These results collectively indicate that projection is not always the preferred mechanism. A plausible implication is that projection is most natural when feasibility sets are convex and explicit, such as \(\Delta(A)\), \(\Delta(\mathcal A)^S\), or closed convex sets of feedback gains, whereas projection-free methods are preferred when the feasible region is nonconvex, open, or only implicitly characterized by dynamics, stability, or coupled uncertainty structure [2201.07443], [2305.19004], [1911.04672].

## 6. New geometric formulations: response simplices, Wasserstein proximal maps, and Hilbert projections

Recent work expands policy gradient projection beyond Euclidean simplex geometry. On finite response sets for LLM post-training, the projection target is a listwise distribution \(w^\ast\) on the response simplex, and exact projection gradients follow from divergence minimization. For forward KL,
\[
\nabla_\theta D_{\mathrm{KL}}(w^\ast\|P_\theta)
=
\sum_{k=1}^K (P_{\theta,k}-w_k^\ast)\,\nabla_\theta \log \pi_\theta(y_k|x),
\]
which yields bounded, zero-sum, self-correcting coefficients on the simplex [2605.06139]. This is projection in an empirical probability simplex associated with a single prompt and sampled response group.

"Wasserstein Proximal Policy Gradient" replaces KL/Bregman geometry with a Wasserstein proximal step. For each state,
\[
\pi_{k+1}(\cdot\mid s)
\in
\argmax_{\pi(\cdot\mid s)}
\left\{
\langle Q_\tau^{\pi_k}(s,\cdot),\pi(\cdot\mid s)\rangle
-\frac{1}{2\eta}W_2^2\big(\pi(\cdot\mid s),\pi_k(\cdot\mid s)\big)
-\tau H^\pi(s)
\right\}.
\]
An operator-splitting derivation decomposes this into an optimal-transport step
\[
T_s(b)\in\argmax_{a\in\mathcal A}
\left\{Q(a)-\frac{1}{2\eta}\|a-b\|^2\right\},
\qquad
\pi_{k+\frac12}(\cdot\mid s)=(T_s)_\#\pi_k(\cdot\mid s),
\]
followed by a heat step
\[
\pi_{k+1}(\cdot\mid s)
=
\pi_{k+\frac12}(\cdot\mid s)*\mathcal N(0,2\tau\eta I).
\]
This is a genuine proximal policy update in Wasserstein space, with a global linear convergence theorem under a \(T_2(\lambda)\) transportation-information inequality [2603.02576].

A different geometric reparameterization appears in "Group Orthogonalized Policy Optimization: Group Policy Optimization as Orthogonal Projection in Hilbert Space" [2602.21269]. Fixing a reference policy \(\pi_k\), it defines the density fluctuation field
\[
v(y)=\frac{\pi(y)}{\pi_k(y)}-1
\]
in the Hilbert space
\[
\mathcal H=L^2(\pi_k),\qquad
\langle f,g\rangle_{\pi_k}=\mathbb E_{\pi_k}[f(y)g(y)].
\]
Normalization becomes the orthogonality constraint
\[
\mathbb E_{\pi_k}[v]=\langle v,\mathbf 1\rangle_{\pi_k}=0,
\]
so feasible updates lie in the codimension-one subspace
\[
\mathcal H_0=\{f\in\mathcal H:\langle f,\mathbf 1\rangle_{\pi_k}=0\}.
\]
The work-dissipation functional
\[
\mathcal J(v)=\langle g_\alpha,v\rangle_{\pi_k}-\frac{\mu}{2}\|v\|_{\pi_k}^2
\]
has optimizer given by the Hilbert projection theorem:
\[
v^\ast=P_{\mathcal H_0}(u^\ast)
=
\frac{1}{\mu}\Bigl(g_\alpha-\mathbb E_{\pi_k}[g_\alpha]\mathbf 1\Bigr),
\qquad
u^\ast=\frac{g_\alpha}{\mu}.
\]
With positivity enforced, the bounded projection becomes
\[
v^\ast(y)=\max\!\left(-1,\frac{g_\alpha(y)-\lambda^\ast}{\mu}\right),
\]
inducing exact sparsity for sufficiently poor actions [2602.21269].

Taken together, these works show that policy gradient projection now spans Euclidean projection on policy simplices, KL- and \(\alpha\)-divergence projection, action-level Euclidean projection onto safe sets, projection of uncertainty or weights in robust optimization, Wasserstein proximal maps on action distributions, and orthogonal projection in Hilbert spaces. A common structure persists: an unconstrained improvement direction is first defined in a larger linear or distributional space, and learning then proceeds by mapping that target back into the admissible policy class or safe region under a specified geometry [2006.11266], [2605.06139], [2603.02576], [2602.21269].

Source: https://www.emergentmind.com/topics/policy-gradient-projection