Group Orthogonalized Policy Optimization

• The paper introduces GOPO, a novel method that uses Hilbert space projection to reformulate policy improvement and ensure probability conservation via linear orthogonality.
• It replaces KL-based geometry with an orthogonal projection approach, avoiding gradient saturation and enabling a smooth, quadratic work-dissipation update.
• Empirical evaluations on mathematical reasoning benchmarks demonstrate competitive reward metrics, robust gradient norms, and enhanced entropy compared to alternative methods.

Group Orthogonalized Policy Optimization (GOPO) is a policy-alignment method for LLMs that formulates group policy optimization as orthogonal projection in the Hilbert space $L^2(\pi_k)$ of square-integrable functions with respect to a reference policy $\pi_k$, rather than as optimization on the probability simplex with Kullback–Leibler geometry. In this formulation, policy change is represented by the density-fluctuation field $v(y)=\pi(y)/\pi_k(y)-1$, probability conservation reduces to the linear orthogonality condition $\langle v,\mathbf 1\rangle_{\pi_k}=0$, and policy improvement is derived from a quadratic work-dissipation functional. The resulting group-level objective uses reward-normalized advantages and a quadratic restoring penalty, and the method is evaluated against GRPO, DAPO, GSPO, and OPO on mathematical reasoning benchmarks (Zixian, 24 Feb 2026).

1. Geometric reformulation of policy alignment

GOPO is presented as a policy-alignment method for LLMs trained from reward signals or preference feedback, especially in the setting of group-based rollout sampling used by methods like GRPO. Its starting point is a geometric claim: prior methods optimize on the probability simplex, whereas GOPO formulates policy improvement as a projection problem in a Hilbert space. The paper argues that KL-based policy optimization has an intrinsic geometric weakness because KL induces exponential curvature in log-ratio or logit space; as the policy becomes confident, the gradient saturates, leading to plateauing and motivating heuristic interventions such as clipping and entropy bonuses (Zixian, 24 Feb 2026).

The change of variables is central. Let $\pi_k$ denote the current reference policy, and define the fluctuation field

$v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$

so that

$\pi(y)=\pi_k(y)(1+v(y)).$

This lifts policy optimization into

$\mathcal H=L^2(\pi_k).$

Within these coordinates, the simplex constraint becomes linear. Since probability conservation requires $\sum_y \pi(y)=1$, one obtains

$$\sum_y \pi_k(y)(1+v(y)) = 1+\mathbb E_{\pi_k}[v] = 1,$$

hence

$\pi_k$0

The feasible set is therefore the codimension-one closed subspace

$\pi_k$1

This simplex-to-orthogonality reduction is the specific sense in which the method is “orthogonalized.” The normalization constraint is not handled by a nonlinear simplex geometry but by orthogonality to the constant function. A plausible implication is that the method separates normalization from the rest of the alignment objective more cleanly than KL-anchored formulations.

2. Projection formulation and bounded feasibility

The paper introduces an unconstrained target

$\pi_k$2

where $\pi_k$3 is the effective driving field, typically advantage-shaped and possibly metric-modulated by an escort exponent $\pi_k$4, and $\pi_k$5 is a stiffness or compliance parameter. The constrained minimum-distance problem is

$\pi_k$6

Expanding the squared norm yields

$\pi_k$7

so, after dropping the constant term, the problem is equivalent to maximizing the work-dissipation functional

$\pi_k$8

The term $\pi_k$9 is interpreted as external work, and $v(y)=\pi(y)/\pi_k(y)-1$0 as quadratic dissipation or a trust-region penalty (Zixian, 24 Feb 2026).

Because $v(y)=\pi(y)/\pi_k(y)-1$1 is a closed subspace, the Hilbert Projection Theorem gives the unique optimizer:

$v(y)=\pi(y)/\pi_k(y)-1$2

Since $v(y)=\pi(y)/\pi_k(y)-1$3,

$v(y)=\pi(y)/\pi_k(y)-1$4

Therefore,

$v(y)=\pi(y)/\pi_k(y)-1$5

The scalar subtraction term is interpreted as the chemical potential $v(y)=\pi(y)/\pi_k(y)-1$6, i.e. the Lagrange multiplier enforcing probability conservation.

To impose non-negativity of probabilities, GOPO adds the pointwise constraint

$v(y)=\pi(y)/\pi_k(y)-1$7

equivalent to $v(y)=\pi(y)/\pi_k(y)-1$8. The feasible set becomes

$v(y)=\pi(y)/\pi_k(y)-1$9

The KKT solution is

$\langle v,\mathbf 1\rangle_{\pi_k}=0$0

with $\langle v,\mathbf 1\rangle_{\pi_k}=0$1 chosen so that $\langle v,\mathbf 1\rangle_{\pi_k}=0$2. The paper states the corollary that if

$\langle v,\mathbf 1\rangle_{\pi_k}=0$3

then $\langle v,\mathbf 1\rangle_{\pi_k}=0$4, so the target probability is zero. This is the basis for the paper’s exact sparsity claim for catastrophically poor actions.

3. Group-sampled empirical objective

The practical algorithm is derived by projecting the infinite-dimensional Hilbert-space formulation onto a finite empirical subspace induced by group sampling. For a prompt $\langle v,\mathbf 1\rangle_{\pi_k}=0$5, GOPO samples a group

$\langle v,\mathbf 1\rangle_{\pi_k}=0$6

and approximates the inner product empirically by

$\langle v,\mathbf 1\rangle_{\pi_k}=0$7

The empirical zero-mean subspace is

$\langle v,\mathbf 1\rangle_{\pi_k}=0$8

Rewards $\langle v,\mathbf 1\rangle_{\pi_k}=0$9 are normalized into advantages

$\pi_k$0

which implies

$\pi_k$1

The paper emphasizes that this makes the empirical driving vector already lie in the zero-mean subspace:

$\pi_k$2

Consequently, the probability-conservation multiplier vanishes exactly at the group level; there is no separate solve for $\pi_k$3 and no extra constraint-enforcement step (Zixian, 24 Feb 2026).

Replacing the fluctuation field by the policy ratio

$\pi_k$4

the practical GOPO loss is

$\pi_k$5

The paper also gives a bounded variant,

$\pi_k$6

which acts like a ReLU floor and halts gradients for highly suppressed actions.

Operationally, the algorithm proceeds by on-policy anchoring $\pi_k$7, group sampling, scoring and normalization, ratio computation, loss evaluation, and gradient update

$\pi_k$8

The paper states that no critic network is required and that the extra computational overhead is negligible, essentially just the $\pi_k$9 term.

4. Optimization properties and theoretical claims

For the pointwise loss

$v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$0

GOPO has

$v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$1

The Hessian is therefore the constant scalar $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$2, independent of the advantage $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$3, the current policy state, and the data distribution. This constant-curvature property is one of the method’s main theoretical claims (Zixian, 24 Feb 2026).

The gradient magnitude is proportional to the distance from equilibrium:

$v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$4

The paper uses this to argue that gradients remain linear and non-saturating, unlike KL or logistic gradients. It further gives the ratio-space update relation

$v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$5

so for $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$6, gradient descent contracts globally to $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$7.

The bounded version introduces what the paper calls an intrinsic dead-zone mechanism. Once an action’s ratio is driven to zero, or equivalently $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$8, its gradient is shut off exactly. The paper distinguishes this from PPO-style clipping: the dead zone is attributed to projection onto the feasible boundary of probability space rather than to a flat interval around ratio $v(y)=\frac{\pi(y)}{\pi_k(y)}-1,$9.

The paper also gives a divergence interpretation:

$\pi(y)=\pi_k(y)(1+v(y)).$0

and notes the total-variation bound

$\pi(y)=\pi_k(y)(1+v(y)).$1

This suggests that the quadratic penalty can be read as a $\pi(y)=\pi_k(y)(1+v(y)).$2-style trust region controlling distributional drift. The paper’s entropy-preservation claim is stated more cautiously: GOPO is said to preserve higher entropy than token-level clipping methods because updates are applied at the sequence or group level, it does not micromanage every token with dense clipped supervision, and the smooth quadratic penalty avoids aggressive mode collapse.

5. Implementations and reported empirical behavior

The reported implementation uses Qwen3-1.7B, VERL, 4× RTX 4090, group size $\pi(y)=\pi_k(y)(1+v(y)).$3, learning rate $\pi(y)=\pi_k(y)(1+v(y)).$4, 8 epochs, GOPO stiffness $\pi(y)=\pi_k(y)(1+v(y)).$5, escort exponent $\pi(y)=\pi_k(y)(1+v(y)).$6, and on-policy anchoring. The experiments are conducted on mathematical reasoning benchmarks, training on about 10% of MATH Level 3 and validating on 100 MATH Level 4 problems, explicitly to test generalization rather than only in-distribution training reward (Zixian, 24 Feb 2026).

The compared methods are GRPO, DAPO, GSPO, OPO, and GOPO. Reported mean reward and validation accuracy are as follows:

Method	Mean reward	Val acc
GRPO	0.544	44%
DAPO	0.548	44%
GSPO	0.553	48%
OPO	0.558	48%
GOPO	0.555	47%

The paper also reports gradient norm and entropy. For GRPO, these are 0.674 and 0.115; for DAPO, 0.213 and 0.126; for GSPO, 0.623 and 0.128; for OPO, 1.279 and 0.126; and for GOPO, 1.029 and 0.134. The stated observations are that GOPO reaches 47% validation accuracy with monotonic improvement over training, OPO slightly outperforms GOPO on validation in this experiment, clipping-based methods plateau at 44% validation accuracy, gradient norms are healthier for OPO and GOPO than for GRPO and DAPO, and entropy is highest for GOPO. BHP is described theoretically, but the implementation uses a soft ReLU approximation rather than the exact hard threshold. Further validation on tasks such as instruction following or code generation is explicitly left as future work.

6. Terminological scope, related work, and common confusions

A recurring source of ambiguity is that the acronym GOPO is also used for Group Ordinal Policy Optimization, introduced in “GOPO: Policy optimization using ranked rewards” (Choi et al., 1 Feb 2026). That method is a modification of GRPO for non-verifiable tasks such as summarization, instruction following, and chat completion. It replaces standardized reward advantages with within-prompt ordinal ranks, discarding reward magnitudes entirely. Its central claim is different from Group Orthogonalized Policy Optimization: the former is a rank-based policy-optimization method motivated by the mismatch between pairwise preference-trained reward models and magnitude-based RL updates, whereas the latter is a Hilbert-space projection method motivated by the geometry of policy optimization.

There is also a second, distinct use of “orthogonalized” in adjacent reinforcement-learning literature. “Orthogonalized Estimation of Difference of $\pi(y)=\pi_k(y)(1+v(y)).$7-functions” develops a dynamic generalization of the R-learner for finite-horizon offline RL, using residualized losses, nuisance estimation, and cross-fitting to estimate $\pi(y)=\pi_k(y)(1+v(y)).$8-function contrasts such as

$\pi(y)=\pi_k(y)(1+v(y)).$9

with a universal orthogonality condition ensuring that nuisance perturbations enter only through second-order product terms (Cao et al., 2024). That work is described as conceptually close to a GOPO-style approach, but it is not the same algorithmic object as Group Orthogonalized Policy Optimization.

Accordingly, “orthogonalized” should not be treated as a single technical doctrine across these papers. In Group Orthogonalized Policy Optimization, orthogonalization refers to the reduction of probability conservation to orthogonality against the constant function in $\mathcal H=L^2(\pi_k).$0. In the offline RL contrast-estimation literature, orthogonality refers to score or loss constructions whose first derivatives with respect to nuisance components vanish. The shared terminology indicates a family resemblance around constraint handling and robustness, but the mathematical mechanisms are different.