Intra-Anchor GRPO: Relative Policy Optimization

• The paper introduces intra-anchor GRPO as a method where policy updates are computed within a shared conditioning anchor, enabling stable relative normalization without an external baseline.
• It details various implementations including within-instance whole-trajectory GRPO, P-GRPO with historical preference anchoring, and dual-anchor methods that adjust reward scales.
• Empirical results across domains such as routing, visual generation, and speech reconstruction demonstrate improved stability, faster convergence, and enhanced performance.

“Intra-Anchor Group Relative Policy Optimization” is best treated as an Editor’s term for a class of GRPO interpretations in which relative policy updates are computed inside a shared conditioning anchor—or against anchor-conditioned reference statistics—rather than against a learned critic or a detached rollout baseline. The phrase itself is not standardized in the cited literature. Instead, the relevant papers speak of within-instance normalization, preference-group historical normalization, dual-anchor advantages, semantic prior anchors, value anchoring, or anchor-gated rewards. Across these formulations, the common structure is that an anchor determines which samples are comparable, which statistics are valid baselines, or which absolute reference prevents purely relative optimization from becoming degenerate (Sepúlveda et al., 9 Jun 2026, Wang et al., 17 Feb 2026, Liu et al., 24 Nov 2025).

1. Standard GRPO and the intra-anchor reading

In standard GRPO, a policy samples a group of outputs for the same conditioning context and computes normalized advantages from the rewards inside that group. In the language-model formulation used in alignment work, for a prompt $q$, GRPO samples $G$ outputs and forms

$$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$

The update is therefore not based on an absolute reward scale alone, but on how an output ranks relative to sibling outputs generated for the same prompt (Vojnovic et al., 25 Feb 2025).

This same-prompt structure is what motivates the intra-anchor interpretation. Theoretical treatments make the anchor explicit as the conditioning variable $X$: for each sampled prompt $X^{(b)}$, GRPO draws a group $\{Y^{(b,g)}\}_{g=1}^G$ and uses a leave-one-out baseline

$$\bar Z^{(b,-g)}=\frac{\sum_{k\neq g}Z^{(b,k)}}{G-1},$$

so rewards from one prompt are never used as baselines for another (Zhou et al., 1 Mar 2026). A related stationary-policy analysis likewise characterizes GRPO as optimizing a within-prompt, group-relative preference objective, rather than raw expected reward in the RLHF sense (Vojnovic et al., 25 Feb 2025).

This suggests that “intra-anchor” is most naturally understood as within-context relative optimization: a fixed prompt, graph instance, input image, speech recording, or other conditioning object serves as the anchor, and only samples generated under that same anchor enter the comparison set. That reading is explicit in the neural combinatorial optimization formulation, where all trajectories in a group come from the same routing instance $\mathbf{x}_b$ and there is “no cross-instance normalization” (Sepúlveda et al., 9 Jun 2026).

2. Anchor semantics across the literature

The literature does not use a single anchor notion. Instead, several distinct anchor mechanisms appear, each changing a different part of GRPO.

Formulation	Anchor interpretation	Mechanism
Standard GRPO / NCO / GRPO-RM	Shared conditioning input	Normalize rewards within samples from one prompt, instance, or input
P-GRPO	Preference-group historical anchor	Normalize by $(\mu_p,\sigma_p)$ from historical rewards of group $p$
MDP-GRPO	Dual anchor	Combine intra-group $G$0 with goal-aware $G$1
BPGO	Semantic prior anchor	Reweight groups and transform rewards by deviation from $G$2
VGPO	Temporal and group value anchors	Use per-step $G$3, $G$4, and an absolute-value term
AP-GRPO	Audible anchor structure	Partition rewards by matched anchors and inter-anchor spans

In the most direct anchor-local reading, the anchor is simply the shared conditioning object. In neural combinatorial optimization, a fixed TSP or CVRP instance plays that role, and GRPO computes

$G$5

using only the $G$6 trajectories sampled for that one instance (Sepúlveda et al., 9 Jun 2026). In GRPO-RM, the same pattern is transferred to representation learning by treating the input image as the anchor and the predefined class set as its candidate group; advantages are then normalized inside that input-conditioned candidate set (Xu et al., 19 Nov 2025).

Other papers redefine the anchor as a more persistent statistical reference. P-GRPO replaces concurrent batch statistics with preference-group historical statistics,

$G$7

where $G$8 are maintained online for preference group $G$9. The paper does not call this an anchor, but it explicitly frames it as preference-specific historical normalization, which is the closest anchor-like baseline in that setting (Wang et al., 17 Feb 2026). BPGO goes further and introduces a semantic prior anchor $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$0 for both inter-group trust weighting and intra-group reward transformation (Liu et al., 24 Nov 2025). VGPO similarly introduces absolute anchors in the form of timestep-specific cumulative values $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$1 and value-weighted advantages (Shao et al., 13 Dec 2025).

A distinct use of anchors appears in pathological speech reconstruction. AP-GRPO preserves reliable audible anchors in clear regions and scores candidate inter-anchor spans by phonetic alignment to the corresponding corrupted speech spans. Here the anchor does not define the GRPO group itself; instead, it structures the reward inside each sampled candidate (Zhang et al., 14 Jun 2026).

3. Representative algorithmic forms

The simplest intra-anchor form is within-instance whole-trajectory GRPO. In routing problems, a policy $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$2 generates a full feasible trajectory $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$3, receives reward $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$4, and updates from groups of trajectories drawn for the same instance. The loss is a PPO-style clipped surrogate over complete trajectories,

$$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$5

with $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$6 the old-policy trajectory ratio. The baseline removed by GRPO is the external frozen-policy rollout baseline used by REINFORCE; the old policy is retained only for PPO-style clipping (Sepúlveda et al., 9 Jun 2026). This is the cleanest example of intra-anchor relative normalization as an alternative to an external baseline.

P-GRPO modifies only the normalization source, not the surrounding PPO/GRPO objective. Standard GRPO uses current-generation-group statistics $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$7, whereas P-GRPO uses historical preference-group statistics $$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$8. The paper gives the exact relation

$$A_i=\frac{r_i-\mathrm{mean}(r_1,\ldots,r_G)}{\mathrm{std}(r_1,\ldots,r_G)}.$$9

making the anchor-like role of $X$0 explicit as a bias correction relative to transient batch statistics (Wang et al., 17 Feb 2026).

MDP-GRPO addresses low-dispersion reward regimes by introducing dual-anchor advantages. Standard GRPO’s intra-group term is

$X$1

while the new goal-aware anchor is

$X$2

Both are passed through a prospect-theoretic shaping map and then mixed as

$X$3

This is not an intra-anchor method in the narrow sense, because the second anchor is external to the group. It is, however, a clear example of GRPO moving from a single relative anchor to a multi-anchor estimator (Salmani-Zarchi et al., 4 Jun 2026).

BPGO introduces an explicit intra-group prior-anchored renormalization. At the group level, Reliability-Adaptive Scaling weights group losses by

$X$4

Within groups, Contrastive Reward Transformation maps each reward $X$5 to a prior-centered transformed reward $X$6, after which GRPO is applied again to the transformed group. The paper explicitly describes this as intra-group prior-anchored renormalization (Liu et al., 24 Nov 2025).

VGPO anchors GRPO in both time and group dimensions. It defines an instant reward by one-step ODE projection,

$X$7

then a cumulative value

$X$8

and finally the anchored advantage

$X$9

The numerator can be read as a relative intra-group term plus an absolute value anchor, which the paper uses to prevent stagnation when group reward diversity collapses (Shao et al., 13 Dec 2025).

4. Objective geometry, comparison structure, and group size

A central theoretical result is that GRPO is not merely “reward minus baseline” in an informal sense. Under fixed-context analysis, the GRPO gradient is exactly a second-order U-statistic with kernel

$X^{(b)}$0

Its first-order Hoeffding projection coincides with the oracle actor-critic gradient that would subtract the true prompt-conditional value $X^{(b)}$1, which is why GRPO is asymptotically equivalent to an oracle policy-gradient method (Zhou et al., 1 Mar 2026).

This pairwise structure also clarifies why small groups can still matter. In the binary-reward RLVR setting, the two-rollout case reduces to a positive-negative comparison for the same prompt, and the paper “It Takes Two: Your GRPO Is Secretly DPO” shows that 2-GRPO is a valid minimal pairwise instance of GRPO. The analysis derives that, under Bernoulli rewards, the expected normalized signal in 2-GRPO differs from large-group GRPO mainly by a scaling factor, and experiments show performance on par with 16-GRPO while using $X^{(b)}$2 of the rollouts and reducing training time by over $X^{(b)}$3 (Wu et al., 1 Oct 2025).

A complementary analysis of verifiable rewards shows that unclipped GRPO can be written as a KL-regularized contrastive loss, with positive and negative weights determined by the old policy’s success probability $X^{(b)}$4. Under binary rewards, successful outputs receive weight $X^{(b)}$5 and failures receive weight $X^{(b)}$6, making explicit that GRPO is performing adaptive contrastive amplification rather than ordinary absolute-reward maximization (Mroueh, 9 Mar 2025).

This theoretical picture aligns with the stationary-policy analysis of GRPO’s alignment objective. There, GRPO is characterized as optimizing a reference-regularized, within-prompt, group-relative preference objective, with a regularizer whose stationary effect is essentially reverse-KL-like rather than the direct-KL logarithmic pooling familiar from standard RLHF (Vojnovic et al., 25 Feb 2025).

5. Empirical behavior across domains

In neural combinatorial optimization, intra-instance GRPO was evaluated on TSP-50, TSP-100, CVRP-50, and CVRP-100 using the RL4CO Attention Model. At 100 training epochs it achieved the best average test costs among the compared methods—$X^{(b)}$7 on TSP-50, $X^{(b)}$8 on TSP-100, $X^{(b)}$9 on CVRP-50, and $\{Y^{(b,g)}\}_{g=1}^G$0 on CVRP-100—and, crucially, it avoided the REINFORCE training collapse observed on TSP-100 after rollout-baseline warmup (Sepúlveda et al., 9 Jun 2026). The paper interprets this as evidence that relative signal within a hard instance can remain informative even when an external frozen-policy baseline becomes poor.

In heterogeneous preference alignment, P-GRPO replaces batch-relative normalization with preference-group historical normalization and reports faster convergence and higher rewards than standard GRPO. On MovieLens with Qwen3-8B, for example, P-GRPO achieved $\{Y^{(b,g)}\}_{g=1}^G$1 at 4 choices versus $\{Y^{(b,g)}\}_{g=1}^G$2 for GRPO; its gains also depended on meaningful clustering, since random cluster assignments removed the benefits (Wang et al., 17 Feb 2026). This supports anchor-conditioned normalization when reward distributions are non-exchangeable.

For low-dispersion verifiable rewards, MDP-GRPO identifies three GRPO pathologies—low-variance amplification, mean-centering blindness, and zero-variance collapse—and reports improvements in strict constraint satisfaction of up to $\{Y^{(b,g)}\}_{g=1}^G$3 on Llama-3.2-3B. Its dual-anchor and multi-temperature design is presented as especially important when group size is small and homogeneous reward groups are common (Salmani-Zarchi et al., 4 Jun 2026).

Anchor-based reward structuring also appears effective in pathological speech reconstruction. AP-GRPO combines anchor preservation and inter-anchor phonetic alignment; on TORGO with Qwen2.5-Omni, the full model reduced WER to $\{Y^{(b,g)}\}_{g=1}^G$4, while anchor-gate only reached $\{Y^{(b,g)}\}_{g=1}^G$5 and phonetic reward only reached $\{Y^{(b,g)}\}_{g=1}^G$6. The learned dual variable $\{Y^{(b,g)}\}_{g=1}^G$7 varied by disease condition, with higher anchor enforcement reported for severe articulatory degradation (Zhang et al., 14 Jun 2026).

In visual generation, BPGO reports consistently stronger semantic alignment and faster convergence than standard GRPO and recent variants, attributing the gain to inter-group trust allocation and intra-group prior-anchored renormalization (Liu et al., 24 Nov 2025). VGPO likewise reports state-of-the-art image quality together with improved task-specific accuracy on GenEval, OCR, and PickScore, explicitly framing its value anchoring as a way to mitigate reward hacking and optimization stagnation (Shao et al., 13 Dec 2025).

6. Limitations, ambiguities, and scope

The first limitation is terminological. “Intra-anchor GRPO” is not a formal algorithm name in the cited works. Some papers treat the anchor as the conditioning input, others as a persistent historical baseline, others as an external absolute reference, and still others as a reward-structuring device inside each candidate. Any unified use of the term is therefore interpretive rather than canonical (Sepúlveda et al., 9 Jun 2026, Zhang et al., 14 Jun 2026).

A second limitation is that pure group-relative normalization can fail when diversity collapses. In routing GRPO, if all $\{Y^{(b,g)}\}_{g=1}^G$8 trajectories for an instance become identical, then $\{Y^{(b,g)}\}_{g=1}^G$9 and the relative signal degenerates; the reported mitigation is higher sampling temperature early in training (Sepúlveda et al., 9 Jun 2026). MDP-GRPO formalizes the analogous low-dispersion problem in RLVR as low-variance amplification, mean-centering blindness, and zero-variance collapse (Salmani-Zarchi et al., 4 Jun 2026). VGPO generalizes the same concern to flow matching, arguing that exclusive reliance on intra-group relative rewards causes optimization stagnation when reward diversity is depleted (Shao et al., 13 Dec 2025).

A third limitation is anchor quality itself. P-GRPO depends on meaningful preference-group structure; coarse grouping hurts and random clustering removes the gains (Wang et al., 17 Feb 2026). FGRPO, although not an intra-anchor method in the narrow sense, shows that even when local GRPO normalization works within each client, cross-client aggregation can still be unstable on non-IID data; its solution is a client-level personalized EMA anchor $$\bar Z^{(b,-g)}=\frac{\sum_{k\neq g}Z^{(b,k)}}{G-1},$$0 and relative performance gain weighting rather than a new local advantage estimator (Chen et al., 2 Jun 2026). This suggests that anchor mechanisms solve different comparability problems depending on whether the instability is intra-group, cross-time, or cross-client.

A final ambiguity concerns what exactly is being anchored: rewards, values, groups, or policies. Some formulations still use an old policy only for PPO clipping while removing baseline dependence, as in trajectory-level GRPO for routing (Sepúlveda et al., 9 Jun 2026). Others retain explicit KL anchors, as in MDP-GRPO or the stationary analyses of GRPO (Salmani-Zarchi et al., 4 Jun 2026, Vojnovic et al., 25 Feb 2025). Still others anchor only the reward transformation, as in BPGO, or only the span-level reward decomposition, as in AP-GRPO (Liu et al., 24 Nov 2025, Zhang et al., 14 Jun 2026). The literature therefore supports no single universally accepted definition of intra-anchor GRPO; rather, it supports a family resemblance centered on anchor-conditioned comparability and anchor-stabilized relative optimization.