Selective-Advantage AH-GRPO for RL Language Models

• The paper introduces SA-AH-GRPO, a GRPO variant that applies token-level entropy discounts only to negative-advantage trajectories while fully reinforcing positive outcomes.
• SA-AH-GRPO leverages an entropy-adaptive horizon by computing cumulative token-level discount weights using approximated top-K logits to modulate gradient updates.
• Empirical results on GSM8K show that SA-AH-GRPO improves final accuracy and reduces training variance compared to standard GRPO, highlighting its potential for modular RL refinements.

Selective-Advantage Entropy-Adaptive Horizon GRPO, abbreviated SA-AH-GRPO, is a Group Relative Policy Optimisation (GRPO) variant for reinforcement learning of LLMs that introduces an entropy-based, token-level discount on the policy gradient and applies that discount only to trajectories with negative advantage, while leaving positive-advantage trajectories unattenuated (Chawla et al., 3 Jun 2026). In this formulation, successful or above-average rollouts retain the full gradient signal, whereas underperforming rollouts are discounted more strongly at token positions associated with higher policy entropy. The method was introduced together with Adaptive-Horizon GRPO (AH-GRPO) and evaluated on GSM8K using Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct fine-tuned with LoRA (Chawla et al., 3 Jun 2026).

1. Position within the GRPO family

SA-AH-GRPO is defined relative to standard GRPO. For each prompt $q$, GRPO samples a group of $G$ completions $\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$, assigns each completion a scalar reward $r_i$, and computes a group-normalised advantage

$$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$

Using the per-token ratio

$$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$

GRPO optimises the clipped surrogate

$$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$

In this baseline, all tokens in a rollout share the same $A_i$, and all token positions are weighted symmetrically (Chawla et al., 3 Jun 2026).

AH-GRPO modifies this symmetry by introducing an entropy-adaptive horizon: token-level gradients are multiplied by cumulative entropy-derived weights that shorten the effective horizon in uncertain regions. SA-AH-GRPO adds a further asymmetry: the entropy-based attenuation is applied only when $A_i<0$, so that positive-advantage rollouts are not discounted at all (Chawla et al., 3 Jun 2026).

This places SA-AH-GRPO within a broader line of work that addresses coarse credit assignment in GRPO. Other contemporaneous proposals redistribute advantages across blocks in structured outputs (Pavlenko et al., 10 Feb 2026), across tokens using outcome-grounded attribution (Li et al., 12 Jan 2026), or across sign classes of trajectories in sparse-reward settings (Sana et al., 28 May 2026). SA-AH-GRPO differs by retaining a trajectory-level advantage while making token weighting depend jointly on entropy and the sign of the rollout advantage.

2. Entropy-adaptive horizon

The entropy-adaptive component is inherited from AH-GRPO. For rollout $i$ and token position $G$0, the method defines a normalised token entropy

$G$1

where $G$2 is vocabulary size. In implementation, entropy is approximated using the top-$G$3 logits, with $G$4 in the reported experiments (Chawla et al., 3 Jun 2026).

From this entropy, SA-AH-GRPO inherits the per-step discount

$G$5

and the cumulative weight

$G$6

The parameter $G$7 controls discount strength. When $G$8, all $G$9 and the method reduces to standard GRPO; when $\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$0, high-entropy positions cause cumulative decay, so later tokens after uncertain prefixes receive smaller gradient weights (Chawla et al., 3 Jun 2026).

AH-GRPO uses these weights for all rollouts. Its loss is

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$1

where

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$2

and $\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$3 is a padding mask (Chawla et al., 3 Jun 2026).

The conceptual motivation is that high-entropy positions are treated as less reliable gradient carriers. This suggests an implicit temporal trust region: the more uncertainty accumulated in a prefix, the more heavily later negative updates are discounted.

3. Selective-advantage mechanism

SA-AH-GRPO’s defining step is to condition the entropy-adaptive weight on the sign of the rollout advantage. It introduces the negative-advantage indicator

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$4

The selective token weight is then

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$5

Hence, underperforming rollouts receive the full entropy-adaptive discount, while successful or above-average rollouts are left untouched (Chawla et al., 3 Jun 2026).

The resulting objective is

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$6

Equivalently,

$\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$7

When all rollouts are negative, SA-AH-GRPO reduces to AH-GRPO; when $\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$8, it reduces to GRPO (Chawla et al., 3 Jun 2026).

The selective aspect is therefore not a change to the reward or baseline, but an asymmetric routing of token-level attenuation. The paper’s interpretation is that correct or above-average trajectories should be fully reinforced even if they contain uncertain tokens, whereas failed trajectories should be penalised more cautiously at positions where the policy itself was uncertain (Chawla et al., 3 Jun 2026). This is closely related in spirit to hysteretic weighting of negative updates in sparse-reward GRPO variants (Sana et al., 28 May 2026), but SA-AH-GRPO localises the asymmetry at token level through cumulative entropy discounting rather than only at rollout level.

4. Training procedure and implementation

The reported training loop follows a standard GRPO pipeline with the selective weighting inserted after advantage computation. For each prompt $\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)$9, the old policy $r_i$0 samples $r_i$1 completions. Each completion receives a scalar reward

$r_i$2

with the composite reward specified in Eq. (10) of the paper (Chawla et al., 3 Jun 2026). Group-relative advantages $r_i$3 are then computed, token entropies are estimated from top-$r_i$4 logits, cumulative weights are formed, and the PPO-style loss is evaluated with $r_i$5.

The experiments use Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct, with parameter-efficient fine-tuning via LoRA. The LoRA configuration is rank $r_i$6, $r_i$7, dropout $r_i$8, targeting all $r_i$9 and MLP projections $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$0 (Chawla et al., 3 Jun 2026).

The common RL hyperparameters for GRPO, AH-GRPO, and SA-AH-GRPO are: learning rate $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$1, cosine scheduler with warmup, weight decay $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$2, gradient clip $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$3, PPO clip $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$4, KL penalty $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$5, batch prompts per step $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$6, completions per prompt $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$7, gradient accumulation $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$8, max completion length $$A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G).$$9, and entropy top-$$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$0 (Chawla et al., 3 Jun 2026). SA-AH-GRPO uses $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$1 in all main experiments and is trained for 180 steps, whereas GRPO and AH-GRPO are reported at 150 steps for the principal comparison (Chawla et al., 3 Jun 2026).

The entropy-adaptive weights are computed in log-space for numerical stability: $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$2 The loss is normalised by the weighted valid-token count $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$3, rather than by a raw token count, so selective attenuation changes both numerator and denominator consistently (Chawla et al., 3 Jun 2026).

5. Empirical behavior on GSM8K

The empirical evaluation is confined to GSM8K, with 800 training examples and 500 evaluation examples, using greedy decoding and Pass@1 as the primary metric (Chawla et al., 3 Jun 2026). The reported zero-shot baselines are $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$4 for the 1.5B model and $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$5 for the 3B model (Chawla et al., 3 Jun 2026).

For the 3B model, the comparison is:

• GRPO ($$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$6): final Pass@1 $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$7, peak $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$8, training variance $$\rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,<t})} {\pi_{\theta_{\mathrm{old}}}(o_{i,t} \mid q, o_{i,<t})},$$9.
• AH-GRPO ($$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$0): final $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$1, peak $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$2, variance $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$3.
• SA-AH-GRPO ($$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$4): peak $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$5 at step 30, final $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$6 at step 180, variance $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$7, described as a 3.6 times reduction relative to GRPO while matching its peak accuracy (Chawla et al., 3 Jun 2026).

For the 1.5B model, the comparison is:

• GRPO: final $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$8, peak $$L_{\mathrm{GRPO}}(\theta) = -\mathbb{E}\!\left[ \frac{1}{|o_i|}\sum_{t=1}^{|o_i|} \min\!\left( \rho_t^{(i)}\,A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)\,A_i \right) - \beta\,\mathrm{KL}(\pi_\theta \,\|\, \pi_{\mathrm{ref}}) \right].$$9, variance $A_i$0.
• AH-GRPO: final $A_i$1, peak $A_i$2, variance $A_i$3.
• SA-AH-GRPO: peak and final $A_i$4 at step 180, improving over the zero-shot baseline of $A_i$5 (Chawla et al., 3 Jun 2026).

The 3B results emphasise variance reduction and stability. The 1.5B results emphasise better final accuracy, although the paper notes that variance for SA-AH-GRPO is higher than GRPO and AH-GRPO at this smaller scale, indicating that the stability benefit is scale-dependent (Chawla et al., 3 Jun 2026).

The AH-GRPO $A_i$6-ablation on the 1.5B model shows that entropy-amplifying $A_i$7 performs worst, with final Pass@1 $A_i$8, below the GRPO baseline. Positive $A_i$9 values outperform negative $A_i<0$0, and SA-AH-GRPO with $A_i<0$1 exceeds all tested AH-GRPO variants, suggesting that advantage-sign selectivity matters in addition to the entropy discount itself (Chawla et al., 3 Jun 2026).

Training logs reported in the paper show declining mean normalised entropy over time, so $A_i<0$2 tends toward 1 as the policy becomes more confident. The authors describe this as a self-annealing curriculum: the selective discount is strongest when uncertainty is high and naturally weakens later in training (Chawla et al., 3 Jun 2026).

6. Relation to neighboring methods, interpretation, and limitations

SA-AH-GRPO belongs to a wider family of attempts to refine GRPO’s coarse sequence-level credit assignment. OAR redistributes a sequence-level advantage across tokens using outcome-grounded perturbation or gradient saliency while preserving total advantage mass (Li et al., 12 Jan 2026). Blockwise Advantage Estimation routes objective-specific advantages to structured text blocks and uses outcome-conditioned baselines for later blocks (Pavlenko et al., 10 Feb 2026). HPO and A-HPO reduce the weight of negative-advantage updates and replace per-response length normalization with mean-length normalization in sparse-reward regimes (Sana et al., 28 May 2026). GRAIL reweights token-level advantages by gradient-activation saliency with respect to final-answer tokens (Pala et al., 3 Jun 2026). Compared with these methods, SA-AH-GRPO keeps the rollout-level scalar advantage of GRPO intact and changes only the token-level weighting, using entropy and advantage sign rather than attribution or block structure.

This suggests that SA-AH-GRPO is best understood as an asymmetric trust mechanism rather than a full fine-grained credit assignment method. It does not infer which specific token caused success or failure; rather, it discounts negative updates in uncertain regions and leaves positive updates untouched. In that respect it is simpler than token-attribution methods (Li et al., 12 Jan 2026, Pala et al., 3 Jun 2026) and more local than block-structured methods (Pavlenko et al., 10 Feb 2026).

The paper’s limitations are explicit. Evaluation is restricted to two Qwen 2.5 models, one dataset, and a single random seed. There is no $A_i<0$3-ablation specific to SA-AH-GRPO, only to AH-GRPO. Entropy is approximated with top-$A_i<0$4 logits using $A_i<0$5, without a dedicated $A_i<0$6-ablation. Some metric differences are within 95% confidence intervals, so the paper advises caution in interpretation (Chawla et al., 3 Jun 2026).

A broader implication, though not established experimentally in the paper, is that SA-AH-GRPO offers a modular asymmetry that could be combined with other GRPO refinements. Because it leaves group-relative advantage computation, PPO clipping, and KL regularisation intact, it is plausibly composable with structured advantage routing (Pavlenko et al., 10 Feb 2026), sign-asymmetric rollout weighting (Sana et al., 28 May 2026), or token-level attribution modules (Li et al., 12 Jan 2026, Pala et al., 3 Jun 2026). The published evidence, however, is limited to the entropy-selective mechanism itself on GSM8K (Chawla et al., 3 Jun 2026).