--- title: Selective-Advantage AH-GRPO for RL Language Models url: https://www.emergentmind.com/topics/selective-advantage-ah-grpo-sa-ah-grpo type: topic --- # Selective-Advantage AH-GRPO for RL Language Models Selective-Advantage Entropy-Adaptive Horizon GRPO, abbreviated SA-AH-GRPO, is a Group Relative Policy Optimisation (GRPO) variant for reinforcement learning of language models that introduces an entropy-based, token-level discount on the policy gradient and applies that discount only to trajectories with negative advantage, while leaving positive-advantage trajectories unattenuated [2606.05434]. In this formulation, successful or above-average rollouts retain the full gradient signal, whereas underperforming rollouts are discounted more strongly at token positions associated with higher policy entropy. The method was introduced together with Adaptive-Horizon GRPO (AH-GRPO) and evaluated on GSM8K using Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct fine-tuned with LoRA [2606.05434]. ## 1. Position within the GRPO family SA-AH-GRPO is defined relative to standard GRPO. For each prompt \(q\), GRPO samples a group of \(G\) completions \(\{o_i\}_{i=1}^{G}\sim \pi_\theta(\cdot \mid q)\), assigns each completion a scalar reward \(r_i\), and computes a group-normalised advantage \[ A_i = \frac{r_i - \mathrm{mean}(\bm{r})}{\mathrm{std}(\bm{r}) + \epsilon}, \qquad \bm{r} = (r_1,\dots,r_G). \] Using the per-token ratio \[ \rho_t^{(i)} = \frac{\pi_\theta(o_{i,t} \mid q, o_{i,0\), high-entropy positions cause cumulative decay, so later tokens after uncertain prefixes receive smaller gradient weights [2606.05434]. AH-GRPO uses these weights for all rollouts. Its loss is \[ L_{\mathrm{AH}}(\theta) = -\frac{ \displaystyle\sum_{i=1}^{G}\sum_{t=1}^{|o_i|} w_t^{(i)}\,\tilde{\ell}_t^{(i)}\,m_t^{(i)} }{ \displaystyle\sum_{i=1}^{G}\sum_{t=1}^{|o_i|} w_t^{(i)}\,m_t^{(i)} } \;+\; \beta\,\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}), \] where \[ \tilde{\ell}_t^{(i)} = \min\!\big( \rho_t^{(i)}A_i,\; \mathrm{clip}(\rho_t^{(i)},1{-}\epsilon,1{+}\epsilon)A_i \big), \] and \(m_t^{(i)}\) is a padding mask [2606.05434]. The conceptual motivation is that high-entropy positions are treated as less reliable gradient carriers. This suggests an implicit temporal trust region: the more uncertainty accumulated in a prefix, the more heavily later negative updates are discounted. ## 3. Selective-advantage mechanism SA-AH-GRPO’s defining step is to condition the entropy-adaptive weight on the sign of the rollout advantage. It introduces the negative-advantage indicator \[ n_i = \mathbf{1}[A_i < 0]. \] The selective token weight is then \[ \tilde{w}_t^{(i)} = n_i \cdot w_t^{(i)} + (1 - n_i) \cdot 1 = \begin{cases} w_t^{(i)} & A_i < 0 \\ 1 & A_i \ge 0 . \end{cases} \] Hence, underperforming rollouts receive the full entropy-adaptive discount, while successful or above-average rollouts are left untouched [2606.05434]. The resulting objective is \[ L_{\mathrm{SA}}(\theta) = -\frac{ \displaystyle\sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \tilde{w}_t^{(i)}\,\tilde{\ell}_t^{(i)}\,m_t^{(i)} }{ \displaystyle\sum_{i=1}^{G}\sum_{t=1}^{|o_i|} \tilde{w}_t^{(i)}\,m_t^{(i)} } \;+\; \beta\,\mathrm{KL}(\pi_\theta \| \pi_{\mathrm{ref}}). \] Equivalently, \[ \tilde{w}_t^{(i)} = \begin{cases} \exp\!\left(-\alpha \sum_{s=1}^t H_s^{(i)}\right) & \text{if } A_i < 0, \\ 1 & \text{if } A_i \ge 0. \end{cases} \] When all rollouts are negative, SA-AH-GRPO reduces to AH-GRPO; when \(\alpha=0\), it reduces to GRPO [2606.05434]. The selective aspect is therefore not a change to the reward or baseline, but an asymmetric routing of token-level attenuation. The paper’s interpretation is that correct or above-average trajectories should be fully reinforced even if they contain uncertain tokens, whereas failed trajectories should be penalised more cautiously at positions where the policy itself was uncertain [2606.05434]. This is closely related in spirit to hysteretic weighting of negative updates in sparse-reward GRPO variants [2605.30201], but SA-AH-GRPO localises the asymmetry at token level through cumulative entropy discounting rather than only at rollout level. ## 4. Training procedure and implementation The reported training loop follows a standard GRPO pipeline with the selective weighting inserted after advantage computation. For each prompt \(q\), the old policy \(\pi_{\theta_{\mathrm{old}}}\) samples \(G\) completions. Each completion receives a scalar reward \[ r = r_{\mathrm{correct}} + r_{\mathrm{format}} + r_{\mathrm{present}} + r_{\mathrm{steps}}, \] with the composite reward specified in Eq. (10) of the paper [2606.05434]. Group-relative advantages \(A_i\) are then computed, token entropies are estimated from top-\(K\) logits, cumulative weights are formed, and the PPO-style loss is evaluated with \(\tilde{w}_t^{(i)}\). The experiments use Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct, with parameter-efficient fine-tuning via LoRA. The LoRA configuration is rank \(r=16\), \(\alpha_{\mathrm{LoRA}}=32\), dropout \(=0.05\), targeting all \(q,k,v,o\) and MLP projections \(\{\mathrm{gate}, \mathrm{up}, \mathrm{down}\}\) [2606.05434]. The common RL hyperparameters for GRPO, AH-GRPO, and SA-AH-GRPO are: learning rate \(5\times 10^{-6}\), cosine scheduler with warmup, weight decay \(0.01\), gradient clip \(1.0\), PPO clip \(\epsilon=0.2\), KL penalty \(\beta=0.04\), batch prompts per step \(=4\), completions per prompt \(G=4\), gradient accumulation \(=2\), max completion length \(=512\), and entropy top-\(K=500\) [2606.05434]. SA-AH-GRPO uses \(\alpha=0.5\) in all main experiments and is trained for 180 steps, whereas GRPO and AH-GRPO are reported at 150 steps for the principal comparison [2606.05434]. The entropy-adaptive weights are computed in log-space for numerical stability: \[ \log w_t^{(i)} = -\alpha\,\mathrm{cumsum}(H_{1:t}^{(i)}). \] The loss is normalised by the weighted valid-token count \(\sum_{i,t}\tilde{w}_t^{(i)}m_t^{(i)}\), rather than by a raw token count, so selective attenuation changes both numerator and denominator consistently [2606.05434]. ## 5. Empirical behavior on GSM8K The empirical evaluation is confined to GSM8K, with 800 training examples and 500 evaluation examples, using greedy decoding and Pass@1 as the primary metric [2606.05434]. The reported zero-shot baselines are \(0.637 \pm 0.042\) for the 1.5B model and \(0.831 \pm 0.032\) for the 3B model [2606.05434]. For the 3B model, the comparison is: - **GRPO (\(\alpha=0\))**: final Pass@1 \(0.846 \pm 0.032\), peak \(0.852 \pm 0.031\), training variance \(0.0885\). - **AH-GRPO (\(\alpha=0.5\))**: final \(0.848 \pm 0.032\), peak \(0.862 \pm 0.030\), variance \(0.0630\). - **SA-AH-GRPO (\(\alpha=0.5\))**: peak \(0.858\) at step 30, final \(0.846\) at step 180, variance \(0.0246\), described as a 3.6 times reduction relative to GRPO while matching its peak accuracy [2606.05434]. For the 1.5B model, the comparison is: - **GRPO**: final \(0.660\), peak \(0.668\), variance \(0.0885\). - **AH-GRPO**: final \(0.650\), peak \(0.676\), variance \(0.0630\). - **SA-AH-GRPO**: peak and final \(0.686 \pm 0.041\) at step 180, improving over the zero-shot baseline of \(0.637\) [2606.05434]. The 3B results emphasise variance reduction and stability. The 1.5B results emphasise better final accuracy, although the paper notes that variance for SA-AH-GRPO is higher than GRPO and AH-GRPO at this smaller scale, indicating that the stability benefit is scale-dependent [2606.05434]. The AH-GRPO \(\alpha\)-ablation on the 1.5B model shows that entropy-amplifying \(\alpha=-0.25\) performs worst, with final Pass@1 \(0.634\), below the GRPO baseline. Positive \(\alpha\) values outperform negative \(\alpha\), and SA-AH-GRPO with \(\alpha=0.5\) exceeds all tested AH-GRPO variants, suggesting that advantage-sign selectivity matters in addition to the entropy discount itself [2606.05434]. Training logs reported in the paper show declining mean normalised entropy over time, so \(w_t^{(i)}\) tends toward 1 as the policy becomes more confident. The authors describe this as a self-annealing curriculum: the selective discount is strongest when uncertainty is high and naturally weakens later in training [2606.05434]. ## 6. Relation to neighboring methods, interpretation, and limitations SA-AH-GRPO belongs to a wider family of attempts to refine GRPO’s coarse sequence-level credit assignment. OAR redistributes a sequence-level advantage across tokens using outcome-grounded perturbation or gradient saliency while preserving total advantage mass [2601.07408]. Blockwise Advantage Estimation routes objective-specific advantages to structured text blocks and uses outcome-conditioned baselines for later blocks [2602.10231]. HPO and A-HPO reduce the weight of negative-advantage updates and replace per-response length normalization with mean-length normalization in sparse-reward regimes [2605.30201]. GRAIL reweights token-level advantages by gradient-activation saliency with respect to final-answer tokens [2606.04889]. Compared with these methods, SA-AH-GRPO keeps the rollout-level scalar advantage of GRPO intact and changes only the token-level weighting, using entropy and advantage sign rather than attribution or block structure. This suggests that SA-AH-GRPO is best understood as an asymmetric trust mechanism rather than a full fine-grained credit assignment method. It does not infer which specific token caused success or failure; rather, it discounts negative updates in uncertain regions and leaves positive updates untouched. In that respect it is simpler than token-attribution methods [2601.07408; 2606.04889] and more local than block-structured methods [2602.10231]. The paper’s limitations are explicit. Evaluation is restricted to two Qwen 2.5 models, one dataset, and a single random seed. There is no \(\alpha\)-ablation specific to SA-AH-GRPO, only to AH-GRPO. Entropy is approximated with top-\(K\) logits using \(K=500\), without a dedicated \(K\)-ablation. Some metric differences are within 95% confidence intervals, so the paper advises caution in interpretation [2606.05434]. A broader implication, though not established experimentally in the paper, is that SA-AH-GRPO offers a modular asymmetry that could be combined with other GRPO refinements. Because it leaves group-relative advantage computation, PPO clipping, and KL regularisation intact, it is plausibly composable with structured advantage routing [2602.10231], sign-asymmetric rollout weighting [2605.30201], or token-level attribution modules [2601.07408; 2606.04889]. The published evidence, however, is limited to the entropy-selective mechanism itself on GSM8K [2606.05434]. Source: https://www.emergentmind.com/topics/selective-advantage-ah-grpo-sa-ah-grpo