Adaptive-Horizon GRPO (AH-GRPO) in Reinforcement Learning

• The paper introduces AH-GRPO, which applies cumulative entropy-based discounting to mitigate gradient variance and stabilize training.
• AH-GRPO adapts the effective optimization horizon by down-weighting tokens following high-entropy prefixes, ensuring more reliable credit assignment.
• Empirical evaluations on GSM8K demonstrate that AH-GRPO and its variant SA-AH-GRPO reduce training variance by up to 1.4× while preserving competitive accuracy.

Adaptive-Horizon GRPO (AH-GRPO) is a variant of Group Relative Policy Optimisation (GRPO) for reinforcement learning of LLMs in which each token’s policy-gradient contribution is weighted by a cumulative entropy-based discount. In the formulation introduced for reasoning-task alignment, the effective optimization horizon contracts when the model is uncertain, so later tokens following high-entropy prefixes receive reduced weight; with $\alpha=0$, the method recovers standard GRPO exactly (Chawla et al., 3 Jun 2026). A related 2026 line of work uses an “adaptive horizon” idea in a different sense—tree-local rollout depth in dialogue optimization under Adaptive Tree-based GRPO (AT-GRPO)—which makes the term easy to conflate across papers despite the mechanisms being distinct (Peng et al., 9 Feb 2026).

1. Definition and scope

AH-GRPO was proposed as one of two complementary extensions to GRPO, alongside Selective-Advantage AH-GRPO (SA-AH-GRPO), in order to address a symmetry built into standard GRPO: conventional GRPO treats every token position and every sampled rollout symmetrically (Chawla et al., 3 Jun 2026). AH-GRPO breaks this symmetry by introducing token-level weights derived from predictive entropy. The central idea is that high-entropy positions correspond to exploratory or ambiguous token choices, so any single sampled token at those positions is a noisy estimate of the gradient. Uniform weighting therefore forces unreliable contributions to count equally, which the report identifies as a source of inflated gradient variance and unstable learning.

Under this view, “adaptive horizon” refers to the length of the useful gradient pathway rather than the generated sequence length itself. When the model is uncertain early in a completion, the cumulative discount suppresses the contribution of downstream tokens, effectively shortening the gradient horizon under uncertainty. When the model is confident, the cumulative weights remain closer to $1$, and the optimization behaves more like standard GRPO. The proposal is framed for reinforcement learning with verifiable rewards on structured generation tasks, and its reported experiments focus on GSM8K with Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct fine-tuned with LoRA (Chawla et al., 3 Jun 2026).

2. Formal entropy-adaptive weighting

The formalization begins at the token level. For token position $t$ in rollout $i$, the model’s predictive entropy over the next-token distribution is normalized by $\log V$, where $V$ is the vocabulary size:

$$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$

Here $H(p)=-\sum_v p(v)\log p(v)$ is measured in nats; the report states that in practice one estimates $H_t^{(i)}$ over the top-$K$ logits, for example $1$0, for efficiency (Chawla et al., 3 Jun 2026).

Given discount strength $1$1, AH-GRPO defines a per-step entropy-adaptive discount

$1$2

and a cumulative horizon weight up to position $1$3,

$1$4

This construction yields three regimes. If $1$5, then $1$6 for all tokens and the method reduces to GRPO. If $1$7, tokens following high-entropy prefixes are down-weighted, which shortens the effective horizon where the model is most uncertain. If $1$8, the method enters an entropy-amplifying regime in which high-entropy positions are up-weighted.

The weighting is cumulative rather than local. That detail is important: the method does not merely suppress an individual uncertain token; it discounts the gradient contribution of all later tokens that depend on an uncertain prefix. This suggests that AH-GRPO is designed as a bias–variance intervention on sequence-level credit assignment rather than as a local confidence heuristic.

3. Objective, advantage normalization, and training procedure

AH-GRPO preserves the group-relative structure of GRPO. Given $1$9 sampled completions $t$0, scalar rewards $t$1 are converted into group-normalized advantages

$t$2

For each token $t$3 in completion $t$4, the clipped surrogate is

$t$5

where

$t$6

and $t$7 masks valid token positions (Chawla et al., 3 Jun 2026).

The AH-GRPO loss is the AH-weighted analogue of the GRPO objective:

$t$8

An equivalent unclipped form given in the report is

$t$9

The one-step training procedure is correspondingly direct. Prompts $i$0 are sampled; for each prompt, $i$1 continuations are drawn from $i$2; scalar rewards and normalized advantages are computed; for each rollout and token position, token-prediction logits are obtained, normalized entropy is estimated over top-$i$3, $i$4 and $i$5 are accumulated, and the PPO ratio and clipped surrogate are formed; finally, the AH-weighted loss is evaluated and a gradient step is taken (Chawla et al., 3 Jun 2026).

Relative to standard GRPO, the mathematical distinction is minimal but consequential. Standard GRPO sets $i$6, so every valid token contributes equally. AH-GRPO retains the same clipped-advantage machinery, KL regularization, and rollout grouping, but injects a tokenwise multiplicative weight determined by cumulative uncertainty. This makes the method an architectural modification of the policy-gradient estimator rather than a replacement of GRPO’s basic optimization scaffold.

4. Empirical results on GSM8K

The reported empirical study evaluates standard GRPO with $i$7, AH-GRPO with $i$8, and SA-AH-GRPO with $i$9 on GSM8K using Qwen 2.5-1.5B-Instruct and Qwen 2.5-3B-Instruct fine-tuned with LoRA (Chawla et al., 3 Jun 2026).

For AH-GRPO on Qwen 2.5-3B over 150 steps, Pass@1 is reported as $\log V$0 at step $\log V$1, $\log V$2 at step $\log V$3 as the peak, then fluctuating between $\log V$4 and $\log V$5 from steps $\log V$6 to $\log V$7, with final Pass@1 $\log V$8 $\log V$9. Training variance, defined as mean reward standard deviation over the last half, is $V$0. The report compares this to GRPO with final $V$1 and variance $V$2. For Qwen 2.5-1.5B over 150 steps, AH-GRPO reaches $V$3 at step $V$4, $V$5 at step $V$6 as the peak, fluctuates between $V$7 and $V$8 from steps $V$9 to $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$0, and ends at final Pass@1 $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$1 $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$2, again with training variance $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$3, compared with GRPO final $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$4 and variance $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$5.

The report summarizes these results as approximately $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$6 training-variance reduction at both scales relative to GRPO, indicating more stable gradient updates under entropy-based token weighting. The abstract adds that, on the 3B model, SA-AH-GRPO achieves Pass@1 $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$7 at its peak at step $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$8 and maintains $$H_t^{(i)} \;=\; \frac{H\bigl(\pi_\theta(\,\cdot\,\mid q, o_{i,<t})\bigr)}{\log V} \;\in [0,1].$$9 at $H(p)=-\sum_v p(v)\log p(v)$0 steps, with training variance reduced to $H(p)=-\sum_v p(v)\log p(v)$1, a $H(p)=-\sum_v p(v)\log p(v)$2 times reduction relative to GRPO while matching its peak accuracy. On the 1.5B model, SA-AH-GRPO achieves a peak Pass@1 of $H(p)=-\sum_v p(v)\log p(v)$3, improving over the zero-shot baseline of $H(p)=-\sum_v p(v)\log p(v)$4 (Chawla et al., 3 Jun 2026).

Setting	Accuracy result	Variance result
AH-GRPO, Qwen 2.5-3B	Peak Pass@1 $H(p)=-\sum_v p(v)\log p(v)$5 at step $H(p)=-\sum_v p(v)\log p(v)$6; final $H(p)=-\sum_v p(v)\log p(v)$7	$H(p)=-\sum_v p(v)\log p(v)$8
AH-GRPO, Qwen 2.5-1.5B	Peak Pass@1 $H(p)=-\sum_v p(v)\log p(v)$9 at step $H_t^{(i)}$0; final $H_t^{(i)}$1	$H_t^{(i)}$2
SA-AH-GRPO, Qwen 2.5-3B	Peak Pass@1 $H_t^{(i)}$3 at step $H_t^{(i)}$4; $H_t^{(i)}$5 at $H_t^{(i)}$6 steps	$H_t^{(i)}$7

The empirical picture is therefore nuanced rather than uniformly monotone. AH-GRPO improves stability clearly, but its final accuracy relative to GRPO depends on scale and evaluation point. SA-AH-GRPO is reported to preserve the stability gain while better maintaining peak accuracy, which the paper links to its asymmetric handling of successful and unsuccessful rollouts.

5. Ablations, asymmetric discounting, and implementation guidance

The ablation study on the 1.5B model over 150 steps examines $H_t^{(i)}$8 and $H_t^{(i)}$9 (Chawla et al., 3 Jun 2026). The reported results are: $K$0 gives peak $K$1 and final $K$2; $K$3 gives peak $K$4 and final $K$5; $K$6 gives peak $K$7 and final $K$8; $K$9 gives peak $1$00 and final $1$01; and $1$02 gives peak $1$03 and final $1$04. The report interprets this as showing that positive $1$05 values, corresponding to entropy discounting, consistently outperform negative $1$06, confirming that down-weighting high-entropy tokens stabilizes learning. It also states that small $1$07 such as $1$08 gives the highest early peak, whereas larger values at or above $1$09 yield more consistent final gains.

The same report identifies a limitation of symmetric entropy discounting. Because AH-GRPO discounts high-entropy positions in all rollouts, it can inadvertently weaken gradients on correct trajectories, which it refers to as “entropy collapse.” SA-AH-GRPO addresses this by applying the discount only to negative-advantage rollouts, thereby preserving full credit on successful samples. The abstract characterizes this as asymmetric discounting that preserves the full gradient signal on correct solutions, prevents entropy collapse, and substantially stabilises training (Chawla et al., 3 Jun 2026).

The implementation recommendations are explicit. For larger-scale stability, particularly at 3B+, $1$10 is reported as effective. For smaller models or faster early gains, $1$11–$1$12 can be tried. Entropy estimation should use top-$1$13 logits, such as $1$14, to balance accuracy and compute. Integration into existing RLHF or RLVR pipelines consists of augmenting the policy-gradient or PPO surrogate with per-token weight $1$15, computing $1$16 online during the forward pass and accumulating $1$17, multiplying the usual log-probability or clipped advantage term by $1$18, and leaving standard KL-penalty and clipping hyperparameters unchanged. The report also recommends monitoring mean normalized entropy $1$19 and mean weight $1$20; as $1$21 decays, $1$22 anneals toward $1$23, yielding what it describes as an implicit curriculum (Chawla et al., 3 Jun 2026).

6. Relation to Adaptive Tree-based GRPO and broader adaptive-horizon formulations

A potential source of confusion is the coexistence of AH-GRPO and AT-GRPO in contemporary GRPO literature. In the dialogue-optimization framework “Dialogue Model Optimization via Agent Game and Adaptive Tree-based GRPO,” the adaptive mechanism is not token-level entropy discounting but an adaptive observation range over a rollout tree (Peng et al., 9 Feb 2026). There, a two-agent game couples a dialogue agent $1$24 with a user agent $1$25, the immediate reward is defined as $1$26 from a turn-level termination probability, and the rollout is reinterpreted as a rooted tree rather than a single chain.

AT-GRPO expands only a local subtree of depth

$1$27

beneath each node at depth $1$28, instead of fully expanding a tree of depth $1$29. The rationale given is stage-aware: early in the dialogue, topics must be explored broadly, so $1$30 should be large; later, immediate coherence dominates, so $1$31 should be small. Aggregated rewards are then computed from local leaves, and the resulting objective retains a clipped PPO-style surrogate with a KL penalty. The stated complexity benefit is a reduction from full-tree exponential cost $1$32 to a polynomial total budget $1$33 while preserving long-term reward capture (Peng et al., 9 Feb 2026).

The relationship between AH-GRPO and AT-GRPO is therefore conceptual rather than procedural. Both seek to mitigate short-horizon bias in GRPO-like training, but they do so at different granularities and for different task structures. AH-GRPO adapts horizon continuously at the token level as a function of model uncertainty during structured generation; AT-GRPO adapts horizon discretely at the node level as a function of dialogue stage in tree-structured rollouts. A plausible implication is that “adaptive horizon” has become a family resemblance term for methods that retain long-term credit assignment while pruning unreliable or computationally expensive parts of the trajectory.