Self-Hint Aligned GRPO (SAGE)

• The paper introduces SAGE, a framework that injects compact privileged hints to overcome advantage collapse in sparse-reward settings.
• SAGE employs online self-hinting and adaptive curriculum scheduling to maintain effective gradient signals and enhance policy learning.
• Empirical results demonstrate SAGE's gains of 2–6 points in accuracy and improved exploration compared to baseline GRPO methods.

Self-Hint Aligned GRPO with Privileged Supervision (SAGE) is an on-policy @@@@1@@@@ (RL) framework developed to address the challenge of aligning LLMs using group-based sparse-reward protocols. SAGE modifies the Group Relative Policy Optimization (GRPO) procedure by injecting privileged, compact hints—such as plans or solution decompositions—during training. These hints reshape the model’s rollout distribution without affecting the verifier reward, overcoming critical bottlenecks in RL from sparse rewards. During inference, the model operates without hints, reflecting only the improved capabilities learned during privileged supervision (Liao et al., 3 Feb 2026).

1. Theoretical Foundation and Training Objective

GRPO is a group-based RL protocol where a policy $\pi_\theta$ generates rollouts $\tau=(y_1, \ldots, y_T)$ given a prompt $x \sim D$, receiving a binary terminal reward $R(x, \tau) \in \{0, 1\}$. Within a group of $G$ rollouts $\{\tau_i\}_{i=1}^G$, normalized advantages are computed as $A_i = (R_i - \bar R)/(\sigma + \epsilon)$, with $\bar R = \frac{1}{G} \sum_i R_i$ and $\sigma^2 = \frac{1}{G} \sum_i (R_i - \bar R)^2$. The GRPO gradient loss is:

$$L_{\text{GRPO}}(\theta) = -\mathbb{E}_{x, \tau_i} \left[ \frac{1}{G} \sum_{i=1}^G A_i \sum_{t=1}^{|\tau_i|} \nabla_\theta \log \pi_\theta(y_{i,t}|x, y_{i,<t}) \right]$$

GRPO often suffers from advantage collapse in the sparse-reward regime ($p_0(x) \ll 1$), where all rollouts within a group receive zero reward, resulting in $\sigma = 0$ and vanishing gradients. SAGE introduces privileged hints $h$ obtained by compressing reference solutions $\tau^*$ into compact forms via a sampled hint-strength variable $\ell$; $h \sim q(h \mid x, \tau^*, \ell)$.

For each prompt $x$, SAGE’s workflow:

• Sample hint level $\ell \sim p(\ell | x, \theta)$
• Sample hint $h \sim q(h | x, \tau^*, \ell)$
• Generate group rollouts $\tau_i \sim \pi_\theta(\cdot | x, h)$, compute rewards $R_i = R(x, \tau_i)$
• Compute advantages $A_i$
• Aggregate policy gradient and optional KL penalty against reference policy $\pi_{\text{ref}}$

The SAGE loss:

$$L_{\text{SAGE}}(\theta) = \mathbb{E}_{x \sim D, \ell, h, \tau_i}\left[ L_{\text{PG}} + \beta L_{\text{KL}} \right]$$

where $L_{\text{PG}}$ is the group advantage-weighted policy gradient and $L_{\text{KL}}$ is the reference KL penalty.

2. Advantage Preservation through Hint Injection

Under Bernoulli rewards, group variance is $\sigma^2 = \bar R (1 - \bar R)$, collapsing when all rewards are identical. The gate-opening probability that group variance is nonzero is $u(p) = 1 - (1 - p)^G - p^G$, maximized at $p=1/2$. SAGE raises the conditional success rate $p(x, h) = \Pr_\tau[R(x, \tau) = 1 | x, h]$ by injecting hints, avoiding $Gp<1$ degeneracy and preserving gradient signals.

Jensen’s inequality shows that, at fixed mean $p$, mixing across many hint distributions dilutes gate-opening probability. SAGE allocates compute to a single well-calibrated hint per prompt, optimizing non-degenerate group formation and learning efficiency.

3. On-Policy SAGE Training Workflow

The SAGE training loop operates as follows:

• Input: Dataset $(x, \tau^*)$, policy $\pi_\theta$, reference $\pi_{\text{ref}}$, group size $G$, hint level cap $L$, KL weight $\beta$, stabilizer $\epsilon$
• Hint Generation: $q(h|x, \tau^*, \ell)$ via $\pi_\theta$ prompting
• Scheduling: per-prompt policy $\ell(x)$, either by epoch-level SAGE-LIGHT or group-degeneracy SAGE full
• Loop:
  • For each batch $(x_b, \tau^*_b)$, select $\ell(x_b)$, sample $h \sim q(h|x_b, \tau^*_b, \ell)$
  • Optionally probe with size $G_0$; if all $R=0$, increment $\ell(x_b)$
  • Roll out $G$ trajectories, aggregate rewards, compute advantages
  • Compute and apply policy gradient and KL penalty
  • Update parameters via gradient descent

SAGE-LIGHT relies on epoch-wide empirical success for hint scheduling, while SAGE monitors per-prompt collapse directly.

4. Adaptive Curriculum and Online Self-Hinting

Self-hinting—the generation of hints by a lagged copy of $\pi_\theta$—establishes an adaptive curriculum, ensuring hints remain matched to the learner’s proficiency. Fixed hint generators (initial policy, external LLM) often produce hints that become suboptimal as $\theta$ evolves, either too easy (no gradient) or too hard (collapses). Online self-hinting maintains $p(x, h)$ near optimal, maximizing update frequency and learning traction.

Empirical ablations demonstrate that online self-hinting is superior to both offline self-hinting and external-teacher hinting (e.g., GPT-5 scaffold), with a 1–2 point accuracy benefit, even when sampling multiple offline hints.

5. Deployment Protocol and Distributional Generalization

During evaluation, SAGE sets $\ell = 0$, enforcing $h = \varnothing$. The model executes its policy $\pi_\theta(\cdot | x)$ without any privileged information, and since the reward function never depended on hints, test-time accuracy directly reflects reasoning improvements acquired during privileged supervision.

6. Empirical Results and Learning Dynamics

SAGE’s performance has been validated across six in-distribution (AIME24, AIME25, AMC23, MATH-500, Minerva-Math, Olympiad) and two out-of-distribution benchmarks (GPQA-diamond, MMLU-Pro). Three backbone LLMs (Llama-3.2-3B-Instruct, Qwen2.5-7B-Instruct, Qwen3-4B-Instruct) serve as baselines, compared against SFT, vanilla GRPO, LUFFY, and Scaf-GRPO.

Model	SAGE Gain vs GRPO	SAGE Gain vs Best Baseline	Fraction Never Correct (GRPO→SAGE)
Llama-3.2-3B-Instruct	+6.1 pts	+2.0 pts	40.2% → 30.0%
Qwen2.5-7B-Instruct	+4.5 pts	n/a	n/a
Qwen3-4B-Instruct	+4.2 pts	n/a	n/a

SAGE achieves the highest average accuracy in both ID and OOD tests. SAGE-LIGHT is approximately 1 point less than full SAGE but only 1.2× slower than GRPO, offering compute efficiency.

Training reward curves show SAGE achieves continuous improvement while GRPO plateaus. SAGE sustains moderate $\pi_\theta$ entropy (on-policy exploration), in contrast to collapsed exploration in Scaf-GRPO and off-policy oscillation in LUFFY. SAGE also increases response length during training, correlating with accuracy enhancement.

–––––––––––––––––––––––––––––––––––––

In summary, Self-Hint Aligned GRPO with Privileged Supervision (SAGE) leverages online self-hint injection and adaptive curriculum scheduling to address advantage collapse in group-based RL. SAGE maintains on-policy update integrity, dynamically calibrates hint difficulty, and delivers empirical gains of 2–6 points in accuracy over strong baselines across multiple math and general knowledge benchmarks for diverse LLM backbones (Liao et al., 3 Feb 2026).