Off-Context GRPO: Learning to Reason on Hard Problems using Privileged Information

Abstract: Reinforcement learning with verifiable rewards (RLVR) improves reasoning in LLMs. Yet, typical RLVR approaches fail on difficult problems: when a model cannot generate any correct solutions, it receives \textit{zero} learning signal. Providing privileged guidance during training, such as solution prefixes, can help overcome this learning cliff by steering the model towards {correct solutions with non-zero reward}. {We call these rollouts \textit{off-context}: they are generated from a training prompt that contains privileged guidance, while the target objective is defined by the original prompt without that guidance.} {We introduce} Off-Context GRPO (OC-GRPO), a minimally modified variant of GRPO that uses guided rollouts but applies an importance-corrected objective to steer the update back toward the original unguided objective, avoiding the mismatch that destabilizes uncorrected guided training. Empirically, our algorithm achieves a 3.9\% absolute improvement (13.8\% relative gain) over vanilla GRPO on average across standard mathematical reasoning benchmarks with negligible additional cost.

• The paper introduces Off-Context GRPO, which uses per-token importance weighting to align guided rollouts with the deployment objective and achieve an unbiased policy-gradient estimate.
• The method controls variance by using the shortest guidance that escapes the learning cliff, while discounting guidance-dependent successes and amplifying failures that persist without robust reasoning.
• OC-GRPO-Fixed reaches 31.7 average Pass@1 on mathematical reasoning benchmarks, delivering a 13.8% relative gain over vanilla GRPO and stronger improvements at 3B and 1.5B model scales.

The learning cliff and privileged guidance

Reinforcement learning with verifiable rewards (RLVR) improves LLM reasoning by optimizing verifier-checked outcomes, and Group Relative Policy Optimization (GRPO) is its dominant workhorse. GRPO's gradient signal comes entirely from within-group reward variance: when every rollout for a problem fails, rewards are uniformly zero, advantages vanish, and the gradient is identically zero regardless of training duration. The paper terms this failure mode the learning cliff, and it is pervasive on hard problems — on MATH Levels 3–5, 595 problems yield zero successes across 64 rollouts from Qwen2.5-7B-Instruct.

A growing family of methods escapes the cliff by injecting privileged guidance — solution prefixes, hints, or output-space prefixes available only at training time — into the rollout prompt so that correct continuations become reachable. The paper identifies a shared flaw in these methods, which it calls the off-context problem: rollouts are sampled under a guided prompt $g(x)$ that never appears at deployment, yet gradients are computed as if sampling and evaluation distributions matched. Concretely, guided-target methods such as POPE, BREAD, PrefixRL, and scaffolded-hint approaches optimize $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$ rather than the deployment objective $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$. The paper's toy example makes the gap sharp: with a "shortcut" mode that succeeds only under $g(x)$ and a "robust" mode succeeding with probability $\beta$ under both prompts, $\arg\max J = 0$ while $\arg\max J^{\mathrm{guide}} = 1$ — the two objectives prefer opposite policies. The practical transfer of guided-target updates to $J$ rests on the empirical, unguaranteed hypothesis of back-generalization.

The OC-GRPO correction

Off-Context GRPO (OC-GRPO) is a minimal modification to GRPO: guided rollouts are retained, but per-token advantages are reweighted by an importance ratio between the unguided target context and the guided behavior context,

$$\rho^{\mathrm{oc}}_{i,t}(\theta) = \frac{\pi_\theta(y_{i,t} \mid x, y_{i,<t})}{\pi_{\theta_{\mathrm{old}}}(y_{i,t} \mid g(x), y_{i,<t})},$$

substituted into the standard clipped PPO-style surrogate. The response-level estimator is provably unbiased for $\nabla_\theta J(\theta)$ under a support condition; when $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$0, the ratio reduces to vanilla GRPO. Notably, masking guidance tokens from the loss does not fix the problem — it changes which tokens receive gradient but not what they condition on, so the update still targets $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$1. The paper demonstrates this empirically: without the importance correction, repeated off-context mismatch inflates gradient norms and produces reward collapse during training.

A key variance result motivates design choices: decomposing the ratio into an on-policy drift factor (controlled by clipping) and an off-context correction factor $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$2, the paper shows under a per-guidance-token regularity assumption that $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$3 is bounded in $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$4 with Rényi-2 variance at most $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$5, where $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$6 is the guidance length. Variance therefore scales with the length of the hint, not the rollout — yielding the principle of using the shortest guidance that breaks the cliff.

Behavior-aware credit assignment

Beyond unbiasedness, the correction induces a credit-assignment mechanism formalized at the response level. With aggregate correction factors $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$7 on successful guided rollouts and $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$8 on failures (where $$J^{\mathrm{guide}}(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid g(x))}[r(x,y)]$$9 and $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$0 are expected verifier rewards under the original and guided prompts), the gradient decomposes into a success branch scaled by $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$1 and a failure branch scaled by $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$2. Three consequences follow:

• Guided successes are discounted in proportion to how much more likely they are under the guidance than under the model's own distribution — a success inherited almost verbatim from a full-solution prefix receives heavily damped credit.
• Failures that persist despite guidance receive amplified penalties, so even on hard problems with $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$3, a nonzero failure-side update exists whenever $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$4.
• No reward shaping is required: the behavior-awareness emerges from the importance ratio itself, since $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$5 is conditioned only on $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$6, keeping training and deployment objectives exactly aligned.

The framework is mechanism-agnostic — any privileged signal works, because what matters is the context mismatch rather than its form.

Empirical results

Training uses MATH Level 3–5 problems unsolvable in 64 base-model rollouts, cascaded solution-prefix guidance at fractions $$J(\theta) = \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)}[r(x,y)]$$7 of the reference solution, LoRA fine-tuning via veRL, and evaluation on AIME (1983–2026), Gaokao2023, and OmniMath. Two variants are studied: OC-GRPO-Fixed selects the minimum guidance level once with the base model before training, and OC-GRPO-Adaptive re-selects guidance each step against the current policy.

On Qwen2.5-7B-Instruct (3 seeds), OC-GRPO-Fixed achieves the best average Pass@1 of 31.7, a 13.8% relative gain over vanilla GRPO and 1.4 absolute points ahead of the strongest guided baseline POPE* (+8.7%); it leads on Gaokao2023 (+17.3%) and OmniMath (+10.8%). The adaptive variant is competitive (+10.7%) but incurs per-step inference cost, so the fixed variant is recommended as default.

Method	Avg Pass@1	Rel. vs GRPO
Vanilla GRPO	27.8	ref.
PrefixRL*	28.4	+1.9%
POPE*	30.3	+8.7%
OC-GRPO-Fixed	31.7	+13.8%
BREAD*	29.5	+6.1%
OC-GRPO-Adaptive	30.8	+10.7%

At smaller scales the contrast sharpens. At 3B, guided baselines degrade below vanilla GRPO (BREAD* −3.4%, PrefixRL* −0.3%) while OC-GRPO-Fixed gains +7.2%; at 1.5B, PrefixRL* drops 7.2% while OC-GRPO retains +10.2%. This supports the claim that back-generalization is capacity-dependent: larger models absorb the objective mismatch, smaller models cannot, making the importance correction most valuable precisely where learning cliffs bind hardest. Hint-family ablations (self-generated hints, frontier hints, self-correction) show smaller but mostly positive gains, confirming mechanism-agnosticism, though fixed hints slightly underperform vanilla GRPO on some benchmarks.

Limitations and open questions

The authors state their limitations plainly. Evaluation is confined to academic scale (Qwen2.5 up to 7B with LoRA adapters); frontier-scale behavior may differ. Results cover mathematical reasoning with verifiable answers, and transfer to code generation, agentic tool use, or multimodal RLVR is untested. Main results rely on ground-truth solution prefixes as the privileged signal; behavior under partial, noisy, or adversarial guidance remains open. The credit-assignment analysis assumes a support condition and a centered advantage without group normalization, and the variance bound depends on a per-guidance-token regularity assumption that could fail for guidance that radically reshapes next-token distributions. Whether adaptive guidance selection can be made cheap enough to dominate the fixed variant is unresolved, as is the extension to multi-turn settings where privileged context takes the form of tool results or oracle subgoals.

Conclusion

This paper reframes privileged-guidance RLVR as off-context sampling, shows that existing guided-target methods optimize a misaligned objective whose benefit depends on an unguaranteed back-generalization hypothesis, and corrects the mismatch with a per-token importance reweighting that is provably unbiased, has variance scaling with guidance length, and doubles as behavior-aware credit assignment. Empirically, the correction yields consistent gains over guided baselines across scales, with the margin widening exactly where the objective mismatch is hardest to absorb.