Scaf-GRPO: Hierarchical Scaffolding for RL

• The paper demonstrates that Scaf-GRPO effectively restores gradient signals in zero-reward plateau scenarios using hierarchical hint interventions.
• The methodology integrates on-policy scaffolding via incremental hint injection and layered outcome rewards to overcome learning cliffs in RL.
• Empirical results show significant performance gains in complex reasoning and agentic code repair, with improved success rates and stable policy gradients.

Scaffolded Group Relative Policy Optimization (Scaf-GRPO) is a training methodology developed to address the failure modes of Group Relative Policy Optimization (GRPO) in reinforcement learning settings with sparse or weak supervision, particularly for enhancing complex reasoning in LLMs and agentic code repair systems. The primary innovation of Scaf-GRPO is the introduction of minimal, hierarchical scaffolding or signal reshaping, selectively applied when independent learning progress has plateaued, which systematically restores gradient signals for difficult problem instances while preserving the theoretical and on-policy characteristics of GRPO (Zhang et al., 22 Oct 2025, Li et al., 8 May 2026).

1. Motivation and Problem Context

Reinforcement learning from verifiable rewards (RLVR) has demonstrated utility in elevating autonomous reasoning in LLMs, but standard algorithms such as GRPO experience a "learning cliff" on tasks far beyond the model's current capability. When faced with these "true-hard" cases, all generated trajectories for a prompt receive zero reward, collapsing the group-normalized advantage $$\hat{A}_i = (R(o_i) - \mu_G) / (\sigma_G + \epsilon_{std})$$ to zero across the batch, stalling learning since the policy gradient vanishes. Vanilla GRPO relies on intra-group variance in rewards to drive policy improvement, and the absence of any successful trajectory precludes gradient updates, rendering high-difficulty problems invisible to the optimizer (Zhang et al., 22 Oct 2025).

In code-agent RL with weak or surface-level feedback signals—such as compilation correctness without semantic verification—the standard GRPO group comparison is undermined by noise and insufficient credit assignment, further exacerbating the difficulty of robust policy optimization (Li et al., 8 May 2026).

2. Scaf-GRPO Algorithmic Framework

Scaf-GRPO operationalizes scaffolding and signal reshaping in a manner that minimally interferes with on-policy exploration and preserves the GRPO loss structure. The approach is characterized by the following phases and interventions:

1. Guidance Exemption Phase: For an initial portion of training (e.g., the first 15%), no scaffolding is provided; learning progresses autonomously with vanilla GRPO. Problems exhibiting improved reward rates are considered within reach, while those plateauing with persistently zero reward are deemed "true-hard."
2. Hierarchical Hint-Guided Exploration: Upon detection of stagnation (observed all-zero rewards post-exemption), the algorithm deterministically injects in-prompt hints drawn from a hierarchically structured hint set, ordered as knowledge, planning, and solution categories. Each category contains multiple granularity levels (typically four), forming a scaffolding ladder from abstract to concrete guidance (Zhang et al., 22 Oct 2025).
3. Progressive Replacement: For every prompt in a batch where the cliff is detected, hints are injected incrementally until a sampled rollout achieves a non-zero reward. That guided trajectory then replaces a randomly chosen failure within the group, restoring batch-level variance and enabling the advantage $\hat{A}_i$ to be nonzero for at least one trajectory, which in turn reactivates policy gradient updates.
4. On-Policy Integrity: All guided rollouts are sampled from the policy $\pi_\theta$ (with augmented inputs), ensuring that probability ratios and surrogate objectives remain on-policy, thus avoiding off-policy correction variance. When no cliff is present, Scaf-GRPO is provably identical to vanilla GRPO.

A similar principle underlies signal-reshaped GRPO for agentic code repair, where signal injection is operationalized as layered outcome rewards (compilation and semantic scores), step-level process credit scoring, and rollout governance to enforce intra-group comparability as detailed below (Li et al., 8 May 2026).

3. Signal Reshaping in Weak-Feedback Settings

In agentic code repair and similar tasks characterized by weak signals, Scaf-GRPO incorporates three key reshaping layers:

• Layered Outcome Rewards: Terminal reward $R_{\mathrm{layered}}(\tau)$ is defined as the sum of compile and semantic (functional equivalence) verifications, each contributing $0.5$, yielding $R_{\mathrm{layered}}(\tau) \in \{0, 0.5, 1\}$. This preserves ordinal distinctions between failure, surface-level success, and full semantic correctness.
• Step-Level Process Scores: Key actions (e.g., tool calls) within a trajectory are assigned process scores $s_i\in[0,1]$ by an LLM judge, evaluating "directional correctness and information gain." The trajectory mean $\bar{s}$ is computed, and step weights $\alpha_i$ scale token-level contributions to the loss. Positive-advantage trajectories emphasize high-signal steps, while negative-advantage cases receive complementary normalization.
• Rollout Governance: Each trajectory is categorized by "exit cause" (e.g., system errors, catastrophic repetition, normal finish), dictating masking strategies in the loss to filter out non-comparable or degenerate runs. Only normal or semi-normal rollouts contribute to group statistics with appropriately assigned rewards and weights.

These interventions collectively maintain meaningful intra-group comparisons under GRPO, localize learning signals to effective sub-trajectories, and stabilize the group-wise advantage signal (Li et al., 8 May 2026).

4. Theoretical Properties

Scaf-GRPO preserves the original GRPO clipped surrogate objective:

$$J_{\mathrm{Scaf}\text{-}\mathrm{GRPO}}(\theta) = \mathbb{E}_{i,t}\left[\min(r_{i,t} \hat{A}_i,\, \mathrm{clip}(r_{i,t},1-\epsilon,1+\epsilon)\hat{A}_i)\right]$$

with probability ratios $\hat{A}_i$0 computed in standard on-policy fashion. During guidance, the only change is the input context augmentation and trajectory replacement within-group. There is no alteration of the fundamental policy gradient form.

Importantly, Scaf-GRPO guarantees a nonzero learning gradient on "cliff" batches whenever any guided rollout achieves task success, without introducing high-variance corrections or off-policy sampling artifacts. No formal convergence guarantees are provided, but empirical stability and learning resumption are observed across benchmarks (Zhang et al., 22 Oct 2025, Li et al., 8 May 2026).

5. Empirical Results

Extensive experimentation validates the effectiveness and generality of Scaf-GRPO in both LLM mathematical reasoning and weak-feedback agentic code repair:

• Mathematical Reasoning (Qwen2.5-Math-7B): Pass@1 on the AIME24 benchmark improves from 30.0% (vanilla GRPO) to 43.3% (+44.3% relative). Aggregate performance gains are observed across all targeted benchmarks, model sizes, and architectures, with a typical absolute improvement of 4–6 points over equivalent baselines. Ablation studies confirm the additive value of the full scaffolded hierarchy, with reductions to partial or no scaffolding yielding lower gains (Zhang et al., 22 Oct 2025).
• Agentic Code Repair: Strict compile-and-semantic accuracy improves from 0.385 (zero-shot base) to 0.480 (GRPO-early), and further to 0.535 with the full layered outcome/process-score/rollout-governance construction—a net increase of 15 percentage points. Additional gains are noted in compile rates, single-edit success, and reduced average evaluation steps. Distillation baselines underperform relative to the GRPO-based schemes, indicating the criticality of outcome/process/rollout components for stable optimization (Li et al., 8 May 2026).
• Generalization: Scaf-GRPO enhances OOD task performance (e.g., GPQA-Diamond) and maintains its advantage across in-domain and out-of-domain settings.

6. Limitations and Future Directions

Scaf-GRPO requires pre-constructed, high-quality hierarchical hints or externally valid signal judges for effective intervention, leading to potential preprocessing overhead and domain specificity. The framework is most naturally applicable to structured, verifiable-reward domains (e.g., mathematics, programming). Extension to open-ended or creative domains is nontrivial.

Directions for future work include automated hint extraction through self-supervised or reward model loops, adaptive scheduling of hint specificity, extension to new domains (logical reasoning, program synthesis), and integration of denser token-level reward structures for improved credit assignment and policy refinement (Zhang et al., 22 Oct 2025).

7. Summary Table: Scaf-GRPO Key Elements and Outcomes

Component	Description	Empirical Effect (reported)
Hierarchical Hint Scaffolding	Tiered in-prompt hints: knowledge → planning → solution	Restores gradients on hard problems
Layered Outcome Rewards	Scalar reward: 0 (fail), 0.5 (compile), 1 (semantic equivalence)	Preserves ranking among trajectories
Step-Level Process Scores	Per-tool-step LLM judgment, trajectory mean normalization	Localizes credit assignment
Rollout Masking/Governance	Exclude or restrict loss contribution for abnormal/failed rollouts	Filters execution noise
Policy Gradient Update	Standard GRPO; unaltered loss form, on-policy throughout	Maintains stability, avoids variance
Aggregate Performance (Qwen2.5-Math-7B)	pass@1: 45.2% (GRPO) → 50.9% (Scaf-GRPO); AIME24: 30.0% → 43.3% (+44.3% rel)	Consistent multi-task improvements

Scaf-GRPO stands as a robust methodology for extending the effective learning frontier of reinforcement learning agents in structured, verifiable domains by integrating minimal, on-policy scaffolding, and outcome/procedure-aware signal reshaping (Zhang et al., 22 Oct 2025, Li et al., 8 May 2026).