---
title: 'Scaf-GRPO: Scaffolded RL for LLMs'
url: https://www.emergentmind.com/topics/scaf-grpo
type: topic
---

# Scaf-GRPO: Scaffolded RL for LLMs

Scaf-GRPO (Scaffolded Group Relative Policy Optimization) is a progressive reinforcement learning framework designed to address the shortcomings of sparse-reward policy optimization algorithms when training large language models (LLMs) on complex, verifiably-scored reasoning tasks. It systematically injects minimal, tiered in-prompt hints only when independent learning has plateaued, thereby restoring learning signal and enabling models to progress on problem instances previously beyond their autonomous reach [2510.19807].

## 1. Motivation: Overcoming the Learning Cliff in GRPO

The central motivation for Scaf-GRPO is the “learning cliff” phenomenon observed in Group Relative Policy Optimization (GRPO) applied to LLMs. In standard GRPO, for each query $q$, a group of $N$ model rollouts $\mathcal{G} = \{o_1, ..., o_N\}$ is produced, with each solution string $o_i$ receiving a sparse binary reward $R(o_i) \in \{0, 1\}$ from a verifier. The normalized advantage for each trajectory is then

$$
\hat{A}_i = \frac{R(o_i) - \mu_{\mathcal{G}}}{\sigma_{\mathcal{G}} + \varepsilon_{std}}
$$

where $\mu_{\mathcal{G}}$ and $\sigma_{\mathcal{G}}$ are the group mean and standard deviation of rewards. If all $R(o_i) = 0$ (i.e., every attempt fails on a hard problem), then $\mu_{\mathcal{G}} = \sigma_{\mathcal{G}} = 0$ and $\hat{A}_i = 0$ for all trajectories, causing the policy gradient to vanish. In this regime, dubbed the learning cliff, “true-hard” examples become invisible to the learning signal, stalling progress.

Off-policy prefix guidance methods (e.g., seeding the model with gold prefixes) avoid zero rewards but create unwanted distribution shifts, high-variance gradient corrections, and restrict the exploration essential for robust skill acquisition.

## 2. Formal Framework and Algorithm

Solution generation is formulated as a Markov Decision Process (MDP):

- **State**: $s_t = (q, o_{<t})$; the prompt and generated tokens so far
- **Action**: $a_t = o_t$; the next token
- **Policy**: $\pi_\theta(a_t|s_t)$; parameterized by the LLM
- **Reward**: Terminal $R(o) \in \{0,1\}$ from a verifier

Within-group normalization yields: 
- $\mu_{\mathcal{G}} = \frac{1}{N}\sum_{i} R(o_i)$
- $\sigma_{\mathcal{G}} = \sqrt{\frac{1}{N}\sum_{i}(R(o_i) - \mu_{\mathcal{G}})^2}$
- GRPO objective (with clipping): 
  $$
  J_{GRPO}(\theta) = \mathbb{E}_{i, t} \left[ \min\left( r_{i,t} \hat{A}_i, \text{clip}(r_{i,t}, 1-\varepsilon, 1+\varepsilon)\hat{A}_i \right) \right]
  $$
where $r_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|o_{i,<t},q)}{\pi_{\theta_{old}}(\ldots)}$.

## 3. The Scaffolded Training Process

Scaf-GRPO is implemented as a two-phase, on-policy extension of GRPO:

**Phase 1: Guidance Exemption**  
For the first 15% of training steps, zero intervention is allowed. This phase lets the model resolve queries with formatting or surface-level deficiencies—a regime dominated by “pseudo-hard” explorations. Zero-reward plateau statistics are monitored; plateauing in the batchwise count signals identification of “true-hard” queries.

**Phase 2: Hierarchical Hint-Guided Exploration**  
A three-tier hint hierarchy $H = \{H_{\text{knowledge}}, H_{\text{planning}}, H_{\text{solution}}\}$ is pre-generated per problem:
- Knowledge hints: key formula or core concept
- Planning hints: high-level solution outline
- Solution hints: concrete calculation step

Each tier is decomposed into four incrementally detailed steps (generated via an offline teacher model prompt). For each true-hard failure, the algorithm incrementally injects hints (starting from abstract, progressing to detailed) into the prompt. If a rollout with the augmented prompt achieves reward $R(o_{h^*}) > 0$, one random failed trajectory is swapped for $o_{h^*}$, yielding $\mathcal{G}_{final}$ with nonzero variance.

The same on-policy update is then performed:
$$
J_{Scaf-GRPO}(\theta) = \mathbb{E}\left[ \min(r'_{i,t}\hat{A}_i', \text{clip}(r'_{i,t}, 1-\varepsilon, 1+\varepsilon)\hat{A}_i') \right]
$$
Notably, all trajectories remain on-policy relative to $\pi_\theta$, as hints only augment the prompt.

**Algorithmic Outline:**

```python
for each training step t:
    sample batch of queries {q}
    for each q:
        roll out N trajectories 𝒢 under π_θ(·|q)
        if any R(o)>0 or t<T_exempt:
            𝒢_final ← 𝒢
        else:
            (h*, o_h*) ← SearchHierarchicalHints(q, π_θ)
            if o_h* found:
                replace a random fail in 𝒢 with o_h* → 𝒢_final
            else:
                𝒢_final ← 𝒢
    compute normalized advantages  on 𝒢_final
    update θ by maximizing clipped-surrogate on 𝒢_final
```

## 4. Key Hyperparameters and Implementation

Primary hyperparameters and settings include:
- Guidance exemption fraction $T_{exempt} = 15\%$ of total training steps
- Rollouts per query $N = 8$
- GRPO clip $\varepsilon = 0.2$; KL penalty $\beta = 0.0$ (favours exploration)
- Hint hierarchy: Three tiers, four levels per tier, incrementally injected
- Dataset filtering: From DeepScaleR-Preview (40K math problems), “too easy” dropped, “potentially solvable” subsampled, “too hard” retained
- Model families: Qwen2.5-Math-1.5B/7B, Qwen2.5-7B, Llama-3.2-3B, DeepSeek-R1-Distill-Qwen-1.5B LongCoT
- Training: 10 epochs, AdamW learning rate $1\times10^{-6}$, global batch size 256, PPO mini-batch 64, max generation length 2048 (8192 for LongCoT)

## 5. Effects and Mechanisms of Scaffolded Guidance

Minimal, progressive scaffolding enables the following properties:
- **Restoration of Reward Variance**: Inserting a single successful trajectory into the group disrupts reward degeneracy, restoring a usable learning signal even on true-hard queries.
- **On-Policy Integrity**: Since all augmented runs are sampled from the current policy with controlled prompts, there is no requirement for high-variance off-policy corrections.
- **Skill Internalization**: Forcing the model to succeed with the most abstract hint possible fosters true generalization, rather than rote memorization.
- **Unhampered Exploration**: Hints serve as “signposts” rather than “rails,” allowing the model to deviate and discover novel solution paths.

A plausible implication is that this approach can systematically elevate LLM performance ceilings on tasks that otherwise yield persistent zero rearwards under standard on-policy RL protocols.

## 6. Empirical Evaluation and Results

Performance was benchmarked on seven challenging math tasks (pass@1, greedy decoding), notably on the AIME24 benchmark and several out-of-distribution settings. Key quantitative results for Qwen2.5-Math-7B:

| Method            | AIME24 | Avg. over 7 benchmarks |
|-------------------|--------|-----------------------|
| Vanilla GRPO      | 30.0   | 45.2                  |
| SimpleRL-Zero     | 23.3   | 42.6                  |
| Oat-Zero          | 30.0   | 46.5                  |
| LUFFY (prefix)    | 33.3   | 46.6                  |
| Scaf-GRPO         | **43.3 (+44.3%)** | **50.9 (+12.6%)**   |

Additional highlights:
- Qwen2.5-1.5B on AIME24: $7.2 \rightarrow 20.0$ (+50% relative)
- Llama-3.2-3B average: $26.1 \rightarrow 28.8$ (+10.3% relative)
- DeepSeek-R1 LongCoT: $50.6 \rightarrow 53.6$ (+5.9% relative)
- Out-of-distribution GPQA-Diamond: vanilla GRPO $=32.3\%$, Scaf-GRPO $=37.3\%$

Ablation studies reveal:
- No guidance exemption reduces performance by $-9.2\%$ (relative)
- Solution-only hints (no progressive hierarchy): $-4.9\%$
- Removing any hint tier: up to $-5.7\%$
- No incremental hint chunking (all hints at once): $-6.3\%$

## 7. Limitations and Extensions

Scaf-GRPO is model-agnostic, applicable to various LLM architectures, sizes, and reasoning styles, and fosters stable skill acquisition. Nonetheless, it requires:
- Offline (teacher model) hint generation
- Reward functions that are sparse but verifiable

Potential extensions include:
- Automated or online hint synthesis
- Adapting scaffolding schedules dynamically
- Application to program synthesis, logic puzzles, and other sequential-decision LLM domains subject to sparse reward structures

These directions suggest a broad future applicability for the underlying pedagogy-inspired principle of incrementally restoring learning variance with minimal, on-policy interventions [2510.19807].

Source: https://www.emergentmind.com/topics/scaf-grpo