Prompt Curriculum Learning for LLMs

• Prompt Curriculum Learning (PCL) is a reinforcement learning method that dynamically curates prompt difficulty by focusing on those with a 50% success rate to maximize the training signal.
• It leverages a lightweight on-policy value model to efficiently filter candidate prompts, avoiding costly rollout-based evaluations while ensuring optimal gradient updates.
• Empirical results show that PCL accelerates convergence and enhances reasoning performance on tasks such as mathematical problem solving compared to traditional prompt selection methods.

Prompt Curriculum Learning (PCL) is a reinforcement learning (RL)–based post-training methodology for LLMs in which the prompt selection policy is explicitly cast as a dynamic curriculum: at each training iteration, prompts of intermediate difficulty (where the model success rate is near 50%) are prioritized, maximizing learning signal and data efficiency. This approach leverages a learned value model to identify informative prompts with optimal difficulty in an on-policy fashion, in contrast to conventional RL methods that uniformly sample prompts or rely on resource-intensive rollout-based filtering. PCL demonstrably accelerates convergence and improves upper-bound performance for reasoning-intensive tasks, such as mathematical problem solving, by systematically curating the prompt curriculum to match the evolving capability of the LLM policy.

1. Motivation and Theoretical Foundations

The central motivation for PCL is rooted in the observation that gradients with respect to the RL objective are maximized for prompts where the model is neither highly proficient nor entirely unskilled. More formally, for a prompt $x$, if the current policy $\pi$ yields a success probability $p_\pi(x)$, then the expected squared advantage (gradient magnitude) over completions is maximized when $p_\pi(x) = 0.5$. Too-easy prompts produce vanishing gradients, and too-hard prompts offer uninformative negative rewards. This insight extends the established principle in curriculum learning that training should focus on the "zone of proximal development," maximizing pedagogical efficiency by operating at the boundary of current mastery.

In PCL, the curriculum is enacted by filtering candidate prompts through a value model $V(x)$ that estimates $p_\pi(x)$, efficiently identifying those with an expected reward close to a predetermined threshold $\tau$ (commonly $\tau=0.5$). The learning pipeline is thus reframed to continuously adapt the prompt selection policy so that the active batch contains prompts that will yield the highest effective learning signal at each step.

2. Value Model–Driven Prompt Filtering

A core innovation of PCL is the use of a lightweight value model, trained online and in tandem with the policy, to directly estimate expected prompt-level rewards:

$$V(x) \approx \mathbb{E}_{y \sim \pi(\cdot | x)}[r(x, y)] = p_\pi(x)$$

At each training step:

• A large pool of candidate prompts is sampled.
• A single forward pass through $V$ computes the expected reward for all candidates.
• The $m$ prompts with $|V(x_i) - \tau|$ minimized are selected as the batch (typically $\tau=0.5$), focusing on intermediate difficulty.

This approach entirely avoids the computational costs associated with rollout-based or dictionary-based estimation (which require generating completions for every candidate prompt to estimate $p_\pi(x)$), and remains fully on-policy—filtered prompts are always scored under the current model parameters.

Value model targets are updated using observed rewards from multiple generation attempts ($n$ per prompt), with a squared error objective:

$$\mathcal{L}_V = \sum_{i=1}^{m} \left( V(x_i) - \frac{1}{n} \sum_{j=1}^n r(x_i, y_{i,j}) \right)^2$$

This continual updating protocol ensures that the value model adapts rapidly to policy improvement, enabling the curriculum to expose the policy to progressively more challenging prompts as mastery advances.

3. Empirical Batching Strategy and Efficiency

The paper details a systematic paper of batching configurations, decomposing total batch size into number of prompts ($m$) and number of generations per prompt ($n$). Key findings include:

• There exists a transition regime: small $m,n$ values lead to sublinear scaling in generation wall time (dominated by the longest sequence), while large $m,n$ values yield linear scaling (dominated by computational saturation).
• Optimal throughput is achieved at the transition point (in the reported experiments, around $8\text{k}$ prompt-response pairs per update).
• By restricting prompt selection to the intermediate-difficulty regime (enforced via the value model), PCL achieves a higher diverse effective ratio—most items in a batch contribute strong learning signal, obviating the need for large $n$ (generations per prompt) and improving data diversity.

Generation and filtering speed benchmarks on MATH and DeepScaleR reveal speedups of $12.1\times$ and $16.9\times$ over rollout-based filtering, respectively, without sacrificing convergence speed or final accuracy.

4. Reinforcement Learning Objective and Mathematical Analysis

PCL adopts a token-level REINFORCE variant as its core RL objective:

$$J(\theta) = \mathbb{E}_{x \sim \mathcal{D},~y \sim \pi_\theta(\cdot|x)}\left[r(x, y)\right]$$

with the advantage function:

$A(x, y) = r(x, y) - p_\pi(x)$

and policy-gradient estimated as:

$$\nabla_\theta J = \mathbb{E}_{x, y}\left[\frac{1}{|y|} \sum_{\ell} \frac{\pi_\theta(y_\ell | x, y_{<\ell})}{\pi_{\text{ref}}(y_\ell | x, y_{<\ell})} A(x, y)\right]$$

A key theoretical result from the analysis:

$$\mathbb{E}_{y \sim \pi(\cdot | x)}[A(x, y)^2] = p_\pi(x)[1 - p_\pi(x)]$$

which is maximized at $p_\pi(x) = 0.5$. This justifies the focus on intermediate-difficulty prompts, as they maximize expected gradient magnitude and hence learning progress.

5. Empirical Performance and Learning Dynamics

PCL was benchmarked on high-difficulty reasoning tasks, including MATH and DeepScaleR, against rollout-based and uniform sampling baselines. Key outcomes:

• PCL attains either the highest final performance on held-out evaluations or reaches comparable performance with significantly less compute.
• During RL, prompt-level training rewards remain stably near 0.5, confirming sustained focus on the intermediate zone.
• Baselines without dynamic filtering (rollout-based or dictionary methods) often drift to easier prompts over time, leading to reduced gradient signals, slower policy improvement, and lower ultimate sample efficiency.
• PCL demonstrates high throughput both in filtering prompts and in converging to optimal test accuracy.

The methodology was further validated with successful application to reasoning benchmarks such as OlympiadBench, Minerva Math, AMC, and AIME, showing consistent generalization and sample efficiency improvements.

6. Practical Implications and Extensions

PCL represents a principled methodology for RL-based LLM post-training, optimizing the trade-off between learning efficiency and ultimate upper-bound performance in domains where prompt difficulty is variable and directly impacts the utility of gradient updates. Its on-policy, value-model–driven filtering can be generalized to tasks beyond reasoning, wherever difficulty estimation translates to actionable data selection.

Extensions to PCL could include:

• Generalization to non-binary reward structures or continuous quality metrics for richer prompt assessment signals.
• Asynchronous variants and adaptation to longer-context settings to further improve training stability and scalability.
• Integration with dynamic batching and adaptive learning rate scheduling as the transition regime shifts with dataset and architecture scale.

A plausible implication is that PCL-type data curation may become a foundational paradigm for RL-based LLM adaptation in tasks requiring high efficiency, high reasoning fidelity, or real-time deployment.

7. Summary Table: Key Features of Prompt Curriculum Learning

Component	Technical Description	Efficiency Impact
Value Model	Predicts expected reward for prompts under current policy	Negligible overhead
Prompt Filtering	Selects prompts with $p_\pi(x)$ closest to a target (usually 0.5)	Maximizes learning signal
Batch Configuration	Balances $m$ (prompts) and $n$ (generations) at transition regime	Accelerates convergence
On-Policy Curriculum	Adapts prompt curriculum as policy improves	Progressive skill acquisition
Rollout-Free Selection	Avoids candidate rollouts per prompt for scoring	$12$–$17\times$ faster filtering
Application Scope	LLM post-training for reasoning (e.g., MATH, DeepScaleR, OlympiadBench)	Outperforms or matches baselines

In conclusion, Prompt Curriculum Learning systematically curates informative prompts at each RL update, increasing sample efficiency and learning signal, and resulting in faster, more robust post-training of LLMs on reasoning-intensive domains (Gao et al., 1 Oct 2025).