Value Model–Driven Prompt Filtering

• Value Model–Driven Prompt Filtering is a method that uses a transformer-based value model to estimate prompt difficulty and select intermediate-difficulty prompts.
• It computes a scalar estimate via a small MLP head, enabling near-instantaneous, on-policy filtering to dramatically reduce computational overhead.
• Empirical results on benchmarks like MATH500 and DeepScaleR show 12–17× speedup and improved sample efficiency in RL post-training.

Value model–driven prompt filtering is a methodology introduced within the Prompt Curriculum Learning (PCL) algorithm for reinforcement learning (RL) post-training of LLMs. It exploits a value model to efficiently estimate prompt difficulty and thereby filters for “intermediate-difficulty” prompts—those most informative for policy gradient optimization. PCL’s value model–driven selection achieves significant improvements in both computational efficiency and sample efficiency for post-training on reasoning-intensive tasks such as MATH500 and DeepScaleR, and provides an implicit on-policy curriculum.

1. The Value Model: Estimation and Training

PCL employs a value model $V_{\phi}(x)$, where $\phi$ denotes value-model parameters and $x$ is a tokenized prompt (e.g., a math question). The model reuses the transformer backbone, omitting chain-of-thought generations, and appends a small MLP “head” to produce a scalar estimate. Formally,

$$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$

where $\pi$ is the current policy, $y$ is a sampled output, and $r(x, y)\in\{0,1\}$ is a binary correctness reward. The value model is trained using mean-squared error regression against empirical average rewards. Given a minibatch of $m$ prompts, each with $n$ sampled rollouts,

$$\hat{Y}_i = \frac{1}{n} \sum_{j=1}^{n} r(x_i, y^{i,j}),\qquad L_v(\phi) = \sum_{i=1}^m \Bigl(V_{\phi}(x_i) - \hat{Y}_i\Bigr)^2$$

This setup ensures that the value model acts as a cheap proxy for expected prompt difficulty under the evolving policy.

2. Difficulty Metrics and Effective Ratio

Prompt difficulty under policy $\phi$0 is quantified by

$\phi$1

Since exact computation is intractable, empirical rollouts give

$\phi$2

PCL leverages $\phi$3 rather than direct rollouts for nearly-instantaneous difficulty estimation. The effective ratio (ER) is introduced to measure the proportion of informative (nonzero advantage) examples in a batched training step:

$\phi$4

where

$\phi$5

Prompts that are too easy or too hard yield no gradient contribution, highlighting the importance of filtering for intermediate difficulty.

3. On-Policy Filtering Workflow

PCL’s on-policy selection relies on value model–driven prompt filtering as formalized in Algorithm 1:

1. Sample Candidates: Uniformly sample $\phi$6 candidate prompts.
2. Compute Values: In one forward pass, compute $\phi$7 for each candidate.
3. Select by Threshold: Identify $\phi$8 prompts whose value estimates are closest to $\phi$9 ($x$0 by default),

$x$1

1. Rollouts: For each selected prompt, generate $x$2 outputs using current policy $x$3.
2. Reward and Advantage: Compute rewards and estimate advantages, as above.
3. Policy Update: Update $x$4 by a standard on-policy gradient (pure GRPO, no KL-regularization),

$x$5

1. Value Model Update: Minimize $x$6 over the same batch.

Critically, the value model is always updated concurrently and lagged by one RL step.

4. Computational Efficiency Versus Rollout-Based Filtering

Traditional approaches such as DS and SPEED require generating rollouts for all $x$7 candidates to estimate prompt difficulty, resulting in substantial computational overhead. By contrast, PCL achieves prompt filtering by a single forward pass of $x$8 over $x$9 candidates, with rollouts performed only for the selected $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$0. Empirical measurements indicate, for Qwen3-1.7B-Base:

• Rollout-based filtering over 2048 prompts (3 rollouts each): $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$1288 s (MATH), $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$2396 s (DeepScaleR)
• PCL value-model forward + backward: $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$323 s per step

This results in a $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$4 speedup for MATH and $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$5 for DeepScaleR in the prompt-filtering phase (Gao et al., 1 Oct 2025).

Dataset	Rollout Filtering Time (s)	PCL Filtering Time (s)	Speedup
MATH	288	23	12.1×
DeepScaleR	396	23	16.9×

5. Empirical Performance on Reasoning Benchmarks

PCL has been benchmarked on MATH500 and DeepScaleR using Qwen3-8B-Base and Qwen3-4B-Base. Key results include:

• MATH500, Qwen3-8B-Base: PCL achieves 88.2% accuracy in 37.2 h, outperforming DS (87.8% in 37.8 h) and unfiltered GRPO (86.4% in 28.3 h).
• MATH500, Qwen3-4B-Base: PCL reaches 83.4% in 14.0 h, versus 83.0% (GRPO, 29.2 h).
• DeepScaleR (average of six benchmarks), Qwen3-8B-Base: PCL achieves 52.0% at 41.8 h, compared to 51.4% (GRPO, 43.0 h).
• DeepScaleR, Qwen3-4B-Base: PCL matches peak performance with 28% less compute time.

PCL maintains an effective ratio close to $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$6 throughout training, indicating two-fold gains in sample efficiency over uniform GRPO.

6. Curriculum Dynamics via Value Model–Driven Filtering

By fixing $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$7, the prompt selection mechanism always targets prompts at the policy’s $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$8 “success frontier.” Early training iterations involve relatively easy or medium-difficulty prompts (with $$V_{\phi}(x) \approx p_{\pi}(x) \equiv \mathbb{E}_{y \sim \pi(\cdot|x)}[r(x, y)]$$9), but as the policy improves, only harder examples maintain value near $\pi$0. The transition is empirically confirmed: when prompt difficulty is measured with a frozen reference policy, the difficulty of selected prompts decreases as training progresses. This demonstrates that value-model–driven filtering produces an automatic, on-policy curriculum that adapts as the policy evolves, with the highest policy gradient magnitudes concentrated near $\pi$1.

7. Summary and Theoretical Implications

Value model–driven prompt filtering in PCL exploits: (a) the fact that policy gradient magnitude is maximized for intermediate-difficulty prompts ($\pi$2 for binary reward tasks), (b) an efficiently trained, policy-lagged value estimator, and (c) greedy selection of prompts nearest a target threshold. This approach circumvents wasted computation, delivering $\pi$3–$\pi$4 faster prompt filtering than rollout-based methods while consistently matching or exceeding prior baselines on reasoning tasks—demonstrating an efficacious trade-off between upper-bound performance and wall-clock efficiency (Gao et al., 1 Oct 2025).

A plausible implication is that value model–driven filtering could generalize to other RL frameworks requiring adaptive, difficulty-sensitive sampling in domains with high rollout cost and uncertain task distributions.