Learning with a Single Rollout via Monte Carlo Pass@k Critic

Abstract: Estimating token-level advantages in reinforcement learning (RL) for LLMs remains challenging because scaling up episodic experience collection is expensive. The difficulty intensifies for baseline advantage estimation methods, where repeated sampling causes trajectories to diverge into substantially different reasoning prefixes. In this context, RL algorithms such as GRPO prove limited: an outcome reward is too sparse to be attributed to specific actions like intermediate steps, and comparisons across sampled traces are non-trivial because they are heterogeneous. To mitigate both the computational cost of repeated sampling and the difficulty of credit assignment, we study single-rollout proximal policy optimization (SR-PPO) featuring token-level credit assignment in RL for LLMs. Instead of estimating advantages by normalizing episodic returns within the candidate group, we train a calibrated token-level credit critic using Monte Carlo outcomes from one rollout per prompt. Specifically, we use the critic to predict the Pass@k success probability at the prompt prefix, which is derived from a Pass@1 attempt. This choice yields a more selective learning signal than Pass@1: it discounts easily solved prefixes while prioritizing hard ones whose success probability remains marginal. We show that as $k$ increases, Pass@k converges to a reachability indicator, reflecting whether a prefix can lead to at least one successful continuation. In an explicit state graph, the limit ($k \rightarrow \infty$) can be computed in $O(|V|+|E|)$ time, offering a promising surrogate for direct credit assignment without the need to sample contrastive traces. As an initial validation, SR-PPO exhibits stable learning dynamics, along with consistent gains in Pass@128 success rates on mathematical reasoning benchmarks such as HMMT26 and AIME24.

• The paper introduces SR-PPO, leveraging token-level credit assignment via a Monte Carlo Pass@k critic to enhance language model post-training.
• It demonstrates that SR-PPO outperforms traditional group-based methods, achieving superior Pass@k metrics with only a single rollout per prompt.
• Empirical tests in mathematical reasoning tasks show robust stability and improved quality of solution distributions with reduced computational cost.

Learning with a Single Rollout via Monte Carlo Pass@k Critic: An Academic Analysis

Motivation and Problem Setting

Traditional reinforcement learning approaches for LLM (LM) post-training, especially in mathematical reasoning and code generation domains, rely on sparse outcome-based rewards and multiple sampled rollouts per prompt. This design enables comparative group-based methods such as Group Relative Policy Optimization (GRPO). However, increasing rollout cost, trajectory heterogeneity, and the difficulty in aligning reasoning prefixes across disparate rollouts present critical limitations for high-fidelity credit assignment. Moreover, in agentic systems, tool latency and long-horizon tasks further exacerbate these issues. The SR-PPO framework targets the constrained single-rollout regime, aiming to extract token-level credit from only one sampled trajectory per prompt, thus addressing sample efficiency and tractable credit assignment.

Methodology: Token-Level Credit Assignment via Pass@k Monte Carlo Critic

The SR-PPO algorithm departs from group-based reward normalization by introducing a token-level credit assignment mechanism utilizing a Monte Carlo Pass@k critic. For each prompt, a prefix-level critic model predicts the Pass@k success probability at every intermediate prefix state, calibrated from single-rollout outcome labels. The Pass@k metric denotes the probability that at least one out of $k$ continuations succeeds, with Pass@1 being the traditional measure and Pass@k providing a selective, sparse signal. The critic network's outputs are calibrated to model success probabilities, with Brier score and binary cross-entropy losses promoting discriminativeness and calibration.

Token-level advantage is computed as the local shift in predicted Pass@k value across token transitions, supplemented by a terminal correction term aligning the credit signal with the observed terminal outcome. The PPO objective is formulated as a dense, on-policy policy-gradient update, using token-level advantages as weights, in a training setup where each batch consists of a single rollout per prompt.

Pass@k and Reachability: Theoretical Connections

A central theoretical contribution is the elucidation of Pass@k's connection to reachability. As $k$ increases, Pass@k converges to a binary indicator of whether any successful continuation exists from a given prefix—termed reachability. This insight establishes Pass@k as an interpolation between outcome probability and structural solvability. While estimating numerical success probabilities requires substantial sampling, reachability can be efficiently computed via reverse graph traversal in $O(|V| + |E|)$ time if the support graph is known. In practice, Pass@k (with moderate $k$) serves as a tractable surrogate, maintaining gradient sensitivity for hard yet reachable prefixes and promoting selective credit assignment.

Comparative Analysis with Prior Methods

Group-based methods, such as GRPO and tree-structured credit assignment algorithms, rely on multiple completions per prompt and suffer from computational inefficiency and statistical noise in heterogeneous long-horizon environments. Process reward models (PRMs) and implicit process rewards offer dense supervision but often necessitate auxiliary networks or considerable manual labeling. Likelihood-based rewards further abstract task-specific supervision, at the cost of interpretability. SR-PPO innovates by leveraging outcome-only supervision and learned prefix-level progress, aligning with the emerging paradigm of dense credit assignment from sparse terminal rewards.

Empirical Validation and Results

Experimental results on DeepScaleR mathematical reasoning data and competition-level benchmarks (AIME24, AIME25, HMMT26) demonstrate that Pass@4 SR-PPO achieves consistent improvements in Pass@k metrics despite being strictly more sample-efficient compared to GRPO—requiring only a single rollout per prompt, whereas GRPO uses 8 rollouts. Key findings include:

• Stable optimization dynamics and superior Pass@8 success rates: Pass@4 SR-PPO matches GRPO on Pass@8 metrics with less computational cost.
• Selective credit assignment: Empirical ablations show higher sparsity in token-level advantages under Pass@4 SR-PPO, enhancing learning stability and reducing noisy updates.
• Quality of sampled solution distributions: Improvements are observed linearly as the number of completions increases, indicating that SR-PPO improves not just greedy behavior but sampled solution quality.

The visualizations of token-level advantages further corroborate the qualitative stability and selectivity imparted by Pass@k-based credit assignment, with substantial fractions of tokens receiving near-zero advantage.

Implications and Future Directions

The SR-PPO framework presents practical implications for post-training LMs in settings where generation cost and trajectory heterogeneity preclude group-based optimization. The Pass@k critic offers a robust mechanism for token-level credit assignment, facilitating scalable RL fine-tuning on mathematical reasoning and complex agentic tasks. Theoretically, the Pass@k-reachability connection motivates further structural credit assignment methods, potentially leveraging explicit graph-based representations. Future research may extend Pass@k SR-PPO to broader domains, incorporate dynamic $k$ selection, and develop more explicit graph reasoning models to further bridge outcome probability and reachability for systematic trajectory credit assignment.

Conclusion

SR-PPO with Monte Carlo Pass@k critic presents a principled approach for single-rollout RL, assigning calibrated token-level credit from sparse terminal outcomes while maintaining computational efficiency and learning stability. By incorporating Pass@k value families and leveraging their reachability properties, the method provides selective learning signals and addresses longstanding challenges in credit assignment for long-horizon LM reasoning tasks. The framework is empirically validated and theoretically grounded, setting the stage for future advances in sample-efficient RL for LLM reasoning and agentic systems.

(2606.25451)