Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought

Abstract: State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting.

• The paper presents a calibrated reinforcement learning method (CAL) that leverages attempt-level verifier feedback to maximize the correct answer probability over multiple attempts.
• It derives an unbiased, variance-reduced policy gradient estimator that overcomes the limitations of traditional trajectory-level methods in dense feedback scenarios.
• Empirical evaluations on benchmarks like MATH, Maze Navigation, and Synthetic Markov Chain demonstrate CAL's superior performance in iterative correction and robust reasoning.

Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought: An Expert Review

Problem Formulation and Motivations

This paper systematically addresses the optimization of long-horizon chain-of-thought (CoT) reasoning in LLMs, particularly when iterative multi-attempt procedures with verifier feedback are employed. The authors formalize the Verification@K (Ver@K) setting, wherein a model is allowed up to $K$ attempts to solve a problem, each time receiving binary (pass/fail) feedback from an external hard verifier, with the objective being to maximize the frequency with which the model produces a correct answer within these $K$ attempts.

Traditional trajectory-level reinforcement learning (RL) approaches fail to exploit the dense per-attempt feedback available in this iterative process, instead propagating sparse trajectory-level rewards across all attempts, which leads to noisy credit assignment and confounded learning signals for reasoning and correction behaviors. Simple attempt-level reward assignment heuristics, such as naïvely giving per-attempt pass/fail as rewards, yield biased policy gradients and suboptimal optimization of Ver@K.

The core motivation of this work is to construct a principled RL method that leverages attempt-level (within-trajectory) signals to enable finer-grained, variance-reduced, and unbiased optimization for models that can revise their answers based on intermediate verifier feedback—a learning scenario that is now prevalent for advanced reasoning LLMs.

Technical Contributions

The principal technical contributions of the paper are:

1. Attempt-Level Policy Gradient Derivation: The authors rigorously derive the form of the policy gradient for the Ver@K objective at the attempt level, showing that, in contrast to the trajectory-level gradient, the correct gradient should credit each attempt by its marginal improvement in the probability of Ver@K success, conditioned on previous failed attempts.
2. Calibrated Attempt-Level Algorithm (CAL): The paper introduces the CAL algorithm, a generalized attempt-level policy gradient estimator using theoretically derived sample-based weights per attempt, which ensures the estimator is unbiased and exhibits lower variance compared to trajectory-level REINFORCE and group-based policy optimization (GRPO) estimators. CAL corrects the systematic bias present in the Naive Attempt-Level (NAL) variant, which disproportionately credits early attempts and in effect optimizes a discounted loss, as proven in the paper.
3. Variance Reduction Guarantees: Theoretical analysis demonstrates that the attempt-level estimator (and CAL in particular) provably has lower variance than the trajectory-level estimator under mild assumptions, enabling more sample-efficient and stable RL training, especially as $K$ increases.
4. Practical Estimation for Conditional Success Probabilities: Recognizing the practical challenge of computing future conditional success probabilities required for calibration, the paper proposes a sample-based estimation directly from the current group of rollouts, yielding a computationally efficient and unbiased estimator under an attempt-index independence assumption.

Experimental Results

The authors present comprehensive empirical evidence using three benchmarks:

• MATH Dataset: CAL-GRPO yields the highest Ver@2 (and Ver@4) accuracy, outperforming both TL-GRPO (trajectory-level) and NAL-GRPO (naive attempt-level) baselines. Gains are driven primarily by improved conditional second-attempt success, confirming the theoretical analysis that CAL appropriately maintains learning signal on corrections rather than overoptimizing first-attempt accuracy.
• Maze Navigation: CAL-GRPO consistently reaches high Ver@K accuracy earlier in training compared to baselines and provides efficient exploration-exploitation trade-off. Notably, NAL-GRPO reduces the average number of attempts due to its bias toward earlier attempts, illustrating the practical distinction in objective alignment between NAL and CAL.
• Synthetic Markov Chain: In a controlled setting with deterministic verification and a trap-state variant introducing adversarial late-stage failure, CAL-GRPO is the only method that robustly breaks through the non-trivial performance plateau, indicating its efficacy in scenarios where late-attent corrections are critical for Ver@K improvements.

Quantitative Highlights:

• MATH (Ver@2): CAL-GRPO achieves the best validation accuracy across training, with the difference magnified on harder cases where second-attempt correction meaningfully contributes to overall success rate.
• Maze (Various K and Sizes): CAL-GRPO exhibits the most robust accuracy across maze sizes and attempt budgets, with clear gains in marginal attempt success and variance-reduced learning curves.
• Markov Chain (Trap Variant): Only CAL-GRPO escapes the 0.75 plateau induced by the random trap, by successfully optimizing for robust correction across attempts.

Theoretical and Practical Implications

The introduction of the calibrated attempt-level framework provides a unified and sample-efficient method for RL fine-tuning in the presence of dense and structured verifier feedback, applicable to verifier-guided multi-attempt learning scenarios in both synthetic environments and real-world reasoning tasks. The methodology precisely aligns the training objective with the deployment-time metric (Ver@K), eliminating the conflation of first-try and correctional learning signals, and thus enables policymakers to directly control the accuracy-compute trade-off in systems supporting iterative refinement and self-correction.

Critically, the theoretical guarantee that CAL is unbiased (as opposed to NAL), and minimizes variance compared to trajectory-level estimators, provides a solid foundation for scaling to longer-horizon reasoning and larger verification budgets ($K \gg 2$), where traditional trajectory-level gradients would suffer from overwhelming variance and diminishing signal-to-noise.

Limitations and Future Directions

The practical calibration presented here is contingent on the ability to reliably estimate marginal future success probabilities from rollout samples, an assumption that may be weakened in non-deterministic or highly multi-modal settings. Additionally, extension to soft/noisy verifiers or to tasks involving non-binary, structured feedback is left open.

Key future directions include:

• Noisy/Learned Verifiers: Generalizing CAL to settings where verifier feedback is probabilistic, learned, or continuous.
• Adaptive Attempt Budgets: Developing policies which dynamically allocate or expand $K$ based on problem difficulty or intermediate feedback.
• Scalability: Application and analysis at scale ($K > 4$) and in open-ended settings like program synthesis and automated theorem proving.
• Fine-Grained Credit Assignment: Integration with fine-grained token-level or step-level credit assignment using richer process-level rewards or embedded intrinsic objectives.

Conclusion

This paper provides a rigorous and practical solution to optimization and credit assignment in multi-attempt, verifier-guided RL for long chain-of-thought reasoning. The Calibrated Attempt-Level approach yields unbiased, variance-reduced gradient estimates and empirically delivers superior performance in complex reasoning and structured planning benchmarks. The framework will directly inform the design of future RL fine-tuning regimes for LLMs, particularly as multi-turn, self-corrective, and verifier-augmented reasoning becomes the dominant paradigm in large-scale LLM deployment (2604.17912).