Learning to Self-Verify Makes Language Models Better Reasoners

Abstract: Recent LLMs achieve strong performance in generating promising reasoning paths for complex tasks. However, despite powerful generation ability, LLMs remain weak at verifying their own answers, revealing a persistent capability asymmetry between generation and self-verification. In this work, we conduct an in-depth investigation of this asymmetry throughout training evolution and show that, even on the same task, improving generation does not lead to corresponding improvements in self-verification. Interestingly, we find that the reverse direction of this asymmetry behaves differently: learning to self-verify can effectively improve generation performance, achieving accuracy comparable to standard generation training while yielding more efficient and effective reasoning traces. Building on this observation, we further explore integrating self-verification into generation training by formulating a multi-task reinforcement learning framework, where generation and self-verification are optimized as two independent but complementary objectives. Extensive experiments across benchmarks and models demonstrate performance gains over generation-only training in both generation and verification capabilities.

• The paper shows that verification-only RL training produces math reasoners with accuracy comparable to generation-only training while substantially shortening reasoning traces, including a 75% token reduction for the 7B model.
• Self-verification training improves error detection and correction, raising held-out judgment accuracy by up to 16.73 percentage points and enabling verification-based candidate voting to improve results.
• Decoupled schedules that alternate generation and verification training outperform generation-only and mixed-objective baselines, although they add training cost and remain untested beyond 7B math models.

Overview

"Learning to Self-Verify Makes LLMs Better Reasoners" (2602.07594) investigates the asymmetry between generation and self-verification in LLMs trained with Reinforcement Learning with Verifiable Rewards (RLVR). The authors document two findings: first, that improving a model's generation ability through RL training does not improve its ability to verify the correctness of its own solutions, even on identical tasks; second, and more consequentially, that the reverse direction does not hold — training a model solely to verify its own answers improves its generation performance to a level comparable with standard generation-only RL training, while producing substantially shorter reasoning traces. Building on this, they formulate self-verification and generation as complementary objectives within a multi-task RL framework and show consistent gains over generation-only and mixed-objective baselines.

The work positions itself against prior observations that LLMs cannot reliably self-verify their outputs, that apparent "aha moment" verification behaviors are largely decorative ("fake verification"), and that verification ability does not improve with scale. Prior joint-training approaches treat verification as an auxiliary signal dominated by the generation objective; this paper instead isolates the verification objective and studies its effect on the generator.

The asymmetry between generation and self-verification

The paper's empirical starting point is a training-dynamics study on Qwen2.5-1.5B-Instruct: as the model is optimized for task-solving accuracy via GRPO, its accuracy on the same tasks improves while its self-verification accuracy remains flat. This confirms at the level of a single training run what earlier work had suggested across scales — generation gains do not transfer to verification.

The reverse experiment is the core contribution. The authors construct a self-verification training pipeline under the same RLVR umbrella: at each iteration, the current policy samples $G$ candidate solutions per query; a rule-based verifier labels each $(x_i, y_{i,j})$ pair with a binary correctness label $c_{i,j}$; and the policy is then optimized with GRPO to predict $\hat{c}_{i,j}$ matching $c_{i,j}$, using a verification reward $$r^v_{i,j} = \mathrm{Verifier}(\hat{c}_{i,j}, c_{i,j})$$. Crucially, no generation reward appears in this objective. A post-processing pipeline — filtering invalid samples and queries where all candidates fail, query-level diversity sampling, and explicit balancing of correct/incorrect labels per batch — stabilizes training and keeps compute comparable to generation training.

Verification-only training yields competitive generators

Trained for 1,000 steps on DAPO-Math-17K and evaluated on six mathematical benchmarks (AIME24, AIME25, AMC23, Minerva, MATH500, OlympiadBench) across Qwen2.5-Instruct models of 1.5B, 3B, and 7B parameters, verification-only training matches or exceeds generation-only training:

Model	Avg Acc (Generate)	Avg Acc (Self-Verify)	Avg Tokens (Generate)	Avg Tokens (Self-Verify)
Qwen2.5-1.5B	20.2	21.7	1604	1227
Qwen2.5-3B	28.5	29.5	3273	1936
Qwen2.5-7B	38.9	38.4	4458	1152

Two results stand out. For Qwen2.5-3B, self-verification surpasses generation training by 4.7 points on OlympiadBench (32.1 vs. 27.4) and 6.0 points on MATH500 (65.6 vs. 59.6). More strikingly, the 7B verification-trained model achieves near-parity accuracy while consuming roughly 25% of the tokens of the generation-trained model (1,152 vs. 4,458 average tokens). The authors attribute the token reduction to genuine error detection: the model learns when its reasoning has gone wrong and corrects course rather than emitting redundant or decorative verification steps.

This result carries a direct implication: verification is not merely an auxiliary capability but an independently sufficient training signal for producing competent reasoners, and it produces more inference-efficient ones.

What verification training actually teaches

Three analyses characterize the acquired capability. First, on a held-out verification set built from DeepSeek-R1-Distill-Qwen-7B solutions, the self-verification-trained Qwen2.5-1.5B improves from 45.58% (base) to 62.31% judgment accuracy (+16.73), whereas generation training yields only +0.37; notably, generation training degrades the 3B model's verification accuracy by 4.63 points. The learned verifier also generalizes to judging other models' outputs.

Second, on a corrupted-prefix evaluation (GPT-4.1 rewrites reasoning prefixes with injected mistakes across 1,545 problems), the verification-trained model substantially outperforms both base and generation-trained models at detecting and correcting mid-reasoning errors, supporting the claim that shorter traces reflect real error correction rather than truncated reasoning.

Third, improved self-verification unlocks a test-time scaling mechanism unavailable to generation-trained models: aggregating per-candidate verification scores with majority voting consistently helps only the Self-Verify model (e.g., +3.4 on AIME25), while it is neutral or harmful for base and generate-trained variants. This indicates the benefit is contingent on actually possessing reliable verification ability — a dependency worth noting, since naive application of verification-weighted voting to untrained verifiers can hurt performance (as seen in the base-model rows).

Integrating verification into generation training

The paper then proposes two decoupled multi-task strategies, both implemented in GRPO with the same hyperparameters: Verify-Init, which uses a 400-step verification-trained checkpoint as the initialization for 600 steps of generation training; and Verify-Alter, which alternates blocks of $n$ generation steps with verification phases built from the preceding phase's samples. Both are compared against Generate (generation-only) and Mixed-Train (joint optimization within each step).

Verify-Alter delivers the strongest averages: 22.7 vs. 20.2 for the 1.5B model (including 36.4 on AMC23, +5.9 over Generate, and 4.2 vs. 0.8 on AIME25), 29.4 vs. 28.5 for the 3B model, and 40.3 vs. 38.9 for the 7B model. Decoupled optimization outperforms Mixed-Train, which matches or trails the generation baseline on several settings (e.g., 37.2 average for the 7B model). The implication is that how verification and generation objectives are scheduled matters: simply mixing rewards per step dilutes the verification signal, whereas staged or alternating exposure preserves both.

Limitations and open questions

The paper is candid about several constraints. The added verification phases incur extra inference and optimization cost, so the token savings at inference come at a training-time overhead. Experiments are limited to models up to 7B parameters, leaving open whether the asymmetry and its reversal persist at larger scale. Only one verification task formulation (binary correctness judgment on mathematical solutions) and two heuristic schedules are explored; adaptive or principled scheduling of the two objectives remains unresolved. Finally, all evaluation is on mathematical reasoning — whether the generation–verification asymmetry and the benefits of verification-first training hold in planning, code, or multimodal domains is an open question the paper explicitly does not answer.

Conclusion

This paper establishes, through controlled RLVR experiments, that the generation–verification asymmetry is directional: generation training fails to produce verifiers, but verification training alone produces competitive, markedly more efficient generators, and integrating the two objectives in a decoupled multi-task schedule outperforms both generation-only and mixed-objective training. The central claim — that rewarding self-verification alone suffices to train a strong reasoner — challenges the treatment of verification as a secondary signal and suggests concrete value for inference efficiency and test-time scaling, subject to the stated caveats of scale, domain coverage, and schedule design.