Propose, Solve, Verify: Self-Play Through Formal Verification (2512.18160v1)

Abstract: Training models through self-play alone (without any human data) has been a longstanding goal in AI, but its effectiveness for training LLMs remains unclear, particularly in code generation where rewards based on unit tests are brittle and prone to error propagation. We study self-play in the verified code generation setting, where formal verification provides reliable correctness signals. We introduce Propose, Solve, Verify (PSV) a simple self-play framework where formal verification signals are used to create a proposer capable of generating challenging synthetic problems and a solver trained via expert iteration. We use PSV to train PSV-Verus, which across three benchmarks improves pass@1 by up to 9.6x over inference-only and expert-iteration baselines. We show that performance scales with the number of generated questions and training iterations, and through ablations identify formal verification and difficulty-aware proposal as essential ingredients for successful self-play.

• The paper introduces the Propose, Solve, Verify (PSV) paradigm that employs formal verification to ensure semantic correctness in self-play for code synthesis.
• It details a dual-model system with a difficulty-aware proposer and a verifier-enhanced solver that collaboratively generate an autonomous training curriculum.
• Experiments on multiple benchmarks show significant pass@1 improvements, highlighting the advantage of formal verification over traditional unit test approaches.

Propose, Solve, Verify: A Framework for Self-Play Through Formal Verification

Motivation and Background

The paper "Propose, Solve, Verify: Self-Play Through Formal Verification" (2512.18160) addresses a core challenge in self-improving LLMs: how to reliably train LLMs via self-play in domains where usual reward signals (e.g., unit test-based correctness) are fundamentally unreliable. Previous successes in domains like board games are not directly transferable due to the vastness of the action space and the difficulty of property verification in language and program synthesis. In code generation, test-suite-based evaluation leads to reward hacking and error propagation, undermining self-improvement and curriculum learning (Figure 1).

Figure 1: Comparison between the guarantees of unit test-based verification and formal verification in code generation tasks.

To address these limitations, the authors introduce the Propose, Solve, Verify (PSV) paradigm, leveraging formal verification frameworks (specifically, Verus for Rust) to provide a sound, deterministic correctness signal for code synthesis. Unlike prior proposer-solver self-play systems where verification is weak, PSV grounds all curriculum generation and solver improvement in formal specification and machine-checked proof.

The PSV Algorithmic Framework

PSV instantiates a dual-model system:

• Proposer Model ($P_{\phi_t}$): Generates formal specifications as novel coding problems, with difficulty levels conditioned on recent solver performance.
• Solver Model ($S_{\theta_t}$): Generates candidate program solutions for given specifications, which are accepted by the formal verifier only if all required postconditions hold for all allowed inputs.

The self-play proceeds iteratively:

1. The proposer creates $B$ new problem specs.
2. The solver attempts $k_{trn}$ solution samples per spec.
3. Verified solutions are added to the data pool, and only these inform further fine-tuning of $S_{\theta}$ via rejection fine-tuning (RFT).
4. The proposer is updated, and the loop continues.

Figure 2: The PSV self-play loop couples a difficulty-aware proposer with the solver model, relying exclusively on formal verification feedback.

Figure 3: Iterative improvement in the PSV framework where solving and formal verification constrain the self-play curriculum.

The PSV approach ensures only semantically correct solutions augment solver capability, eliminating reward leakage. Hard negative samples are similarly filtered by the spec-verifier, improving supervision efficiency.

Experimental Protocols, Datasets, and Baselines

PSV-Verus, a Qwen2.5-Coder-3B model fine-tuned under PSV, is evaluated against strong baselines on three benchmarks:

• Dafny2Verus: 274 diverse programs from the AlphaVerus corpus.
• MBPP-Verified: 78 problems with fully verified specifications.
• HumanEval-Verified: 85 functions across 49 programs, derived from HumanEval.

Evaluations are split into transfer learning and test-time training (TTT), with pass@k metrics ($k=1,5,10$) computed using up to 100 samples per problem.

Results: Performance Scaling and Component Ablations

PSV-Verus achieves substantial improvements over prior state-of-the-art:

• Transfer Learning: Pass@1 on MBPP: 25.25% vs. AlphaVerus 6.48% and RFT 10.99%; HumanEval: 16.18% vs. AlphaVerus 7.24%.
• Test-Time Training: Dafny2Verus pass@1: 65.63%; MBPP pass@1: 36.78%; HumanEval pass@1: 19.07%.

It is observed that performance scales positively both with the number of unique questions proposed per iteration and with the number of self-play iterations, even under a fixed total question budget (Figure 4, Figure 5, Figure 6):

Figure 4: Pass@k improvement across self-play iterations for multiple test sets.

Figure 5: Direct correlation between test-time training performance and the number of questions per self-play iteration.

Figure 6: For fixed total question budgets, increasing the number of self-play iterations yields higher pass@1.

Ablation experiments confirm that:

• Formal Verification is essential: removing it incurs up to 50% drops in pass@1.
• Difficulty-Aware Proposal and Prompt Diversity substantively improve coverage and model learning, supporting the approach's generalization properties.

Practical and Theoretical Implications

The results demonstrate that formal verification enables robust exploratory data generation for solver self-training, sidestepping the incompleteness and hackability of unit test-based supervision. PSV achieves high pass@1 on multi-domain code verification benchmarks without any human-written solutions in the training loop, highlighting the feasibility of fully autonomous curriculum generation.

Theoretically, the PSV paradigm opens study into scaling laws for self-play reasoning under sound (but incomplete) reward signals, especially in Turing-complete domains. PSV allows for precise curriculum shaping and difficulty adaptive training, which may be extensible to other domains with reliable verifiers (e.g., formal mathematics, system verification).

Future Directions

Potential research avenues include:

• On-Policy RL Algorithms: Investigating more sophisticated RLVR variants (e.g., GRPO) compatible with incomplete verifiers.
• Extension to Other Formal Domains: Adapting PSV to mathematical theorem proving, agentic tool use, or other areas with accessible formal semantics.
• Improved Difficulty Estimation and Sampling: Combining PSV with more advanced difficulty modeling (e.g., non-parametric score functions or meta-learned proposal heads).
• Scaling Model and Corpus Sizes: Testing with larger architectures and rapid synthetic corpus expansion.

Conclusion

The PSV methodology exemplifies a rigorous route to self-play improvement for code generation with LLMs: leveraging machine-checked formal verification for specification and solution correctness. This paradigm overcomes brittleness in unit test based approaches, demonstrates strong empirical scaling with data and compute, and reveals the central role of sound reward in autonomous curriculum building.

PSV stands as a reference algorithmic blueprint for future work in sound self-play for reasoning domains, with practical impact for trustworthy code generation and theoretical significance for advancing self-improving machine intelligence.