Process-Verified Reinforcement Learning for Theorem Proving via Lean

Abstract: While reinforcement learning from verifiable rewards (RLVR) typically has relied on a single binary verification signal, symbolic proof assistants in formal reasoning offer rich, fine-grained structured feedback. This gap between structured processes and unstructured rewards highlights the importance of feedback that is both dense and sound. In this work, we demonstrate that the Lean proof assistant itself can serve as a symbolic process oracle, supplying both outcome-level and fine-grained tactic-level verified feedback during training. Proof attempts are parsed into tactic sequences, and Lean's elaboration marks both locally sound steps and the earliest failing step, yielding dense, verifier-grounded credit signals rooted in type theory. We incorporate these structured rewards into a GRPO-style reinforcement learning objective with first-error propagation and first-token credit methods that balances outcome- and process-level advantages. Experiments with STP-Lean and DeepSeek-Prover-V1.5 show that tactic-level supervision outperforms outcome-only baselines in most settings, delivering improvements on benchmarks such as MiniF2F and ProofNet. Beyond empirical gains, our study highlights a broader perspective: symbolic proof assistants are not only verifiers at evaluation time, but can also act as process-level reward oracles during training. This opens a path toward reinforcement learning frameworks that combine the scalability of LLMs with the reliability of symbolic verification for formal reasoning.

• The paper introduces a reinforcement learning framework that leverages Lean’s process-level verification to assign dense, tactic-level rewards.
• It integrates outcome-level and tactic-level reward signals through first-error propagation and first-token credit mapping to enhance training stability.
• Experiments on MiniF2F and ProofNet benchmarks demonstrate measurable gains and efficient proof generation with minimal computational overhead.

Process-Verified Reinforcement Learning for Theorem Proving via Lean

Motivation and Background

Automated theorem proving (ATP) in formal mathematics is advancing rapidly, propelled by both interactive theorem provers (ITPs) and LLM-based agents. While traditional RL approaches in ATP often rely on sparse, outcome-level reward signals, modern ITPs such as Lean can provide dense, fine-grained feedback at the level of individual proof tactics. However, most prior methods underutilize this structure, focusing on binary whole-proof verification and failing to exploit dense process-level feedback for policy optimization. The paper "Process-Verified Reinforcement Learning for Theorem Proving via Lean" (2606.20068) directly addresses this gap, proposing a framework in which Lean's symbolic verification system not only serves as a global proof checker but as a process-level oracle that supplies verifiable, structured rewards during RL training.

Framework and Methodology

The approach formalizes Lean's feedback by parsing LLM-generated proofs into tactic sequences and leveraging Lean's elaboration process to mark locally sound steps and identify failure points. This yields two reward streams: an outcome-level signal reflecting entire proof correctness, and tactic-level signals specifying the validity of individual steps. These rewards are integrated into a GRPO RL objective, utilizing first-error propagation and first-token credit assignment strategies:

• First-error propagation: Upon detecting the first erroneous tactic, all subsequent steps are treated as invalid for credit assignment, aligning with the causal structure of Lean proofs.
• First-token credit mapping: Tactic-level advantages are assigned to the first token of each tactic, enhancing the model’s ability to select the correct proof strategy at decision points crucial for downstream reasoning.

The tactic-level rewards are normalized by problem difficulty, stabilizing training and providing dynamic baselines. The tactic-level MDP formulation enables dense, semantically faithful credit assignment, bypassing the need for natural-language PRMs or manual annotations.

Empirical Results

Experiments on MiniF2F and ProofNet benchmarks demonstrate clear improvements:

• STP-Lean + Process-Verified RL: Achieves up to +2.5%p on MiniF2F (pass@64) and +1.4%p on ProofNet (pass@32) relative to outcome-only RL and baseline models.
• DeepSeek-Prover-V1.5: Marginal yet consistent gains across all evaluation metrics.
• Comparison with strong baselines: The method approaches the accuracy of search-based paradigms (e.g., InternLM2.5-StepProver) in single-shot, whole-proof generation, while avoiding computational overhead at inference.

Ablation studies confirm that integrating both outcome-level and tactic-level advantages yields more stable and robust policy improvements than using either source in isolation. Assigning credit to the first token of each tactic (as opposed to all, last, or entropy-selected tokens) produces maximal and consistent performance gains, substantiated by both quantitative and qualitative analyses.

Theoretical Implications

The design is theoretically grounded via potential-based reward shaping. Under the first-error propagation assumption, the tactic-level reward sequence can be interpreted as a coarse yet semantically aligned quantization of an ideal value function in MDPs: monotone growth along error-free prefixes and irreversible collapse after the first error. This ensures that credit assignment is not only dense but also type-theoretically sound, adhering to the semantics of Lean proofs.

The method enables reinforcement learning frameworks to combine the scalability of LLMs with the rigor of symbolic verification, advancing the reliability of formal reasoning in neural models. It circumvents the limitations of annotation-dependent PRMs, demonstrating that symbolic verifiers can serve as process-level reward sources during training.

Practical Implications and Limitations

Practically, the framework mitigates reward sparsity and enables more efficient policy optimization:

• Dense verifier-grounded credit assignment allows rapid convergence and efficient exploration, as reflected in decreased policy entropy and stable proof lengths.
• The computational overhead of tactic-level credit assignment is negligible, given the pre-existing reliance on REPL interfaces to Lean for both outcome verification and node-level parsing.
• Shorter verification timeouts bias the models toward generating concise, canonical proofs, often enhancing generalization on formal mathematics benchmarks.

Limitations include sensitivity to tactic reward scales (d1, d2) and the lack of large annotated datasets for explicit process-based reward model comparison. The current method does not address natural-language chain-of-thought supervision or learned PRMs, leaving open questions for hybrid formal/natural-language reasoning.

Future Directions

Directions for future research include:

• Scaling to longer-context proofs and reasoning tasks where process-level reward signals can be supplied at both token and tactic granularity.
• Developing general advantage estimators and large-scale tactic-level datasets for more robust process supervision.
• Investigating hybrid reward models that combine Lean-based process supervision with natural-language reasoning for tasks requiring informal logic and formal verification.

Conclusion

The paper presents a principled RL framework where Lean serves not only as an inference-time verifier but as a process-level oracle for dense, verifiable rewards. This enables rigorous credit assignment in LLM-based theorem proving, substantially improving empirical performance and training stability on established benchmarks. The results establish that symbolic proof assistants are potent reward sources for RL in formal reasoning, setting the stage for more stable, scalable, and reliable ATP frameworks integrating symbolic and neural paradigms.