DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Search
DeepSeek-Prover-V1.5 endeavors to enhance the capabilities of LLMs in formal theorem proving through a novel integration of reinforcement learning and Monte-Carlo tree search (MCTS). This model builds upon the foundation laid by DeepSeek-Prover-V1 by incorporating new techniques in pre-training, fine-tuning, and reward-based search strategies.
Model Overview
DeepSeek-Prover-V1.5 augments its predecessor by optimizing both training and inference layers. The model, based on the DeepSeekMath-Base, undergoes supervised fine-tuning with an extensive formal theorem proving dataset derived from DeepSeek-Prover-V1. Enhanced through reinforcement learning from proof assistant feedback (RLPAF), this model integrates proof-step granularity into the whole-proof generation, offering a scalable approach for theorem proving.
Methodological Enhancements
- Pre-Training:
- Built on DeepSeekMath-Base, the pre-training involves extensive datasets encompassing mathematics and formal systems languages such as Lean, Isabelle, and Metamath.
- Supervised Fine-Tuning:
- An enriched dataset compiling formal theorem proofs from multiple projects, enhanced through data augmentation techniques including natural language reasoning annotations by DeepSeek-Coder V2 236B and intermediate tactic states in Lean 4 code.
- The fine-tuning process includes techniques like thought-augmented proof generation and syntax state augmentation to improve alignment between formal mathematical reasoning and natural language problem-solving methods.
- Reinforcement Learning:
- Using the GRPO algorithm, the RLPAF approach refines the model based on the Lean prover's feedback, optimizing the alignment with formal specifications via reward-driven learning.
- Monte-Carlo Tree Search:
- Introducing RMaxTS, an innovative variant of MCTS. The method uses a truncate-and-resume approach to integrate proof-step information into whole-proof generation, allowing for effective use of compiler feedback to guide the exploration of diverse proof paths.
- RMaxTS leverages intrinsic rewards to address reward sparsity, fostering exploration of the proof search space with curiosity-driven strategies.
Numerical Results and Claims
DeepSeek-Prover-V1.5 demonstrates substantial improvements over its predecessor:
- MiniF2F Benchmark: Achieving a pass rate of 63.5% on the test set; a significant upsurge from DeepSeek-Prover-V1’s 50.0%.
- ProofNet Benchmark: Demonstrating a pass rate of 25.3% on verification tests; a marked improvement from previous versions and strong comparable baselines.
Theoretical and Practical Implications
Theoretical Implications:
- The innovation in merging RLPAF and MCTS with intrinsic-reward-based exploration opens new avenues in automated reasoning and formal theorem proving, providing a robust framework for improving long-horizon predictions in mathematical proofs.
- The ability to integrate intermediate tactic states enhances the LLM's capacity to handle complex proofs efficiently, mitigating the inherent risk of error propagation in long proof sequences.
Practical Implications:
- This approach holds potential for enhancing automated proof-assistant tools, making them more reliable and capable in solving complex mathematical problems.
- The advancements can be applied to domains requiring rigorous formal verification such as software verification, cryptographic protocol analysis, and formal methods in hardware design.
Future Directions
The research points towards several promising future directions:
- Expanding the model's synthetic data generation and expert iteration processes to continuously refine theorem proving capabilities.
- Integrating critic models for temporal credit assignment, which could provide intermediate rewards and more granular feedback during the proof generation process, thus improving the exploitation aspect of reinforcement learning.
- Extending the application to larger scales, including theory proving across multiple interconnected theorems within comprehensive Lean files, advancing the model’s utility in real-world mathematical formalization projects.
Conclusion
DeepSeek-Prover-V1.5 represents a substantial advancement in the domain of machine learning for formal theorem proving. By implementing sophisticated techniques in reinforcement learning and Monte-Carlo tree search, this model not only sets a new benchmark but also paves the way for further innovations in automated reasoning and proof generation. The model's ability to integrate and leverage proof-assistant feedback effectively renders it a versatile tool, poised to impact both theoretical research and practical applications in formal verification and automated theorem proving.