Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs (2210.12283v3)

Abstract: The formalization of existing mathematical proofs is a notoriously difficult process. Despite decades of research on automation and proof assistants, writing formal proofs remains arduous and only accessible to a few experts. While previous studies to automate formalization focused on powerful search algorithms, no attempts were made to take advantage of available informal proofs. In this work, we introduce Draft, Sketch, and Prove (DSP), a method that maps informal proofs to formal proof sketches, and uses the sketches to guide an automated prover by directing its search to easier sub-problems. We investigate two relevant setups where informal proofs are either written by humans or generated by a LLM. Our experiments and ablation studies show that LLMs are able to produce well-structured formal sketches that follow the same reasoning steps as the informal proofs. Guiding an automated prover with these sketches enhances its performance from 20.9% to 39.3% on a collection of mathematical competition problems.

• The paper demonstrates that leveraging informal proofs improves formal theorem proving success from 20.9% to 39.3% on the miniF2F dataset.
• It introduces a three-tiered approach—drafting informal proofs, sketching formal proof outlines, and automated proving—to bridge human intuition and formal logic.
• This framework offers practical insights for automated verification and educational tools in advanced mathematics.

An Overview of Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofs

The formalization of mathematical proofs, a task essential for robust verification of complex theorems, remains a domain that demands significant expertise and is often resistant to full automation. Traditional approaches focus heavily on advanced search mechanisms to automate this task. In contrast, Jiang et al. propose a method named Draft, Sketch, and Prove (DSP), which uniquely harnesses informal proofs to guide formal theorem provers, demonstrating a promising alternative approach to this challenge. This essay reviews the DSP method and its implications, addressing both its methodological innovations and quantitative achievements.

Methodological Insights

DSP introduces a three-tiered approach:

1. Drafting Informal Proofs: The initial step involves drafting informal proofs either written by humans or generated by LLMs. This step acknowledges the wealth of informal mathematical resources available relative to formal data, which can be used to guide the formalization process.
2. Sketching Formal Proofs: These informal drafts are then converted into formal proof sketches using an autoformalizer equipped with LLMs. The proof sketches retain the structure of informal reasoning, providing a skeleton for further formal proof completion. This is crucial as it bridges informal and formal logic through a semi-formal representation, which can directly interface with theorem provers.
3. Proving with Automated Provers: The last stage employs automated provers to fill in the gaps of these proof sketches, working on intermediate conjectures that remain open in the sketches. This differs from conventional approaches, which focus on searching complete proofs without leveraging the structure provided by informal narratives.

Quantitative Achievements

The proposed DSP framework demonstrates significant improvements in the capability of solving mathematical problems by combining structured informal reasoning with formal automation tools. For instance, guiding a theorem prover with informal proofs enhances its performance from 20.9% to 39.3% on the miniF2F dataset, a collection of mathematical competition problems. This stark increase signifies the potential of informal proofs to enhance the theorem proving process, especially when facilitated by state-of-the-art LLMs such as Minerva and Codex.

Theoretical and Practical Implications

The central theoretical contribution of Jiang et al.'s work is the validation of translating informal proof strategies into formal sketches to leverage the strengths of both human intuition and formal rigor. This approach broadens the scope of problems accessible to automated proving, suggesting potential applications in verifying extensive mathematical theories that have been informal but lack formal proofs due to complexity barriers.

Practically, by shifting the focus to incorporate informal proofs, DSP provides a novel pathway for educational tools, theorem verification, and automated mathematical assistance across various domains. The approach could further be extended with advanced models and hybrid strategies that include symbolic and neural methods for enhanced performance.

Speculating on Future Developments

Future developments in AI might see the rise of more sophisticated DSP-like systems, potentially incorporating deep reinforcement learning and advanced search algorithms, to deploy even more substantial computational power towards automated theorem proving. Integration with collaborative platforms where human mathematicians can actively shape and refine informal drafts might also enhance the richness and usability of this framework in real-world applications.

Jiang et al.'s DSP framework thus represents a meaningful progression towards more approachable formal theorem proving, harnessing informal proofs to bolster formal logic systems. The successes recorded suggest a substantial potential for broader impact as the interplay between informal understanding and formal rigor continues to evolve in the landscape of computational theorem proving.