Proving Theorems Recursively (2405.14414v1)

Abstract: Recent advances in automated theorem proving leverages LLMs to explore expanded search spaces by step-by-step proof generation. However, such approaches are usually based on short-sighted heuristics (e.g., log probability or value function scores) that potentially lead to suboptimal or even distracting subgoals, preventing us from finding longer proofs. To address this challenge, we propose POETRY (PrOvE Theorems RecursivelY), which proves theorems in a recursive, level-by-level manner in the Isabelle theorem prover. Unlike previous step-by-step methods, POETRY searches for a verifiable sketch of the proof at each level and focuses on solving the current level's theorem or conjecture. Detailed proofs of intermediate conjectures within the sketch are temporarily replaced by a placeholder tactic called sorry, deferring their proofs to subsequent levels. This approach allows the theorem to be tackled incrementally by outlining the overall theorem at the first level and then solving the intermediate conjectures at deeper levels. Experiments are conducted on the miniF2F and PISA datasets and significant performance gains are observed in our POETRY approach over state-of-the-art methods. POETRY on miniF2F achieves an average proving success rate improvement of 5.1%. Moreover, we observe a substantial increase in the maximum proof length found by POETRY, from 10 to 26.

• The paper introduces POETRY, a recursive approach that decomposes complex proofs into verifiable subgoals using a novel best-first search algorithm.
• It refines traditional methods by generating high-level proof sketches that are iteratively detailed, enhancing the search efficiency.
• Empirical tests on miniF2F and PISA datasets show improved proving success and extended proof lengths, demonstrating significant performance gains.

Analysis of "Proving Theorems Recursively"

In their paper titled "Proving Theorems Recursively," Haiming Wang et al. introduce a novel approach to automated theorem proving called POETRY (PrOvE Theorems RecursivelY). This research work addresses the limitations of existing step-by-step methods by employing a recursive, level-by-level strategy in the Isabelle theorem prover. POETRY achieves impressive gains in performance, particularly in solving longer and more complex proofs.

Methodological Innovations

POETRY's primary innovation lies in its recursive strategy for theorem proving. The method generates a high-level proof sketch for each theorem, consisting of intermediate conjectures. The detailed verification of these conjectures is deferred to subsequent levels using a placeholder tactic called 'sorry', allowing the approach to iteratively refine and solve each subgoal.

Recursive Best-First Search (BFS)

The authors introduce a recursive best-first search (BFS) algorithm that is responsible for discovering these proof sketches at each level before diving deeper into verifying intermediate conjectures. This algorithm, termed recursive BFS, iteratively generates proof steps and navigates the proof space level by level, only expanding deeper levels as necessary. This hierarchical decomposition is inspired by human problem-solving techniques, where complex problems are broken down into simpler, more manageable sub-problems.

Empirical Results

The effectiveness of POETRY is evaluated through extensive experiments on the miniF2F and PISA datasets. The results demonstrate a significant improvement over state-of-the-art methods:

• MiniF2F: POETRY achieves average proving success rate improvements of 5.1%.
• Proof Length: The method significantly increases the maximum proof length found, from 10 steps to 26 steps in the PISA dataset.

Key Findings and Implications

1. Performance Enhancements: POETRY's recursive approach allows it to effectively tackle longer proofs that traditional step-by-step methods struggle with. This is due to the method's ability to avoid becoming trapped in suboptimal or distractive subgoals.
2. Representation and Learning: The decomposition of proofs into verifiable sketches at each level improves the tractability of the search space. Rather than searching for a complete proof in one go, the recursive approach manages the exponential growth of the search space more efficiently.
3. Generalization: While the current implementation focuses on Isabelle, the methodology can be adapted to other formal proof environments like Lean, Coq, or HOL with some engineering adjustments.

Theoretical and Practical Implications

From a theoretical perspective, POETRY contributes to the foundational understanding of automated theorem proving by showcasing the benefits of a recursive problem-solving strategy. Practically, it enhances the capabilities of automated theorem provers, making them more suitable for tackling complex theorems that require extensive proof steps.

Future Directions

The research opens several avenues for further exploration:

• Integration with Other Tools: The recursive strategy could be combined with tools like Sledgehammer or Magnushammer to further boost performance.
• Generalization to Other Formal Systems: Extending the recursive methodology to other formal environments could validate its generality and stimulate improvements in those systems.
• Enhanced Heuristics for Proof Search: Developing more accurate heuristics beyond log probabilities or value functions could yield even better performance in guiding the recursive BFS.

In summary, "Proving Theorems Recursively" makes a substantial contribution to the field of automated theorem proving. The recursive, level-by-level approach of POETRY, supported by rigorous empirical validation, presents a robust framework that enhances the proving capabilities of existing automated systems, particularly for more complex theorems requiring longer proofs.