Learning to Prove Theorems by Learning to Generate Theorems (2002.07019v2)

Abstract: We consider the task of automated theorem proving, a key AI task. Deep learning has shown promise for training theorem provers, but there are limited human-written theorems and proofs available for supervised learning. To address this limitation, we propose to learn a neural generator that automatically synthesizes theorems and proofs for the purpose of training a theorem prover. Experiments on real-world tasks demonstrate that synthetic data from our approach improves the theorem prover and advances the state of the art of automated theorem proving in Metamath. Code is available at https://github.com/princeton-vl/MetaGen.

Learning to Prove Theorems by Learning to Generate Theorems

The field of automated theorem proving continues to bridge artificial intelligence with formal logical reasoning, and the paper "Learning to Prove Theorems by Learning to Generate Theorems" presents an intriguing enhancement to the research in this domain. This work addresses the prominent limitation of prior endeavors: the insufficient availability of human-written theorems and proofs necessary for supervised learning approaches to training theorem provers. The proposed method augments existing supervised learning techniques by utilizing a neural generator to automatically generate theorems and proofs, thus enriching the training dataset for theorem provers.

Key Insights

The core contribution of this paper is the introduction of a framework wherein a neural theorem generator, termed "MetaGen," produces synthetic theorems. MetaGen facilitates the training of theorem provers by generating high-quality synthetic data and advancing performance metrics in the Metamath system—a widely-used formal language in automated reasoning. This approach is methodically broken down into several components where each plays a vital role in synthesizing meaningful theorems:

1. Theorem Synthesis: MetaGen generates new theorems via forward reasoning by selecting an invocable theorem and executing substitution-based inferences. This effectively leverages existing proofs to generate coherent synthetic data.
2. Generator Training: Two primary training strategies for the generator are explored. In the presence of human proofs, imitation learning is applied. Alternatively, in situations with theorem statements but no explicit proofs, reinforcement learning is employed to maximize the similarity between synthetic and human-generated theorems.
3. Integration with the Prover: Holophrasm, an established theorem prover for Metamath, is utilized and enhanced with synthetic data. The relevance, substitution, and payoff networks are refined using a combination of human and synthetic proofs, showing measurable improvements in performance.

Experimental Results

The empirical evaluation reveals notable results across multiple configurations:

• On the Metamath knowledge bases, substantial improvements are observed in the relevance and substitution networks of the theorem prover when trained with synthetic data, versus solely human-written proofs.
• When trained on both human and synthetic proofs, Holophrasm proves more theorems in the test set than when trained on just human proofs, establishing synthetic data as a beneficial complement.

The paper demonstrates a significant advancement on the set.mm benchmark, competing with state-of-the-art systems within the constraints of the evaluation protocol. The improvements in theorem proving abilities underline the potential of integrating generative strategies in this field.

Implications and Future Directions

Practically, this work suggests increased robustness and scalability for theorem provers by diminishing their reliance on scarce human-curated datasets. Theoretical implications point towards a deepened integration of generative models and reinforcement learning techniques in logical reasoning domains.

Future expansions could explore extending this generative framework to other formal systems beyond Metamath, potentially involving interactive theorem provers such as HOL Light or Coq. Another promising avenue involves exploring more sophisticated deep learning architectures like Transformers to conduct both theorem synthesis and proving, potentially yielding even greater proof discovery capabilities.

In conclusion, this paper provides a robust methodological framework, convincing empirical validation, and a meaningful leap towards autonomous and scalable theorem proving systems.