Effectiveness of LLMs in Automating Proofs with Proof Assistants

Determine the effectiveness of large language models in automating proofs for formal verification using proof assistants.

Background

The paper investigates whether LLMs can effectively automate proof generation within proof assistants. As motivation, the authors note that despite the promise of LLMs for verification, their actual effectiveness is unclear. To address this uncertainty, they conduct a case study on two mature Rocq (Coq) projects—hs-to-coq and Verdi—performing quantitative and qualitative analyses of LLM-generated proofs.

Their study examines factors such as external dependencies, context within source files, proof sizes, and project-specific differences, reporting empirical findings on where LLMs succeed and fail. The open question in the abstract frames the need for systematic evaluation of LLM performance in real verification workflows.

References

However, it is unclear how effective LLMs are in this task.

— A Case Study on the Effectiveness of LLMs in Verification with Proof Assistants  (2508.18587 - Bayazıt et al., 26 Aug 2025) in Abstract (Page 1)

For instance, how can we automate with sums in our proofs ?

— Retro-synthetic Riemannian geometry in Rocq I : Taylor series of a metric  (2609.20127 - Clemente et al., 17 Sep 2026) in Section 4, item 3 (Proof automation)

Whether classical search with small models remains competitive with large generic LLMs at equal compute is an open question, and it can only be studied if this class of system exists in an open and reproducible form.

— NanoProof: Open and Efficient Automated Theorem Proving in Lean 4  (2610.11605 - Kripner et al., 8 Oct 2026) in Section 1, Introduction