IsabeLLM: LLM-Enhanced Isabelle Verification

• IsabeLLM is a framework that integrates LLMs with Isabelle/HOL to guide proof search and repair using a verifier-in-the-loop architecture.
• It leverages error feedback, Sledgehammer repair, and stepwise planning to automate theorem proving, successfully verifying blockchain consensus properties.
• Empirical results show effective proof automation despite LLM unreliability, with ongoing improvements aimed at parallel search and broader applications.

Searching arXiv for the specified IsabeLLM papers and closely related context papers. IsabeLLM is the name used for Isabelle/HOL systems that place a LLM inside a verifier-in-the-loop proving workflow, with Isabelle remaining the final authority on proof correctness. In the current literature, the label covers at least two closely related formulations: a general-purpose interface between Isabelle and an LLM, demonstrated on mechanized verification of Bitcoin’s Proof of Work consensus protocol (Jones et al., 12 Jan 2026), and an LLM-powered Isabelle/HOL theorem prover aimed at fully automatic proof synthesis through stepwise search and higher-level proof planning (Hou, 8 Jan 2026). Taken together, these works define IsabeLLM as a family of semi-automated or fully automated proof-synthesis architectures in which LLMs generate candidate proof actions, while the trusted proof assistant accepts, rejects, repairs, or completes them.

1. Terminological scope and problem setting

One paper defines IsabeLLM as “an interface between the Isabelle proof assistant and an LLM,” designed to extract lemmas and proofs from an Isabelle theory file, send the proof context to an LLM, inject the returned proof candidate back into Isabelle, and iterate with Sledgehammer-based repair until success or budget exhaustion (Jones et al., 12 Jan 2026). A second paper, styled “Isabellm,” describes an “LLM-powered theorem prover for Isabelle/HOL that performs fully automatic proof synthesis,” combining bounded proof search with a higher-level proof planner (Hou, 8 Jan 2026).

Formulation	Primary objective	Characteristic mechanism
IsabeLLM (Jones et al., 12 Jan 2026)	Assist and automate proofs in Isabelle	LLM proof attempts plus Sledgehammer repair
Isabellm (Hou, 8 Jan 2026)	Fully automatic proof synthesis	Stepwise prover plus proof planner

Both formulations address the same structural difficulty: higher-order theorem proving in Isabelle/HOL involves large and branching proof-search spaces, thousands of available lemmas, context-sensitive proof modes, and nontrivial choices among induction, simplification, premise selection, and proof methods. The shared response is to treat the LLM as a proposal engine rather than a source of truth. Isabelle’s trusted kernel remains the final verifier, so soundness is preserved even when the model proposes invalid tactics, malformed proof fragments, or hallucinated facts. This suggests that the core contribution of IsabeLLM is not unchecked natural-language reasoning, but search guidance under formal verification constraints.

2. Core architecture: verifier-in-the-loop proof generation

In the blockchain-oriented formulation, the architecture is an explicit feedback loop. A user provides a .thy theory file and a ROOT file; IsabeLLM asks Isabelle to build the theory; if a lemma is unproven, Isabelle’s error output identifies the failing statement; the lemma and its theory context are sent to the LLM via API; the returned proof is injected into the theory file; Isabelle rebuilds the file; if the proof fails, IsabeLLM calls Sledgehammer on the problematic lines; if Sledgehammer does not solve everything, the remaining proof state and error messages are sent back to the LLM; and the loop repeats (Jones et al., 12 Jan 2026). The intended division of labor is explicit: the LLM handles high-level proof structure, while Isabelle and Sledgehammer handle low-level proof completion.

The fully automatic formulation decomposes the same problem into two cooperating layers: a stepwise prover and a higher-level proof planner (Hou, 8 Jan 2026). The stepwise prover models a proof script as

$S = \langle s_0, s_1, \dots, s_k \rangle$

where $s_0$ is the lemma declaration and later $s_i$ are proof commands. Beam states are tuples

$(\sigma, S, h, n),$

with $S$ the accepted proof prefix, $\sigma$ a score, $h$ the textual proof-state hint from print_state, and $n$ the number of remaining subgoals. Search is bounded by beam width $B$, depth $D$, and a wall-clock timeout. The planner works at the Isar level, generating structured outlines, filling holes, repairing blocks, and regenerating larger fragments when local repair fails.

The architectural commonality is stronger than the implementation differences. Both systems rely on repeated interaction with Isabelle, both use proof-state feedback to guide further generation, and both reject the idea that LLM output should be trusted without kernel validation. A plausible implication is that IsabeLLM is best understood as a search-and-repair framework for formal proof engineering rather than as a standalone generative prover.

3. Search, prompting, retrieval, and repair mechanisms

The blockchain-verification system uses two prompt types. The initialisation prompt includes the theory file so far, the target lemma, and a request for raw Isabelle code only. The error prompt includes the current proof state, the exact failed line, the Isabelle error message, and a request to amend the proof (Jones et al., 12 Jan 2026). Error handling is specialized: the tool detects Sorry, failed proofs, undefined facts or methods, lexical or syntax errors, and timeouts, the latter often associated with tactics such as metis or blast. The implementation tracks chat history and resets it after each successfully proven lemma to keep context manageable.

The fully automatic theorem prover makes prompting more granular. It constrains the LLM to short commands in step mode, such as apply ..., and finish mode, such as by simp, by auto, by (metis ...), or done, and sanitizes outputs by removing code fences, bullets, numbering, and extraneous text (Hou, 8 Jan 2026). Candidate tactics may then be reranked by a learned model

$s_0$0

with adjusted score

$s_0$1

The feature vector includes search-context features, structural goal flags such as is_listy, is_natty, is_sety, has_quantifier, and is_boolean, tactic-prefix encodings, and premise-interaction features. Training targets take the form

$s_0$2

where $s_0$3 iff Isabelle accepts the step.

Premise handling is also explicit. The system uses a two-stage retrieval pipeline with a recall-oriented select stage and an optional rerank stage, and defines bi-encoder similarity as

$s_0$4

At the planner level it uses micro-RAG over AFP, combining context-derived hints from the current proof state with lexicon-derived hints mined offline from AFP and Isabelle corpora more generally. Counterexample handling is equally central: Quickcheck and Nitpick can prefilter candidate steps, while the planner employs a bounded CEGIS-style repair loop with counterexample-guided hints, deduplication of failed candidates via SHA1 fingerprints, and staged repair over have/show/obtain blocks, case blocks, and subproofs. The search policy is beam-based rather than Monte Carlo tree search, because the latter was tested but not used due to slower performance in small-step theorem proving.

4. Blockchain consensus verification and the n-ary Proof of Work model

A major application of IsabeLLM is formal verification of blockchain consensus, motivated by the role of consensus protocols in allowing distributed nodes to agree on blockchain state in a potentially adversarial environment (Jones et al., 12 Jan 2026). The stated risks of incorrect consensus include 51% attacks, double spending, and chain reorganization. The paper emphasizes that “test and patch” remediation is often inadequate because consensus bugs may require a hard fork, which is disruptive and controversial, and smart contracts are often immutable once deployed. Formal verification is therefore framed as valuable because it can provide correctness-by-construction, even though it is expensive and requires specialized expertise.

The case study develops a novel mechanized model of Bitcoin’s Proof of Work consensus protocol in Isabelle, extending earlier work by generalizing the blockchain structure from a binary tree to an n-ary tree. The point of the generalization is representational fidelity: the n-ary model allows an arbitrary number of forks from a block and therefore a more realistic account of blockchain branching. The cost is proof complexity, since inductive arguments must range over many branches rather than only left and right cases.

The verified consensus statement is given as:

$s_0$9

The interpretation supplied for this theorem is the common prefix style consensus property: if $s_0$5, and both $s_0$6 and $s_0$7 are longest chains in the initial state, then the two chains agree on their first $s_0$8 blocks. The paper frames this as a safety property and states that it follows from Bitcoin Backbone-style assumptions: majority honest network, synchronisation, omission of chain quality under the majority-honest assumption, and establishment of consensus through the common prefix property.

The n-ary verification uses 16 lemmas, including subtree_height, height_mono, obtain_max, foldr_max_eq, branch_height, sub_longest, sub_branch, weaken_distance, weaken_depth, common_prefix, height_add in mining and honest variants, check_add in mining and honest variants, bounded_check, and consensus. The automated proofs cover height and monotonicity lemmas, longest-chain reasoning, branch and subtree lemmas, weakening lemmas for distance and depth, mining and honest node update lemmas, boundedness checks, and the final consensus theorem.

5. Empirical behavior and proof-effort results

For the n-ary Bitcoin PoW verification, the authors tested each of the 16 lemmas 10 times with a maximum of 5 iterations per attempt (Jones et al., 12 Jan 2026). The reported outcome is that IsabeLLM was able to prove each lemma multiple times and to generate correct proofs for every non-trivial lemma in the verification. Some lemmas, such as subtree_height, were often solved in one iteration; larger lemmas required more iterations and showed more variability; branch_height and bounded_check were among the hardest; and the final consensus theorem was successfully proved in all 10 attempts. The summary figures given include subtree_height: 10 successful attempts, average 1 iteration; height_mono: 10/10, average 1 iteration; common_prefix: 6 successful attempts, average 1.5 iterations; branch_height: 3 successful attempts; bounded_check: 7 successful attempts; and consensus: 10 successful attempts, average 1 iteration.

The proof-complexity comparison uses Lines of Proof (LoP) for the binary-tree and n-ary-tree models. The key observation is that the n-ary model is substantially more complex and that many lemmas require many more lines of proof, with total proof effort increasing from 175+193 LoP in the binary-tree model to 395 LoP in the n-ary model. The notation x+y denotes symmetric proof parts repeated across cases.

The fully automatic theorem-proving formulation reports a different kind of empirical result. Rather than a broad benchmark table, it emphasizes qualitative cases in which the system can prove certain lemmas that defeat Isabelle’s standard automation, including Sledgehammer (Hou, 8 Jan 2026). It also stresses deployment constraints: the system is designed to run on consumer-grade hardware, specifically tested on a 2021 MacBook Pro with an M1 Pro and 32 GB RAM, and supports local LLMs through Ollama as well as API or CLI models such as Gemini CLI. This establishes a distinct evaluation axis: not only proof capability, but proof capability under modest hardware assumptions.

6. Limitations, misconceptions, and future directions

The first recurring limitation is LLM unreliability. The blockchain-oriented system reports syntax mistakes, hallucinated facts or methods, and looping on the same failed proof step; DeepSeek R1 via OpenRouter was sometimes slow and occasionally returned empty outputs; and Sledgehammer could struggle with some intermediate steps, consume significant memory, and in remote usage not always provide counterexamples like nitpick (Jones et al., 12 Jan 2026). The paper also states that some steps that were hard for Sledgehammer in Isabelle2022 would succeed reliably in Isabelle2025, indicating that part of the observed behavior depends on the underlying Isabelle release rather than on the LLM layer alone.

A second limitation concerns scope. IsabeLLM does not generate the theorem statement or the full formal model automatically in the blockchain setting; the user must provide the theory structure and property to prove. This directly excludes a common misconception that LLM-assisted theorem proving automatically replaces the modeling phase. The system automates proof search and proof repair over supplied formal artifacts; it does not eliminate the need for specification design.

The fully automatic theorem prover identifies a different bottleneck: fill-and-repair is hard (Hou, 8 Jan 2026). Even with effective-goal extraction, staged repair, CEGIS-style loops, whole-proof regeneration, and counterexample-guided hints, state-of-the-art LLMs such as GPT 5.2 Extended Thinking and Gemini 3 Pro still struggle to reliably implement the intended fill-and-repair mechanisms with complex algorithmic designs. The difficulty is not merely isolated proof failure; it lies in maintaining a large syntax-sensitive, indentation-sensitive codebase whose logic must respect existing APIs and context-dependent Isabelle proof modes. The paper therefore suggests that improvement will likely require better training data, more symbolic abstraction, or tighter coupling between planner and stepwise prover, rather than simply using a stronger model.

The future directions named in the blockchain-verification paper include parallel proof-tree search, using multiple LLMs, compatibility with newer Isabelle releases, retrieval-augmented generation or static prompt templating, application beyond blockchain, and fine-tuning on proof corpora such as AFP (Jones et al., 12 Jan 2026). In aggregate, these proposals point toward a broader research program: semi-automated formal verification in which LLMs provide heuristic search bias, retrieval support, and repair proposals, while Isabelle provides trusted semantics and final acceptance.