---
title: AutoReal-Prover in Industrial Isabelle
url: https://www.emergentmind.com/topics/autoreal-prover
type: topic
---

# AutoReal-Prover in Industrial Isabelle

AutoReal-Prover is a **7B fine-tuned Isabelle theorem prover** derived from the broader **AutoReal** methodology for **LLM-driven theorem proving** on **real-world industrial-scale verification**, with **seL4-Isabelle** as the primary case study and additional evaluation on **security-related Isabelle projects from the Archive of Formal Proofs (AFP)**. It operates in a theorem-plus-context setting: the model receives a target theorem together with theorem-relevant proof context, generates a complete Isabelle proof script along with step-aligned natural-language rationales, and counts as successful only when **Isabelle accepts the proof, all subgoals are discharged, and no placeholders such as `sorry` or `oops` remain** [2602.08384].

## 1. Industrial verification setting

AutoReal-Prover is defined against a markedly different target domain from mathematics-oriented benchmarks such as miniF2F. The motivating environment is the **seL4 microkernel verification in Isabelle/HOL**, where proofs are **context-dependent**, **long-horizon**, **interactive and state-sensitive**, and **maintenance-heavy**. The seL4 proof base is reported as about **8.7 KLOC of C**, roughly **600 lines of assembly**, and around **200,000 lines of Isabelle scripts**, and the original verification required about **20 person-years** of proof effort. In this setting, theorem proving is not treated as isolated tactic prediction over short mathematical lemmas, but as proof construction inside a large, evolving verification project with strong dependence on local theory context and project-specific proof idioms [2602.08384].

The system is also motivated by deployment constraints. Prior work on industrial verification is described as often relying on **closed-source models** such as GPT-4 or other very large commercial systems, which are problematic for local deployment because of **security and confidentiality concerns**, **API usage costs**, and the practical requirement for **on-premise, lightweight deployment**. AutoReal-Prover addresses this by using a **compact 7B model** rather than a hundreds-of-billions-parameter hosted system. A common misconception is that the work is primarily a miniature adaptation of benchmark-oriented theorem proving; the paper instead frames it as an attempt to move LLM-driven proving into **industrial-scale Isabelle developments** [2602.08384].

## 2. AutoReal methodology and proof-generation model

The paper distinguishes between **AutoReal** and **AutoReal-Prover**. **AutoReal** is the methodology: a pipeline comprising **proof CoT data construction**, **CoT-based proof training**, **context augmentation**, and **proof generation with formal checking**. **AutoReal-Prover** is the concrete prover obtained by fine-tuning **Qwen2.5-Coder-7B** under that methodology for industrial Isabelle theorem proving [2602.08384].

At inference time, the system takes a **target theorem** and a **proof context** consisting of theorem-relevant definitions and lemmas. Unlike stepwise interactive provers, AutoReal-Prover is trained to generate the **whole proof in a single inference**, together with **step-by-step rationales**. The paper motivates this as an efficiency decision: repeated theorem-prover interaction can be costly in developments such as seL4. The formal viewpoint is organized around Isabelle subgoals of the form
$$
A_1 \Longrightarrow \cdots \Longrightarrow A_n \Longrightarrow C,
$$
with proof-state evolution written as
$$
G_0 \xrightarrow{a_1} G_1 \xrightarrow{a_2} \cdots \xrightarrow{a_T} G_T,
$$
and the structured reasoning trace as
$$
(x, G_0) \Rightarrow (z_1, a_1) \Rightarrow \cdots \Rightarrow (z_T, a_T).
$$
Here \(a_t\) denotes a proof command and \(z_t\) the aligned rationale for that command [2602.08384].

This architecture implies another important clarification. AutoReal-Prover is **not** a search-based interactive prover in the style of tactic-by-tactic best-first expansion. Proof verification occurs **after generation**: a candidate proof script is inserted in place of the original proof and then checked by Isabelle. The system allows **up to 256 proof attempts per theorem**, and a theorem is considered solved if at least one attempt verifies. The paper does not provide a separate retrieval score, ranking formula, or pass@\(k\) estimator; the operational success criterion is simply verifier acceptance within the bounded attempt budget [2602.08384].

## 3. Chain-of-thought proof training

The central methodological novelty is **step-level chain-of-thought supervision** extracted from real Isabelle proof traces. The source traces come from **FVELER**, which records per-step proof states before and after each proof command. For each individual proof step, the construction pipeline packages four ingredients: the **source lemma**, the **proof state before applying the step**, the **proof step itself**, and the **proof state after applying the step**. A model is then prompted to explain the reasoning of that single step in a concise technical paragraph, focusing on methods used, proof-state change, and the role of the step in the larger proof [2602.08384].

The resulting dataset contains about **200k step-level CoT instances** built from seL4 proofs. These instances are then folded into full-proof instruction-tuning data whose input is the theorem statement plus proof context and whose output is the entire proof sequence interleaved with rationales. The intended effect is twofold: first, the model is exposed to why a command is applicable in a given proof state rather than merely seeing final proof text; second, the resulting prover can emit explanations aligned with the formal derivation [2602.08384].

The paper also imposes a strict anti-leakage protocol. Proof steps associated with evaluation theorems are excluded from CoT data construction, so the CoT dataset contains **no proof steps from the evaluation targets**. This is significant because a superficial reading might suggest that the model is trained directly on the very proofs it is later evaluated against. The paper explicitly rules that out for the benchmark theorems [2602.08384].

## 4. Context augmentation

The second major component is **context augmentation**. In large Isabelle developments, the theorem statement alone frequently underdetermines the proof strategy because essential information resides in local definitions, refinement lemmas, or project-specific infrastructure. AutoReal therefore supplies the model with a compact **proof context** consisting of theorem-relevant definitions and lemmas rather than entire transitive theory closures [2602.08384].

This context is **not automatically retrieved** in the current implementation. It is **explicitly provided by the verifier**, and in evaluation it is kept consistent with the lemmas and definitions used in the corresponding human-written proofs. That design isolates theorem proving from premise-selection noise, but it also means the pipeline is not yet end-to-end automated. A common misconception is that AutoReal-Prover solves industrial Isabelle proving together with premise retrieval; the paper instead evaluates the theorem-plus-context problem, with context discovery left open [2602.08384].

The role of context is illustrated by the running example `lec_ex_cap_to[wp]`, whose proof depends on the definition `lookup_extra_caps_def` and the traversal lemma `mapME_set`. With that local context available, the model can unfold the function and apply weakest-precondition reasoning. Without it, the paper argues, both humans and LLMs would struggle to recover the proof structure. Context augmentation is therefore not presented as optional prompt engineering, but as a structural requirement for proving in large Isabelle verification projects [2602.08384].

## 5. Empirical performance

The main evaluation covers two settings: **seL4 Important Theories** and three **security-related AFP projects**. On seL4, the benchmark contains **660 theorems** drawn from one designated theory in each of the **10 seL4 proof categories**: **access-control**, **asmrefine**, **bisim**, **capDL-api**, **crefine**, **drefine**, **infoflow**, **invariant-abstract**, **refine**, and **sep-capDL**. On AFP, the evaluation covers **451 theorems** across **CRYSTALS-Kyber_Security**, **RSAPSS**, and **Elliptic_Curves_Group_Law** [2602.08384].

| Evaluation set | Theorems | Success rate |
|---|---:|---:|
| seL4 Important Theories | 660 | 51.67% |
| AFP security-related projects | 451 | 53.88% |
| `Syscall_AI.thy` ablation, base model | 105 | 0.95% |
| `Syscall_AI.thy` ablation, CoT only | 105 | 22.86% |
| `Syscall_AI.thy` ablation, context only | 105 | 5.71% |
| `Syscall_AI.thy` ablation, full method | 105 | 54.29% |

The headline seL4 result is **341/660 = 51.67%**. Category-wise, the system reaches **89/123 = 72.36%** on **capDL-api / `KHeap_DP.thy`**, **57/105 = 54.29%** on **invariant-abstract / `Syscall_AI.thy`**, **101/233 = 43.35%** on **infoflow / `Noninterference.thy`**, **13/46 = 28.26%** on **crefine / `Refine_C.thy`**, **1/1 = 100%** on **asmrefine / `SEL4GraphRefine.thy`**, and **0/1 = 0%** on **sep-capDL / `AbstractSeparation_SD.thy`**. The paper compares the overall seL4 result to a previously reported **27.06%** from **Selene**, while noting that the comparison is not fully apples-to-apples because Selene used **340 sampled seL4 theorems**, whereas AutoReal-Prover evaluates **all theorems** in the selected Important Theories [2602.08384].

The attempt-budget sensitivity shows that the success rate saturates well before the maximum budget: on seL4 the system reaches **37.12%** with **5 attempts**, **46.36%** with **64 attempts**, **50.00%** with **128 attempts**, and **51.67%** with **256 attempts**. On AFP, the corresponding success rates are **36.36%**, **51.00%**, **52.55%**, and **53.88%**. The ablation on `Syscall_AI.thy` is particularly diagnostic: the **base model** solves **1/105 = 0.95%**, **CoT-based proof training** alone reaches **24/105 = 22.86%**, **context augmentation** alone reaches **6/105 = 5.71%**, and the **full method** reaches **57/105 = 54.29%**. This makes the complementarity of CoT training and context augmentation one of the clearest empirical claims in the paper [2602.08384].

## 6. Generated proofs and explanation fidelity

AutoReal-Prover is not evaluated only by theorem-level success. The paper also analyzes the structure of the successful proofs and the quality of the aligned rationales. Among the **341** successfully proved seL4 theorems, **93.55%** of generated proofs differ from the original human-written proofs, while the average number of applied methods per proof is similar: **6.018** for generated proofs versus **6.123** for human proofs. The system therefore usually does not reproduce the original script verbatim; it often constructs an alternative proof with comparable apparent tactical complexity [2602.08384].

The qualitative examples reinforce that point. For `thread_set_has_no_reply_cap`, the generated proof unfolds `has_reply_cap_def`, explicitly applies `hoare_pre`, and then uses `wpsimp` with general Hoare/WP lemmas, whereas the human proof starts with `clarsimp` and uses the more specialized lemma `thread_set_cte_wp_at_trivial`. Both scripts verify, but they expose different decompositions of the same obligation. This supports the interpretation that the model is synthesizing Isabelle proofs rather than merely replaying memorized tactic sequences [2602.08384].

The rationales were manually analyzed over **670 proof steps and aligned rationales** from the successful seL4 proofs. The major source of explanation error is the terminal command `done`: **84** `done` rationales are reported as irrelevant or incorrect. Outside `done`, only **10** additional inaccurate rationales are observed, yielding a reported non-`done` rationale inaccuracy rate of **1.49%**. The remaining errors involve minor mismatches for `by simp`, overly generic explanations for `apply auto`, and spurious references under `apply wpsimp+`. This matters because it shows that explanation generation is largely accurate for substantive proof steps, but not uniformly reliable for semantically underspecified terminal automation [2602.08384].

## 7. Relation to surrounding automated proving and open issues

AutoReal-Prover occupies a distinct design point within recent automated proving systems. In **Isabelle/HOL**, **AUTO2** is an embedded **saturation-based heuristic prover** implemented as an Isabelle tactic, with best-first search over derivable propositions and user-specified proof steps rather than LLM-based proof generation [1605.07577]. In **Lean 4**, **REAL-Prover** is a **retrieval-augmented stepwise theorem prover** with best-first search over proof states and semantic premise selection over Mathlib, reporting **23.7%** on **ProofNet** and **56.7%** on **FATE-M** [2505.20613]. **ImProver** addresses a different task—**automated proof optimization**—by rewriting existing Lean proofs to improve criteria such as length or readability rather than proving industrial Isabelle theorems from scratch [2410.04753]. In **Rust verification**, **AutoVerus** is a verifier-in-the-loop proof-synthesis system for Rust+Verus that proves **137 of 150 tasks (91.3%)** by staged generation, refinement, and debugging [2409.13082]. In **Coq**, **QEDCartographer** guides search with a learned state-value heuristic and proves **21.4%** of the CoqGym test set [2408.09237]. Against that background, AutoReal-Prover is characterized by **industrial Isabelle/seL4 evaluation**, **whole-proof generation in a single inference**, **step-aligned CoT training**, and **explicit proof context supplied by the verifier** rather than interactive state-space search [2602.08384].

Two misconceptions are particularly important. First, AutoReal-Prover is not an end-to-end autonomous industrial prover, because **theorem-relevant proof context is not automatically extracted** in the current implementation. Second, it is not an interactive stepwise system: the proof is generated as a whole and only then checked by Isabelle. These choices help explain both the strengths and the current limits of the method [2602.08384].

The open problems are correspondingly concrete. The paper identifies **automating theorem-relevant proof-context extraction** as a major future direction, because the current theorem-plus-context setting leaves premise discovery outside the automated pipeline. It also leaves substantial headroom in proof coverage, with success rates of **51.67%** on seL4 and **53.88%** on AFP. Finally, while the model is argued to support **lightweight local deployment** because it is a **compact 7B model**, the paper does not provide detailed runtime, latency, or memory measurements. A plausible implication is that AutoReal-Prover presently demonstrates that industrial Isabelle proof generation is feasible under explicit context supply, but not yet that industrial-scale proof maintenance has become fully automatic [2602.08384].

Source: https://www.emergentmind.com/topics/autoreal-prover