Harnessing Code Agents for Automatic Software Verification

Abstract: Formal verification offers the strongest guarantee of software correctness, but it does not scale: the proofs demanded by interactive theorem provers such as Coq require enormous expert effort. LLMs promise to generate these proofs automatically, yet existing approaches wire a fixed, human-designed proof strategy into the system and constrain the model to follow it (retrieving premises and predicting tactics one step at a time, or splitting goals by divide-and-conquer), and still prove only a fraction of their target theorems. We show that imposing such a strategy is unnecessary and limiting. Handing the whole lemma to a general LLM code agent (for example, Claude Code), free to choose its own approach, and wrapping it in a verification harness is both simpler and more effective, achieving full coverage: every targeted lemma proved, with no failures and no Coq expert intervention. The agent writes the proofs under feedback and hard constraints from the harness that keep each one sound (accepted only when the prover's kernel closes it), complete (no obligation left unproved or silently dropped), and terminating (no divergent tactics). We evaluate this harness plus code agent along three dimensions. (1) Core logic: on Iris, the state-of-the-art separation logic for concurrent and memory-manipulating programs, Aria proves all 4,257 lemmas of the four core modules and the 217 lemmas verifying Rust's standard libraries built on it, fully automatically. (2) Comparison with prior LLM provers: on reglang, where prior provers manage barely one in eight, Aria proves all 318. (3) Generality: on iris-lean, the unfinished Lean 4 port of Iris, it proves 72 not-yet-ported lemmas, showing the approach is not specific to Coq. A state-of-the-art model (Claude Opus 4.7) can write proofs for verified software development fully and automatically.

• The paper demonstrates that harness-constrained LLM agents can fully automate verification, achieving 100% lemma coverage without expert intervention.
• It presents a layered architecture integrating model, agent, and harness modules that iteratively synthesizes proofs with high first-try success rates.
• The approach uses a closed-loop feedback system enforcing soundness, completeness, and style, paving the way for unattended, expert-grade formal verification.

Fully Autonomous Software Verification with LLM Code Agents

Problem Statement and Motivation

Despite the maturity of proof assistants such as Coq and the existence of advanced separation logics like Iris for reasoning about concurrent, mutable, and higher-order programs, formal verification of realistic software remains limited by the need for expert-driven interactive proofs. State-of-the-art systems like CompCert have demonstrated the feasibility of full formal verification, but scaling this methodology is stymied by the manual effort required for proof construction—particularly in domains involving concurrency and heap manipulation. Recent LLM-based systems have attempted automated proof synthesis but plateau at partial coverage and lower-order targets, with extensive reliance on human-designed strategies and limited reach on benchmarks reflective of real-world verification complexity.

Approach: Unconstrained Code Agents with Harness Verification

This work demonstrates that hand-engineered proof strategies are not necessary for effective automatic verification. The methodology involves handing the complete lemma statement to a general-purpose LLM code agent—specifically Claude Code (Opus 4.7)—and wrapping it in a programmatic verification harness. The code agent chooses its own proof approach, iteratively synthesizes proof candidates, and reacts to structured feedback from the harness, which serves as a correctness, completeness, and termination oracle. The harness is specified declaratively using the Harness Hook Language (HHL), streamlining integration and ensuring auditable, reusable enforcement of policy and proof constraints.

Critically, this design leverages two proof-assistant invariants:

• Soundness is kernel-enforced: LLM hallucinations or incomplete constructs are rejected by the Coq/Lean kernel.
• Agent-harness feedback loop: By transforming verification into a closed-loop repair process, the agent iteratively converges on a valid proof.

System Architecture

The architecture is factored into three layers:

• Model Layer: Stateless LLM backend (Claude Code, Codex, etc.) generating candidate proofs.
• Agent Layer: Orchestrates proof construction, session management, and context assembly; can interface with different agent backends.
• Harness Layer: Trusted verifier mediating every agent action; enforces soundness (kernel acceptance), completeness (obligation not dropped/weakened), and style/safety policies (linter, shell restrictions, resource caps).

The entire proof-generation loop executes non-interactively, is driven by declarative harness policies, and is robust to both model errors and divergent tactics.

The Harness Hook Language (HHL)

HHL supports:

• Pre- and post-action hooks with programmable blocking/rejection.
• Composite workflows for proof orchestration, error handling, and feedback-driven repair.
• Workspace scoping, session management, and precise enforcement of theorem-specific policies. HHL compiles directly to harness code targeting the Claude Code SDK, but is designed for portability to additional LLM agent runtimes.

Experimental Results

Iris Core Modules

On the four core Iris modules (algebra, bi, base_logic, program_logic):

• Total lemmas proved: 4,257 (100%)
• Failures: 0
• First-try success: 79.2%
• Mean retries per lemma: 0.51
• Maximum retries: 28 (out of 30 allowed)
• Mean model time per lemma: 321.1 s
• No Coq expert intervention required

The proofs produced were not simple reproductions of upstream proofs, indicating true synthesis. The retry/correction loop efficiently pruned incorrect strategies and eliminated divergence.

Rust standard library (RustBelt) and Downstream Verification

On RustBelt's concurrent and interior mutability libraries:

• Lemmas proved: 217 (100%)
• First-try success: 73%
• Max retries: 13

reglang (Regular-language theory library)

A syntactic, non-concurrent Coq library previously intractable for automated LLM provers:

• Lemmas proved: 318 (100%)
• First-try success: 75.5%
• Max retries: 6

Lean 4 Port (iris-lean)

On 72 unported Lean versions of Iris lemmas (no possibility of training-data leakage or overfitting):

• Lemmas proved: 72 (100%)
• First-try success: 90.3%
• Mean retries: 0.10

The approach generalized out-of-the-box to Lean 4 with equivalent completeness.

Model Comparison

Open-source models (Kimi K2.6, DeepSeek V4 Pro) proved feasible on small proofs but exhibited significantly lower one-shot rates and up to 7x slower convergence compared to Claude Opus 4.7. For large, nontrivial proofs, SOTA commercial models retained a qualitative advantage in both success rate and efficiency.

Proof Quality and Style

LLM-synthesized proofs, once accepted, were further polished by a rubric-driven agent module to align with human-expert style: compactness (try done idioms), robust structuring, and avoidance of brittle proof scripts. The style-harnessed agent matched human-written proofs on both correctness and pedagogical criteria, with only trivial feedback required for most lemma obligations.

Theoretical and Practical Implications

The principal contribution is empirical evidence that current SOTA LLM code agents, minimally orchestrated by an auditable harness, can synthesize complete, correct, and stylistically competitive proofs for the most advanced libraries in concurrent separation logic. This directly challenges previous work that hard-wired tactics, premise selection, or tree search, showing that with a suitable verification harness, such structuring is unnecessary. The harness+agent methodology also robustly enforces soundness, completeness, and style, closing loopholes where a naively powerful agent might satisfy formal acceptance by silently dropping the lemma, weakening the obligation, or failing to terminate.

Implications include:

• Fully automated synthesis is tractable at the proof engineering frontier, opening the door to unattended, push-button verification for high-value concurrent and systems software.
• The limiting factor is now specification, not proof search; agent-with-harness should be deployable upstream for formal spec generation as well.
• For increasingly capable open-source LLMs, the architecture lowers the cost of entry for community-driven verified libraries and refactoring/maintenance.
• The harness approach, encoding correctness and policy as modular infrastructure, enables future-proof adaptation to new agent ecosystems and evolving proof assistant kernels.

Discussion and Future Directions

The results eliminate the bottleneck of manual proof writing for concurrent separation logic developments like Iris, with clear implications for the broad swath of verified systems built atop such frameworks (from concurrency safety to verified VMs, distributed protocols, verified storage, and compiler backends). The locus of required intelligence shifts from human-engineered search policies to LLM strategic planning; empirical logs suggest the agents often imitate human-style analogical reasoning: reusing proof ideas, seeking similar lemmas, and refining failed attempts in context.

The next empirical frontier is automatic specification synthesis—where strong harnesses could provide the same tight feedback loop for candidate specs, moving toward end-to-end verified software pipelines. The architecture is also readily adaptable to new proof assistants, new LLMs, and, via HHL, to diverse deployment, contextual, and safety policies.

Conclusion

This work demonstrates—in the context of high-order, concurrent, stateful program verification—the full viability of harness-constrained LLM code agents for unattended, expert-grade proof synthesis. By abstracting human proof strategy and focusing on harness-enforced correctness, this approach achieves complete coverage on the most challenging formal verification artifacts presently available, with generalizability across proof assistants and LLM platforms, and with no reliance on domain-expert intervention or hand-engineered automation strategies. The implications for scaling verified software engineering and accelerating verification practice are substantial, particularly as LLMs continue to advance in capability and open-source accessibility.

Reference: "Harnessing Code Agents for Automatic Software Verification" (2607.06341)