PAT-Agent: Autoformalization for Model Checking

• PAT-Agent is an end-to-end framework that autoformalizes natural language system requirements into verifiable formal models using LLM-based planning, code generation, verification, and repair.
• It decomposes the autoformalization process into distinct stages to improve transparency and reliability by introducing an interpretable intermediary plan between informal requirements and formal models.
• Experimental evaluations show that PAT-Agent achieves 100% compilation and correctness on benchmark tasks, significantly outperforming direct LLM generation methods.

Searching arXiv for the specified PAT-Agent paper and closely related work to ground the article. arXiv Search Tool Input: {"query":"PAT-Agent Autoformalization for Model Checking PAT-Agent arXiv (Zuo et al., 28 Sep 2025)", "max_results": 5} PAT-Agent is an end-to-end framework for natural language autoformalization and formal model repair that couples LLMs with formal verification to automate the construction of verifiable formal models. It is designed for settings in which natural-language system descriptions and expected properties must be translated into formal models, checked with a model checker, and iteratively corrected when the generated model diverges from the intended behavior. In PAT-Agent, a Planning LLM produces a structured model plan, a Code Generation LLM synthesizes CSP# code and assertions for the Process Analysis Toolkit (PAT), and a verification-guided Repair Loop uses counterexamples to revise the model until requirements are satisfied or a repair budget is exhausted (Zuo et al., 28 Sep 2025).

1. Problem setting and design objective

PAT-Agent addresses the application of LLMs to formal verification, a setting characterized in the source paper by the complexity of specification languages, the risk of hallucinated output, and the semantic gap between natural language and formal logic. Its primary target is autoformalization: the automatic translation of natural-language system requirements into verifiable formal models, together with iterative repair when initial generations fail verification.

A central design choice is that PAT-Agent does not perform a direct natural-language-to-code mapping. Instead, autoformalization is decomposed into planning, code generation, verification, and repair. This decomposition is intended to improve reliability and transparency by introducing an interpretable intermediate representation between informal requirements and executable formal models. The generated formalism is CSP# for the PAT model checker, and the verified properties include safety, liveness, deadlock-freedom, reachability, and LTL-style requirements (Zuo et al., 28 Sep 2025).

This structure suggests a broader methodological position: formal verification is treated not as a post hoc evaluator of LLM output, but as an active component in a closed corrective loop.

2. System architecture and formal pipeline

The PAT-Agent pipeline contains four principal components: a Planning LLM, a Code Generation LLM, the PAT model checker, and a Repair Loop. The Planning LLM analyzes the natural-language description and extracts modeling elements. The Code Generation LLM converts the resulting plan into formal code. PAT then verifies the model against user-specified properties. If discrepancies occur, the Repair Loop revises the model using model-checker feedback.

The paper presents the stages formally as

$$\mathcal{T}_{\mathrm{plan}}: \mathcal{L}_\mathrm{NL} \rightarrow \Pi \rightarrow \mathcal{P},$$

$$\mathcal{T}_{\mathrm{gen}}: \mathcal{P} \rightarrow \mathcal{M},$$

$$\mathcal{V}: \mathcal{M} \times \mathcal{Q} \rightarrow \{\mathrm{MATCH},\mathrm{MISMATCH}\}^m \times \mathcal{C},$$

$$\mathcal{T}_{\mathrm{repair}}: \mathcal{M} \times \mathcal{C} \times \mathcal{R} \rightarrow \mathcal{M}.$$

Here, natural-language specifications are mapped into prompt slots and then into a detailed plan; the plan guides formal code generation; verification yields matches or mismatches together with counterexamples; and repair transforms the model using the counterexamples and repair directives.

The significance of this formulation lies in its explicit separation of semantic interpretation from syntactic realization. The plan is JSON-serializable, auditable, and editable, which gives the pipeline a degree of inspectability that direct prompting lacks (Zuo et al., 28 Sep 2025).

3. Semantic planning and formal code generation

The Planning LLM uses structured semantic prompts to extract constants, variables, actions, guarded conditions, and processes. Prompt templates include role assignment, structured supporting context, structural guidance, and naming-consistency rules. The output is a structured plan, exemplified in the paper with JSON-style entries for constants and variables. These extracted elements are then used to produce a line-by-line natural-language annotation of the formal model, yielding a detailed plan.

An example given for a planned action is: if engineStatus is engineOn, carMotion is stop, fuel > 0, and ownerPos is inCar, then start_driving makes carMotion become drive. This illustrates the intended role of the planning stage: not merely entity extraction, but semantic organization of state, preconditions, and effects.

The Code Generation LLM consumes the detailed plan and produces valid CSP# models and PAT assertions. Its prompting is augmented with syntax cues, PAT documentation snippets, and retrieval-augmented generation based on similar plan-code pairs used as one-shot exemplars. If no close example is available, the best available exemplar is used. The paper presents representative PAT assertions such as:

#assert car deadlockfree; #define driveWithoutDriver (carMotion==drive && ownerPos!=inCar); #assert car reaches driveWithoutDriver;

This plan-guided generation process is presented as the main mechanism by which PAT-Agent enforces both syntactic correctness and semantic faithfulness relative to the natural-language description (Zuo et al., 28 Sep 2025).

4. Verification-guided repair and counterexample use

Verification is performed by the PAT model checker, which compiles the generated CSP# model and checks each assertion against the expected result, such as VALID or INVALID. When any requirement is not satisfied, PAT returns counterexample traces of the form $C = t_0 \to t_1 \to \ldots \to t_\ell$, which are then used for repair.

The Repair Loop performs fault localization over the trace, assigning higher responsibility to actions near the end of the trace or to frequently occurring actions. From this analysis it produces repair directives. The paper gives concrete examples: guard conditions may be tightened for incorrect reachability or liveness behavior, while constraints may be relaxed to resolve deadlocks. The repair prompts include both the counterexample details and these directives, and the Code Generation LLM then performs a local, patch-based revision rather than full regeneration.

The repair process iterates, typically up to 5 rounds, until all requirements are met or the repair budget is exhausted. The paper reports that this localized repair strategy improves efficiency and result stability. A notable conceptual point is that counterexamples are not treated merely as diagnostics for humans; they are elevated to machine-readable supervisory signals within the generation loop itself (Zuo et al., 28 Sep 2025).

5. Experimental evaluation and ablation findings

PAT-Agent is evaluated on 40 examples drawn from three datasets: PAT, Alloy4Fun, and UCS. On these examples, the full system achieves 100% compilation, full-pass, and average pass rates, reported as CSR, FPR, and APR all equal to 1.0. By contrast, direct-generation LLM baselines did not exceed 75% FPR, and several failed to compile consistently.

The ablation study isolates the contribution of planning and repair:

Variant	CSR	FPR	APR
Full Pipeline	1.0	1.0	1.0
w/o Repair Loop	1.0	0.75	0.8045
w/o Planning LLM	0.75	0.60	0.7594
w/o Both	0.75	0.475	0.6316

These results show two distinct failure modes. Removing the Repair Loop preserves compilation but reduces formal correctness. Removing the Planning LLM degrades both compilation and property satisfaction, with the paper noting more than 25% syntactic failure when planning is omitted.

In efficiency terms, the full pipeline has a median runtime of approximately 4.34 minutes per task. The paper further reports that planning with o3 and code generation with Claude was both fastest and most accurate, and that planning and repair times were modest with low variance. The experimental evidence therefore supports the paper’s claim that the framework improves both correctness and efficiency relative to direct LLM generation (Zuo et al., 28 Sep 2025).

6. Web interface, user interaction, and system position

PAT-Agent includes a web-based interface intended to let users, particularly non-FM-experts, describe, customize, and verify system behaviors through user-LLM interactions. The interface includes a chatbot for top-level descriptions, verified-example customization for model reuse, guided extraction and editing of constants, variables, and actions in table form, and requirement specification via drag-and-drop and selection menus. LTL expressions can be generated automatically from natural language or UI input.

The interface also exposes plan, code, and verification viewers, together with counterexample visualization and a repair dashboard. This modularity is important because it makes each transformation stage visible and editable rather than collapsing the workflow into a single opaque generation step.

The user study reported in the paper compares an experimental group using the PAT-Agent interface with a control group. The experimental group achieved assertion accuracy of 0.9958 versus 0.7500 and system accuracy of 0.9688 versus 0.6633, while also completing tasks faster. The paper states that all usability feedback was positive, and that users specifically highlighted the “modular layout and structured tables” as aiding understanding and interaction (Zuo et al., 28 Sep 2025).

PAT-Agent is therefore positioned not only as an autoformalization engine but also as an interaction framework for verification-centered modeling. Its defining characteristic is the integration of semantic planning, syntax-aware generation, model-checking feedback, and repair into a single formalization workflow.