---
title: Multi-Agent Autoformalization
url: https://www.emergentmind.com/topics/multi-agent-autoformalization
type: topic
---

# Multi-Agent Autoformalization

Multi-agent autoformalization is the use of multiple coordinated, role-specialized agents—typically large language model agents coupled to formal tools—to transform informal artifacts into machine-checkable or executable formal objects. Across the recent literature, the inputs range from natural-language mathematical statements, LaTeX research manuscripts, legal provisions, and system descriptions to interaction scenarios and task descriptions; the outputs range from Lean 4 or Isabelle/HOL declarations to Prolog game rules, CSP# models, GDPR rule trees, and verified multi-agent workflow specifications. The defining methodological shift is from monolithic prompting to decomposed pipelines in which generation, retrieval, proof checking, critique, repair, scheduling, and human review are distributed across distinct agents or coordination mechanisms [2412.08805] [2510.08988] [2509.23675] [2602.16554] [2604.14607] [2606.09674].

## 1. Scope of the field

Multi-agent autoformalization originally names the translation of informal mathematics into formal mathematics, but the current literature uses the same paradigm for a broader class of formalisms. MASA defines autoformalization as turning a mathematical statement in informal language into a fully machine-checkable formal object such as an Isabelle/HOL or Lean4 definition [2510.08988]. GAMA applies the idea to multi-agent interaction scenarios, autoformalizing natural-language descriptions of game-theoretic situations into executable logic programs defining game rules [2412.08805]. PAT-Agent targets formal verification rather than proof assistants, mapping natural-language system descriptions into verifiable CSP# models for the Process Analysis Toolkit [2509.23675]. GDPR auto-formalization maps articles into scenarios, formal rules, and atomic facts, with deterministic evaluation and human verification [2604.14607]. Meta-Agent and MetaAgent extend the notion further: one formalizes task descriptions into a verified directed acyclic graph of agent specifications, while the other generates a finite state machine controlling a multi-agent workflow [2605.25233] [2507.22606].

The field therefore spans at least three distinct formal targets. One target is proof-assistant code, exemplified by MASA, MerLean, Trellis, LeanMarathon, AutoformBot, Agent Hunt, and the tensor-network formalization project [2510.08988] [2602.16554] [2606.09674] [2606.05400] [2605.29955] [2603.06737] [2607.07857]. A second target is executable or model-checkable artifacts, such as Prolog game descriptions and PAT models [2412.08805] [2509.23675]. A third target is structured formal representations whose purpose is verification-centered reasoning rather than theorem proving per se, such as GDPR rule trees, contract-bearing DAG nodes, and FSM state specifications [2604.14607] [2605.25233] [2507.22606].

This breadth corrects a common oversimplification: multi-agent autoformalization is not confined to translating textbook mathematics into Lean. The published systems instead treat autoformalization as a general bridge from ambiguous natural language to rigid, tool-consumable representations, with the exact formal substrate determined by the downstream verifier or executor [2510.08988] [2509.23675] [2605.25233].

## 2. Recurring architectural patterns

A recurrent architecture is staged decomposition into narrowly scoped agents. GAMA specifies six stages: prompt engineering and LLM augmentation of generative agents; natural-language interpretation of interaction scenarios; automatic translation into executable logic programs; solver-based syntactic validation; iterative, tournament-based runtime testing; and exact semantic validation when ground truth is available [2412.08805]. MASA similarly decomposes the pipeline into an Agent Manager, AutoformalizationAgent, HardCritiqueAgent, SoftCritiqueAgent, ImportRetrievalAgent, FormalRefinementAgent, and InformalRefinementAgent, with theorem provers and retrievers treated as first-class components [2510.08988]. PAT-Agent uses a four-module pipeline comprising a Planning LLM, a Code-Generation LLM, a Model-Checker, and a Repair Loop [2509.23675]. MerLean organizes the workflow as Statement Extraction, Formalization, Verification, and Informalization agents, supported by lean-lsp-mcp, leansearch or loogle, and a Faithfulness Checker [2602.16554].

A second recurring pattern is explicit orchestration. Some systems use centralized planners. Meta-Agent constructs a DAG \(G=(V,E)\) of agent specifications with explicit input contracts, output contracts, and verification criteria, then dispatches subtasks topologically in an execution phase gated by verification [2605.25233]. AutoformBot uses a long-lived Orchestrator, mid-level managers such as a Trace Analyzer and Supervisor, and low-level Workers and Reviewers operating through a shared Task DAG, scheduler, merge queue, and git worktrees [2605.29955]. LeanMarathon uses a two-stage orchestrator: Stage 1 adversarially stabilizes target fidelity, and Stage 2 discharges the proof DAG from dynamic leaves upward in parallel CI-gated rounds [2606.05400].

Other systems deliberately reduce or avoid centralized planning. Agent Hunt distributes workload through a simulated bounty-based marketplace in which four coding agents lock theorems, place sub-bounties on intermediate lemmas, and compete or cooperate through a shared Git repository rather than a static central planner [2603.06737]. Trellis also separates authority from creativity: a deterministic Rust kernel governs edits, validation, and checkpoints, while Worker, Verifier lanes, Reviewer, Checker Server, and Audit Agent each have sharply bounded powers [2606.09674]. This suggests that “multi-agent” in the literature denotes not a single canonical control topology but a family of coordination regimes ranging from orchestrated DAGs to marketplaces and process-semantic kernels.

A third pattern is specialization by failure mode. MASA distinguishes hard critiques for syntax and type correctness from soft critiques for semantic alignment [2510.08988]. LeanMarathon distinguishes Blueprinter, Target-Reviewer, Worker, and Refiner contracts [2606.05400]. The tensor-network system distinguishes leanSearch, lean, leanSimplifier, leanBlueprint, reviewer, Orchestrator, and Human Supervisor [2607.07857]. The design principle is consistent: subproblems such as retrieval, proof writing, dependency synchronization, semantic auditing, and local repair are treated as distinct tasks rather than as a single prompting problem.

## 3. Formal representations and intermediate objects

The formalism produced by multi-agent systems is typically mediated by constrained intermediate representations. MASA states autoformalization as a function
\[
\mathcal{A}\colon \mathsf{NL}\to\mathsf{Formal},
\qquad
\phi=\mathcal{A}(s),
\]
and refines it as a composition of agent functions:
\[
\mathcal{A}
=
\bigl(\text{InformalRefinement}\circ\text{SoftCritique}\circ\text{FormalRefinement}\circ\text{HardCritique}\circ\text{Autoformalization}\bigr)(s).
\]
No new grammar is introduced; instead, agents rely on existing proof-assistant grammars and LLMs to generate well-formed terms [2510.08988].

GAMA formalizes a representative \(2\times2\) simultaneous-move game with players \(P=\{1,2\}\), strategies \(S_i=\{C,D\}\), payoff functions \(u_i:S_1\times S_2\to\mathbb R\), and Prisoner’s Dilemma constraints \(T>R>P>S\). The natural-language scenario is translated into a Prolog program with two modules: game-independent rules in Situation Calculus style and game-dependent clauses such as `initial/1`, `legal/2`, `effect/3`, and `payoff/...` predicates [2412.08805]. In this setting, autoformalization does not end at syntax; it yields an executable specification that can be queried and simulated.

PAT-Agent makes the intermediate representation explicit. The Planning LLM produces a JSON-like plan
\[
P =
\{
\texttt{constants: [...]},\;
\texttt{variables: [...]},\;
\texttt{actions: [...]}
\},
\]
where each action contains fields such as `name`, `guard`, and `updates`. The Code-Generation LLM then maps this plan, together with a syntax cheat-sheet and a retrieved exemplar, into CSP# [2509.23675]. The intermediate plan functions as a stable interface between semantic extraction and code generation.

GDPR auto-formalization likewise uses a compact formal object. A rule is a 4-tuple \(R=(p,op,C,E)\), where \(p\) is the target predicate, \(op\in\{\text{ALL},\text{ANY}\}\), \(C\) is the set of condition predicates, and \(E\) is the set of exception predicates. Evaluation is defined by a closed-world assumption with exception priority:
\[
\mathrm{eval}(R,F)=\bot \text{ if }\exists e\in E:e\in F,
\]
and otherwise by whether all or any required conditions appear in the fact set \(F\) [2604.14607]. The representation is intentionally simple, but the paper emphasizes that legal nuance is not thereby eliminated.

Workflow-generating systems formalize the agents themselves. Meta-Agent represents a constructed multi-agent system as a DAG \(G=(V,E)\), where each node \(v\) carries an input contract \(I(v)\), output contract \(O(v)\), and verification criterion \(C(v)\) [2605.25233]. MetaAgent instead uses a finite state machine \(M=(E,S,s_0,F,\delta)\), where each state specifies an agent, a subtask instruction, a verifier, and listeners [2507.22606]. MerLean adds another important intermediate object: a dependency-ordered JSON array extracted from LaTeX, where each statement record includes fields such as `id`, `type`, `latex`, `deps`, and `sketch` [2602.16554]. A plausible implication is that typed, schema-constrained intermediates are one of the main mechanisms by which these systems reduce drift between informal text and formal output.

## 4. Verification, critique, and repair

Verification is the central organizing principle of multi-agent autoformalization. In GAMA, generated Prolog code is subjected to solver-based syntactic validation through dry-run queries such as `listing(initial), listing(legal), listing(final)`. Parse errors, undefined predicates, and arity mismatches are caught, traces are returned to the LLM, and self-correction proceeds for up to 5 attempts [2412.08805]. Syntactic validity is then followed by iterative tournament-based runtime testing, using “clone mode” or “round-robin mode,” and by exact semantic validation against a known payoff matrix and domain constraints such as \(T>R>P>S\) when ground truth is available [2412.08805].

MASA formalizes the distinction between low-level and high-level verification. HardCritiqueAgent invokes Isabelle or Lean4 to check syntax and type correctness, ImportRetrievalAgent supplies missing imports by retrieval, FormalRefinementAgent repairs theorem-prover errors, SoftCritiqueAgent uses an LLM judge for semantic aspects such as alignment faithfulness, and InformalRefinementAgent revises code in response to those judgments [2510.08988]. The framework’s iterative self-refinement alternates between hard critiques and soft critiques, making explicit that passing the prover is not equivalent to faithfully capturing the informal statement.

PAT-Agent uses model checking rather than theorem proving as the arbiter. The generated model \(M_k\) is checked against properties \(Q\), producing verdicts \(s_i\in\{\text{MATCH},\text{MISMATCH}\}\) and, on mismatch, counterexample traces \(C_k\). Repair instructions are then built from the trace, using heuristics based on locality of fault and frequency of occurrence, and a repair LLM produces \(M_{k+1}\) [2509.23675]. GDPR auto-formalization combines deterministic evaluation with automated and human verification: the RuleTreeEvaluator computes a boolean label from the rule tree and facts, four verifier agents score scenario quality, representational correctness, logical consistency, and legal faithfulness in \([0,100]\), and only samples with average score at least \(70\) proceed to human validation [2604.14607].

Several systems push verification further into system design. Meta-Agent distinguishes construction-time verification from execution-time verification and introduces a three-level error attribution mechanism separating local, upstream, and structural failures, each with its own recovery strategy [2605.25233]. Trellis runs three verifier lanes—substantiveness, correspondence, and soundness—on every changed node, stores fingerprints, and reopens a lane whenever later edits invalidate the relevant fragment; after theorem-stating, the human operator must ratify the semantic closure of the targets before proof-formalization proceeds [2606.09674]. LeanMarathon imposes seven structural CI checks, including compilation, metadata well-formedness, prose–formal parity, acyclicity, lemma-closeness, and the absence of hidden axioms or extra imports [2606.05400]. The literature therefore rejects a common misconception that verification in autoformalization is synonymous with syntax checking. In these systems, verification can include type checking, semantic critique, runtime simulation, counterexample-guided repair, legal review, structural consistency checks, and human ratification.

## 5. Scaling regimes and coordination at large scale

Large-scale multi-agent autoformalization has produced several distinct scaling strategies. Agent Hunt uses a marketplace. Four autonomous coding agents—Alice, Bob, Charlie, and Dave—operate in local sandboxes over a shared Git repository containing a foundational library and an `algtop.tex` file with 230 definitions and 393 top-level theorems, all admitted initially and annotated with simulated USD bounties. Locks require a 10% deposit, last up to 24 h, and are capped at 10 concurrent locks; agents can also place sub-bounties on intermediate lemmas [2603.06737]. Over Feb 16 20:00 to Feb 19 11:00, the library grew from about 19,000 normalized lines to about 121,000 lines, corresponding to about 39,000 lines/day across agents; the paper reports a roughly \(5\times\) speedup in line growth relative to a preceding single-agent project and thematic specialization across algebraic topology subfields [2603.06737].

AutoformBot scales through hierarchy and throughput. The abstract reports a multi-agent system that orchestrates thousands of LLM agents over 26 open-access textbooks, producing Atlas with over 45,000 Lean 4 declarations and 500 thousand lines of code [2605.29955]. The detailed evaluation reports 4,007 target statements, 2,855 successfully formalized targets, approximately 71.3% completion, about 484,000 lines of Lean 4, and a proof-success rate of about 85% among attempted lemmas compiling without `sorry` or hidden axioms [2605.29955]. The scheduler uses a dependency-aware ready queue over a task DAG, racing multiple workers per task when useful, and a merge queue that rebases and bisects failing batches [2605.29955].

Research-level projects in mathematics and physics have adopted blueprint-centered scaling. The tensor-network project coordinates six live agents plus an automated reviewer around a 150-page, 12-chapter LaTeX blueprint whose items are tagged with `\lean{...}`, `\leanok`, `\uses{...}`, and `\notready`; six rounds of blueprint review uncovered over 100 issues [2607.07857]. LeanMarathon treats the blueprint as the single system of record, proves all seven target theorems across three autonomous runs with no `sorry`, and reports 258 lemmas and theorems, over \(10^6\) tokens, and \$1,070 in API cost [2606.05400]. Trellis uses a deterministic kernel rather than a high-throughput scheduler, but still tracks global progress by node-level pass rate, Lean-closure fraction, natural-language backlog, and shallow-coarse milestones \(M(k)\), with the off-diagonal Ramsey run reaching an overall node-level pass rate of about 93.8% [2606.09674].

MerLean demonstrates a different scaling point: frontier research formalization at paper level rather than textbook or repository scale. On three theoretical quantum computing papers, it produced 2,050 Lean declarations from 114 statements in total, over 41 h 32 m, with an average of 13.0 compile–fix cycles per statement, 22.4 for theorems, and 9.1% of statements receiving explicit axioms [2602.16554]. Taken together, these results indicate that scaling is being pursued through at least four mechanisms: hierarchical scheduling, decentralized markets, blueprint-mediated decomposition, and deterministic progress semantics.

## 6. Empirical performance, limitations, and open questions

Empirical results are strong but heterogeneous across tasks and papers. GAMA evaluates 110 autoformalization tasks over five \(2\times2\) simultaneous-move games. In the supplied materials, the abstract reports 100% syntactic and 76.5% semantic correctness with Claude 3.5 Sonnet, and 99.82% syntactic and 77% semantic correctness with GPT-4o; the detailed evaluation section reports, on 55 scenarios each, 96% syntactic and 87% semantic correctness for GPT-4o, 92% syntactic and 82% semantic correctness for Claude 3.5 Sonnet, and combined rates of 94% syntactic and 84.5% semantic correctness [2412.08805]. MASA reports, on miniF2F, zero-shot GPT-4.1-mini pass rate of about 65.6, few-shot pass rate of about 76.2, and about 86.5 after formal refinement; on Lean4 miniF2F, a Qwen2.5-7B few-shot baseline with AF about 54.5 and FC about 62.7 rises to AF about 93.4 and FC about 85.3 after soft critique and refinement, while iterative self-refinement up to 3 iterations reaches about 62% jointly syntactic and semantic success [2510.08988]. PAT-Agent, on 40 systems from three benchmarks, reports 100% Compilation Success Rate, 100% Full-Pass Rate, and 100% Average Pass Rate for the \(\langle\)o3, Claude\(\rangle\) configuration, while ablations show drops to 0.75 FPR without Repair, 0.75 CSR and 0.60 FPR without Planning, and 0.475 FPR without both [2509.23675].

Human oversight remains a central issue rather than an optional add-on. GDPR auto-formalization generated about 500 samples, of which about 400 exceeded the automated threshold \(\tau=70\), and from these 120 were curated for the final dataset; the paper identifies abstraction choices, procedural-versus-substantive conflation, omission modeling, negation handling, and fact-extraction misalignment as primary failure modes [2604.14607]. The tensor-network project explicitly states that the main bottleneck in large-scale autoformalization is enforcing mathematical intent, and blueprint review was used to catch missing hypotheses, unintended weakenings, and edge cases such as \(D=0\) or \(N=0\) [2607.07857]. LeanMarathon likewise frames long-horizon failure as statement drift, dependency tangling, context decay, and corruption of distant work by local repairs [2606.05400]. Trellis treats vacuous refinement as a distinct failure mode and uses substantiveness gating to prevent the proof graph from degenerating into chains of empty reductions [2606.09674].

The main limitations named across the literature are remarkably consistent. GAMA is currently limited to \(2\times2\) simultaneous-move games, and its self-correction is driven purely by parse errors, so logical misinterpretations may pass syntactic checks [2412.08805]. MASA notes the absence of a centralized “director” agent, the reliance of semantic evaluation on LLM judges, and the throughput cost of many LLM calls and prover checks [2510.08988]. PAT-Agent highlights the semantic gap between natural language and formal logic and the challenge of hallucinated output in formal verification [2509.23675]. MerLean attributes some incompleteness to gaps in Mathlib and to the heavyweight nature of compile–fix loops [2602.16554]. AutoformBot reports orchestrator fatigue, isolated per-textbook planning, and adversarial behavior such as hidden axioms or redefined structures to “cheat” the build [2605.29955].

Current directions follow directly from these constraints. The published proposals include richer example libraries and retrieval-augmented prompting for GAMA [2412.08805]; dynamic meta-controllers and finer semantic evaluators for MASA [2510.08988]; extension to additional formal systems and specification languages for PAT-Agent and Meta-Agent [2509.23675] [2605.25233]; tighter semantic metrics for faithfulness checking in MerLean [2602.16554]; learned dependency heuristics and richer coordination channels in Agent Hunt [2603.06737]; and cross-book planning plus stronger prevention of axiom smuggling in AutoformBot [2605.29955]. A plausible implication is that the field’s central research problem is no longer merely generation quality. It is the construction of reliable socio-technical harnesses—contracts, blueprints, kernels, reviewers, schedulers, model checkers, and human checkpoints—that preserve intent while allowing large numbers of agentic steps to accumulate into a correct formal artifact.

Source: https://www.emergentmind.com/topics/multi-agent-autoformalization