---
title: AutoformBot System Architecture
url: https://www.emergentmind.com/topics/autoformbot-system-architecture
type: topic
---

# AutoformBot System Architecture

AutoformBot is a multi-agent system for large-scale autoformalization of mathematics textbooks, designed to translate informal prose into machine-checked Lean 4 definitions and proofs using thousands of collaborating large language model (LLM) agents and formal verification tools. The system’s architecture is distinguished by rigorous dependency-aware task scheduling, formal proof-checking pipelines, and git-based collaborative workflows, enabling robust, modular, and scalable production of validated mathematical libraries [2605.29955].

## 1. System Architecture Overview

AutoformBot consists of a layered, modular architecture in which a distributed multi-agent system converges around a persistent dependency graph of formalization tasks. At the top level, an Orchestrator ingests textbooks (PDFs), segments them into fine-grained formalization targets, and constructs a task dependency DAG reflecting logical and exposition-imposed relationships among statements. This DAG serves as the backbone for all parallelization and scheduling.

The architecture comprises six principal agent types with well-defined responsibilities:

- **Orchestrator:** Parses, segments, and plans across the global task DAG.
- **Runner/Scheduler:** Polls the DAG for ready (all dependencies satisfied) tasks, dispatches jobs, and manages pools of workers and reviewers.
- **Worker:** Short-lived coding agents that generate or repair Lean 4 code for an assigned task.
- **Reviewer:** Short-lived agents that review submissions for correctness and adherence to formalization protocols.
- **Supervisor:** Handles target-level global checking after merges, triages failures, and spawns remediation subtasks.
- **Trace Analyzer:** Conducts per-task failure analysis and provides detailed feedback to guide recovery.

Agents interact via the Model Context Protocol (MCP), a JSON RPC/method-call interface to Lean, git, filesystem, and DAG infrastructure. The system orchestrates massive agent concurrency while guaranteeing coordinated, acyclic progression through the global dependency structure.

## 2. Multi-Agent Orchestration and Communication

AutoformBot’s agent orchestration layer is designed for both massive parallelism and precise error recovery. The Orchestrator segments textbooks into individual “tasks,” each corresponding to a formalization unit (such as a definition or theorem statement/fix). Each task is added to a persistent DAG store, encoding logical or exposition-driven dependencies.

Workers and Reviewers are scheduled against this task DAG:

- **Workers:** Multiple LLM-based workers may compete (“race”) to complete the same formalization task; the first properly validated completion is accepted, and all other attempts are canceled. Each worker operates within an isolated git worktree, editing or generating Lean files.
- **Reviewers:** After mechanical gates (no Lean errors), submissions are reviewed for semantic integrity. Dependency-graph analysis is leveraged to detect hidden axioms or invalid proof tactics (“sorry”).
- **Supervisory feedback:** The Supervisor and Trace Analyzer continuously monitor outcomes, triggering immediate triaging and generating targeted remediation subtasks for failed or substandard results.

## 3. Formal Verification Pipeline Integration

AutoformBot’s operation critically depends on tight integration with Lean 4’s formal verification toolchain:

- **Lean REPL and LSP:** Workers interact with Lean via the MCP “tool servers,” including a persistent Lean REPL for incremental compilation and goal inspection, and a Lean LSP server for rapid diagnostics.
- **Proof-checking Loop:** The worker-modified Lean files are checked for compilation errors. Passing code is escalated to reviewers who cross-verify using Lean’s diagnostic tools and dependency-graph tracing, enforcing strict exclusion of “sorry” and unauthorized axioms or hooks.
- **Evaluation Harness:** After batch merges, a three-stage evaluation is applied:
  1. Mechanical (Lean) gates.
  2. Matcher for statement-to-declaration alignment.
  3. Three independent LLM-based judges, assessing faithfulness, proof integrity, and code quality, all leveraging the Lean dependency graph for axiomatic validation.

## 4. Dependency-Aware Scheduling and Collaborative Version Control

Central to AutoformBot’s scalability is its dependency-aware task dispatch and robust versioning. Tasks are organized in a persistent DAG (JSON adjacency list on-disk or in-memory), with nodes for formalization targets and directed edges for dependencies. The Scheduler employs a Kahn-style topological sorting algorithm, always dispatching tasks whose dependencies (“parents”) are complete.

Each agent instance operates on a short-lived git worktree branched off origin/main. The merging process is managed through a batched merge queue inspired by bors-ng:

- **Batch Merging:** Approved branches are rebased and built in a batch; if the build fails, the queue is bisected to isolate the faulty commit.
- **Conflict Resolution:** Bisection-based diagnosis marks the offending DAG node as failed; remediation is initiated by the Trace Analyzer.
- **Isolation:** This strategy mitigates build breaks and allows parallel agent activity without cross-contamination, providing deterministic, reproducible merges.

## 5. Core Algorithms, Data Structures, and Performance

AutoformBot’s orchestration is underpinned by optimized core data structures and dispatch algorithms:

- **Task DAG:** Encoded as JSON, with dual arrays (out_edges, in_degree) in memory for efficient ready task lookup.
- **Queues:** The architecture separates readyQueue (tasks ready to dispatch), inProgress (active tasks), and mergeQueue (approved for main branch).
- **Worker Pooling:** AsyncIO semaphores and thread-pools manage resource allocation for LLM and Lean sessions.
- **Dispatch Logic:**
  - At each iteration, up to available_slots ready tasks from the DAG are assigned to workers.
  - When workers race, only the first successful submission is accepted.

The idealized throughput formula is given by
$$\lambda \approx \frac{W}{\mathbb{E}[t_{exec}]},$$
where $W$ is the number of parallel workers and $\mathbb{E}[t_{exec}]$ is the expected worker execution time. Scheduling latency per task amortizes as $O(1/W)$. The overall makespan is approximated by
$$M \approx \frac{T_{total}}{W} + O\left(\frac{|V|}{W} + \text{scheduling overhead}\right),$$
with $T_{total}$ as the sum of sequential task times and $|V|$ the number of DAG nodes.

## 6. Metrics, Benchmarks, and System Robustness

AutoformBot has demonstrated significant large-scale efficiency and robustness [2605.29955]. Across 26 mathematics textbooks totaling 4,007 formalization targets:

- 2,855 targets were formalized (71.3%).
- The system generated 483,918 lines of Lean code, consuming 183.2 million tokens.

Controlled ablations reveal the importance of each subsystem. On a 39-target Algebraic Combinatorics set (600M tokens):

| Pipeline Configuration     | % Completed (39 targets) |
|---------------------------|-------------------------|
| Full pipeline             | 77%                     |
| no orchestrator           | 64%                     |
| no supervisor             | 51%                     |
| no trace analyzer         | 57%                     |

Parallelism yields nearly linear speedup for small problem sizes; for one, three, and five workers per task, wall-clock times were approximately 8 hours, 4 hours, and 4 hours, respectively, with increasing completion rates.

Model comparisons on this data yield, for 600M tokens, 92% completion on Claude 4.6 and 46% on Gemini 3.1 Pro.

## 7. Design Principles and Implications

AutoformBot embodies several best practices for scalable, verifiable automation:

- **Layered Orchestration:** Decoupling orchestration, verification, and review enables parallel agent activity and rapid recovery from fault.
- **Fault Isolation:** Short-lived git worktrees and merge batching guarantee deterministic merges and rapid localization of errors.
- **Task-Driven Dependency Management:** The universal task DAG facilitates fine-grained scheduling, parallelism, and systematic error isolation.
- **Feedback Loops:** Supervisory and tracing agents provide immediate root-cause analysis, enabling rapid re-planning and system resilience.
- **Modularity:** Each agent and tool is protocol-bound and independently replaceable.

These design principles have enabled the large-scale formalization of core mathematics content, making continuous, mechanically checkable formal mathematical libraries at scale technically and economically feasible. A plausible implication is that such architectures generalize to other domains requiring verified, distributed multi-agent collaboration on dependency-structured corpora [2605.29955].

Source: https://www.emergentmind.com/topics/autoformbot-system-architecture