ARc Autoformalization-Verification Pipeline

• The paper introduces an ARc pipeline that converts informal LaTeX statements into machine-verified Coq proofs using a Universal Transformer with an explicit copying mechanism.
• It employs systematic tokenization and placeholder normalization to align LaTeX and Coq vocabularies, ensuring accurate semantic parsing across mathematical and programatic domains.
• Experimental results indicate high semantic accuracy in controlled domains, though challenges remain with longer inputs and out-of-vocabulary token issues.

An ARc (Autoformalization–Verification) Pipeline is an end-to-end system that begins with informal, natural-language (often LaTeX) mathematical proofs or annotated program fragments and produces a mechanically verified formal development in a proof assistant, notably Coq. The ARc approach detailed in "Towards Autoformalization of Mathematics and Code Correctness: Experiments with Elementary Proofs" operationalizes a semantic parsing framework built on a Universal Transformer architecture, integrating data preprocessing, sequence-to-sequence translation with copy mechanisms, formal script generation, and proof assistant verification (Cunningham et al., 2023).

1. High-Level System Architecture

The ARc pipeline executes a linear workflow with four principal stages:

1. Data Preprocessing: LaTeX theorem–proof pairs (or Hoare-annotated code) are tokenized. Every variable or numeric constant is replaced with a placeholder (<var1>, <nat2>) for out-of-vocabulary generalization. The output is paired token sequences—LaTeX (placeholders) ↔ Coq (placeholders)—ready for ingestion.
2. Semantic Parsing: The main engine is a Universal Transformer–based encoder–decoder with an explicit copying mechanism for placeholder alignment. Input is the tokenized LaTeX sequence; output is a Coq token stream with a probability distribution over both the Coq vocabulary and input placeholders.
3. Proof Script Generation: Decoding proceeds via beam search or greedy search over the combined generate-vs-copy token distribution, yielding a linearized, syntactically valid Coq script comprising imports, definitions, theorem statements, and proof tactics.
4. Coq Verification: The generated script is submitted to the Coq prover (coqc or coqtop), which checks both syntax and proof correctness. Successful outputs are fully machine-checked Coq developments; failures can be interpreted for model correction.

This architecture supports both mathematical proofs and imperative program correctness statements via Hoare logic, achieving end-to-end semantic transformation and automated verification.

2. Universal Transformer–CopyNet Semantic Parsing

The core model is a Universal Transformer–style encoder–decoder network, adapted for mathematical and programmatic translation:

• Tokenization: Inputs consist of LaTeX words, mathematical symbols, and placeholders (e.g., <var1>), while output tokens are Coq keywords, tactics, operators, and the same placeholders.
• Encoder/Decoder: Both components are stacks of self-attention and feed-forward layers, each block recurrently applied over a "time" axis without absolute positional encodings. Instead, relative positional encodings are employed, as in Shaw et al. (2018).
• Copying Mechanism: At each decoding step, the probability of output token $y_t$ is partitioned:
  • If $y_t$ is in the Coq vocabulary, $p(y_t, g) = (1/Z_t) \exp \psi_g(y_t)$.
  • If $y_t$ is an input placeholder, $$p(y_t, c) = (1/Z_t)\sum_{x_j \in X: x_j = y_t} \exp \psi_c(x_j)$$.
  • The normalization factor $Z_t$ ensures a valid probability distribution over the mixed vocabulary-plus-placeholder output set.

This design specifically addresses the compositional and symbolic demands of mathematical formalization, ensuring consistent variable binding and minimizing hallucinated identifiers.

3. Data Preparation and Representation

Training data consists of aligned pairs between placeholder-tokenized LaTeX and their formal Coq representations. Each pair is:

• Source: A concatenation of (optional) definition, theorem, and proof in LaTeX, placeholders for variables and numerals consistently applied across the sequence.
• Target: A linearized Coq import, (optional) definition, theorem, and proof block sequence, with identical placeholders mapped to LaTeX.

Example:

• LaTeX input: “We define that <var1> ∈ ℕ is composite if … <var2>, <var3> ≥ <nat1> and <var2> × <var3> = <var1>.”
• Coq output:

  Require Import Lia. Definition composite (<var1> : nat) := exists <var3> <var2> : nat, (<var2> >= <nat1>) /\ (<var3> >= <nat1>) /\ (<var2> * <var3> = <var1>).

  This ensures out-of-vocabulary generalization and enables placeholder-aware model learning.

4. Scripting Conventions and Tactic Usage

The outputted Coq proofs feature:

• Standard Imports: e.g., Require Import Arith., Require Import Lia.
• Definition Patterns: Definitions such as composite, even_or_odd, or square.
• Proof Tactics: Tactics commonly include intros., unfold ..., repeat rewrite ..., assert ..., lia., structural proof manipulations, and Hoare logic combinators for imperative programs.

Explicit use of these tactics ensures that generated proof scripts fit the expectations of the Coq kernel and are generally robust to in-domain perturbations.

5. Experimental Protocol and Metrics

Datasets:

• Artificial proof domains: even/odd, composite, power, and Hoare logic programs—up to several thousand instances each.
• Hand-written: 45 human-written theorem–proof pairs (15 per domain).

Hyperparameters:

• Arithmetic: embeddings/state size 32, 4 blocks, T=4, 4 attention heads, Adam optimizer.
• Poly (Hoare logic): embeddings/state size 32, 8 blocks, T=8, batch=1.

Metrics:

• Sequence-level accuracy: strict exact match (theorem plus proof).
• Semantic-level accuracy: correct theorem statement and successful Coq verification.

Key results (semantic accuracy % by expression length $n$):

n	even-odd	composites	poly (Hoare)
2	99.8	97.6	100.0
3	99.6	94.2	100.0
4-9	≥94.0	88.3-94.4	45.1-99.2
10-12+	≤7.0	≤1.6	0–93.5

Hand-written semantic accuracy: even-odd 53.3%, composites 53.3%, powers 73.3%.

6. Limitations, Error Modes, and Extensions

Primary limitations:

• Generalization degrades sharply for input lengths beyond seen training ranges.
• Out-of-vocabulary token copying errors can induce sequence-level mismatch.
• The system relies on a manually curated grammar, constraining linguistic variability.
• Only a single proof form is generated, omitting alternate valid derivations.

Common error types:

• Placeholder misalignment (off-by-one errors), swapped or missing assertions, tactic sequence misordering.

Potential enhancements:

• Pre-training on larger LaTeX–Coq corpora to extend linguistic/semantic coverage.
• Adaptive computation time in the Transformer for increased reasoning steps on difficult proofs.
• Expansion to richer mathematical domains, integration with tactic prediction modules, and joint modeling with automated search methods.

7. Context, Impact, and Future Prospects

The ARc Autoformalization-Verification Pipeline demonstrates that Universal Transformer–based semantic parsing—with explicit copying, placeholder normalization, and a grammar-driven data process—can produce fully machine-verifiable formal mathematics and code correctness certificates from informal inputs in elementary domains. Coq verification anchors the workflow in foundational soundness, and exact/semantic success rates approach 100% on in-domain tasks of moderate length.

However, substantial challenges remain in scaling such systems to longer proofs, highly varied language, or advanced mathematical domains, as sequence-level performance decays and the need for alternative proof strategies becomes acute. Future work must address broader generalization, richer source domains, interactive or incremental proof synthesis, and improved tolerance to input and output diversity (Cunningham et al., 2023).