HITL Autoformalization Pipeline

• HITL autoformalization pipeline is a structured workflow that integrates automated LLM stages and human interventions to convert informal or semi-formal descriptions into precise, verifiable models.
• It divides the process into modular steps—such as drafting, entity linking, and type adjustment—to ensure robust translation and maintain semantic integrity.
• The approach supports diverse applications including theorem proving, optimization, and model checking, harnessing both automated coding and targeted expert corrections.

Human-in-the-loop (HITL) autoformalization pipelines are systems that leverage LLMs and formal verification tools, integrating human expertise at targeted stages to translate informal or semi-formal mathematical, optimization, or rule-based statements into machine-verifiable artifacts. These pipelines address the current limitations of fully automated approaches by explicitly structuring the workflow into modular subtasks—each amenable to partial automation and expert intervention—thereby increasing robustness, scalability, and correctness across domains such as mathematical theorem formalization, energy optimization, software verification, and map transformation validation.

1. Core Principles and Motivation

Autoformalization aims to bridge the semantic and syntactic gap between human-authored, natural-language or semi-formal problem descriptions and the rigorously specified inputs required by formal verification environments (e.g., Lean, Dafny, CSP♯ model checkers, FOL-based rule engines). The task is especially nontrivial for research-level mathematics or systems with substantial domain complexity: informal statements omit required formal definitions, conflate distinct formal entities, or underspecify critical parameters.

HITL design partitions the autoformalization problem into structured stages, each supported by appropriate automation (LLMs for language understanding/generation; parsers/checkers; search/ranking algorithms), but maintains explicit checkpoints for human review and correction. This scaffolding ensures system reliability despite current generative model limitations, while also providing informative feedback for benchmarking, error analysis, and future tool improvement.

2. Canonical Pipelines: Three Representative Architectures

The field is characterized by a set of standardized, stage-wise pipelines, with each domain tailoring the process to its formal target language and verification objectives.

2.1. Mathematical Proofs (Lean) (Patel et al., 2023)

Stages:

1. Unlinked Formalization: Given a LaTeX statement, an LLM emits Lean code capturing only structural features, using placeholders for all references. No type checking is required at this stage.
2. Entity Linking: Placeholder identifiers are mapped to concrete Lean/mathlib symbols using string/embedding matching and learning-to-rank approaches.
3. Type Adjustment: Code is minimally edited to type-check, leveraging Lean's own elaboration/unification and heuristic patching strategies.

Expert review occurs after each subtask, allowing targeted correction of naming, mis-linked definitions, and unresolved type signatures. The arXiv2Formal dataset benchmarks initial steps, and annotations for linking/type-correction enable precise evaluation of HITL effort.

2.2. Optimization via Conversational LLMs (Jin et al., 2023)

Stages:

1. Query Classification and Parameter Solicitation: An LLM determines if the problem is an instance of mathematical optimization and interactively queries the user for missing information (e.g., charger ratings, energy needs).
2. Problem Instantiation and Code Generation: Given parameters, the LLM emits Python optimization code (CVXPY/SciPy).
3. Solver Integration and Automated Debugging: Errors from code execution are sent back to the LLM, which proposes patches; this auto-debug loop continues until successful compilation/solution.
4. Interpretation (Auto-Informalism): The LLM explains the solution and details its constraints/objectives in user-accessible language.

Human intervention is central at the points of parameter clarification and solution validation, ensuring intent is preserved over multiple iterations, especially when ambiguity, missing data, or modeling assumptions arise.

2.3. Formal Model Synthesis and Checking (PAT-Agent) (Zuo et al., 28 Sep 2025)

Stages:

1. Planning via LLM: Extraction and planning of constants, variables, and process actions from NL spec into a structured plan (JSON, annotated steps).
2. Code Generation and Syntax Reinforcement: The plan feeds an LLM for syntactically and semantically faithful CSP♯ code, guided by RAG examples and syntax cues.
3. Automated Verification: PAT model checker validates the generated model against user-specified properties.
4. Iterative Repair Loop: On verification failure, counterexample traces inform in-context code edits, triggering up to 5 repair cycles before abortion.
5. User Dashboard and HITL Feedback: A rich UI enables human edits at each model and requirement stage, visibility into counterexamples, and fast cycle-resubmission.

Ablation studies show the critical impact of both planned decomposition and repair, with full pipeline execution yielding 100% verification success on non-trivial datasets.

3. Role and Positioning of Human-in-the-Loop Interventions

HITL design is not monolithic; rather, its implementation is tailored to subtask properties and failure modes, ranging from low-level token corrections to high-level semantic guidance.

Pipeline Stage	Typical Automation	Human-in-the-Loop Role
NL→Draft Formalization	LLM, prompt-based LLMs	Accept/reject/correct code; assign placeholder splits; correct types/names
Entity Linking	Lexical/embedding-based matching, ranking	Shortlist review; create new definitions where library coverage fails
Type/Constraint Adjustment	Unification, heuristic edits	Manual fix on error messages; accept auto-suggestions; final signature edits
Semantic Validation	Test scenario execution	Review constraints/objectives for intent alignment; expose/override assumptions

A plausible implication is that, although ML and formal environment toolchains can automate substantial portions of the process, domain expert verification is indispensable, especially at bottlenecks involving semantic ambiguity, underspecified parameters, or subtle domain-specific conventions.

4. Benchmarking, Datasets, and Evaluation Metrics

Benchmark datasets have become foundational in quantifying HITL-enabled pipeline performance:

• arXiv2Formal (Patel et al., 2023): Contains 50 parallel (LaTeX, Lean) samples for unlinked formalization; under extension to include entity linking and type adjustment annotations, as well as per-edit HITL logs for simulating expert time/effort.
• DafnyComp (Yan et al., 22 Jul 2025): Supports out-of-domain benchmarking (beyond Python-to-Dafny translations) for autoformalized compositional specifications, with metrics including syntactic validity, verification rate (% passing Dafny), and Spec Superiority Rate (SSR).
• Synthetic scenario testbeds (He et al., 3 Nov 2025, Zuo et al., 28 Sep 2025): Used to validate correctness, engineering effort saved, coverage, and rate of human intervention versus error-free outputs.

Metrics span standard translation scores (BLEU, TER), time-to-accept, edit distance, compile/verification success, optimality gap (for optimization tasks), and human satisfaction.

5. Error Modes, Limitations, and Best Practices

Empirical results across pipelines indicate substantial error diversity:

• Naming and Typing Errors: LLMs hallucinate types ("irrational_number" vs. explicit $\mathbb{R}$ with hypothesis), or generate non-canonical names.
• Under/Over-constrained Code: Entity linking may select suboptimal definitions or omit latent parameters required for type- or semantics-correctness.
• Semantic Drift: Optimization code may not accurately instantiate user intent, even if syntactically correct.
• Parsing/Grammar Compliance: Generated formulas may narrowly miss domain grammar or require minor prompt refinements for recurrent errors.

Best practices include prompt templates enforcing hard grammar constraints, explicit examples in context, and targeted feedback mechanisms capturing recurrent human corrections for downstream prompt/template refinement. Prompt clarity, joint formula/code generation, and maintaining a shared context for both LLM and human reviewers are essential.

6. Domain Extensions and Scalability

Pipelines extend beyond a single formal verification setting:

• Proof Assistants: Methodology generalizes to Isabelle, Coq, or any system supporting entity linking and type inference.
• Model Checking: Techniques based on CSP♯, TLA⁺, Alloy, or B-Method are adaptable given grammar exposure and plan-code pair curation for RAG.
• Optimization/Control Problems: Conversational approaches and solver integration apply to sectors beyond energy (e.g., logistics, scheduling), when embedded in modular parameter-extraction and code-validation loops.
• Rule-based Verification: Any computationally tractable FOL framework can benefit from joint formula and predicate generation with grammar-in-the-prompt discipline.

The evidence suggests that, while RL-based pipelines and formal feedback (as in (Yan et al., 22 Jul 2025)) can further reduce human dependence, domain-specific oversight and corrective feedback remain essential for scalable, trustworthy automation.

7. Outlook and Open Challenges

The HITL autoformalization paradigm raises the prospect of quantifying and minimizing expert correction burden while exploiting LLM advances. Ongoing challenges include extending to higher-order or more abstract domains, developing more principled exploration and reward designs for RL-driven formalization, and building open benchmarks with comprehensively annotated HITL logs. The HITL framework stands as a key intermediary, alchemizing model flexibility, formal rigor, and human expertise into practical workflows for formal verification and beyond.