MMFormalizer: Multimodal Autoformalization in the Wild

Abstract: Autoformalization, which translates natural language mathematics into formal statements to enable machine reasoning, faces fundamental challenges in the wild due to the multimodal nature of the physical world, where physics requires inferring hidden constraints (e.g., mass or energy) from visual elements. To address this, we propose MMFormalizer, which extends autoformalization beyond text by integrating adaptive grounding with entities from real-world mathematical and physical domains. MMFormalizer recursively constructs formal propositions from perceptually grounded primitives through recursive grounding and axiom composition, with adaptive recursive termination ensuring that every abstraction is supported by visual evidence and anchored in dimensional or axiomatic grounding. We evaluate MMFormalizer on a new benchmark, PhyX-AF, comprising 115 curated samples from MathVerse, PhyX, Synthetic Geometry, and Analytic Geometry, covering diverse multimodal autoformalization tasks. Results show that frontier models such as GPT-5 and Gemini-3-Pro achieve the highest compile and semantic accuracy, with GPT-5 excelling in physical reasoning, while geometry remains the most challenging domain. Overall, MMFormalizer provides a scalable framework for unified multimodal autoformalization, bridging perception and formal reasoning. To the best of our knowledge, this is the first multimodal autoformalization method capable of handling classical mechanics (derived from the Hamiltonian), as well as relativity, quantum mechanics, and thermodynamics. More details are available on our project page: MMFormalizer.github.io

• The paper introduces a recursive grounding pipeline that decomposes multimodal inputs into formal logical statements for machine theorem proving.
• It employs adaptive recursive termination to halt decomposition at key empirical or axiomatic milestones, ensuring valid and efficient formalization.
• Experimental results reveal state-of-the-art models excel in physics yet struggle with multimodal geometry, indicating clear directions for future research.

Multimodal Autoformalization: A Technical Overview of MMFormalizer

Introduction

"MMFormalizer: Multimodal Autoformalization in the Wild" (2601.03017) presents a comprehensive framework for converting multimodal mathematical and physical problems—spanning natural language, images, and diagrams—into formal logical statements suitable for machine theorem proving. The work responds to core challenges in current autoformalization systems, notably their limited capacity to link perceptual input with type-theoretic formalizations, especially in domains beyond pure geometry and into theoretically dense fields like physics. The paper grounds its contributions in both theoretical design and an extensive benchmark, PhyX-AF, that spans classical and modern physics, as well as complex synthetic geometry.

Technical Contributions

Recursive Multimodal Grounding

The principal innovation of MMFormalizer is its recursive grounding pipeline, which performs structured decomposition of multimodal problems:

• SceneGraph Construction: Input images or visual scenes are decomposed into primitive geometric or physical entities, formalized as dependent types (e.g., points, lines, regions, or, in physics, masses, charges, reference frames).
• Propositional Hierarchy: Formal dependency graphs are constructed such that each node consists of a lemma—a pair of a formal proposition and a proof term—grounded in both perceptual input and prior subpropositions.
• Alignment with Formal Libraries: Intermediate hypotheses generated during recursion are mapped onto reusable, machine-verifiable lemmas from formal libraries (mathlib and PhysLean) using semantic retrieval mechanisms.

This compositional and type-theoretic approach ensures physical and geometric constraints (e.g., dimension, regularity, conservation laws) are encoded directly into the formal chain, going beyond traditional first-order symbolic reasoning and capturing critical dependencies such as dimensional consistency and axiomatic closure.

Adaptive Recursive Termination

A significant technical advancement is the adaptive recursive termination mechanism. Rather than indefinitely descending to primitive entities or axioms, the recursion halts when the reasoning chain is anchored in either:

• Primitive Dimensional Groundings: (e.g., in physics, mass, charge, or fundamental constants like the speed of light).
• Axiomatic Closure: (e.g., Newton’s laws, the principle of relativity, Maxwell equations).

This ensures abstraction depth is justified by empirical or axiomatic necessity, enabling MMFormalizer to bridge from observable evidence to provable statements without spurious over-decomposition—a failure mode observed under naive recursive strategies.

Axiom Chain Composition and Semantic Validation

All grounded lemmas are subject to formal axiom chain composition, ensuring closure within the type system and full alignment with domain axioms. Each candidate solution undergoes semantic checking across modalities—image, text, and formal code—with both automated and expert human (PhD-level) evaluation, ensuring that grounded statements are not merely syntactically valid but semantically sound with respect to the evidence.

PhyX-AF Benchmark

To rigorously evaluate multimodal autoformalization, the authors introduce PhyX-AF: 115 curated samples spanning four domains (MathVerse, PhyX, Synthetic Geometry, Analytic Geometry). Key benchmark features include:

• Enforced visual dependency, ensuring models must utilize images rather than rely on text-only shortcuts.
• Inclusion of out-of-distribution synthetic geometry and modern physics (relativity, quantum mechanics), targeting the generalization and grounding limits of current LLMs.

Experimental Findings

Model Comparison

Evaluation on PhyX-AF yields several nontrivial insights:

• Frontier Model Performance: GPT-5 and Gemini-3-Pro exhibit highest compile and semantic accuracy, with GPT-5 demonstrating superior physical reasoning in the PhyX subset—including relativity and quantum mechanics. Notably, geometry, especially synthetic and analytic, remains a persistent challenge for all systems, with semantic and compile accuracy substantially lower than for physics.
• Open-source Model Gaps: Qwen3-VL-235B, the leading open-source model, fails to achieve meaningful performance in physical or out-of-distribution geometry domains, reflecting architectural and data constraints relative to closed-source LLMs.
• Modal Gap: Across all models, performance on genuinely multimodal (image-dependent) tasks substantially lags text-only tasks, underscoring the rigidity of current visual–symbolic integration.

Ablative Analysis

Ablation studies highlight critical sensitivities:

• Termination Condition: Absence leads to uncontrolled recursive depth and combinatorial explosion in the dependency graph, undermining synthesis and computational tractability.
• Code Synthesis vs. Retrieval: In synthetic geometry, preventing the use of retrieved reference code (forcing generation by synthesis) increases out-of-distribution generalization, whereas code retrieval benefits in-distribution, less novel settings.
• Image Grounding: Direct grounding on visual input improves performance on high-complexity cases (e.g., modern physics, synthetic geometry), indicating LLMs’ capacity for aligning perceptual cues with formal semantics is critical but still underdeveloped.
• Sampling Strategies: Increasing decoding diversity ($pass@k$) systematically boosts success rates on challenging instances, suggesting future directions for scaling test-time synthesis.

Implications and Future Directions

Practical Impact

MMFormalizer establishes a scalable blueprint for enabling machine reasoning across previously disconnected domains—geometry, classical/modern physics—by providing a robust bridge from multimodal input to type-theoretic formalization. This unlocks applications such as:

• Automated Science Assistants: Systems capable of converting textbook-style questions, research diagrams, or even physical-world video into machine-checkable proofs or parameterized models.
• Verifiable STEM Education Tools: Platforms for interactive, verifiable reasoning in mathematical and physical problem-solving settings.

Theoretical Impact

The reliance on dependent type theory and the explicit alignment of dimensional, axiomatic, and perceptual groundings advances the rigor and semantic interpretability of autoformalization systems. The approach also identifies key open challenges:

• Visual–Symbolic Bridging: Persistent low accuracy in geometry indicates the need for more robust perceptual parsing and symbolic–visual alignment.
• Generalization: High dependency on pre-existing corpora for code retrieval curtails zero-shot and out-of-distribution generalization, a critical barrier for broad scientific autoformalization.
• Termination Control: Failure cases demonstrate that principled, perhaps learned, recursive termination criteria are essential for tractable, semantically coherent formalization.

Forward Outlook

Anticipated future developments arising from this work include:

• Enhanced neuro-symbolic models with improved perception-to-proof pipelines, leveraging advances in vision transformers and self-supervised geometric relational reasoning.
• Expansion of multimodal benchmarks and formal libraries to nurture research at the intersection of perception, language, and logic, especially in the natural sciences.
• Development of agentic formalization systems capable of on-the-fly procedural construction and verification, further automating the translation from scientific observation to formal understanding.

Conclusion

MMFormalizer introduces an adaptive, compositional framework that systematically links visual and textual perception with formal, verifiable reasoning across mathematics and physics. Empirical results on the PhyX-AF benchmark illustrate both the present capabilities and clear limitations of the current generation of multimodal LLMs in unified autoformalization. The formal recursive grounding, axiom composition, and semantic checking infrastructure presented here lay critical groundwork for expanding the reach of machine reasoning into scientific domains characterized by complex multimodal dependencies, theoretical abstraction, and empirical anchoring.