FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning

Abstract: Breakthroughs in frontier theory often depend on the combination of concrete diagrammatic notations with rigorous logic. While multimodal LLMs (MLLMs) show promise in general scientific tasks, current benchmarks often focus on local information extraction rather than the global structural logic inherent in formal scientific notations. In this work, we introduce FeynmanBench, the first benchmark centered on Feynman diagram tasks. It is designed to evaluate AI's capacity for multistep diagrammatic reasoning, which requires satisfying conservation laws and symmetry constraints, identifying graph topology, converting between diagrammatic and algebraic representations, and constructing scattering amplitudes under specific conventions and gauges. To support large-scale and reproducible evaluation, we developed an automated pipeline producing diverse Feynman diagrams along with verifiable topological annotations and amplitude results. Our database spans the electromagnetic, weak, and strong interactions of the Standard Model, encompasses over 100 distinct types and includes more than 2000 tasks. Experiments on state-of-the-art MLLMs reveal systematic failure modes, including unstable enforcement of physical constraints and violations of global topological conditions, highlighting the need for physics-grounded benchmarks for visual reasoning over scientific notation. FeynmanBench provides a logically rigorous test of whether AI can effectively engage in scientific discovery, particularly within theoretical physics.

• The paper introduces FeynmanBench, a large-scale benchmark of over 2,000 Feynman diagram tasks to systematically evaluate multimodal LLMs on diagrammatic physics reasoning.
• It employs an automated pipeline combining FeynArts and FeynCalc to generate and annotate visual tasks for robust evaluation of graph topology, conservation laws, and amplitude computation.
• Experimental results highlight a significant accuracy drop in global reasoning phases, emphasizing the gap between local visual recognition and rigorous symbolic analysis.

FeynmanBench: A Systematic Evaluation Framework for Multimodal LLMs on Diagrammatic Physics Reasoning

Motivation and Benchmark Design

The capacity to perform rigorous diagrammatic reasoning is indispensable for advanced physics, where visual notations encapsulate nontrivial topological, algebraic, and physical constraints. Existing MLLM benchmarks overwhelmingly focus on straightforward pattern recognition and local information extraction, thereby failing to probe the requisite global structural logic and mapping between visual formalisms and mathematical representations essential for real-world scientific reasoning.

"FeynmanBench: Benchmarking Multimodal LLMs on Diagrammatic Physics Reasoning" (2604.03893) addresses this evaluative gap by introducing the first large-scale, systematically annotated benchmark centered on Feynman diagram tasks. FeynmanBench targets diagrammatic tasks that require enforcement of conservation laws, identification of graph topology, mapping between diagrams and scattering amplitudes, and the application of algebraic and combinatorial rules under physical conventions and gauge choices. The benchmark derives over 2,000 tasks spanning QED, electroweak, and QCD processes, with a diversity of topologies (tree, one-loop reducible/irreducible, and two-loop), and over 100 task types.

The data is generated through an automated pipeline combining FeynArts and FeynCalc, ensuring verifiable topological and amplitude annotations, enabling reproducible, large-scale quantitative evaluation. The workflow comprises (1) diagram generation, (2) particle insertion, (3) annotation of vertex and connectivity information, (4) mapping to amplitudes via Feynman rules, and (5) automated scoring against theoretical ground truth.

Figure 1: Overview of FeynmanBench workflow, from automated diagram generation and annotation to symbolic amplitude computation and automated evaluation.

Task Formulation and Prompt Engineering

Each benchmark problem in FeynmanBench includes a Feynman diagram (as visual input) with standardized, unambiguous task prompts that specify interaction type, diagram orientation, and external states. Conventional ambiguities—such as identical-looking fermion lines or multiple valid amplitude interpretations—are systematically removed via explicit conventions for external momenta and field content, enabling unique adjudication and robust model evaluation. The accompanying annotation framework catalogs all salient topological features and the full analytic amplitude in a machine-verifiable format.

Figure 2: Schematic of task prompt specification and annotation structure, ensuring disambiguation for both human and machine evaluation.

Evaluation Protocol and Progressive Checkpoints

The evaluation protocol is structured as a cascade of five checkpoints (CP1–CP5), each assessing distinct levels of diagrammatic understanding and formal reasoning:

• CP1: Vertex counting and identification
• CP2: Internal line and propagator inventory
• CP3: Topological connectivity and graph isomorphism
• CP4: Momentum routing and conservation law verification
• CP5: Complete algebraic structure including index contractions, Dirac traces, and symmetry factors

Correct answers at each stage are a prerequisite for subsequent evaluation, enforcing logical consistency and spotlighting failure propagation.

Baseline models include GPT-5.1, GPT-5.2, Gemini-3, Claude 4.5 Opus/Sonnet, and Qwen-VL-Plus. Scoring for CP1/2 is automated via symbolic matching; CP3–CP5 are adjudicated via LLM-as-a-judge protocols with majority voting, coupled with manual spot checks for robustness.

Experimental Results and Analysis

Recognition vs. Reasoning: Systemic Findings

The evaluation reveals that while SOTA MLLMs (especially Gemini-3) achieve high accuracy in local recognition (vertex and propagator counts, e.g., CP1 up to 95.8%), all models suffer catastrophic collapse on the global topological and algebraic reasoning phases. Gemini-3, for instance, transitions from 89% at CP2 to 3.3% at CP3, indicating a lack of mechanisms for mapping visual object sets to graph-structured representations needed for topology and conservation-law enforcement.

Figure 3: Checkpoint progression of model accuracy, indicating high front-end recognition but rapid decay in graph and algebraic reasoning.

This pattern generalizes: GPT and Claude models show slightly more balanced but consistently insufficient performance, particularly at CP2/3 transitions (e.g., GPT-5.1: 46.7% at CP1 to 23.0% at CP2, then 29% at CP3). Qwen-VL-Plus and Claude Sonnet fail primarily due to unreliable low-level parsing, especially for high-complexity topologies.

Systematic Bottlenecks by Task Category

The sharpest accuracy degradation occurs as diagrammatic complexity (loop order, reducibility) increases. The dataset includes granular topological (A1–A5) and interaction (B1–B6) categorizations, highlighting that even basic QED tree-level diagrams are not robustly parsed for global constraints, and performance is worse for reducible or higher-loop graphs. Even canonical electron-photon systems do not yield high consistency in topology or conservation rates (CP3/CP4).

Figure 4: Model/category heatmaps for CP1–CP3, demonstrating steep accuracy decline for non-textbook topologies and complex interaction classes.

Direct analysis finds rare successes at CP5 correspond to memorized textbook cases, not genuine abstraction. No model approaches nontrivial symmetry factor or group theoretical calculations as required for full amplitude derivation.

Qualitative Error Modes

The failure patterns are diagnostically instructive. Many errors are low-level—misclassifying vertex–line crossings, miscounting propagators in two-dimensional projections—but the most critical failures are logical: inability to enforce physical constraints (e.g., charge, flavor, baryon number conservation in identifying internal lines), disregard of fundamental topological invariants ($L=I-V+1$), failure to reconstruct graphs from local features, and total neglect of nonlocal algebraic quantities such as chiral structures and symmetry factors.

Figure 5: Classical errors in Feynman diagram parsing, including topological misidentification, symmetry factor omission, and graph-theoretic constraint violations.

This evidences a fundamental disconnect in current MLLMs between local visual parsing and the higher-order symbolic reasoning mandatory in scientific domains. Models treat feature detection as an isolated pipeline, not as input to constraint-driven consistency propagation.

Theoretical and Practical Implications

FeynmanBench empirically demonstrates that MLLMs, despite success in visually and linguistically rich open domains, fundamentally lack "physical intuition"—the ability to synthesize global topological and algebraic structure from symbolic diagrams in tightly constrained scientific settings. The results spotlight that performance on general vision-language benchmarks does not extrapolate to authentic scientific reasoning, where correctness is rigidly topological, algebraic, and subject to physical invariants.

The practical implication is that deploying MLLMs for automated or assisted calculation in physics—even for tasks as "simple" as Feynman diagram parsing—is premature given current architectures. Theoretical implications are profound: current MLLMs, even with self-attention-based visual encoders, don't instantiate true diagrammatic world-models or graph-based reasoning modules capable of bottom-up, constraint-governed symbol manipulation. This exposes the need for architectures unifying visual representation, inductive (physics-based) bias, and symbolic manipulation.

Future Directions

FeynmanBench is extensible by construction, enabling benchmark expansion to beyond-the-standard-model scenarios, effective field theories, and high-loop, high-symmetry contexts. Critical future research axes include (1) augmentation with physics-informed inductive biases or explicit symbolic modules, (2) evaluation of generative tasks (diagram synthesis from algebraic/textual specification), and (3) generalization to other domains of formal diagrammatic reasoning (e.g., tensor networks, category theory).

These advances are prerequisites for moving from image–text extraction toward AI agents capable of engaging in genuine scientific reasoning and discovery in physics and related fields.

Conclusion

FeynmanBench establishes a new paradigm for evaluating MLLMs as scientific reasoners. The revealed decoupling between local pattern recognition and global diagrammatic reasoning sets a high bar for future research and highlights the indispensability of constraint-based, topological, and algebraic reasoning in AI for science. Progress in this direction will require new architectures that natively encode the logic of physical diagrams, symbolic constraints, and nonlocal consistency in both perception and reasoning modules.