SeePhys: Benchmark for Physics Reasoning

• SeePhys is a large-scale multimodal benchmark for physics reasoning that integrates diagram interpretation with symbolic and mathematical analysis.
• It consists of 2,000 rigorously validated physics problems sourced from textbooks and international exams, stratified across eight difficulty levels from middle school to PhD qualifying challenges.
• The benchmark evaluates models under varied input settings and has led to a successful caption-assisted reasoning approach that significantly enhances multimodal performance.

Searching arXiv for the SeePhys benchmark and the associated technical report to ground the article in the cited literature. SeePhys is a large-scale multimodal benchmark for physics reasoning in which correct solutions frequently depend on extracting analytically indispensable information from diagrams rather than relying on text alone. It was introduced to measure whether LLMs and multimodal LLMs can couple diagram interpretation with symbolic and mathematical reasoning across physics problems ranging from middle school to PhD qualifying exams, and it subsequently served as the basis for a challenge on which a caption-assisted reasoning system achieved 1st place (Xiang et al., 25 May 2025, Liang et al., 7 Sep 2025).

1. Scope, task definition, and benchmark structure

SeePhys was designed to expose shortcomings in multimodal reasoning on physics problems whose visual content is not merely illustrative. The benchmark contains 2,000 rigorously validated, open-ended questions with a single ground-truth answer. Its difficulty spectrum is stratified into eight levels: Middle-school (5.1%), High-school (12.5%), Beginner Olympiad (5.4%), Advanced Olympiad (22.6%), Undergraduate (17.8%), Senior Undergraduate (11.0%), Master’s (7.3%), and PhD Qualifying (18.6%). The benchmark covers seven physics domains: Classical Mechanics; Electromagnetism; Astrophysics/Cosmology/Gravitation; Optics & Acoustics; Atomic/Molecular/Nuclear/Particle Physics; Quantum Mechanics/Information Technology; and Thermodynamics/Statistical Mechanics (Xiang et al., 25 May 2025).

Its visual heterogeneity is central to the task definition. SeePhys includes 21 diagram categories, with examples such as Feynman diagrams, free-body force-analysis, circuit schematics, position-time and velocity-time plots, coordinate systems, photon-emission spectra, thermodynamic cycles, and wave interference patterns. The dataset also distinguishes two vision-enrichment levels. In the Vision-Essential subset, which comprises 75% of problems, key problem-solving information is split between text and figure. In the Vision-Optional subset, comprising 25%, the diagram serves only to illustrate a scenario already fully specified in text. This distinction makes SeePhys a benchmark of multimodal dependency rather than a generic science QA collection (Xiang et al., 25 May 2025).

SeePhys also appeared in challenge form. In the challenge setup reported by the winning technical report, evaluation was conducted on SeePhys-mini, a set of 200 random examples spanning the same eight difficulty tiers from Middle School to PhD. In that configuration, the data retained the benchmark’s characteristic multimodal format: line drawings with labels such as forces, masses, and angles, paired with question prompts describing the physical scenario and the unknown to solve (Liang et al., 7 Sep 2025).

2. Corpus construction, annotation, and standardization

The benchmark was assembled from over 7,000 PDF pages drawn from open-source textbooks, international competitions including IPhO and CPhO, Cambridge IGCSE and AS–A-level exams, and U.S. university PhD qualifying exams. The final corpus is bilingual, with English (1,039 questions) and Chinese (961 questions). This breadth of sourcing was paired with a standardization pipeline intended to remove superficial formatting differences while preserving the underlying reasoning task (Xiang et al., 25 May 2025).

The preprocessing workflow begins with Mathpix OCR to produce Markdown, followed by GPT-4.1 for line-break and LaTeX syntax cleanup, and then manual LaTeX parsing and verification. Compound questions are split into atomic sub-questions, multiple-choice items are converted to open-ended format, and significant-figure requirements are annotated. Subject labels for the seven physics fields and diagram-type labels for the 21 classes are assigned by physics experts. Difficulty levels are assigned per international curriculum and problem-solving time benchmarks, and Vision-Essential versus Vision-Optional labels are assigned through dual-expert consensus: a problem is Vision-Essential if “key problem-solving information is split between text and figure,” and Vision-Optional otherwise (Xiang et al., 25 May 2025).

SeePhys also incorporates explicit data-leakage prevention. The reported procedure has two phases: first, GPT-4o with web-search toggled on and off is used to flag items whose accuracy depends on search access; second, all “easy” items with perfect scores are manually verified through Google search. In addition, the benchmark includes a pure-vision enhancement in which each question and diagram are rendered into a single composite image of up to $4096\times4096$ pixels with randomized font styles and sizes. Auto-generated captions produced via o4-mini are then used to create a Text+Caption version. This makes the benchmark suitable for comparing raw visual input against caption-mediated alternatives within a common evaluation frame (Xiang et al., 25 May 2025).

3. Evaluation protocol and empirical performance profile

SeePhys defines four evaluation settings. In Text + Vision (TV), the model receives the full problem text and diagram images. In Text + Caption (TC), it receives the problem text and an auto-generated detailed caption describing the diagram. In Text-Only (TO), vision is stripped away. In Vision-Only (VO), the input is a composite image of text plus diagram without separate text input. Accuracy is the principal metric and is defined as

$$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$

A composite judgment pipeline first applies SymPy for numeric-value matching; if uncertainty remains, a secondary LLM-based judge, DeepSeek-V3, extracts and compares answers. For Vision-Essential problems, the benchmark also reports $\Delta_1=(TV-TC)/TV$, $\Delta_2=(TV-TO)/TV$, and $\Delta_3=(TV-VO)/TV$, respectively measuring dependence on raw images versus captions, gain from any visual input over text only, and reliance on text-diagram parsing versus pure visual layout (Xiang et al., 25 May 2025).

The evaluation covers 28 models: 9 text-only LLMs and 19 MLLMs. Prompting uses zero-shot Chain-of-Thought templates in English and Chinese, with a hint about required significant figures. On the full 2,000-question benchmark, the best reported TV accuracy is 54.9% for Gemini-2.5-Pro, followed by 51.9% for o4-mini, 45.6% for o1, 43.9% for Doubao-1.5-Pro, and 34.6% for Claude-3.7 Sonnet. The best LLM result is 42.2% for DeepSeek-R1 in the TC setting. Even with chain-of-thought prompting, no model exceeds 55% accuracy on the full benchmark (Xiang et al., 25 May 2025).

Performance is also highly nonuniform across diagram classes and difficulty levels. For o4-mini in the TV setting, the accuracy range across the 21 diagram types spans about 31 percentage points between maximum and minimum. Categories such as Wave Motion, Circuit Diagram, and Coordinate System exhibit the largest TV–TO gaps, indicating strong dependence on diagram topology. The reported difficulty trend is non-monotonic: Senior Undergraduate and Advanced Olympiad questions are sometimes harder than PhD qualifiers, which the benchmark authors interpret as evidence that current models often rely on memorized cues rather than stable structural reasoning (Xiang et al., 25 May 2025).

4. Error taxonomy, misconceptions, and research significance

A central misconception addressed by SeePhys is that diagrams in physics benchmarks are mostly auxiliary. The benchmark is explicitly constructed to test the opposite condition for most of its instances: in 75% of problems, the diagram contains analytically indispensable information. This is reflected empirically. On the Vision-Optional subset, many MLLMs still gain significantly by seeing the diagram; examples reported include o3-mini with $\Delta_2=29.5\%$ and Claude-3.7 with $\Delta_2=56.1\%$. On the Vision-Essential subset, all models suffer large drops when visuals are removed; for example, o4-mini has $\Delta_2=35.7\%$ (Xiang et al., 25 May 2025).

The benchmark paper provides a manual analysis of a 10% stratified subset of o4-mini outputs and identifies four failure modes: Visual Misinterpretation, Modeling Flaws, Oversimplification, and False Assumptions. Visual Misinterpretation covers errors such as missing symbols, units, or graph scales. Modeling Flaws involve incorrect physical model selection, such as using the wrong circuit topology or misapplying conservation laws. Oversimplification refers to skipped constraints or omitted intermediate steps. False Assumptions involve unwarranted simplifications or conditions. For 100 common errors, the reported breakdown is: o4-mini, 15 VM / 61 MF / 8 FA / 6 OS; Gemini-2.5-Pro, 17 VM / 49 MF / 13 FA / 0 OS; Qwen2.5-VL-3B, 11 VM / 48 MF / 8 FA / 4 OS (Xiang et al., 25 May 2025).

These findings frame SeePhys as a benchmark of coupling failure rather than isolated OCR weakness. The paper identifies two central challenges: rigorous coupling of pixel-level diagram interpretation with downstream symbolic reasoning, and persistent reliance on textual shortcuts whenever text alone is partially sufficient. The proposed future directions are correspondingly structural: pre-training objectives that bind visual primitives such as nodes, edges, and units to symbolic physics concepts; hybrid neuro-symbolic architectures with learned OCR and graph extraction feeding formal reasoning engines; process-level evaluation metrics beyond outcome accuracy; and stronger OCR and layout parsing to reduce the $\Delta_3$ bottleneck (Xiang et al., 25 May 2025).

5. Caption-assisted reasoning and the winning SeePhys solution

The winning SeePhys solution, reported in “Multimodal Reasoning for Science: Technical Report and 1st Place Solution to the ICML 2025 SeePhys Challenge” (Liang et al., 7 Sep 2025), reorganizes multimodal reasoning into a two-stage caption-assisted pipeline. Given a figure-question pair $(I,Q)$, a caption generator first produces a structured natural-language description

$$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$0

and a textual reasoner then answers from the caption and question,

$$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$1

optionally with the original image reintegrated. The reported motivation is that converting sparse but critical visual relations into text improves cross-modal alignment and reduces spurious visual noise. Four prompt-based caption styles are explored—Default, Grounding, Structured, and Rephrasing—and the system may either rely on the caption alone or reinsert the image. When visual complexity is low, the caption alone may suffice; otherwise an adaptive answer-routing strategy selects between caption-only and image+text pipelines on a per-category basis (Liang et al., 7 Sep 2025).

The architecture uses an off-the-shelf ViT-based visual encoder, a causal Transformer LLM backbone, and a lightweight cross-modal fusion layer for optional vision reintegration. In the reintegrated case, a cross-attention block is inserted into decoder layers so that textual queries attend to visual token features. The report also formulates the method from a multi-task optimization perspective despite relying exclusively on pre-trained models and using no end-to-end fine-tuning. The objectives are caption generation loss $$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$2, optional contrastive alignment loss $$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$3, reasoning loss $$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$4, and a combined objective $$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$5, with the $$\mathrm{Acc}=\tfrac1N\sum_{i=1}^{N}\mathbf{1}\{\hat y_i=y_i\}.$$6 values emphasizing reasoning and caption quality (Liang et al., 7 Sep 2025).

On SeePhys-mini, the reported accuracy progression is as follows:

Method	Total accuracy (%)
Direct Multimodal (no caption)	58.0
Rephrasing	50.5
Default Captioning	58.5
Grounding Captioning	59.0
Structured Captioning	61.5
Structured + Image Reintegration	63.5
Structured + Img + Format Optimization (FO)	65.5
Structured + Img + FO + Critical Review (CR)	66.0

Across curricula, simple problems at the Middle School and Undergraduate levels already exceed 75% with Structured captions, while advanced problems at Olympiad and PhD levels require Critical Review and Format Optimization to reach their peaks of approximately 57–69%. Within the reported ablations, moving from Default to Grounding to Structured captions yields gains of +3 percentage points and then +6 percentage points; image reintegration adds about 2 points; answer-format standardization adds +2 points by reducing parse errors; and a second-pass Critical Review adds about 0.5 points. Adaptive Answer Routing helps only when the caption pipeline is weaker; once Structured captioning is robust, it degrades performance by reintroducing weaker image-only answers (Liang et al., 7 Sep 2025).

6. Transfer beyond physics, limitations, and prospective extensions

The caption-assisted method was also evaluated on MathVerse, a visual geometry benchmark, to test cross-domain transfer. The only reported change was replacement of physics-prompt templates with geometry-style prompts. In the selected results, Claude-Opus-4 goes from 60.2 without captions to 59.4 with caption and no image, but to 85.5 with caption and image; Gemini-2.5-Pro goes from 64.5 to 60.3 to 83.0; Qwen2.5-VL-72B goes from 70.2 to 67.4 to 71.6; and DeepSeek-R1 on the Vision-Only setting reaches 68.2 in the captioned text-only condition. The stated takeaway is that strong captioning yields gains of +15–25 percentage points over end-to-end multimodal input, and that even in a text-only setting, LLMs with captions can rival their vision-augmented counterparts (Liang et al., 7 Sep 2025).

The solution report also specifies its limitations. Overly dense figures, such as circuit diagrams with many small symbols, can overwhelm captioners; in such cases, direct vision-language reasoning may dominate. This suggests that the efficacy of caption mediation depends on whether the figure can be compressed into a concise but complete textual abstraction without losing fine-grained topology or symbolic detail. The report proposes several possible improvements: dynamically adjusting caption granularity, integrating OCR modules for text-heavy diagrams, adding Program-of-Thought chaining or symbolic solvers, introducing adaptive caption-length controllers to balance completeness against token cost, and using human-in-the-loop refinement for high-stakes scientific reasoning (Liang et al., 7 Sep 2025).

Taken together, the benchmark and the winning solution define SeePhys as both a diagnostic instrument and a methodological testbed. The benchmark isolates failures in multimodal physics reasoning under diagram dependence, while the challenge results indicate that a caption-assisted decomposition of perception and inference can materially improve performance without end-to-end fine-tuning. The reported evidence therefore positions SeePhys as a focal resource for studying how visual structure, OCR, prompt design, symbolic reasoning, and cross-modal alignment interact in scientific problem solving (Xiang et al., 25 May 2025, Liang et al., 7 Sep 2025).