VisPhyBench: Executable Physical Reasoning

• VisPhyBench is a benchmark that challenges models to generate executable simulator code, enabling precise physical scene reconstruction.
• It comprises 209 evaluation scenes from 108 templates across 2D and 3D environments, clearly separating semantic understanding from physical dynamics.
• The framework emphasizes reproducibility and interpretability by using detailed metrics to evaluate both scene parsing and physical plausibility.

VisPhyBench is a benchmark within the VisPhyWorld framework for evaluating physical reasoning in multimodal LLMs by requiring executable simulator code rather than recognition-style answers. It comprises 209 evaluation scenes derived from 108 physical templates and is designed to test whether models can reconstruct appearance and reproduce physically plausible motion from visual observations. The benchmark operationalizes physical reasoning as code-driven video reconstruction: given visual evidence, a model must produce runnable code whose behavior is directly inspectable, editable, and falsifiable, thereby separating physical reasoning from rendering (Liang et al., 9 Feb 2026).

1. Conceptual basis and problem formulation

VisPhyBench is motivated by the claim that evaluating whether MLLMs genuinely reason about physical dynamics remains challenging because many existing protocols are recognition-style, such as Visual Question Answering and Violation of Expectation, and can often be answered without committing to an explicit, testable physical hypothesis (Liang et al., 9 Feb 2026). In response, VisPhyBench adopts an execution-based protocol in which success depends on generating simulator code that can be run and compared against reference dynamics.

The benchmark’s central task is not merely scene description. The objective is: given initial scene information, specifically two visual frames and optionally a detection context, the model must output executable simulator code that recreates the scene and generates a physically plausible video of subsequent dynamics (Liang et al., 9 Feb 2026). This establishes a stricter notion of physical reasoning than answer selection or textual explanation, because the model’s inferred world representation must support rollout under deterministic simulation.

A key methodological consequence is the explicit separation between semantic scene understanding and physical inference. The benchmark therefore distinguishes recognition of objects and gross movement from inference of physical parameters such as position, size, velocity, and impulse, and from the simulation of physically governed dynamics (Liang et al., 9 Feb 2026). This suggests that VisPhyBench is intended not only as a leaderboard instrument but also as an analytical framework for decomposing failure modes.

2. Benchmark composition and scene construction

VisPhyBench contains 209 reconstructed evaluation scenes derived from 108 unique physical templates, with both 2D and 3D scenarios (Liang et al., 9 Feb 2026). The 2D scenes are extended from Phy World and rendered using the PHYRE engine, whereas the 3D scenes are curated with Three.js and Cannon.js for rigid-body simulation (Liang et al., 9 Feb 2026). The benchmark therefore spans both planar and volumetric settings while preserving executable structure.

Scenes incorporate multiple object categories, including circles, lines, rectangles, U-shapes, triangles, and composites (Liang et al., 9 Feb 2026). Each template is a stochastic executable script, with object positions and sizes sampled using seeds. This means that the benchmark is built around parameterized physical templates rather than static visual exemplars, which supports controlled variation without changing the underlying physical structure.

Each scene is human-rated on a 1–5 scale by STEM-graduate annotators and stratified into easy, medium, and hard (Liang et al., 9 Feb 2026). The inputs for each sample include two key visual frames, denoted $\mathbf{I}_{\text{start}}$ and $\mathbf{I}_{\text{later}}$, and may also include an optional JSON-based object detection annotation on the first frame giving object type, position, geometry, and color (Liang et al., 9 Feb 2026).

The following table summarizes the benchmark’s structural elements.

Component	Description	Source
Evaluation scenes	209 reconstructed scenes	(Liang et al., 9 Feb 2026)
Physical templates	108 unique templates, including 2D and 3D	(Liang et al., 9 Feb 2026)
Inputs	$\mathbf{I}_{\text{start}}$, $\mathbf{I}_{\text{later}}$, optional JSON detection context	(Liang et al., 9 Feb 2026)

3. Execution-based protocol and code generation pipeline

The evaluation protocol proceeds through a multi-stage pipeline. First, the model receives $$(\mathbf{I}_{\text{start}}, \mathbf{I}_{\text{later}}, D)$$, where the frames capture early and later scene state and $D$ denotes the optional detection context (Liang et al., 9 Feb 2026). Second, the model produces an English textual summary of the scene and observed motion, which is used to gauge general semantic understanding. Third, the model outputs complete executable code in HTML/JavaScript/Three.js or P5.js (Liang et al., 9 Feb 2026).

The generated code is then sanitized, canonicalized, and checked for sanity, including correct structure, callables, canvas, and finite state (Liang et al., 9 Feb 2026). After validation, the code is executed under tightly controlled deterministic conditions with a fixed physics step, static camera, set frame rate, and fixed duration. If execution fails, an error-conditioned retry is attempted; if the second attempt also fails, a minimal fallback template is used to ensure that an output is produced (Liang et al., 9 Feb 2026). The resulting artifacts include scene analysis text, a JSON object layout for the first frame, an executable code file, and a rendered video.

The preferred backends are Three.js and P5.js, which are described as physics-aware. SVG and Manim are treated as scripted or non-physical reference backends and often produce non-physical outputs (Liang et al., 9 Feb 2026). The prompting protocol is single-call and structured: the model is instructed to analyze objects, movement, and plausible causes, then emit only an HTML+JavaScript code block that constructs a simulator scene matching the input and uses the engine’s physics to roll forward (Liang et al., 9 Feb 2026).

Determinism is an explicit part of the protocol. All rendering and simulation use fixed camera, physics time-step, frame rate, and clip duration; 2D scenes additionally constrain motion by initializing $z = 0$ and clamping non-planar motion, while 3D scenes omit this planar constraint but keep the other parameters fixed (Liang et al., 9 Feb 2026). A plausible implication is that determinism is used to reduce evaluation variance and to make cross-model comparisons attributable to reasoning quality rather than execution stochasticity.

4. Metric design and the separation of semantics from dynamics

VisPhyBench measures two distinct capabilities. The first is semantic scene understanding, defined as accurate verbal description of object layout, interactions, and gross movement between key frames. This is evaluated using BERTScore-F1, ROUGE-L, and CLIP-Cap, comparing the model’s English analysis against a GPT-generated reference description or against the video via CLIP-based similarity (Liang et al., 9 Feb 2026).

The second is physical parameter and dynamics inference, defined as the accuracy of physical parameterization and the plausibility of the resulting simulation. These measurements are grouped into several categories (Liang et al., 9 Feb 2026):

• Reconstruction/perceptual quality: PSNR, SSIM, LPIPS, FSIM, VSI, DISTS.
• Visual semantic consistency: CLIP-Img, DINO.
• Motion/physical plausibility: RAFT-EPE, RAFT Angular Error, Temporal Alignment.
• Subjective holistic judgment: Gemini (1–10 scale).

The formalization given in the source emphasizes frame-sequence comparison. Let $X^{GT} = (\mathbf{I}_t)_{t=1}^T$ denote ground-truth frames and $X = (\hat{\mathbf{I}}_t)_{t=1}^T$ generated frames. The paper defines, for example, frame-averaged LPIPS and CLIP-based similarity, and it defines optical-flow error using RAFT after temporal alignment (Liang et al., 9 Feb 2026):

$$\operatorname{LPIPS}(X, X^{GT}) = \frac{1}{T} \sum_{t=1}^T \text{LPIPS}(\mathbf{I}_t, \hat{\mathbf{I}}_t)$$

$\mathbf{I}_{\text{later}}$0

$\mathbf{I}_{\text{later}}$1

SSIM is also defined in the source in its standard form as a structural similarity measure averaged across frames and channels (Liang et al., 9 Feb 2026). Gemini is prompted as an MLLM judge to penalize physically impossible events such as missed collisions or unexpected static behavior, with 10 corresponding to a perfect physical and visual match and 1 to severe violation of physics (Liang et al., 9 Feb 2026).

This metric design makes the benchmark diagnostically granular. High CLIP-Img, DINO, or BERTScore-F1 can indicate competent scene parsing, while poor RAFT-EPE or low Gemini scores can reveal weak physical inference despite semantic competence (Liang et al., 9 Feb 2026). That distinction is central to the benchmark’s stated goal of separating visual pattern matching from grounded physical reasoning.

5. Baseline performance and empirical findings

The pipeline produces valid reconstructed videos in approximately 97.7% of cases on the benchmark when one retry is allowed (Liang et al., 9 Feb 2026). The paper reports that the repair step meaningfully increases execution success. This high execution rate is operationally important because the evaluation depends on the model being able to emit runnable code rather than merely plausible-looking syntax.

Representative baseline results cover multiple MLLMs and engines. The reported figures show that semantic scene understanding is relatively strong, but physical dynamics modeling remains weak (Liang et al., 9 Feb 2026).

Model/Engine	RAFT-EPE ↓	Gemini ↑	Success Rate
GPT-5, Three.js	33.65	3.50	99.0%
Gemini-3-Pro, Three.js	36.20	3.80	95.7%
Claude Sonnet 4.5, Three.js	36.20	2.39	99.5%
Qwen3-VL-Plus, Three.js	35.05	2.12	93.6%
Veo-3.1	32.71	2.62	100.0%
SVD	45.46	1.43	100.0%

The source also reports additional metrics for these systems, including LPIPS, CLIP-Img, DINO, and BERTScore-F1. For example, GPT-5, Three.js obtains LPIPS 0.1736, CLIP-Img 0.8930, DINO 0.8556, and BERTScore-F1 0.8436, while Gemini-3-Pro, Three.js obtains LPIPS 0.1399, CLIP-Img 0.8973, DINO 0.8405, and BERTScore-F1 0.8460 (Liang et al., 9 Feb 2026). Despite these relatively strong semantic and perceptual indicators, Gemini scores remain low, with the reported maximum around 3.8–4.0 on a 10-point scale (Liang et al., 9 Feb 2026).

Several empirical conclusions follow directly from the reported results. First, semantic understanding is strong in MLLMs, as indicated by high CLIP, DINO, and BERTScore values. Second, physical dynamics modeling is weaker, as indicated by substantial optical-flow error and low subjective physical-plausibility judgments (Liang et al., 9 Feb 2026). Third, Three.js outperforms P5.js across all LLM backbones because its physics solver is more expressive (Liang et al., 9 Feb 2026). Fourth, pixel-space baselines may achieve good appearance metrics, but they do not provide interpretable or editable simulators and are still exposed by the physics-oriented evaluation, particularly the Gemini judge and qualitative analysis (Liang et al., 9 Feb 2026).

A common misconception would be to interpret good perceptual similarity as evidence of physical reasoning. The benchmark explicitly argues against this inference: high-level layout and object identity can be recovered without accurate inference of force, collision, timing, or causal structure (Liang et al., 9 Feb 2026).

6. Relation to adjacent benchmarks and broader significance

VisPhyBench occupies a distinct position within the broader landscape of physics-oriented multimodal evaluation. In SeePhys, the primary task is open-ended physics question answering grounded in diagrams, with 75% of questions categorized as Vision-Essential and evaluation performed under modalities such as Text+Vision, Text+Caption, Text Only, and Vision Only (Xiang et al., 25 May 2025). The comparison provided in the source states that SeePhys emphasizes vision-essentiality, high-level coverage up to PhD and advanced Olympiad difficulty, and fine-grained diagram labels, whereas VisPhyBench is grouped with related multimodal benchmarks that include physics diagrams (Xiang et al., 25 May 2025). This suggests that SeePhys probes diagram-grounded reasoning, while VisPhyBench probes executable reconstruction of dynamic physical scenes.

VideoPhy addresses a different setting: physical commonsense in text-to-video generation. It evaluates generated videos using human judgments of semantic adherence and physical commonsense, and reports that the best model achieves only 19.7% joint success (Bansal et al., 2024). Its proposed auto-evaluator, VideoCon-Physics, predicts Yes/No judgments for caption adherence and physical-law conformity. Relative to VisPhyBench, VideoPhy remains centered on evaluating generated video outputs rather than requiring an explicit executable world model (Bansal et al., 2024).

PCMDE targets still-image evaluation for multimodal synthetic images, especially aircraft side-view images, through a pipeline combining object detection, confidence-weighted fusion, deterministic rules, and LLM reasoning (Gupta et al., 19 Nov 2025). It yields a [0, 100] realism score and pass/fail decisions based on structural and relational correctness. Its relevance to “a VisPhyBench” is described in the source as stemming from its use of explicit domain constraints rather than generic semantic similarity (Gupta et al., 19 Nov 2025). In contrast, VisPhyBench evaluates simulator-based reconstruction of motion and dynamics, not only component-level structural realism.

Taken together, these benchmarks illuminate complementary axes of physics evaluation. SeePhys emphasizes diagram-dependent reasoning, VideoPhy emphasizes physical commonsense in generated video, PCMDE emphasizes physics-constrained structural realism in synthetic images, and VisPhyBench emphasizes executable, falsifiable reconstruction of dynamic scenes. A plausible implication is that VisPhyBench’s distinctive contribution lies in making physical hypotheses auditable through code rather than treating model judgments or generations as terminal outputs.

7. Interpretability, falsifiability, and research implications

VisPhyBench is designed to promote interpretability and falsifiability. Because the model produces code, errors can be traced directly either to misunderstanding of objects and layout or to incorrect parameterization and application of physics (Liang et al., 9 Feb 2026). The code is inspectable, editable, and amenable to controlled ablation; the source explicitly notes that one can intervene on variables such as mass and friction to isolate failure modes (Liang et al., 9 Feb 2026).

This property differentiates the benchmark from pixel-level generation settings, where a plausible output may conceal the reasoning process. In VisPhyBench, the executable scene serves as a testable hypothesis about the underlying world. If a reconstructed object fails to collide, moves with an implausible impulse, or remains static when it should move, the failure is exposed both in rendered behavior and in the simulator specification itself (Liang et al., 9 Feb 2026).

The benchmark’s empirical findings support a broader conclusion: current MLLMs can often “see” scenes well enough to describe them, but they struggle to infer precise physical parameters and to simulate consistent physical dynamics (Liang et al., 9 Feb 2026). This should not be read as a claim that semantic competence is unimportant; rather, the benchmark isolates a narrower but more demanding capability. High performance on recognition-style tasks or perceptual similarity metrics does not entail reliable physical reasoning when models must commit to an executable world model.

In this sense, VisPhyBench frames physical reasoning as an auditable systems problem. Its benchmark design, deterministic execution protocol, repair mechanism, and metric decomposition collectively define a rigorous setting in which physical understanding is evaluated through reconstruction, simulation, and falsification rather than through answer plausibility alone (Liang et al., 9 Feb 2026).