---
title: 'VisPhyBench: Executable Physical Reasoning'
url: https://www.emergentmind.com/topics/visphybench
type: topic
---

# VisPhyBench: Executable Physical Reasoning

VisPhyBench is a benchmark within the VisPhyWorld framework for evaluating physical reasoning in multimodal large language models by requiring executable simulator code rather than recognition-style answers. It comprises 209 evaluation scenes derived from 108 physical templates and is designed to test whether models can reconstruct appearance and reproduce physically plausible motion from visual observations. The benchmark operationalizes physical reasoning as code-driven video reconstruction: given visual evidence, a model must produce runnable code whose behavior is directly inspectable, editable, and falsifiable, thereby separating physical reasoning from rendering [2602.13294].

## 1. Conceptual basis and problem formulation

VisPhyBench is motivated by the claim that evaluating whether MLLMs genuinely reason about physical dynamics remains challenging because many existing protocols are recognition-style, such as Visual Question Answering and Violation of Expectation, and can often be answered without committing to an explicit, testable physical hypothesis [2602.13294]. In response, VisPhyBench adopts an execution-based protocol in which success depends on generating simulator code that can be run and compared against reference dynamics.

The benchmark’s central task is not merely scene description. The objective is: given initial scene information, specifically two visual frames and optionally a detection context, the model must output executable simulator code that recreates the scene and generates a physically plausible video of subsequent dynamics [2602.13294]. This establishes a stricter notion of physical reasoning than answer selection or textual explanation, because the model’s inferred world representation must support rollout under deterministic simulation.

A key methodological consequence is the explicit separation between semantic scene understanding and physical inference. The benchmark therefore distinguishes recognition of objects and gross movement from inference of physical parameters such as position, size, velocity, and impulse, and from the simulation of physically governed dynamics [2602.13294]. This suggests that VisPhyBench is intended not only as a leaderboard instrument but also as an analytical framework for decomposing failure modes.

## 2. Benchmark composition and scene construction

VisPhyBench contains **209 reconstructed evaluation scenes** derived from **108 unique physical templates**, with both **2D and 3D scenarios** [2602.13294]. The 2D scenes are extended from Phy World and rendered using the PHYRE engine, whereas the 3D scenes are curated with Three.js and Cannon.js for rigid-body simulation [2602.13294]. The benchmark therefore spans both planar and volumetric settings while preserving executable structure.

Scenes incorporate multiple object categories, including **circles, lines, rectangles, U-shapes, triangles, and composites** [2602.13294]. Each template is a stochastic executable script, with object positions and sizes sampled using seeds. This means that the benchmark is built around parameterized physical templates rather than static visual exemplars, which supports controlled variation without changing the underlying physical structure.

Each scene is human-rated on a **1–5** scale by **STEM-graduate annotators** and stratified into **easy, medium, and hard** [2602.13294]. The inputs for each sample include two key visual frames, denoted $\mathbf{I}_{\text{start}}$ and $\mathbf{I}_{\text{later}}$, and may also include an optional **JSON-based object detection annotation** on the first frame giving object type, position, geometry, and color [2602.13294].

The following table summarizes the benchmark’s structural elements.

| Component | Description | Source |
|---|---|---|
| Evaluation scenes | 209 reconstructed scenes | [2602.13294] |
| Physical templates | 108 unique templates, including 2D and 3D | [2602.13294] |
| Inputs | $\mathbf{I}_{\text{start}}$, $\mathbf{I}_{\text{later}}$, optional JSON detection context | [2602.13294] |

## 3. Execution-based protocol and code generation pipeline

The evaluation protocol proceeds through a multi-stage pipeline. First, the model receives $(\mathbf{I}_{\text{start}}, \mathbf{I}_{\text{later}}, D)$, where the frames capture early and later scene state and $D$ denotes the optional detection context [2602.13294]. Second, the model produces an English textual summary of the scene and observed motion, which is used to gauge general semantic understanding. Third, the model outputs complete executable code in **HTML/JavaScript/Three.js or P5.js** [2602.13294].

The generated code is then sanitized, canonicalized, and checked for sanity, including correct structure, callables, canvas, and finite state [2602.13294]. After validation, the code is executed under tightly controlled deterministic conditions with a fixed physics step, static camera, set frame rate, and fixed duration. If execution fails, an error-conditioned retry is attempted; if the second attempt also fails, a minimal fallback template is used to ensure that an output is produced [2602.13294]. The resulting artifacts include scene analysis text, a JSON object layout for the first frame, an executable code file, and a rendered video.

The preferred backends are **Three.js** and **P5.js**, which are described as physics-aware. **SVG** and **Manim** are treated as scripted or non-physical reference backends and often produce non-physical outputs [2602.13294]. The prompting protocol is single-call and structured: the model is instructed to analyze objects, movement, and plausible causes, then emit only an HTML+JavaScript code block that constructs a simulator scene matching the input and uses the engine’s physics to roll forward [2602.13294].

Determinism is an explicit part of the protocol. All rendering and simulation use fixed camera, physics time-step, frame rate, and clip duration; 2D scenes additionally constrain motion by initializing $z = 0$ and clamping non-planar motion, while 3D scenes omit this planar constraint but keep the other parameters fixed [2602.13294]. A plausible implication is that determinism is used to reduce evaluation variance and to make cross-model comparisons attributable to reasoning quality rather than execution stochasticity.

## 4. Metric design and the separation of semantics from dynamics

VisPhyBench measures two distinct capabilities. The first is **semantic scene understanding**, defined as accurate verbal description of object layout, interactions, and gross movement between key frames. This is evaluated using **BERTScore-F1**, **ROUGE-L**, and **CLIP-Cap**, comparing the model’s English analysis against a GPT-generated reference description or against the video via CLIP-based similarity [2602.13294].

The second is **physical parameter and dynamics inference**, defined as the accuracy of physical parameterization and the plausibility of the resulting simulation. These measurements are grouped into several categories [2602.13294]:

- **Reconstruction/perceptual quality**: **PSNR**, **SSIM**, **LPIPS**, **FSIM**, **VSI**, **DISTS**.
- **Visual semantic consistency**: **CLIP-Img**, **DINO**.
- **Motion/physical plausibility**: **RAFT-EPE**, **RAFT Angular Error**, **Temporal Alignment**.
- **Subjective holistic judgment**: **Gemini (1–10 scale)**.

The formalization given in the source emphasizes frame-sequence comparison. Let $X^{GT} = (\mathbf{I}_t)_{t=1}^T$ denote ground-truth frames and $X = (\hat{\mathbf{I}}_t)_{t=1}^T$ generated frames. The paper defines, for example, frame-averaged LPIPS and CLIP-based similarity, and it defines optical-flow error using RAFT after temporal alignment [2602.13294]:

$$
\operatorname{LPIPS}(X, X^{GT}) = \frac{1}{T} \sum_{t=1}^T \text{LPIPS}(\mathbf{I}_t, \hat{\mathbf{I}}_t)
$$

$$
\operatorname{CLIP\text{-}Img}(X, X^{GT}) = \frac{1}{T} \sum_{t=1}^T \cos(\phi(\mathbf{I}_t), \phi(\hat{\mathbf{I}}_t))
$$

$$
\text{RAFT-EPE}(X, X^{GT}) = \frac{1}{T-1} \sum_{t=1}^{T-1} \left\| F_{\text{raft}}(\mathbf{I}_t, \mathbf{I}_{t+1}) - F_{\text{raft}}(\hat{\mathbf{I}}_t, \hat{\mathbf{I}}_{t+1}) \right\|_2
$$

SSIM is also defined in the source in its standard form as a structural similarity measure averaged across frames and channels [2602.13294]. Gemini is prompted as an MLLM judge to penalize physically impossible events such as missed collisions or unexpected static behavior, with **10** corresponding to a perfect physical and visual match and **1** to severe violation of physics [2602.13294].

This metric design makes the benchmark diagnostically granular. High **CLIP-Img**, **DINO**, or **BERTScore-F1** can indicate competent scene parsing, while poor **RAFT-EPE** or low **Gemini** scores can reveal weak physical inference despite semantic competence [2602.13294]. That distinction is central to the benchmark’s stated goal of separating visual pattern matching from grounded physical reasoning.

## 5. Baseline performance and empirical findings

The pipeline produces valid reconstructed videos in approximately **97.7%** of cases on the benchmark when one retry is allowed [2602.13294]. The paper reports that the repair step meaningfully increases execution success. This high execution rate is operationally important because the evaluation depends on the model being able to emit runnable code rather than merely plausible-looking syntax.

Representative baseline results cover multiple MLLMs and engines. The reported figures show that semantic scene understanding is relatively strong, but physical dynamics modeling remains weak [2602.13294].

| Model/Engine | RAFT-EPE ↓ | Gemini ↑ | Success Rate |
|---|---:|---:|---:|
| GPT-5, Three.js | 33.65 | 3.50 | 99.0% |
| Gemini-3-Pro, Three.js | 36.20 | 3.80 | 95.7% |
| Claude Sonnet 4.5, Three.js | 36.20 | 2.39 | 99.5% |
| Qwen3-VL-Plus, Three.js | 35.05 | 2.12 | 93.6% |
| Veo-3.1 | 32.71 | 2.62 | 100.0% |
| SVD | 45.46 | 1.43 | 100.0% |

The source also reports additional metrics for these systems, including **LPIPS**, **CLIP-Img**, **DINO**, and **BERTScore-F1**. For example, **GPT-5, Three.js** obtains **LPIPS 0.1736**, **CLIP-Img 0.8930**, **DINO 0.8556**, and **BERTScore-F1 0.8436**, while **Gemini-3-Pro, Three.js** obtains **LPIPS 0.1399**, **CLIP-Img 0.8973**, **DINO 0.8405**, and **BERTScore-F1 0.8460** [2602.13294]. Despite these relatively strong semantic and perceptual indicators, **Gemini** scores remain low, with the reported maximum around **3.8–4.0** on a 10-point scale [2602.13294].

Several empirical conclusions follow directly from the reported results. First, **semantic understanding is strong in MLLMs**, as indicated by high **CLIP**, **DINO**, and **BERTScore** values. Second, **physical dynamics modeling is weaker**, as indicated by substantial optical-flow error and low subjective physical-plausibility judgments [2602.13294]. Third, **Three.js outperforms P5.js** across all LLM backbones because its physics solver is more expressive [2602.13294]. Fourth, **pixel-space baselines** may achieve good appearance metrics, but they do not provide interpretable or editable simulators and are still exposed by the physics-oriented evaluation, particularly the **Gemini** judge and qualitative analysis [2602.13294].

A common misconception would be to interpret good perceptual similarity as evidence of physical reasoning. The benchmark explicitly argues against this inference: high-level layout and object identity can be recovered without accurate inference of force, collision, timing, or causal structure [2602.13294].

## 6. Relation to adjacent benchmarks and broader significance

VisPhyBench occupies a distinct position within the broader landscape of physics-oriented multimodal evaluation. In **SeePhys**, the primary task is open-ended physics question answering grounded in diagrams, with **75%** of questions categorized as **Vision-Essential** and evaluation performed under modalities such as **Text+Vision**, **Text+Caption**, **Text Only**, and **Vision Only** [2505.19099]. The comparison provided in the source states that SeePhys emphasizes vision-essentiality, high-level coverage up to PhD and advanced Olympiad difficulty, and fine-grained diagram labels, whereas VisPhyBench is grouped with related multimodal benchmarks that include physics diagrams [2505.19099]. This suggests that SeePhys probes diagram-grounded reasoning, while VisPhyBench probes executable reconstruction of dynamic physical scenes.

**VideoPhy** addresses a different setting: physical commonsense in text-to-video generation. It evaluates generated videos using human judgments of **semantic adherence** and **physical commonsense**, and reports that the best model achieves only **19.7%** joint success [2406.03520]. Its proposed auto-evaluator, **VideoCon-Physics**, predicts Yes/No judgments for caption adherence and physical-law conformity. Relative to VisPhyBench, VideoPhy remains centered on evaluating generated video outputs rather than requiring an explicit executable world model [2406.03520].

**PCMDE** targets still-image evaluation for multimodal synthetic images, especially aircraft side-view images, through a pipeline combining object detection, confidence-weighted fusion, deterministic rules, and LLM reasoning [2511.15204]. It yields a **[0, 100] realism score** and pass/fail decisions based on structural and relational correctness. Its relevance to “a VisPhyBench” is described in the source as stemming from its use of explicit domain constraints rather than generic semantic similarity [2511.15204]. In contrast, VisPhyBench evaluates simulator-based reconstruction of motion and dynamics, not only component-level structural realism.

Taken together, these benchmarks illuminate complementary axes of physics evaluation. SeePhys emphasizes diagram-dependent reasoning, VideoPhy emphasizes physical commonsense in generated video, PCMDE emphasizes physics-constrained structural realism in synthetic images, and VisPhyBench emphasizes executable, falsifiable reconstruction of dynamic scenes. A plausible implication is that VisPhyBench’s distinctive contribution lies in making physical hypotheses auditable through code rather than treating model judgments or generations as terminal outputs.

## 7. Interpretability, falsifiability, and research implications

VisPhyBench is designed to promote **interpretability and falsifiability**. Because the model produces code, errors can be traced directly either to misunderstanding of objects and layout or to incorrect parameterization and application of physics [2602.13294]. The code is inspectable, editable, and amenable to controlled ablation; the source explicitly notes that one can intervene on variables such as **mass** and **friction** to isolate failure modes [2602.13294].

This property differentiates the benchmark from pixel-level generation settings, where a plausible output may conceal the reasoning process. In VisPhyBench, the executable scene serves as a testable hypothesis about the underlying world. If a reconstructed object fails to collide, moves with an implausible impulse, or remains static when it should move, the failure is exposed both in rendered behavior and in the simulator specification itself [2602.13294].

The benchmark’s empirical findings support a broader conclusion: current MLLMs can often “see” scenes well enough to describe them, but they struggle to infer precise physical parameters and to simulate consistent physical dynamics [2602.13294]. This should not be read as a claim that semantic competence is unimportant; rather, the benchmark isolates a narrower but more demanding capability. High performance on recognition-style tasks or perceptual similarity metrics does not entail reliable physical reasoning when models must commit to an executable world model.

In this sense, VisPhyBench frames physical reasoning as an auditable systems problem. Its benchmark design, deterministic execution protocol, repair mechanism, and metric decomposition collectively define a rigorous setting in which physical understanding is evaluated through reconstruction, simulation, and falsification rather than through answer plausibility alone [2602.13294].

Source: https://www.emergentmind.com/topics/visphybench