---
title: 'VisPhyWorld: Code-Based Physical Reasoning'
url: https://www.emergentmind.com/topics/visphyworld
type: topic
---

# VisPhyWorld: Code-Based Physical Reasoning

to=arxiv_search.search  อาคารจีเอ็มเอ็ม json
{"query":"VisPhyWorld physical reasoning code-driven video reconstruction 2602.13294 related VideoPhy ComPhy Phys4D", "max_results": 10}
to=arxiv_search.search  彩票天天乐json
{"query":"ti:\"VisPhyWorld\" OR ti:\"VideoPhy\" OR ti:\"ComPhy\" OR ti:\"Phys4D\" OR ti:\"PhyEditBench\"", "max_results": 10}
VisPhyWorld is an execution-based framework for probing physical reasoning in multimodal large language models by requiring them to generate executable simulator code from visual observations rather than merely answering recognition-style questions. The generated code is then sanitized, executed, and evaluated through reconstructed video, making the inferred world representation directly inspectable, editable, and falsifiable. In the formulation introduced with VisPhyBench, the framework operates on 209 evaluation scenes derived from 108 physical templates, and its pipeline produces valid reconstructed videos in 97.7% on the benchmark [2602.13294].

## 1. Conceptual basis

VisPhyWorld is motivated by a critique of dominant evaluation protocols for physical reasoning. Visual Question Answering and Violation of Expectation typically reward recognition and surface pattern matching, and can often be answered without committing to an explicit physical hypothesis. This is especially limiting for code-capable multimodal models, whose outputs are not predictive likelihoods over futures but structured artifacts such as scene descriptions, programs, or agentic plans. VisPhyWorld replaces answer selection with executable reconstruction: a model must transform visual observations into simulator code and reproduce scene dynamics under a deterministic execution contract.

This design yields three properties that the framework treats as central. First, the hypothesis is **inspectable**, because object states, forces, and parameters are explicit in code. Second, it is **editable**, because parameters such as friction or mass can be modified and the scene rerun. Third, it is **falsifiable**, because failure is observed by executing the program rather than inferred from linguistic inconsistency. This separates physical reasoning from rendering: the model is not primarily judged on whether it can describe a scene, but on whether it can instantiate a consistent world model that reproduces motion [2602.13294].

A recurrent misconception addressed by the framework is that strong semantic scene understanding is equivalent to strong physical reasoning. The reported experiments indicate the opposite pattern: state-of-the-art models often reconstruct object identity and coarse layout well, yet still struggle to infer physical parameters and to simulate consistent physical dynamics. Another misconception is that low motion error alone suffices; VisPhyWorld explicitly notes that RAFT-EPE must be interpreted jointly with holistic physics judgments because low EPE can arise from degenerate outputs.

## 2. Execution pipeline and world representation

The VisPhyWorld pipeline takes as input two key frames, denoted \(I_{\text{start}}\) and \(I_{\text{later}}\), together with an optional detection context \(D\) containing object categories, bounding boxes, and colors in pixel space. An LLM agent then produces three intermediate artifacts: a motion analysis \(A\), a first-frame JSON scene specification \(S\), and executable code \(C\). The code is validated, optionally repaired once using error-conditioned retry, and then executed deterministically to generate a reconstructed video \(\hat{X}\), which is evaluated against ground truth \(X\) [2602.13294].

The framework prioritizes physics-capable backends. For the 3D subset it uses **Three.js + Cannon.js**, with a fixed perspective camera and white background. **P5.js** is also used as a code-generation target. For comparison and diagnosis, non-physics backends such as **SVG** and **Manim** are included; these often produce implausible contacts, including interpenetration. The 2D subset is generated with **PHYRE**. Coordinate conventions are explicit: the detection context uses image pixel coordinates with origin at the top left, the 2D subset uses a fixed orthographic camera, and the 3D subset uses a fixed perspective camera.

The generated program is expected to expose a structured world representation. Environment properties include gravity, time step, camera intrinsics, and background. Object definitions include primitive shapes such as spheres, boxes, lines or ramps, and U-shaped containers; geometry; material properties such as friction \(\mu\) and restitution \(e\); and mass \(m\). Initial states align positions and orientations to the first frame, with optional initial velocities or impulses used to match motion between \(I_{\text{start}}\) and \(I_{\text{later}}\). The simulator then steps the scene with a fixed time step and deterministic recording. The paper explicitly references rigid-body dynamics through \(F = ma\) and Coulomb-like friction through \(F_f = \mu N\) [2602.13294].

This representation makes parameter-level diagnosis possible even though VisPhyBench v1 does not yet report parameter inference accuracy as a standalone metric. A plausible implication is that VisPhyWorld is less a benchmark of free-form video realism than a benchmark of whether a model can externalize a mechanically meaningful latent world model.

## 3. VisPhyBench benchmark design

VisPhyBench consists of **108 physical templates instantiated into 209 videos with first-frame JSON annotations**. The 2D subset is derived from and extends PhyWorld’s 2D data and is rendered via PHYRE-style scripts. The 3D subset is rendered with Three.js and simulated with Cannon.js. Scene types include stacks, ramps, collisions, slides, bounces, and topples, with objects such as spheres, boxes, planks, lines or ramps, and U-shaped containers [2602.13294].

The benchmark includes a full “sub” split of 209 scenes with difficulty labels assigned by eight annotators. Difficulty is partitioned as **114 easy, 67 medium, and 28 hard**. A separate test split contains **49 scenes**, comprising **29 easy, 17 medium, and 3 hard**. Rendering is standardized through fixed camera settings, white background, deterministic time step, and fixed recording protocol.

For the 2D subset, object-type prevalence is reported directly: **circle 100.0% of scenes, line 83.2%, rectangle 62.8%, u-shape 24.6%, triangle 6.3%, composite 7.3%**. These statistics emphasize that the benchmark is not narrowly restricted to a single collision archetype but spans a controlled repertoire of rigid-body configurations.

The construction philosophy is programmatic. Templates are executable programs whose seeds sample placements, sizes, and sometimes material parameters. This preserves controllability while producing diversity across instantiations. The dataset is accompanied by prompt templates, validation and canonicalization procedures, a single-retry repair protocol, and a fallback template that guarantees well-defined evaluation. The project page is listed as `https://tiger-ai-lab.github.io/VisPhyWorld/` [2602.13294].

## 4. Evaluation protocol

VisPhyWorld uses a multi-metric protocol that evaluates reconstructed videos after temporal alignment. A reconstructed output counts as a valid reconstructed video if the generated code executes and produces a non-empty clip without resorting to fallback; one automatic repair attempt is allowed. At the pipeline level, the reported validity is **97.7%** [2602.13294].

The metric suite is deliberately heterogeneous. **Appearance reconstruction** is measured with PSNR and SSIM. **Perceptual quality** uses LPIPS, FSIM, VSI, and DISTS. **Visual semantic consistency** uses CLIP-Img and DINO cosine similarity. **Text-video and analysis-text consistency** uses CLIP-Cap, ROUGE-L, and BERTScore-F1. **Motion and physical plausibility** are assessed with RAFT-EPE, computed after coarse-to-fine temporal alignment using offset search and DTW. In addition, a **Gemini-2.5-Pro holistic judge** assigns a score from 1 to 10 and explicitly penalizes physical violations.

No single composite score is defined. This choice reflects the framework’s diagnostic orientation: different metrics isolate different parts of the failure surface. The paper notes, for example, that parameter inference accuracy is not reported in the current release, even though the code representation would make a metric of the form \(E_\theta = (1 / |\Theta|)\sum_{\theta \in \Theta} |\theta - \hat{\theta}|\) feasible in future work.

The evaluation philosophy differs from purely perceptual benchmarking in one important respect: a plausible video is not sufficient if it emerges from an opaque generator. VisPhyWorld evaluates the executable hypothesis itself. This makes failure attribution more precise, because errors can be localized to scene initialization, parameter choice, or constraint modeling rather than collapsed into a single visual score [2602.13294].

## 5. Empirical results and failure modes

The evaluated code-generation backbones are **GPT-5, GPT-4.1, Gemini-3-Pro, Claude Sonnet 4.5, and Qwen3-VL-Plus**, used with Three.js and P5.js. Pixel-space baselines are **Stable Video Diffusion (SVD img2vid)** and **Veo-3.1**. The central empirical finding is that models exhibit strong semantic scene understanding but difficulty with precise physical parameterization and consistent dynamics [2602.13294].

The reported results show a consistent backend effect. Three.js generally outperforms P5.js in reconstruction fidelity, and non-physics backends such as SVG and Manim reveal what happens when rigid-body solvers are absent: scripted motion, interpenetration, no response to impacts, or stationary objects when motion is expected. Among the Three.js systems, **Gemini-3-Pro** attains **PSNR 21.26, SSIM 0.9445, success 0.957**, the **lowest LPIPS 0.1399**, and the **highest CLIP-Img 0.8973**. **GPT-5 (Three.js)** reaches **PSNR 20.54, SSIM 0.9370, success 0.990**, **DINO 0.8556**, and **RAFT-EPE 33.65**. The holistic Gemini score is highest for **Gemini-3-Pro (Three.js) at 3.80**, followed by **GPT-5 (Three.js) at 3.50**; the pixel-space baselines score lower on this axis, with **Veo-3.1 at 2.62** and **SVD at 1.43** [2602.13294].

| System | Selected results | Observed significance |
|---|---:|---|
| Gemini-3-Pro (Three.js) | PSNR 21.26, SSIM 0.9445, LPIPS 0.1399, holistic 3.80 | Strongest overall reconstruction and holistic physics judgment |
| GPT-5 (Three.js) | PSNR 20.54, SSIM 0.9370, success 0.990, RAFT-EPE 33.65 | High execution success and competitive motion alignment |
| Veo-3.1 | PSNR 20.04, SSIM 0.9354, DINO 0.8839, holistic 2.62 | Strong semantic similarity without executable diagnosis |
| SVD img2vid | RAFT-EPE 45.46, holistic 1.43 | Weaker motion reconstruction and physics plausibility |

The qualitative error analysis is as important as the leaderboard. Models often reproduce object identities and layouts while assigning incorrect friction or restitution, producing premature stopping or excessive bouncing. Syntax and runtime failures still occur, though the single-retry repair mechanism improves success from **0.979 to 0.990** for Three.js and from **0.853 to 0.979** for P5.js. The framework also identifies degenerate scenes—such as static or empty outputs—that can misleadingly improve motion metrics. For this reason, VisPhyWorld argues that RAFT-EPE must be read jointly with the holistic judge rather than in isolation [2602.13294].

## 6. Position within the broader physics-reasoning landscape

VisPhyWorld occupies a specific niche within recent work on physical reasoning: it evaluates whether a model can reconstruct an **executable** world hypothesis from observation. Nearby benchmarks and systems probe adjacent but distinct capabilities.

| Work | Primary target | Distinguishing emphasis |
|---|---|---|
| VisPhyWorld [2602.13294] | Code-driven video reconstruction | Inspectable, editable, falsifiable simulator code |
| VideoPhy [2406.03520] | Physical commonsense in text-to-video | Binary Semantic Adherence and Physical Commonsense on 688 captions |
| ComPhy [2205.01089] | Compositional physical reasoning from videos | Hidden mass and charge inference from reference videos |
| Phys4D [2603.03485] | Physics-consistent 4D world representations | Geometry-motion consistency and 4D trajectory rewards |
| PhyEditBench [2606.26551] | Physics-aware image editing | Multi-stage real-world editing with 238 real and 35 Anti-Physics instances |

**VideoPhy** evaluates whether generated videos follow physical commonsense for real-world activities, using two independent binary axes: Semantic Adherence and Physical Commonsense. Its human evaluation shows that current text-to-video models are far from accurate world simulation; the best joint performance reported in that benchmark is **Pika at 19.7%** for \(SA=1\) and \(PC=1\) [2406.03520]. Relative to VideoPhy, VisPhyWorld does not merely judge whether a video “looks physically right”; it demands an executable account of why.

**ComPhy** targets a different reasoning bottleneck: the inference of hidden intrinsic properties such as mass and electric charge from few-shot reference videos. The proposed Compositional Physics Learner achieves **80.5% factual accuracy**, **75.3% predictive per-option accuracy / 56.4% per-question**, and **68.3% counterfactual per-option / 29.1% per-question**, outperforming end-to-end video-QA baselines that struggle to use reference videos effectively [2205.01089]. This suggests a complementarity: ComPhy isolates latent property discovery, whereas VisPhyWorld externalizes physical hypotheses as runnable programs.

**Phys4D** addresses physics consistency through explicit 4D scene representations learned from video diffusion models. It uses a three-stage training paradigm—pseudo-supervised pretraining, physics-grounded supervised fine-tuning, and simulation-grounded reinforcement learning—and evaluates 4D world consistency through geometric coherence, motion stability, and long-horizon plausibility. On its 4D world evaluation, it reports **4D Chamfer 0.4626**, **Worldline L2 Error 0.4928**, **Fail Rate 9.54%**, and **Trajectory Length 87.21** [2603.03485]. Compared with VisPhyWorld, Phys4D emphasizes learned geometry-motion coupling rather than executable code synthesis.

**PhyEditBench** shifts the problem from video continuation or reasoning to instruction-based image editing under physical constraints. It defines **4 primary classes and 12 subclasses**, with **238 real-world instances** and **35 synthetic Anti-Physics instances**, and shows that current editing methods have substantial limitations in physics-based reasoning. Its training-free baseline **PhyWorld** leverages image-to-video generation as a reasoning mechanism for editing [2606.26551]. This line of work is adjacent to VisPhyWorld in its emphasis on causally coherent transitions, but its unit of evaluation is the edited image or transition sequence rather than the explicit simulator hypothesis.

Taken together, these works suggest that “physical reasoning” is not a single capability. It can mean hidden-property inference, commonsense judgment, world-consistent generation, physics-aware editing, or executable reconstruction. VisPhyWorld’s contribution is to make one of these dimensions—code-level causal commitment—directly testable.

## 7. Limitations, significance, and future directions

The current scope of VisPhyWorld is explicitly bounded. The benchmark is built from **synthetic scenes**, is **predominately rigid-body**, uses **relatively short clips**, and has **moderate motion complexity** with **white backgrounds**. The paper notes that **3D is harder**, that current multimodal models struggle to autonomously generate stable code for more complex engines such as Unreal or Blender without human intervention, and that **fluids, soft bodies, heavy occlusions, and long-horizon tasks are out of scope** [2602.13294].

The framework also exposes several methodological trade-offs. Execution-based evaluation improves falsifiability, but it inherits the expressivity limits of the chosen simulator and renderer. Strong scene semantics do not imply correct parameterization. Non-physics renderers may superficially reconstruct layouts while failing mechanically. Holistic physics evaluation remains necessary because low-level motion alignment can be gamed by degenerate outputs. A plausible implication is that future systems will need stronger priors for 3D initialization, more capable repair loops, and explicit parameter-estimation metrics in addition to current reconstruction scores.

Within the broader literature, VisPhyWorld marks a shift from answer-based to hypothesis-based evaluation. In contrast to benchmarks that ask whether a model can label, rank, or narrate physical phenomena, it asks whether a model can instantiate them in code and survive execution. That makes the framework particularly relevant for applications where unverifiable physical claims are insufficient, including simulation-backed reasoning, controllable scene editing, and downstream embodied systems. Its core claim is therefore methodological as much as empirical: physical understanding becomes substantially more legible when it is forced into an executable, editable, and falsifiable world model [2602.13294].

Source: https://www.emergentmind.com/topics/visphyworld