Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning
Abstract: Physical understanding and reasoning depend on forming compact and generalizable representations of the world. While modern vision-LLMs can recognize and explain diverse physical events, they often lack explicit representations of the underlying mechanisms-such as object states, physical parameters, and governing dynamics-needed for reliably reasoning how the world evolves and responds to interventions. In this work, we introduce Code-as-World, a paradigm that represents physical worlds through executable world representations. By expressing physical composition, dynamic evolution, and visual appearance as executable code, Code-as-World provides a compact, quantitatively grounded, and controllable abstraction of the physical world. To construct such representations from multimodal observations, such as natural-language descriptions or real-world videos, we develop an agentic discovery loop inspired by abductive reasoning, where an agent proposes, executes, renders, verifies, and iteratively refines executable world hypotheses. As a concrete application, we use verified executable worlds to provide scalable physical supervision for training vision-LLMs on quantitative physical reasoning. Experiments show that Code-as-World-VL achieves state-of-the-art performance on QuantiPhy and surpasses leading proprietary models, highlighting the potential of executable world representations as a scalable foundation for physical intelligence.
First 10 authors:
Summary
- The paper proposes executable world representations (EWRs) that explicitly detail physical composition, dynamics, and visual appearances for efficient physical reasoning.
- The method involves iterative hypothesis discovery using a propose–instantiate–execute–render–verify loop, enhancing reconstruction efficiency and outcomes over independent sampling.
- The EWR approach significantly improves physical reasoning, with 5B, 9B, and 27B models achieving 50.6, 55.4, and 58.6 average MRA on QuantiPhy, respectively, outperforming various open-weight baselines.
Problem formulation and central thesis
The paper addresses a representational problem in physical intelligence: visual observations, geometric reconstructions, and language descriptions each preserve only part of the information required for mechanism-grounded reasoning. Pixel prediction can reproduce plausible observations without identifying their causes; 3D reconstruction can preserve geometry without exposing dynamics; and language can encode entities and relations while remaining imprecise for continuous states and physical parameters. “Code as Worlds: Agentic Discovery of Executable World Representations for Physical Reasoning” (2608.27549) proposes executable world representations (EWRs) as an intermediate abstraction that explicitly specifies physical composition, temporal evolution, and visual appearance.
The central claim is that code is not merely a convenient output format but a representation space with properties directly relevant to physical reasoning: compositionality, inspectability, editability, quantitative state access, and executable consequences. An EWR can encode objects, geometry, mass, friction, gravity, contacts, initial states, trajectories, camera parameters, materials, lighting, and rendering configuration. Execution in a simulator produces state trajectories containing physically meaningful quantities such as positions, velocities, contacts, collisions, and event outcomes. This gives the representation both semantic structure and operational semantics.
The paper therefore distinguishes phenomenon-level competence from mechanism-level competence. A model may describe that an object falls, or generate a visually plausible continuation, without representing the support relation, gravity, object state, or counterfactual response that explains the event. EWRs are intended to make these latent mechanisms explicit and queryable.
Figure 1: Pixels preserve visual detail, 3D representations preserve geometry, language preserves semantic abstraction, and executable code additionally exposes quantitative states and mechanisms.
Executable world representations
The proposed EWR is conceptually decomposed into three interacting components:
- Physical composition specifies persistent entities and their physical properties, including object geometry, dimensions, mass, friction, gravity, floors, tables, walls, and other structures participating in support and collision.
- Dynamic evolution specifies initial states, temporal changes, key events, simulation duration, and the resulting state trajectory.
- Visual appearance specifies how the world is observed, including camera pose, background, materials, lighting, frame rate, resolution, and rendering configuration.
This factorization separates mechanism-bearing state from observation-dependent appearance. The distinction is important because a camera change should alter the rendered video without changing the underlying physical trajectory, whereas modifying mass, friction, or initial velocity should alter the dynamics while leaving the scene’s representational interface intact. The implementation consequently supports intervention on individual components, such as changing an initial velocity, camera configuration, or physical parameter and resimulating the same world.
The authors prioritize physical equivalence over pixel-level duplication. This is a consequential design choice: an EWR is considered useful when it preserves composition, constraints, and evolution, even if its rendering does not reproduce every incidental texture or illumination detail in the source observation. The representation is thus closer to a structured inverse-graphics and simulation hypothesis than to a conventional video reconstruction.
The concrete implementation uses MuJoCo as the execution platform and supports two interchangeable engines. The animation engine directly specifies motion through poses and trajectories, whereas the physics engine derives motion from mechanical bodies, forces, contacts, and constraints. Both expose the same EWR interface, allowing the representation and discovery machinery to be decoupled from the execution regime.
Figure 2: An EWR encodes composition, evolution, appearance, physical parameters, relations, simulator settings, and timeline in a structured executable specification.
Agentic discovery as inverse physical modeling
Recovering an EWR from text or video is treated as an abductive search problem rather than one-shot prediction. The input is first converted into modality-specific evidence, after which an agent repeatedly proposes, executes, and evaluates candidate world hypotheses.
Figure 3: The discovery process converts text or video into evidence and iteratively refines an executable world through proposal, instantiation, execution, rendering, and verification.
For text-driven construction, an LLM extracts entities, spatial relations, events, and intended outcomes. Since language typically underdetermines metric geometry, physical parameters, and camera configuration, the system initializes unspecified quantities using physical priors and defaults. The resulting simulation can subsequently be passed through a video-generation model for sim-to-real rendering, adding visual diversity while retaining simulator-defined trajectories and states.
For video-driven abstraction, the system uses depth maps, instance masks, object tracks, estimated camera geometry, and generated 3D meshes. These observations constrain object positions, scales, geometry, visibility, and image-plane motion. Candidate EWRs are executed and projected back into the source viewpoint, where their rendered RGB frames, depth, masks, and trajectories are compared with the original evidence.
The shared loop is:
- Propose or modify an EWR using the input evidence and previous diagnostic feedback.
- Instantiate the code as simulator-ready parameters.
- Execute the candidate in MuJoCo.
- Render and project the resulting trajectory into visual observations.
- Verify and diagnose discrepancies at selected frames.
- Refine or reject the candidate within a fixed iteration budget.
The loop’s distinctive feature is that verification is not limited to a scalar objective. The verifier produces structured discrepancies that can identify whether the relevant error concerns object placement, geometry, depth, trajectory, appearance, or physical interaction. This permits localized revision of the EWR rather than independent resampling of an entire hypothesis.
The paper reports that this iterative procedure is more compute-efficient than Best-of-5 independent sampling. With a matched budget of five evaluations, agentic refinement improves Visual Alignment, Object IoU, Trajectory-ADE, and Accuracy@ relative to independent candidates, although Velocity-ADE is not reported as improving on every comparison in the principal analysis. The result supports the narrower claim that feedback-conditioned refinement makes better use of a fixed number of executions than unstructured sampling.
Figure 4: Agentic refinement generally improves visual and trajectory fidelity over five rounds and outperforms matched-budget Best-of-5 on most reported metrics.
The same qualitative conclusion is reported with the physics engine: over five rounds, Visual Alignment, Object IoU, and Accuracy@ increase, while Traj-ADE and Velocity-ADE decrease, with the final agentic result exceeding Best-of-5 on all five metrics. This cross-engine result is important because it indicates that the gain is not restricted to kinematic trajectory fitting through the animation engine.
Data construction and representation fidelity
The paper evaluates both the ability to construct EWRs and the ability to use them as sources of physical supervision. The source videos are filtered from WISA-80K using motion-focused criteria. Clips with substantial camera motion, severe editing, incomplete events, insufficient object motion, or ambiguous temporal continuity are removed. The resulting evaluation setting is therefore deliberately constrained toward visible rigid-body motion and collision-like interactions.
The reported fidelity metrics cover complementary dimensions. Visual Alignment combines silhouette overlap, depth agreement, and RGB similarity. Object IoU evaluates object-level segmentation agreement. Traj-ADE measures image-plane position error, Velocity-ADE measures inter-frame displacement error, and Accuracy@ measures the fraction of observations within two percent of the frame diagonal. These metrics assess whether an EWR preserves both scene appearance and temporal structure.
For text-driven worlds, the simulator supplies exact physical states and labels. A subsequent sim-to-real stage improves distributional realism: JEDi decreases from 3.000 for simulator renders to 1.484 for generated videos, while TRAJAN decreases from 406.872 to 185.321. These reductions indicate closer agreement with real-video feature distributions. Motion fidelity remains similar: Traj-ADE changes from 1.682 to 1.677, Velocity-ADE increases modestly from 0.404 to 0.472, and Accuracy@ decreases from 78.81% to 77.49%. The implication is specific: the appearance-generation stage substantially improves distributional realism while approximately preserving object motion, but it introduces a measurable degradation in velocity fidelity.
The dataset used for downstream training is relatively small by contemporary VLM standards: 1,585 text-driven and 988 video-driven VQA samples are reported. This is offset by the fact that each executable world carries synchronized visual observations, state trajectories, geometry, timestamps, and exact physical labels, enabling multiple quantitative questions to be derived from one structured source.
Quantitative physical reasoning
The main application is QuantiPhy, which evaluates numerical reasoning over monocular videos. The task requires a model to infer quantities such as object size, displacement, velocity, and acceleration in either pixel space or world space. World-space questions provide a reference quantity with known scale, allowing the model to calibrate image measurements into metric quantities. In 3D settings, depth context supplies additional spatial information.
The training curriculum has two stages. First, Image-Space supervision is derived from RefCOCO, RefCOCO+, RefCOCOg, RefCLEF, and GOT-10K. The resulting 73,335 question–answer pairs cover target grounding, object extent, displacement, velocity, and acceleration. This phase teaches localization and direct measurement from pixels and tracks. Second, the model is optimized with GRPO on world-space questions generated from verified EWRs. Rewards combine scale-normalized numerical accuracy with unit and format correctness.
The two sources of EWR supervision serve different purposes. Text-driven worlds provide exact simulator states and broad control over physical parameters, while video-driven worlds preserve the appearance and motion statistics of real observations. Their combination is intended to reduce the gap between numerical correctness and visual distributional validity.
The primary QuantiPhy results are strong relative to the listed baselines:
| Model | Size | 2S | 2D | 3S | 3D | Average |
|---|---|---|---|---|---|---|
| Gemini-3.1 Flash | undisclosed | 49.4 | 47.5 | 61.4 | 61.1 | 54.8 |
| Qwen3-VL-32B-Instruct | 32B | 38.1 | 39.7 | 39.8 | 43.0 | 40.2 |
| Code-as-World-VL-4B | 4B | 45.4 | 55.4 | 45.8 | 56.0 | 50.6 |
| Code-as-World-VL-9B | 9B | 55.0 | 52.9 | 55.6 | 58.1 | 55.4 |
| Code-as-World-VL-27B Reasoning | 27B | 48.7 | 62.4 | 60.5 | 62.8 | 58.6 |
The 4B model obtains 50.6, exceeding all listed open-weight baselines, including Qwen3-VL-32B-Instruct at 40.2, and approaches Gemini-3.1 Flash at 54.8. The 9B direct-answer model reaches 55.4, exceeding Gemini-3.1 Flash by 0.6 points. The 27B reasoning variant reaches 58.6, exceeding the strongest listed proprietary baseline by 3.8 points.
These comparisons support the paper’s principal empirical claim that executable-world supervision can produce compact VLMs with competitive quantitative physical reasoning. They do not, however, isolate the contribution of model scale, reasoning traces, or optimization protocol in the 27B comparison. The paper explicitly acknowledges that the 27B system changes both parameter count and response protocol relative to the 9B direct-answer model. Its result therefore demonstrates the effectiveness of the combined configuration, not the causal effect of chain-of-thought reasoning alone.
Figure 5: QuantiPhy requires measurement from video and calibration using a physical prior, with depth context available for 3D cases.
Figure 6: Representative reasoning traces combine target localization, image-space measurement, depth or size calibration, and final world-space estimation.
The scaling trend is also favorable: average MRA increases from 50.6 at 4B to 55.4 at 9B and 58.6 for the 27B reasoning model. The result suggests that the proposed supervision is not limited to a single compact model, although the scaling comparison remains confounded by the change from direct answers to outcome-supervised reasoning.
Cross-stage effects and ablations
World-space training improves not only QuantiPhy performance but also the image-space capabilities established in the first curriculum stage. On the five pixel-level benchmarks, Code-as-World-VL-4B improves over its Image-Space counterpart from 56.1 to 62.9 on RefCOCO, 51.6 to 60.2 on RefCOCOg, 56.2 to 63.3 on RefCOCO+, 31.2 to 39.8 on RefCLEF, and 18.7 to 20.5 on GOT-10K. The 9B model similarly improves from 63.7 to 68.3, 61.1 to 65.6, 61.5 to 66.4, 47.2 to 61.9, and 20.1 to 26.6, respectively.
This result is stronger than a simple additive-supervision interpretation. World-space optimization does not merely improve metric calibration while leaving perception unchanged; it appears to reinforce grounding and pixel-level measurement. The proposed explanation is that physical consistency provides feedback that favors more accurate object localization and temporal measurement.
The data-source ablation supports complementarity. For the 4B model, Image-Space-only training yields an average QuantiPhy score of 44.2, adding text-driven worlds yields 48.5, adding video-driven worlds yields 47.8, and combining both yields 50.6. For the 9B model, the corresponding reported averages are 50.9, 52.5, 53.1, and 55.4 in the main ablation table.
There is, however, an internal numerical inconsistency: the accompanying prose states that the full 9B model improves from 50.9 to 56.8, while the displayed ablation result is 55.4. Since the principal QuantiPhy table also reports 55.4 for Code-as-World-VL-9B, 55.4 is the reproducible value from the supplied content; the claim of 56.8 is unsupported unless it refers to an omitted experiment or an earlier result. This discrepancy should be resolved before the quantitative claim is treated as definitive.
Limitations and open questions
The representation is constrained by the fidelity and coverage of the simulator. Even rigid-body dynamics can be sensitive to terrain variation, contact geometry, material properties, camera uncertainty, and unobserved forces. Consequently, the discovery loop can converge to an EWR that is visually and metrically plausible but mechanistically incorrect. Verification against RGB, depth, masks, and trajectories does not guarantee identifiability: multiple physical explanations may remain consistent with the available evidence.
The empirical task is also narrow. QuantiPhy contains 159 validation examples and focuses primarily on monocular scale calibration for size, displacement, velocity, and acceleration under constrained motion. It does not establish performance on camera motion, deformable objects, fluids, fracture, long-horizon multi-object interactions, severe occlusion, or highly nonlinear contact dynamics.
A further limitation is architectural rather than data-related. Code-as-World-VL is trained on the outputs of discovery and verification, not on the discovery process itself. At inference time, the model does not construct, execute, or diagnose an EWR; it predicts a numerical answer from sampled video frames, a question, and the benchmark’s physical prior. The paper therefore demonstrates transfer from executable-world supervision to VLM behavior, but not native executable-world reasoning inside the trained model.
Finally, several evaluations rely on auxiliary learned systems: depth estimation, segmentation, tracking, mesh generation, video rendering, and feature-based realism metrics. Errors in these components can affect both EWR construction and evaluation. The paper’s conclusions consequently apply to the complete pipeline rather than isolating the representation independently of its perception and rendering stack.
Conclusion
The paper presents executable world representations as a structured alternative to purely pixel-based, geometric, or linguistic physical representations. Its main methodological contribution is the propose–instantiate–execute–render–verify loop, which treats world modeling as iterative hypothesis discovery constrained by multimodal evidence. Its main application result is that verified executable worlds provide effective supervision for quantitative physical reasoning: Code-as-World-VL-4B reaches 50.6 average MRA, the 9B direct-answer model reaches 55.4, and the 27B reasoning configuration reaches 58.6 on QuantiPhy.
The evidence supports three bounded conclusions: executable worlds can be constructed from text and selected real videos; iterative refinement improves reconstruction efficiency under matched evaluation budgets; and simulator-derived physical supervision can improve both world-space reasoning and image-space measurement. The principal unresolved question is whether the same representation and discovery process can remain identifiable and useful when observations involve richer physical regimes than the constrained rigid-body and monocular-calibration settings evaluated here.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
1. What is the paper about?
This paper introduces Code-as-World, a way for computers to understand the physical world by representing it as executable computer code.
Many AI models can look at a video and say, “A ball rolls down a table.” However, they may not truly understand why the ball moves, how fast it is moving, or what would happen if its direction or speed changed.
Code-as-World tries to represent the hidden rules of a scene, including:
- What objects are present
- Where the objects are and how large they are
- Their physical properties, such as mass and friction
- How they move and interact
- How a camera sees the scene
The computer can then run this code in a physics simulator, much like running a small virtual world.
2. What questions are the researchers asking?
The paper focuses on several main questions:
- What is a good way for AI to represent the physical world? Should it use pixels, 3D models, language, or something else?
- Can AI recover the rules of a physical scene from incomplete information? For example, can it watch a video of a ball hitting another ball and figure out the objects, their movement, and the collision?
- Can executable world models be checked and improved? The researchers want the computer to test whether its explanation actually matches what was seen.
- Can these representations help AI answer numerical questions about videos?
For example:
- How far did an object move?
- How fast was it traveling?
- What was its acceleration?
- How large is it in the real world?
3. How does the method work?
Representing a world as code
The proposed executable world representation has three main parts:
| Part | What it describes | Simple example |
|---|---|---|
| Physical composition | What exists and its physical properties | A ball, a table, the ball’s size, mass, and friction |
| Dynamic evolution | How things change over time | The ball starts still, rolls, hits another ball, and stops |
| Visual appearance | How the world looks to a camera | Camera position, lighting, colors, and background |
This is similar to creating a video game scene. The code might say:
- Put a ball on a sloped table.
- Give it a certain size and speed.
- Apply gravity and friction.
- Let it roll for several seconds.
- Show the result from a particular camera angle.
The simulator then runs these instructions and produces both:
- A record of the physical events, such as positions and speeds
- A rendered video showing what the scene would look like
The discovery loop
The computer does not try to guess the complete answer only once. Instead, it uses a repeated process:
- Propose: Make a possible explanation of the scene.
- Instantiate: Turn the explanation into simulator settings.
- Execute: Run the simulated world.
- Render: Create images or video from the simulation.
- Verify: Compare the simulation with the original text or video.
- Improve: Fix parts that do not match and try again.
This is like a student building a model of a science experiment, comparing it with the real experiment, noticing mistakes, and adjusting the model.
For a video, the system uses several kinds of clues:
- Depth maps: Estimates of how far objects are from the camera
- Object masks: Which pixels belong to each object
- Object tracks: Where each object moves from frame to frame
- 3D shapes: Approximate three-dimensional models of the objects
For text descriptions, the system extracts information such as objects, positions, and events. Since text usually does not provide every detail, the system fills in missing information with reasonable physical assumptions.
Training a vision-LLM
The researchers also use these simulated worlds to train a model called Code-as-World-VL.
The training happens in two broad stages:
- First, the model learns to measure objects in images, such as their size, position, and movement in pixels.
- Next, it learns to convert these image measurements into real-world quantities, such as meters per second.
For example, a video may show a ball moving 200 pixels. That does not automatically reveal how far it moved in meters. If the model knows that the ball has a real diameter of 5 centimeters, it can use the ball as a ruler to estimate the distance and speed.
4. What did the researchers find?
Repeated improvement worked better than guessing several times
The researchers compared their discovery loop with two simpler approaches:
- Creating one world model only once
- Creating five separate guesses and choosing the best one
The repeated propose-and-check method generally produced better results. It improved:
- Overall visual similarity
- Overlap between simulated and real objects
- Object movement paths
- Accuracy of predicted motion
This suggests that correcting a model using specific feedback is more useful than simply making many unrelated guesses.
The system could create and edit physical worlds
The researchers showed that the code-based worlds could be changed after they were created.
For example, they could:
- Change the starting direction of a bowling ball
- Change an object’s physical properties
- Move the camera to a different viewpoint
- Simulate the same event under new conditions
This is important because the system is not merely copying a video. It is creating a model that can be tested and used to answer “what if?” questions.
The trained model performed strongly on physical reasoning
The researchers tested Code-as-World-VL on QuantiPhy, a benchmark that asks models to calculate physical quantities from videos.
The model had to estimate things such as size, displacement, velocity, and acceleration.
The reported average scores were:
| Model | Size | Average score |
|---|---|---|
| Code-as-World-VL-4B | 4 billion parameters | 50.6 |
| Code-as-World-VL-9B | 9 billion parameters | 55.4 |
| Code-as-World-VL-27B Reasoning | 27 billion parameters | 58.6 |
| Gemini-3.1 Flash | Not reported | 54.8 |
The 9B model slightly outperformed Gemini-3.1 Flash in the reported comparison, even though it is much smaller than many large commercial models. The 27B reasoning model achieved the highest score among the models shown in the table.
These results suggest that giving AI exact physical information from executable simulations can help it become better at numerical reasoning about real videos.
5. Why are these findings important?
Most video AI systems mainly learn patterns in images. They may know that a ball usually falls downward, but this does not guarantee that they understand gravity, speed, or collisions.
Code-as-World gives the AI a more structured internal model. It separates:
- The objects in a scene
- Their physical states
- The rules controlling their movement
- The way a camera displays them
This can help AI reason about events rather than only describe them. It may also allow AI to make predictions about situations it has never seen before.
For example, instead of only saying “the ball moved left,” a model using this approach might estimate its speed and predict where it will be after another second.
6. Limitations and possible future impact
The approach is promising, but it is not perfect. Real-world physics can be extremely complicated. Small details—such as the exact shape of a surface, the amount of friction, or the material of an object—can change what happens.
A simulator may create an explanation that looks correct but is not the true explanation. For example, it might reproduce a ball’s movement using the wrong amount of friction. The result could look convincing while still being physically inaccurate.
The system also depends on computer tools that estimate depth, identify objects, track movement, and create 3D shapes. Errors in any of these tools can lead to an incorrect world model.
Overall, the paper suggests that executable code could become a useful bridge between seeing the world and understanding its rules. In the future, this may lead to AI systems that are better at:
- Understanding videos
- Predicting physical events
- Operating robots
- Planning actions safely
- Learning science and engineering concepts
- Testing “what would happen if…” scenarios
In simple terms, the paper’s main idea is: rather than teaching AI only what the world looks like, teach it a small, runnable model of how the world works.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
The paper leaves the following issues unresolved:
- Simulator fidelity is not quantified across physical regimes. The paper acknowledges that the simulator may fail on complex contact, friction, terrain, deformable-body, fluid, or articulated-object dynamics, but does not measure reconstruction quality separately for these regimes.
- The scope of supported world representations is underspecified. It is unclear which object categories, interaction types, materials, environmental conditions, and physical laws can be represented by the current EWR interface.
- Mechanistic correctness is not directly evaluated. Agreement with RGB frames, depth, masks, and trajectories does not establish that the recovered EWR contains the true causal mechanism; different physical models may produce similar observations.
- Counterfactual validity is untested. Although the system edits initial velocities and camera configurations, the paper does not evaluate whether interventions on mass, friction, gravity, contact geometry, or object arrangement produce physically correct outcomes.
- Identifiability from monocular video remains unresolved. The method may not uniquely recover depth, metric scale, mass, friction, forces, or camera parameters from a single view, yet the paper does not characterize which quantities are identifiable or how uncertainty is represented.
- Uncertainty is not modeled or reported. The system appears to return a single accepted EWR or reject it, without calibrated distributions over alternative hypotheses, parameter confidence intervals, or ambiguity estimates.
- The verification criterion is insufficiently specified. The paper does not provide the exact scoring functions, thresholds, frame-selection policy, diagnostic procedure, or acceptance rules used by the verifier, making the discovery process difficult to reproduce.
- The role of the LLM agent is not isolated. It is unclear how much performance comes from language-model priors, simulator execution, visual evidence adapters, or iterative refinement, because there is no comprehensive ablation of these components.
- Search efficiency is incompletely characterized. The evaluation uses at most five discovery rounds, but does not report computational cost, wall-clock time, simulator calls, token usage, scaling with scene complexity, or performance under larger search budgets.
- The optimization process may be locally trapped. The paper notes that the loop can converge to locally plausible explanations, but does not investigate recovery from poor initializations, sensitivity to proposal order, or the benefits of global search, differentiable optimization, or multi-hypothesis exploration.
- Best-of-$5$ is a limited baseline. The comparison with independent sampling does not establish superiority over stronger alternatives such as beam search, evolutionary program synthesis, gradient-based simulator fitting, particle-based inference, or learned world-model baselines.
- Text-driven construction depends on unspecified priors and defaults. The paper does not report how default geometries, physical parameters, camera settings, and environmental conditions are selected or how sensitive the resulting worlds are to those choices.
- Synthetic text specifications may introduce benchmark bias. Text inputs are generated by LLMs and reviewed by annotators, but the paper does not test robustness to naturally occurring, ambiguous, incomplete, contradictory, or technically inaccurate descriptions.
- Video preprocessing errors are not propagated through discovery. Depth, segmentation, tracking, camera estimation, and 3D mesh generation are treated as evidence sources, but their error rates and downstream effects on recovered EWRs are not quantified.
- The method is evaluated on a heavily filtered video subset. Clips with camera motion, severe editing, incomplete events, poor visibility, or insufficient object motion are removed, leaving open whether the approach works on unconstrained internet videos and long-form observations.
- Generalization beyond rigid-body motion is unknown. The experiments focus on motion and collision-like interactions, so performance on deformable objects, liquids, cloth, biological motion, granular materials, and complex articulated systems remains unexplored.
- Long-horizon temporal consistency is not established. The evaluation emphasizes selected key frames and short clips; it does not test whether executable worlds remain accurate over extended trajectories or across repeated interactions.
- Occlusion and object permanence are insufficiently examined. The paper discusses these issues conceptually but does not evaluate recovery when objects disappear behind occluders, enter or leave the scene, split, merge, or undergo severe visual changes.
- Camera-motion handling is unvalidated. Videos with substantial camera translation or rotation are excluded, so the method’s ability to disentangle camera motion from object motion remains unknown.
- Visual appearance reconstruction is not clearly separated from physical reconstruction in evaluation. The paper uses simulator renderings for alignment and a video generation model for realistic outputs, but does not determine whether visual-generation artifacts mask errors in the underlying physical trajectory.
- Sim-to-real transfer is not quantitatively validated. The claim that generated videos preserve physical evolution after appearance synthesis is supported primarily qualitatively; temporal, geometric, and causal fidelity after video generation are not measured.
- Physical parameter recovery lacks ground-truth evaluation. The paper reports trajectory and visual metrics but does not assess whether recovered mass, friction, gravity, restitution, or other parameters match the real world.
- The dataset scale and composition are not fully reported. The numbers of text-driven worlds, video-driven worlds, retained clips, generated questions, and train/validation/test instances are not specified in the provided methodology.
- Potential train–test contamination is not addressed. It is unclear whether source videos, generated worlds, templates, or related scenes overlap with QuantiPhy or the baselines’ training data.
- The contribution of text-driven versus video-driven supervision is not isolated. Joint training is reported as beneficial, but separate training, mixture ratios, data scaling curves, and cross-domain transfer experiments are not provided.
- Synthetic supervision may teach dataset-specific shortcuts. The models could exploit rendering styles, question templates, object distributions, or simulator artifacts rather than learning general physical reasoning; controls for these shortcuts are absent.
- The downstream evaluation is narrow. Improvements are demonstrated primarily on QuantiPhy, leaving open whether EWR supervision transfers to physical question answering, prediction, planning, visual intervention, robotics, or novel counterfactual tasks.
- Numerical performance is not analyzed by error type. Aggregate MRA and subset scores do not reveal whether errors arise from object localization, temporal differencing, depth estimation, scale calibration, unit conversion, or physical inference.
- The reliability of chain-of-thought reasoning is unclear. For the 27B reasoning model, only the final answer is scored; the paper does not verify whether intermediate measurements and physical arguments are correct or causally used.
- The calibration assumption is restrictive. World-space estimation relies on a known reference quantity, but the paper does not examine performance when the reference is inaccurate, absent, spatially distant, or subject to perspective and depth variation.
- Acceleration estimates may be highly noise-sensitive. The finite-difference formulation is applied to tracks and simulated trajectories, but the paper does not study frame-rate dependence, tracking noise, temporal smoothing, or uncertainty in second-order derivatives.
- No human or expert assessment of recovered mechanisms is provided. Independent physics experts are not used to judge whether reconstructed EWRs are plausible, parsimonious, interpretable, or useful for intervention.
- Parsimony is asserted but not operationalized. The paper states that hypotheses should remain parsimonious, yet it does not define a complexity measure or evaluate the trade-off between program length, physical accuracy, and explanatory coverage.
- Reproducibility is limited by missing implementation details. The simulator, EWR language/interface, prompts, verifier configuration, training hyperparameters, filtering rules, and generated datasets are not sufficiently documented in the provided text to reproduce the results fully.
- Failure detection is not independently validated. The rejection outcome is described procedurally, but the paper does not report precision, recall, or calibration for distinguishing successful reconstructions from plausible but incorrect EWRs.
- Robustness to adversarial or inconsistent evidence is unknown. The system is not tested on videos or descriptions containing mislabeled objects, physically impossible events, inconsistent timestamps, corrupted tracks, or contradictory multimodal cues.
- The relationship between executable representations and learned latent representations remains open. The paper does not establish whether EWRs are superior to, complementary with, or distillable into latent world models under equal data and compute budgets.
- Real-world deployment constraints are unexamined. The paper does not assess safety, simulator misuse, computational feasibility, or the consequences of using an incorrect EWR for planning and physical intervention.
Practical Applications
Immediate Applications
- Synthetic training data for physical reasoning in AI systems (software, robotics, autonomous systems; deployable now)
- Use verified executable worlds to generate synchronized videos, object states, trajectories, contact events, and exact labels for training vision-LLMs, perception systems, and video agents.
- Practical workflow: define or reconstruct a scene → simulate its trajectory → generate quantitative questions about size, displacement, velocity, or acceleration → train and evaluate a model.
- This can reduce reliance on expensive manual annotation for robotics, autonomous driving, industrial inspection, and embodied AI.
- Dependencies: simulator fidelity, quality of reconstructed geometry and tracks, adequate coverage of real-world appearance, and safeguards against synthetic-data bias.
- Quantitative video analysis and measurement assistance (software, sports, transportation, media analysis)
- Deploy models based on the paper’s image-space and world-space measurement pipeline to estimate object dimensions, motion, speed, and acceleration from monocular video when a reference scale is available.
- Potential products include video analytics APIs, sports-performance tools, traffic-analysis dashboards, and media post-production tools that automatically measure motion.
- A practical workflow would combine object detection/tracking with reference-object calibration and explicit unit-aware output.
- Dependencies: camera stability, visible reference dimensions, reliable object tracking, sufficient frame rate, and appropriate perspective or depth estimation.
- Physics-aware video annotation and dataset curation (academia, AI engineering, data operations)
- Use the propose–instantiate–execute–render–verify loop to convert real videos into structured records containing objects, geometry, physical parameters, trajectories, collisions, and events.
- Dataset builders can retain only reconstructions that satisfy visual, depth, mask, and trajectory consistency thresholds, producing higher-quality training corpora than caption-only pipelines.
- The recorded discovery trace can also provide provenance: proposed hypotheses, simulator outcomes, verification errors, and accepted revisions.
- Dependencies: reliable segmentation, tracking, depth, and 3D reconstruction tools; independent evaluation metrics; and human review for ambiguous cases.
- Controllable counterfactual video generation (media, education, simulation software)
- Edit executable parameters—such as initial velocity, camera position, object mass, or scene configuration—and resimulate the same event under changed conditions.
- Potential tools include interfaces for generating “what-if” videos, camera-angle variants, controlled collision sequences, and physically consistent educational animations.
- Unlike unconstrained video editing, the workflow preserves the specified physical trajectory before applying appearance synthesis.
- Dependencies: valid simulator parameterizations, stable sim-to-real rendering, and explicit handling of uncertainty when the original video does not uniquely determine the mechanism.
- Automated physical-scene documentation (manufacturing, logistics, safety engineering)
- Convert workplace or laboratory videos into executable records of object interactions, motion sequences, and environmental constraints.
- These records could support incident reconstruction, process documentation, collision analysis, and identification of abnormal motion.
- The representation’s editable structure enables analysts to test whether changing an initial condition or physical parameter could have altered the observed outcome.
- Dependencies: sufficiently visible objects and contacts, trustworthy scale calibration, domain-specific simulator models, and human validation before use in safety-critical decisions.
- Physics-aware evaluation and reward construction for AI agents (academia, robotics, reinforcement learning)
- Use executable worlds to create verifiable rewards based on numerical accuracy, unit correctness, trajectory agreement, collision outcomes, or successful intervention effects.
- This can supplement subjective language-based evaluation with measurable physical criteria for multimodal agents and robotic planners.
- Dependencies: reward definitions must reflect the intended task rather than merely simulator agreement; the simulator must not omit important real-world factors.
- Interactive educational demonstrations of mechanics (education, training, daily learning)
- Build applications in which students alter gravity, friction, mass, launch direction, or camera viewpoint and immediately observe the simulated consequences.
- Executable worlds can generate exercises with exact answers for velocity, acceleration, displacement, and scale calibration, while realistic rendering improves engagement.
- Dependencies: pedagogically appropriate interfaces, calibrated parameter ranges, and clear communication that simplified simulations are not complete physical models.
- Research workflows for reproducible physical reasoning (academia and scientific computing)
- Store physical hypotheses as executable programs rather than only prose, images, or latent representations. Researchers can inspect, modify, rerun, and compare hypotheses using common simulator interfaces.
- This supports reproducible experiments in computer vision, robotics, cognitive science, and computational physics.
- Dependencies: standardized EWR schemas, versioned simulator environments, licensing compatibility, and preservation of uncertainty and failed hypotheses.
Long-Term Applications
- World models for autonomous robots and embodied agents (robotics, autonomous vehicles, warehouse automation)
- An extended EWR system could infer a robot’s environment, maintain object identities and physical states, predict the effects of interventions, and expose a queryable model for planning.
- Example applications include predicting whether an object will fall, selecting a grasp point, anticipating collisions, or planning around friction and support constraints.
- The code interface could allow a planner to ask counterfactual questions before acting.
- Dependencies: real-time inference, robust 3D reconstruction, deformable and articulated-object modeling, uncertainty-aware dynamics, sensor fusion, and sim-to-real validation. Current results focus mainly on relatively simple rigid-body scenarios, so direct deployment in safety-critical robotics is not yet established.
- Physics-grounded autonomous driving and advanced driver assistance (transportation and mobility)
- Vehicle systems could use executable scene hypotheses to predict trajectories, reconstruct collision mechanisms, and evaluate hypothetical maneuvers under changing speeds, road conditions, or viewpoints.
- Potential workflows include near-miss analysis, interpretable trajectory prediction, and simulation-based validation of emergency actions.
- Dependencies: accurate modeling of pedestrians, vehicles, weather, road friction, occlusion, non-rigid dynamics, and human behavior; high-confidence uncertainty estimates; and extensive regulatory validation.
- Digital twins for industrial facilities and infrastructure (manufacturing, construction, energy, logistics)
- Video-driven world abstraction could evolve into automatically maintained digital twins that encode equipment geometry, motion, contacts, and operational states as editable executable models.
- These twins could support predictive maintenance, robotic work-cell planning, layout optimization, and virtual commissioning.
- Dependencies: persistent multi-camera sensing, precise calibration, integration with CAD/PLM systems, high-fidelity domain simulators, continuous state estimation, and cybersecurity controls.
- Scientific discovery and automated mechanism inference (physics, biology, materials science, engineering)
- The abductive discovery loop could be adapted to infer compact executable mechanisms from experiments, laboratory video, or sensor data. Agents might propose competing physical explanations, simulate them, and rank them by evidence fit and parsimony.
- Potential outputs include executable hypotheses for material failure, fluid interactions, robotic manipulation, or experimental apparatus behavior.
- Dependencies: stronger causal-identification methods, uncertainty quantification, experiment-design capabilities, domain-specific simulators, and safeguards against accepting visually plausible but mechanistically false explanations.
- Intervention-aware medical and biomechanical modeling (healthcare, rehabilitation, sports medicine)
- With suitable biomechanical models, executable representations could model body motion, assistive-device interactions, falls, gait, or rehabilitation exercises and simulate interventions such as altered support or joint constraints.
- Potential products include patient-specific movement analysis, virtual therapy planning, and clinician-facing explanations of biomechanical outcomes.
- Dependencies: clinically validated anatomical and biomechanical models, privacy-preserving video processing, patient-specific calibration, medical oversight, and evidence that simulated predictions improve outcomes. The paper does not directly evaluate healthcare data or biological dynamics.
- Physical planning and control for household or service robots (daily life, robotics)
- Home robots could construct editable representations of furniture, objects, surfaces, and typical interactions to decide how to pick up, move, stack, or avoid objects.
- A robot might simulate alternative grasp directions or collision outcomes before executing an action.
- Dependencies: reliable handling of clutter, transparent and deformable objects, liquids, fragile items, changing lighting, human activity, and unknown materials; low-latency execution; and strong safety guarantees.
- Regulatory and forensic reconstruction of physical incidents (policy, law enforcement, insurance, safety regulation)
- Verified executable reconstructions could support structured analysis of industrial accidents, traffic collisions, product failures, or disputed event sequences.
- Analysts could compare multiple hypotheses and test counterfactual conditions rather than relying solely on visual inspection or narrative reports.
- Dependencies: chain-of-custody requirements, explainability, calibrated confidence, admissibility standards, independent expert review, and recognition that a best-fitting EWR may not represent the true mechanism when observations are incomplete.
- Energy-system and environmental process simulation from observations (energy, climate, environmental policy)
- A generalized version could reconstruct executable models of equipment motion, water flow, particulate transport, or environmental interactions from video and sensor streams.
- Applications may include wind-turbine inspection, hydropower monitoring, infrastructure resilience analysis, and scenario testing under changed operating conditions.
- Dependencies: extension beyond rigid-body physics, integration with continuous sensor data, scalable high-fidelity solvers, domain calibration, and validation across weather and operating regimes.
- General-purpose physical intelligence platforms (software infrastructure and AI research)
- The paper points toward a platform in which language, video, simulation, and intervention share a common executable representation. Such a system could serve as a foundation for agents that reason about both observed and hypothetical worlds.
- Possible components include an EWR programming language, simulator adapters, verification services, world-model registries, editable scene tools, and APIs for querying trajectories or testing interventions.
- Dependencies: interoperable representation standards, scalable execution, robust compiler and sandboxing infrastructure, multimodal uncertainty handling, and broad validation beyond curated motion-focused videos. A central risk is that the system may converge on locally plausible simulations that reproduce observations without recovering the actual causal mechanism.
Glossary
- Abductive reasoning: Inferring the most plausible explanation or hypothesis from incomplete observations. “scientific discovery has often proceeded through abductive reasoning”
- Agentic discovery loop: An iterative process in which an autonomous agent proposes, tests, evaluates, and revises hypotheses. “we formulate world representation as an agentic discovery loop”
- Appearance synthesis: Generating visual details such as textures, lighting, and materials while preserving underlying structure or motion. “Separating world editing from appearance synthesis enables coherent counterfactual videos”
- Camera geometry: The spatial and projection properties describing how a camera observes a scene. “VGGT-Omega~\citep{wang2026vggt} estimates scene depth and camera geometry”
- Causal factors: Variables or mechanisms that produce or control observed outcomes. “does not automatically uncover the causal factors that govern the evolution of the world”
- Chain-of-thought: A sequence of intermediate reasoning steps generated before a model’s final answer. “which produces a chain-of-thought reasoning before its final answer”
- Compositionality: The ability to construct complex representations by combining simpler entities or relations. “The former provides compositionality, editability, and explicit constraints for reasoning”
- Counterfactual video: A video depicting what might have happened under altered conditions. “enables coherent counterfactual videos without reconstructing every variant”
- Depth map: An image in which each pixel encodes the distance from the camera to the corresponding scene point. “the agent extracts depth maps, instance masks, and object tracks as visual evidence”
- Dynamic evolution: The specification of how a world’s states and events change over time. “ describes how the world unfolds over time”
- Executable world representation (EWR): A programmatic representation of a physical environment that can be run by a simulator. “we develop a concrete implementation of executable world representation (EWR)”
- Frame diagonal: The diagonal distance across an image frame, used here to normalize spatial errors. “with distances normalized by the frame diagonal ”
- Generative model: A model that produces new data, such as images or videos, based on learned patterns or specified conditions. “Video generative models learn temporal dynamics by predicting future observations from past frames”
- GRPO (Group Relative Policy Optimization): A reinforcement-learning optimization method that compares multiple sampled outputs within a group to improve a model’s policy. “We optimize the model on these executable-world examples using Group Relative Policy Optimization (GRPO)”
- Image-plane trajectory: The path of an object’s projected position across successive image frames. “tracks capture temporal correspondences and image-plane motion”
- Inverse graphics: The recovery of scene geometry, materials, lighting, and camera properties from visual observations. “inverse graphics \cite{boss2021nerd,gao2024relightable,jiang2024gaussianshader,NeRFactor}”
- Inverse problem: A problem in which hidden causes or system parameters are inferred from observed effects. “Obtaining such a representation from incomplete observations, however, is fundamentally an inverse problem”
- Kinematic category: A classification based on motion-related quantities such as position, velocity, or acceleration. “We report MRA on the 2S, 2D, 3S, and 3D subsets”
- Latent factor: An unobserved variable that influences the data or physical process being modeled. “or other latent factors”
- Latent representation: An internal model representation whose variables are not directly observable or explicitly interpretable. “A latent representation may faithfully reproduce an observation while entangling persistent structures”
- Mechanistic structure: The explicit organization of entities, states, parameters, and causal processes that explains how a system behaves. “It therefore preserves the mechanistic structure needed for physical reasoning”
- Metric calibration: Converting measurements from an image or other relative coordinate system into real-world units. “For world-space queries, the question additionally provides a reference quantity with a known world-space value to enable metric calibration”
- Monocular video: Video captured from a single camera viewpoint. “this task requires a VLM to infer physical quantities hidden beneath pixel observations in monocular videos”
- Motion fidelity: The degree to which generated or reconstructed motion matches the motion in the reference observation. “static quality and most motion-fidelity metrics improve overall”
- Object IoU (Intersection over Union): A measure of overlap between predicted and reference object regions, calculated as intersection area divided by union area. “Visual Alignment and Object IoU~\citep{perazzi2016benchmark} measure full-video visual agreement and object-region overlap”
- Parsimonious: Using the simplest representation or explanation that adequately accounts for the evidence. “searching the executable hypothesis space for an EWR that best explains the input evidence while remaining as parsimonious as possible”
- Physical equivalence: Consistency between two representations in their world composition, constraints, and evolution, even if their pixels differ. “where physical equivalenceâthe consistency of world composition, constraints, and evolutionâis prioritized over pixel-level duplication”
- Physical prior: предварary knowledge or assumptions about how physical quantities and processes generally behave. “the agent combines physical priors with reasonable default conditions”
- Physical supervision: Training signals derived from physical states, parameters, or trajectories rather than only visual labels. “we use verified executable worlds to provide scalable physical supervision”
- Policy optimization: Adjusting a model’s behavior policy to maximize a reward function. “We optimize the model on these executable-world examples using Group Relative Policy Optimization”
- Quantitative physical reasoning: Inferring numerical physical properties or motion quantities from observations. “As a concrete downstream application, we study quantitative physical reasoning”
- Rendering: Producing visual observations from a scene description, state, camera, and appearance parameters. “The agent subsequently renders into predicted visual observations”
- Rigid-body motion: Motion of an object whose shape and internal distances remain fixed. “Even seemingly simple rigid-body motion can be sensitive to small variations”
- Rollout: The sequence of states generated by executing a simulation from specified initial conditions. “The rollout of each candidate EWR is projected back into the input view”
- Semantic evidence: Structured information extracted from language about entities, relations, events, and outcomes. “The resulting representation externalizes the underlying physical mechanism as an interface”
- Sim-to-real transfer: Adapting simulated content or knowledge to realistic or real-world observations. “we apply a video generation model for sim-to-real transfer”
- State trajectory: The time-ordered sequence of states through which a simulated or physical system evolves. “the simulator executes to produce a complete state trajectory ”
- Supervised fine-tuning: Training a pretrained model on labeled examples to optimize its outputs for a particular task. “We train the model on the resulting pixel-level dataset $\mathcal D_{\mathrm{pix}$ with supervised fine-tuning”
- Temporal correspondence: The association of the same object or point across different video frames. “tracks capture temporal correspondences and image-plane motion”
- Trajectory-ADE (Traj-ADE): Average displacement error measuring the difference between predicted and reference object trajectories. “we use Traj-ADE, Velocity-ADE, and Accuracy@ to measure object-position error”
- VLM (Vision-LLM): A model trained to process and relate visual inputs and natural-language inputs. “We train Code-as-World-VL, a family of vision-LLMs”
- World-space measurement: A measurement expressed in physical units or coordinates rather than image pixels. “This stage teaches the model to localize, measure, and track objects, providing the visual foundation for subsequent world-space physical reasoning”
