Physion: Benchmarking Physical Prediction

• Physion is a benchmark dataset designed to evaluate object contact prediction using realistic, multi-object 3D simulation scenarios.
• It covers diverse scenarios such as collisions, support, and soft-body behaviors with a unified protocol for comparing human and machine predictions.
• Studies show that models with object-centric and physics-infused representations outperform pixel-based approaches, highlighting a key bottleneck in visual dynamics.

Searching arXiv for relevant Physion papers and related benchmark/context work. {"query":"Physion benchmark physical prediction from vision arXiv", "max_results": 10} Physion is a dataset and benchmark designed to rigorously evaluate physical prediction from vision in both humans and machines, using visually realistic, multi-object 3D simulations that span rigid-body interactions, soft-body cloth behavior, and object-object relationships such as support, containment, and attachment (Bear et al., 2021). Its central task is object contact prediction (OCP): given the initial portion of a video, predict whether a designated “agent” object will come into contact with a designated “patient” object by the end of the scene. Subsequent work has used Physion both as a vision benchmark and as a stress test for learned dynamics models with privileged state access, while later variants and extensions have broadened the focus from short-horizon physical prediction to latent property inference and to evaluation of physical realism in generated video (Venkatesh et al., 2023).

1. Origins and benchmark motivation

Physion was introduced to address limitations in prior benchmarks for physical reasoning from video. Compared to datasets such as IntPhys, Shapestacks, and PHYRE, it emphasizes broader physical phenomena coverage within a single framework, realistic 3D visual scenes rendered in the Unity3D-based ThreeDWorld (TDW) engine, and a unified, model-agnostic evaluation protocol that supports direct comparison to human predictions on the same scenarios (Bear et al., 2021).

The benchmark’s motivating question is whether vision algorithms understand the physical dynamics of real-world environments well enough to forecast how scenes will evolve. In the original formulation, a model receives approximately the first $1.5$ seconds of a scene and must infer whether the designated red “agent” will contact the yellow “patient” by the end. This framing operationalizes physical understanding as predictive judgment under partial observability rather than retrospective detection alone (Bear et al., 2021).

A common misconception is that Physion is only a vision benchmark. In the original release it was explicitly designed to compare pixel-based models against privileged-state particle simulators, and later work continued to use it in both modes. The original paper evaluates unsupervised visual dynamics models, supervised object-centric models, pretrained image encoders with learned dynamics heads, and particle-based graph neural network simulators that operate directly on TDW physical state (Bear et al., 2021). This dual use made Physion a benchmark not only for perceptual physical reasoning, but also for the representation bottleneck between raw pixels and structured dynamics.

2. Dataset composition, scenarios, and simulation environment

Physion comprises eight scenario types, each operationalizing a distinct physical phenomenon through the OCP task: Dominoes, Support, Collide, Contain, Drop, Link, Roll, and Drape (Bear et al., 2021). These scenarios cover collisions, stable and unstable multi-object configurations, rolling and sliding conditioned on geometry, falling and bouncing under gravity, attachment constraints, and cloth draping.

For each scenario, the original benchmark provides a Training set of 2000 movies, a Readout Fitting set of 1000 movies, and a Testing set of 150 movies, with Testing balanced at 50% contact versus 50% no-contact outcomes. Across eight scenarios, this yields 16,000 training movies, 8,000 readout-fitting movies, and 1,200 human-tested movies (Bear et al., 2021). Testing videos and human trials are strictly matched.

The simulations are generated in TDW, a Unity3D-based environment. Per-frame outputs include RGB images, depth maps, surface normals, instance segmentation, and optical flow, alongside privileged physical state such as object centroids, poses, velocities, surface meshes, collision locations and normals, and stimulus-level metadata including camera intrinsics and extrinsics, object models, scales, colors, and scenario-specific parameters (Bear et al., 2021). Videos are rendered at $30$ fps and typically last $5$–$10$ seconds. Camera parameters, textures, backgrounds, distractor objects, and colors are randomized, while “toy blocks” and simple shapes are used to avoid semantic confounds.

Later work distinguishes an updated Physion v1.5. In the evaluation protocol used by Counterfactual World Modeling, Physion v1.5 “has improved rendering quality and more physically plausible simulations,” and comprises seven rigid-body scenarios: collide, drop, support, link, roll, contain, and dominoes (Venkatesh et al., 2023). It uses realistic 3D simulations rendered in ThreeDWorld and diverse lighting via HDRI skyboxes. A plausible implication is that the later version deemphasizes the original Drape scenario in the standard v1.5 evaluation setting while tightening rendering and simulation fidelity.

3. Task definitions and evaluation protocols

In the original benchmark, the core mapping is

$$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$

where $X$ denotes the observed RGB movie, $o_a$ the agent identity, and $o_p$ the patient identity (Bear et al., 2021). Pixel-based models use a frozen or trained visual encoder, an optional dynamics predictor, and a task adaptor. The logistic readout fitted on the Readout Fitting set uses cross-entropy,

$$L(\theta)=-\sum_i \left[y_i \log p_i + (1-y_i)\log(1-p_i)\right], \quad p_i=\sigma(w^\top f_i+b),$$

with $f_i$ the feature vector and $30$0 the OCP label (Bear et al., 2021).

The original paper defines three training protocols for models: all, all-but, and only. It also defines three readout protocols for pixel-based models: observed, where the readout uses only features from the visible prefix; observed+simulated, where observed features are concatenated with features from model rollouts; and full, where the readout uses the full movie to test whether the representation contains contact-relevant information even when no prediction is required (Bear et al., 2021). Privileged-state particle-based models instead predict trajectories directly and determine contact by a fixed inter-particle distance threshold at the end of rollout.

Evaluation in the original benchmark reports accuracy, Pearson correlation between model scores and average human responses, and Cohen’s $30$1, where

$30$2

These metrics quantify not only binary correctness but also similarity to human error structure (Bear et al., 2021).

Physion v1.5 introduced an additional binary task, object contact detection (OCD), alongside OCP. In the protocol used by Counterfactual World Modeling, OCP asks whether the red object will touch the yellow surface at any point in the future given the context video, while OCD asks whether they come into contact somewhere in the observed video (Venkatesh et al., 2023). That protocol uses 4 RGB frames sampled 150 ms apart for image-based methods, frozen feature extraction from a pretrained encoder, and a logistic regression linear probe with regularization selected via 5-fold validation on training sets of 5,608 videos for OCP and OCD, with evaluation on a test set of 1,035 videos sampled equally across the seven scenarios (Venkatesh et al., 2023).

4. Human behavior and baseline model results

Physion was unusual among physical reasoning benchmarks in pairing machine evaluation with large-scale human behavioral measurement. The original study recruited 800 participants via Prolific, with 100 per scenario, and each participant saw all 150 stimuli from one scenario after a 10-trial familiarization phase (Bear et al., 2021). Overall human proportion correct across all scenarios was $30$3, with scenario-wise accuracies of Dominoes $30$4, Support $30$5, Collide $30$6, Contain $30$7, Drop $30$8, Link $30$9, Roll $5$0, and Drape $5$1 (Bear et al., 2021).

The main empirical result of the original benchmark is that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance, whereas graph neural networks with direct access to physical state perform substantially better and make predictions more similar to those made by humans (Bear et al., 2021). Among vision models, SVG is near chance across scenarios, OP3 is marginally better, CSWM is significantly better than both, RPIN is competitive with CSWM in several settings, and ImageNet-pretrained DeIT-based encoders with learned dynamics outperform comparable VGG-based variants (Bear et al., 2021). The paper interprets this as evidence that high-quality visual pretraining yields features more conducive to physical prediction, but that physical representation extraction remains the central bottleneck.

Privileged-state particle models approach or exceed human accuracy in some scenarios. For example, in Collide, GNS and DPI both reach $5$2, versus humans at $5$3; in Drop, DPI reaches $5$4 versus humans at $5$5; in Link, GNS reaches $5$6 versus humans at $5$7 (Bear et al., 2021). Drape remains difficult for all such models. The benchmark also shows that observed+simulated readouts do not improve vision-model accuracy overall relative to observed readouts ($5$8), whereas full readouts significantly improve accuracy, implying that the relevant spatial information is often visually available but not converted into adequate physical prediction (Bear et al., 2021).

This result is central to the interpretation of Physion. The benchmark does not show that visual encoders fail to represent contact-relevant scene structure; rather, it suggests that the failure lies in turning those visual features into predictive physical representations.

5. Physion as a testbed for learned dynamics and world models

Later work used Physion as a method-development benchmark for structured dynamics learning. In “Learning Physical Dynamics with Subequivariant Graph Neural Networks,” Physion serves as a stress test for particle-based graph simulators to capture contacts, collisions, friction, and gravity across diverse objects and materials (Han et al., 2022). That study follows Physion’s particle-based splits exactly, with 2000 trajectories for training and 200 for testing per scenario, and evaluates both contact prediction accuracy and rollout MSE. It argues that standard simulators suffer from symmetry-induced generalization failures, especially under gravity, and proposes subequivariance with respect to the subgroup $5$9 that preserves the gravity vector (Han et al., 2022).

SGNN achieves the best contact prediction accuracy on $10$0 scenarios, with an average improvement of approximately $10$1 over the best non-SGNN baseline per scenario. Examples include Dominoes $10$2 versus $10$3, Contain $10$4 versus $10$5, and Link $10$6 versus $10$7 (Han et al., 2022). The same work reports substantially lower rollout MSE for SGNN in long-horizon rollouts, particularly in Dominoes, Link, and Contain. This established Physion not only as an end-task benchmark, but also as a tool for diagnosing inductive biases in dynamics architectures.

Counterfactual World Modeling repositions Physion v1.5 as an out-of-distribution test for self-supervised visual dynamics models trained on real-world video. CWM is pretrained self-supervised on Kinetics-400 and evaluated on Physion without fine-tuning the backbone (Venkatesh et al., 2023). Its core training objective is temporally-factored masked prediction,

$10$8

where the first frame is fully visible and the second frame is sparsely visible, and prediction is conditioned on a high mask ratio over the future frame. The model also supports counterfactual prompting through sparse action frames,

$10$9

which enables zero-shot extraction of keypoints, optical flow, and “Spelke object segments” from a single pretrained predictor (Venkatesh et al., 2023).

On Physion v1.5, CWM reaches state-of-the-art performance. Reported accuracies are $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$0 OCP and $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$1 OCD for CWM (ViT-B), and $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$2 OCP and $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$3 OCD for CWM (ViT-L), outperforming VideoMAE, MAE, DINOv2, TECO, MCVD, R3M, FitVid, and GPT-4V (Venkatesh et al., 2023). GPT-4V performs near chance, at $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$4 OCP and $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$5 OCD. Ablations show that adding counterfactual structures to frozen features improves OCP from $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$6 to $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$7, while OCD remains at $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$8, indicating that such structures are especially useful for the harder forward-prediction task (Venkatesh et al., 2023). The same work also reports that context length matters: 2 frames outperform 1 frame, which in turn outperform 4 frames on OCP, suggesting short, focused contexts are optimal in this setting.

6. Extensions, reinterpretations, and related benchmarks

Physion++ was introduced to address a limitation of the original Physion benchmark: most original scenes did not require inference of latent mechanical properties such as mass, friction, elasticity, or deformability for success (Tung et al., 2023). Physion++ retains the broad object-contact framing but adds an inference period in which latent properties must be inferred from observed interactions, an optional transition phase, and a prediction period in which the standard OCP question is posed. It spans four property families and nine scene types, including Mass-Dominoes, Mass-Waterpush, Elasticity-Wall, Friction-Slide, Friction-Clothslide, and Deform-Roll (Tung et al., 2023).

The core claim of Physion++ is that models trained using standard regimes and datasets do not spontaneously learn to make inferences about latent properties, even when these are required for accurate prediction (Tung et al., 2023). Separate training generally outperforms unified training. Under the separate, with-property condition, overall accuracy is $$F_{\Theta}:(X_{1:t_{vis}}, o_a, o_p)\rightarrow P(\text{contact}),$$9 for MCVD, $X$0 for DeiT-mlp, $X$1 for ResNet-mlp, $X$2 for VGG-mlp, $X$3 for ALOE, $X$4 for SlotFormer, and $X$5 for DPI-Net (Tung et al., 2023). Models without property frames perform similarly to those with property frames, showing that they largely fail to use the inference phase. Human accuracy is approximately $X$6 overall, but model-human Pearson correlations are poor; the best reported is SlotFormer at $X$7, versus split-half human reliability of $X$8 (Tung et al., 2023). Physion++ therefore reframes the challenge from visible-geometry prediction to online latent property inference.

A later diagnostic study of frontier VLMs evaluates Physion and Physion++ in a unified true/false format under two conditions: with property (truncated video) and fully observed (full video) (Bagdonaviciute et al., 3 Oct 2025). On the combined Physion/Physion++ with-property set of 2,390 videos, six VLMs remain near chance: GPT-4o reaches $X$9, GPT-O1 $o_a$0, VideoLLaVA-7B $o_a$1, VideoLLaVANext-7B $o_a$2, Qwen2.5-VL-7B-Instruct $o_a$3, and InternVL-4B $o_a$4 (Bagdonaviciute et al., 3 Oct 2025). Diagnostic subtests for target perception, goal perception, latent-object perception, motion prediction, and spatial or causal relationships show weak and sometimes negative alignment between subskill mastery and correct task prediction. This suggests that current VLMs do not integrate perception and physics into coherent causal understanding.

Within the broader “Physion” ecosystem, Physion-Eval is distinct from the original benchmark. Rather than predicting future contact in simulated scenes, it evaluates physical realism failures in videos generated by five state-of-the-art text-and-image-to-video models, using expert human reasoning traces with temporally localized glitches and structured failure categories (Zhang et al., 20 Mar 2026). The paper explicitly positions Physion-Eval as a new evaluation benchmark and notes that it does not explicitly compare to or re-use the original Physion dataset and task suite. Its relationship to Physion is therefore genealogical rather than task-identical: it extends the name toward evaluation of generated-video physics rather than predictive physical scene understanding (Zhang et al., 20 Mar 2026).

Taken together, these developments clarify the role of Physion in contemporary research. The original benchmark established that non-trivial physical forecasting from video is possible, but that extracting the right physical representations from pixels is the main bottleneck (Bear et al., 2021). Later work used Physion to test inductive biases in particle-based simulators, self-supervised world models, latent property inference, and the integration limits of frontier VLMs (Han et al., 2022). This suggests that Physion’s enduring significance lies less in any single leaderboard and more in its function as a controlled, extensible probe of how visual systems represent dynamics, objects, and causality.