Physion++: Latent Physical Inference Benchmark

• Physion++ is a benchmark that requires inferring unseen mechanical properties such as mass, friction, elasticity, and deformability from video observations.
• It uses paired video trials with fixed geometry and appearance that differ only in a key latent property, forcing models to go beyond superficial cues.
• The evaluation compares model predictions with human judgments on accuracy and trial-level correlation, highlighting gaps in current physical reasoning systems.

Physion++ is a dataset and benchmark for visual physical prediction in which accurate scene understanding depends on online inference of latent mechanical properties rather than on object localization, recognition, or directly observable cues alone. It extends the original Physion benchmark by constructing paired video trials in which geometry, appearance, and initial motion are held fixed while a key latent property differs, thereby changing the final YES/NO contact outcome. The benchmark targets mass, friction, elasticity, and deformability, and compares machine predictions with human judgments through both accuracy and trial-level correlation. Reported results show that models trained with standard regimes and datasets do not spontaneously learn to infer such latent properties, while models with object-centric or state-based structure perform somewhat better yet still remain far from human-like physical prediction (Tung et al., 2023).

1. Benchmark objective and conceptual scope

Physion++ is motivated by the claim that general physical scene understanding requires more than recognizing objects and extrapolating visible trajectories. In the benchmark formulation, accurate prediction depends on estimating hidden properties of individual objects—such as mass, friction, elasticity, and cloth deformability—that are not directly visible and must instead be inferred from motion and interaction patterns. The paper therefore targets cases in which physical prediction relies on observing how objects move and interact with other objects or fluids, rather than on static appearance cues (Tung et al., 2023).

This design distinguishes Physion++ from benchmarks that either do not require latent-property inference or only test properties that are directly observable, such as size or color. The central methodological claim is that a model should not succeed by exploiting superficial scene regularities. The paired-trial construction operationalizes that requirement by keeping the visible scene fixed while changing only the hidden mechanical parameter of a key object. In consequence, success on the benchmark is intended to reflect inference over latent physical state rather than shortcut recognition.

The benchmark also has an explicitly comparative orientation. It evaluates a range of state-of-the-art prediction systems spanning different amounts of learned versus built-in physical structure, and it situates those results against human predictions. The resulting comparison is not limited to aggregate accuracy: it also asks whether models make mistakes on the same trials that humans do, which turns trial-level agreement into part of the definition of physical understanding.

2. Scenario construction and latent-property manipulations

Physion++ comprises nine scenarios grouped by target property. Each scenario is rendered in the Unity-based ThreeDWorld engine, and latent properties are sampled uniformly from physically plausible ranges, with examples given as mass $m \in [0.5, 2.0]$ kg, friction coefficients $\mu \in [0.1, 0.8]$, restitution $e \in [0.2, 0.9]$, and cloth stiffness $k \in [10, 100]$ (Tung et al., 2023).

Property	Scenarios
Mass	Mass-Dominoes; Mass-Waterpush; Mass-Collision
Elasticity	Elasticity-Wall; Elasticity-Platform
Friction	Friction-Slide; Friction-Collision; Friction-Clothslide
Deformability	Deform-Roll

Every trial is paired: two videos share identical geometry, object appearances, and initial motions, yet differ only in the latent property of a key object, yielding opposite YES/NO outcomes in the final contact-prediction task. This pairing is a core part of the benchmark design because it prevents models from relying on superficial visual cues alone (Tung et al., 2023).

Each rendered video is divided into three phases. The inference phase lasts 5–8 s and is designed to reveal the hidden property through scene dynamics, such as a domino chain falling to expose relative masses, a ball bouncing to reveal elasticity, a block sliding to reveal friction, or a ball landing on cloth to reveal deformability. An optional transition phase uses a curtain to occlude the scene while objects are reconfigured. The prediction phase, also 5–8 s, highlights two objects—one in red and one in yellow—and asks whether the red object will collide with the yellow object if physics continues (Tung et al., 2023).

The scenario design therefore separates evidence gathering from the final prediction target. A system must extract information about the latent property during the inference phase and deploy it later under a changed arrangement. This two-phase structure is central to the benchmark’s emphasis on online inference rather than purely reactive forecasting.

3. Tasks, dataset splits, and evaluation criteria

Physion++ provides three disjoint splits per scenario. The dynamics-training split contains 2,000 unlabeled videos per property for self-supervised learning of future-prediction representations. The readout-fitting split contains 192 labeled videos per property for training a lightweight classifier or readout that maps learned representations to the YES/NO contact decision. The testing split contains 192 paired videos per property for final evaluation of both machine models and human participants (Tung et al., 2023).

The benchmark defines two task types. The first is physical prediction, formulated as future-state regression or, more centrally, as a binary contact decision: given the inference- and prediction-phase frames up to time $t_0$, determine whether the highlighted red object will touch the yellow object at any future time $t > t_0$, yielding a binary label $y \in \{0,1\}$. The second is latent-property inference, in which the hidden mechanical parameter $\theta \in \Theta$—for example $m$, $\mu$, $\mu \in [0.1, 0.8]$0, or $\mu \in [0.1, 0.8]$1—is estimated from inference-phase observations (Tung et al., 2023).

Performance is measured along two axes. The first is accuracy, defined as the proportion of correct YES/NO predictions on the test set. The second is correlation with human judgments, defined as the Pearson correlation between a model’s confidence $\mu \in [0.1, 0.8]$2 on trial $\mu \in [0.1, 0.8]$3 and the average human response $\mu \in [0.1, 0.8]$4:

$\mu \in [0.1, 0.8]$5

Although future-state regression can also be evaluated by mean-squared error, the benchmark focuses on binary collision outcome in order to isolate the role of latent-property inference (Tung et al., 2023).

Human judgments were collected via Prolific from $\mu \in [0.1, 0.8]$6 participants, paid \$\mu \in [0.1, 0.8]$760\%$\mu \in [0.1, 0.8]$8r \approx 0.37$ (Tung et al., 2023).

4. Model families and experimental protocol

Physion++ evaluates seven state-of-the-art approaches spanning pure video prediction, object-centric modeling, and oracle 3D simulation. The models are:

1. MCVD (Masked Conditional Video Diffusion), described as a transformer–diffusion model trained end-to-end to predict future pixels.
2. pDEIT-mlp, pRESNET-mlp, and pVGG-mlp, each consisting of an image encoder followed by an MLP readout trained in the readout-fitting stage.
3. ALOE, an object-centric encoder that learns slot representations via STEVE and then learns dynamics on slots to roll out future slots.
4. SlotFormer, combining unsupervised slot decomposition with transformer dynamics on slots.
5. DPI-Net, a graph-neural-network physics simulator operating on ground-truth 3D particle states and latent attributes, evaluated in “w/ property,” “w/o property,” and “fully observed” variants (Tung et al., 2023).

All models share a two-stage evaluation protocol. First, they learn to predict future frames or future slots on the 2,000-video dynamics set. Second, a small logistic-regression or MLP readout is trained on the 192 readout-fitting videos to map the latent representation at time $\mu \in [0.1, 0.8]$9 to the binary contact label. Final evaluation is performed on the 192 test videos in three view conditions: “w/ property”, which provides the full inference-plus-prediction video; “w/o property”, which restricts input to prediction-phase frames and thereby blocks property inference; and “fully observed”, which includes the true future frames $e \in [0.2, 0.9]$0 and serves as an upper bound on representation quality (Tung et al., 2023).

DPI-Net is distinctive because it is given privileged state information. In the “w/ property” condition it receives the true latent parameter $e \in [0.2, 0.9]$1 in its inputs, whereas in the “w/o property” variant $e \in [0.2, 0.9]$2 is masked to zero. The “fully observed” DPI-Net variant is given actual future trajectories and classifies contact by thresholding the minimal interobject distance $e \in [0.2, 0.9]$3. This makes DPI-Net an oracle-style reference rather than a directly comparable video-only model (Tung et al., 2023).

5. Reported performance and model–human divergence

Across all four mechanical properties, the pure video models—MCVD, pDEIT-mlp, pRESNET-mlp, and pVGG-mlp—achieve only slightly above $e \in [0.2, 0.9]$4 accuracy, described as no better than random, regardless of whether they have access to the inference phase. The object-centric models ALOE and SlotFormer perform only marginally better, peaking at approximately $e \in [0.2, 0.9]$5 on mass or elasticity. These results support the paper’s conclusion that training under standard regimes and datasets does not spontaneously produce latent-property inference (Tung et al., 2023).

The oracle simulator establishes a substantially higher ceiling. DPI-Net with ground-truth $e \in [0.2, 0.9]$6 attains approximately $e \in [0.2, 0.9]$7 on mass and approximately $e \in [0.2, 0.9]$8 overall in the “w/ property” regime, but when $e \in [0.2, 0.9]$9 is masked it drops toward approximately $k \in [10, 100]$0 on mass and approximately $k \in [10, 100]$1 across properties. Even in the “fully observed” setting, most models remain below $k \in [10, 100]$2, whereas DPI-Net approaches approximately $k \in [10, 100]$3 overall. The contrast indicates that explicit access to physical state and latent attributes matters strongly for performance, but it also shows that current video-based systems do not recover comparable information from observation alone (Tung et al., 2023).

Human observers average approximately $k \in [10, 100]$4 accuracy and exhibit reliable split-half agreement of approximately $k \in [10, 100]$5. No video-based model matches human accuracy or consistency. The best-of-class DPI-Net with property reaches $k \in [10, 100]$6 overall, versus human $k \in [10, 100]$7, but its trial-by-trial correlation with human judgments is near zero at approximately $k \in [10, 100]$8. SlotFormer has the highest reported model–human correlation, but this is still low at approximately $k \in [10, 100]$9 (Tung et al., 2023).

The pattern of errors is also diagnostic. When stimuli are divided into “easy” trials, where human success exceeds $t_0$0, and “hard” trials, where human success is below $t_0$1, all models underperform humans on easy trials yet outperform humans on hard trials. The paper interprets this as evidence that model and human error patterns are fundamentally dissimilar. Two further analyses reinforce that conclusion. First, increasing the number of dynamics-training trials from 200 to 2,000 yields only modest gains after approximately 1,000 videos, suggesting that data scale alone is not the bottleneck. Second, an auxiliary multitask readout that explicitly regresses $t_0$2 alongside the collision label fails to improve collision-prediction accuracy, implying that these architectures do not encode the latent property even when prompted (Tung et al., 2023).

6. Subsequent evaluations and broader research context

Later work used Physion++ as an evaluation substrate for frontier vision-LLMs. One study benchmarked six such models on Physion++, using 2,390 “with-property” clips and an equal number of fully observed clips, for a total of 4,780 trials. Reported predictive and verification accuracies remained modest: GPT-4o achieved $t_0$3 in the prediction condition and $t_0$4 in the verification condition; GPT-O1 achieved $t_0$5 and $t_0$6; VideoLLaVA-7B achieved $t_0$7 and $t_0$8; VideoLLaVA-Next-7B achieved $t_0$9 and $t > t_0$0; Qwen2.5-VL-7B achieved $t > t_0$1 and $t > t_0$2; and InternVL-4B achieved $t > t_0$3 and $t > t_0$4. That study also introduced diagnostic subtests intended to separate perception from physics reasoning and found a weak and inconsistent link between subtest mastery and success on the main task, reinforcing the view that Physion++ probes failures of causal binding rather than simple perceptual omission (Bagdonaviciute et al., 3 Oct 2025).

A separate line of work on Counterfactual World Modeling reported state-of-the-art performance on Physion v1.5 rather than on Physion++, but it discussed Physion++ as a benchmark for richer forms of physical reasoning requiring online inference of latent physical properties. The proposed extensions included longer-horizon masking policies, a “physics-head” for regressing scalar properties such as inferred mass or friction coefficient, relative-position biases or spatiotemporal attention windows, and fusion with a small Graph Network whose nodes are keypoints or object segments. These were presented as concrete adaptation strategies rather than as reported Physion++ results (Venkatesh et al., 2023).

Taken together, these follow-on studies place Physion++ within a broader research program on learned physical reasoning. The benchmark’s lasting significance lies in its insistence that accurate future prediction is not equivalent to human-like physical understanding. In the reported evidence, success on surface-level video prediction does not guarantee inference of latent mechanical properties, and even comparatively strong systems fail to reproduce human trial structure. Physion++ therefore functions as a targeted test of whether a model can infer hidden physical parameters online and deploy them in later predictive judgments (Tung et al., 2023).