Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence

Abstract: Modern video generation models are increasingly hailed as emerging world models with an internalized grasp of physical law. Yet existing benchmarks largely evaluate physical plausibility only at the output level, without verifying whether the model arrives there through a faithful, law-grounded reasoning process. We introduce Apple-PI, the first benchmark that anchors video-model evaluation explicitly in physical laws. Apple-PI comprises three components. 1) Orchard: a dataset of 400 videos covering ten canonical tasks in classical mechanics. It separates single-law tasks for confounder-free diagnosis from multi-law tasks for probing generalization. 2) Benchmark Protocol: a three-stage protocol based on scientific reasoning, including Perception, Formulation, and Deduction. It uses chain-of-frames prompting on infographic-annotated first frames, treating the generated video as the model's visible reasoning trace. 3) Evaluation Suite: a hybrid evaluation suite that combines MLLM-based subjective scoring with physics-law-grounded objective measures. This enables stage-resolved diagnosis of not only whether a model fails, but where it fails. Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators, with the best video model scoring only 0.473. Our stage-, pillar-, and source-resolved analyses further expose a Perception-to-Formulation-to-Deduction bottleneck, weak multi-law state transfer, and a persistent Sim-to-Real gap. These findings position Apple-PI as a diagnostic foundation for guiding future video models toward world models with law-grounded physical intelligence.

• The paper introduces a 400-case Orchard benchmark that evaluates video models across perception, physical-law formulation, target-state prediction, and trajectory deduction using 6,000 responses and hybrid objective-subjective metrics.
• The results show GPT Image 2 and Nano Banana 2 leading with average scores of 0.704 and 0.699, while the best video model, Seedance 2.0, reaches only 0.473 and 0.315 on deduction.
• The benchmark reveals that models struggle most with multi-law reasoning, real-world videos, state transfer, and law-consistent temporal dynamics, showing that visual plausibility does not establish physical intelligence.

Overview and motivation

Apple-$\pi$ is a benchmark that evaluates whether video generation models reason about physical scenes in a law-grounded manner rather than merely producing visually plausible motion (2607.16401). Its central claim is that existing physical-intelligence benchmarks—PhysBench, VideoPhy-2, PhyWorldBench, Physics-IQ, WorldModelBench, and related suites—evaluate only the output of a generative model, never the reasoning process that produced it. Consequently, a successful generation cannot be attributed to an internalized physical law, and a failure cannot be localized to a misread scene, an incorrect law, or a faulty deduction. Apple-$\pi$ addresses this by turning Newton-style scientific reasoning into an auditable video-generation protocol: the model must perceive annotated physical quantities, formulate the governing law, and deduce law-consistent dynamics, with the generated video serving as a visible chain-of-frames reasoning trace.

The paper makes a strong empirical claim: the best video generation model, Seedance 2.0, scores only 0.473 on the benchmark's $[0,1]$ scale, while the strongest unified understanding-generation models, GPT Image 2 (0.704) and Nano Banana 2 (0.699), substantially outperform all video models. This gap directly challenges the narrative that large-scale video models are emerging world simulators with internalized physical law.

The Orchard dataset

Orchard comprises 400 cases covering ten canonical tasks in classical mechanics, drawn from three complementary sources: 243 simulated cases generated in NVIDIA Isaac Sim with programmatic ground truth (engine states, masks, poses, velocities, contact events), 121 self-recorded real-world cases captured under controlled laboratory conditions with instrumented measurements, and 36 Internet-sourced physics-demonstration clips. Dynamic objects are standardized to four primitive solids—sphere, cube, cylinder, and cone—to control object-specific semantic priors and yield consistent centers, masks, and contact surfaces.

The task taxonomy is two-level. A single-law branch supports confounder-free diagnosis and is organized into three pillars: universal gravitation (free fall, projectile motion, inclined-plane motion, circular motion), conservation of momentum (elastic, perfectly inelastic, and partially inelastic collisions, parameterized by the coefficient of restitution), and Newton's first law (objects at rest and in uniform linear motion). A multi-law branch composes tasks across pillars—for example, an inclined plane feeding into projectile motion—to test whether models can transfer state variables such as position, velocity, and contact status across law transitions. Ground-truth quality is enforced through a three-pass review protocol, instrumented measurement (mass to $\pm0.01$ g, restitution to $\pm0.02$), and an inter-annotator agreement study reporting Cohen's $\kappa$ up to 1.00 and ICC(2,1) up to 0.99.

Benchmark protocol

Every case is evaluated along three reasoning stages, split into five subtracks. All subtracks share a common input—an infographic-annotated first frame in which each physical quantity is placed next to its visual referent—and a common output format: a chain-of-frames video. The subtracks are:

• Perception-Text (P-T): reproduce the numeric annotations as a final-frame artifact on a white background (fade-to-white), analogous to OCR.
• Perception-Graphic (P-G): isolate the experiment-relevant objects, analogous to instance segmentation.
• Formulation-Text (F-T): select the governing law from four options—the correct law, a confusing real law sharing symbols, an unrelated real law, and a fabricated formula—and render the symbolic and numerically substituted forms.
• Formulation-Graphic (F-G): predict the scene configuration at a target instant $t^\star$, with velocity arrows and speed labels overlaid.
• Deduction: generate the full law-consistent trajectory, evaluated frame by frame.

The distractor design in F-T is notable: it distinguishes genuine law selection from superficial symbol matching, since a model that picks a formula merely because it contains the annotated variables should fail on the confusing distractor.

Evaluation suite

Scoring combines MLLM-based subjective judgments (Gemini 3 Flash at temperature 0, with track-specific rubrics over grouped criteria) with physics-law-grounded objective measures wherever ground truth permits. Graphic subtracks use segmentation IoU against ground-truth masks with Hungarian matching for multi-object cases. Deduction uses a weighted fusion with a 0.60 weight on the physics group, comprising MLLM physics accuracy, Spatial IoU, Spatiotemporal IoU, Weighted Spatial IoU (adopted from Physics-IQ but computed on SAM3-extracted motion masks), normalized and masked PSNR, and a velocity-accuracy score $S_{\mathrm{vel}}=1/(1+e_v)$ based on 3D velocity error. A deliberate time-normalization rule treats the decoded generated video as the requested physical duration rather than the provider's nominal container length, ensuring cross-model comparability. A cross-check with the open-weights Qwen3-VL judge preserves the model-tier ranking (Pearson $r=0.95$ on overall scores, Spearman $\rho=1.00$ for F-T and Deduction), supporting that the leaderboard is not an artifact of a single judge.

Main results

Each of 11 models (5 video generation models, 6 unified understanding-generation models) is evaluated on $\pi$0 responses. Key scores:

Model	Type	Avg.	Deduction	Multi-law
Seedance 2.0	Video	0.473	0.315	0.389
Veo 3.1	Video	0.313	0.160	0.235
VBVR-Wan2.2	Video	0.373	0.201	0.387
Wan2.2	Video	0.267	0.149	0.200
GPT Image 2	Unified	0.704	0.406	0.618
Nano Banana 2	Unified	0.699	0.405	0.567
SenseNova-U1-8B-MoT	Unified	0.362	0.304	0.267

Three findings stand out. First, large-scale video training yields physical priors that are insufficient under explicit quantities, laws, and target states: proprietary video models outperform open-source bases, but all remain far from saturation. Second, reasoning-oriented fine-tuning helps interface skills but does not transfer: VBVR-Wan2.2, fine-tuned on the VBVR video-reasoning dataset, reaches 0.923 on Perception-Text yet scores 0.001 on Formulation-Text—a stark dissociation between annotation reading and law selection. Third, the unified-model advantage does not stem from the interface alone: open-source unified models (BAGEL, OmniGen2) score around 0.21, indicating that the advantage of GPT Image 2 and Nano Banana 2 likely reflects stronger multimodal foundations, broader data, and mature post-training. Even the strongest unified models score only ~0.40 on Deduction, identifying law-consistent temporal dynamics as the central bottleneck.

Stage-, pillar-, and source-resolved diagnosis

The stage-resolved analysis reveals a progressive reasoning funnel: Perception is easiest, Formulation harder, Deduction hardest, for every video model. Within stages, P-T is easier than P-G (reading quantities is easier than grounding them to object regions), and F-T versus F-G exposes a gap between symbolic law selection and grounded state prediction—models can choose a plausible equation without correctly placing objects at the target time. The paper is explicit that earlier-stage success is necessary but not sufficient: copying annotations can remain superficial, and selecting a law does not guarantee executing it consistently over time.

Pillar-wise, multi-law cases are consistently harder than any single-law pillar, indicating that models capture isolated motion priors but fail to carry physical states across law transitions, where state transfer and temporal consistency are critical. Source-wise, all models perform worse on real-world than simulated cases despite identical governing laws, isolating a Sim-to-Real gap attributable to grounding and tracking failures under realistic optics rather than to the physics itself. Unified models retain higher absolute scores on real cases, but the gap persists.

Qualitative analysis identifies a characteristic failure mode: models may preserve annotations and track objects correctly yet misbind an annotation to its physical meaning—for instance, assigning an initial-velocity arrow the wrong direction as an initial condition, which then propagates errors through both Formulation-Graphic and Deduction. A second failure class precedes physics entirely: format violations such as blurred text, unreadable formulas, or missing velocity arrows.

Protocol ablations

Two ablations verify that the protocol does not inject the solution. Supplying physical quantities as structured text instead of infographic overlays changes average scores by at most +0.020 (Nano Banana 2) and +0.008 (Veo 3.1), with all track deltas within $\pi$1—evidence that the benchmark is not bottlenecked by reading annotations. A concise-prompt variant lowers averages (−0.018 and −0.037), with degradation concentrated on graphic-output tracks (P-G drops up to −0.085), while F-T improves by +0.051 for both models. The full prompt therefore mainly standardizes visual answer format rather than providing physics content.

Limitations and open questions

The paper concedes several scopes of validity. Physics coverage is restricted to classical rigid-body mechanics; fluids, thermodynamics, electromagnetism, deformable and granular media, and biological motion are excluded. Objects are limited to four primitive solids with a single fixed camera, so articulated objects, multi-view consistency, and novel-view generation are untested. The annotated-first-frame interface means the benchmark does not test whether a model can reconstruct the physical scene from text alone—it tests reasoning given a visually grounded setup. MLLM judges are acknowledged as imperfect physical oracles, and objective metrics depend on segmentation quality; velocity estimation is noisy for real-world clips. Real-world ground truth, while instrumented and multi-pass reviewed, remains less exact than simulator state. The temporal normalization rule assumes generated clips represent the requested physical duration, an assumption necessitated by fixed-length provider containers. An open question the results leave unresolved is whether the Deduction bottleneck (~0.40 even for the strongest models) can be closed by scaling video training, or whether it requires architectural mechanisms that couple explicit physical understanding to temporal generation.

Conclusion

Apple-$\pi$2 contributes a law-grounded, stage-resolved evaluation protocol that makes the physical reasoning of video models auditable, supported by a carefully annotated 400-case mechanics dataset and a hybrid subjective-objective metric suite. Its results quantify a clear gap between visual plausibility and law-grounded physical intelligence: video models perceive physical cues and partially formulate laws but fail to deduce law-consistent dynamics, compose laws, and transfer across visual domains. The benchmark provides a diagnostic foundation for measuring progress on these specific failures rather than on aggregate plausibility.