A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

Abstract: Generative world models hold immense promise as scalable simulators for autonomous systems, particularly for synthesizing rare but safety-critical multi-agent interactions, such as vehicle collisions. However, current evaluation paradigms index heavily on visual fidelity and semantic alignment, leaving a critical blind spot: they cannot reliably quantify whether generated dynamics actually obey the fundamental physical laws required for reliable simulation. Assessing this physical plausibility is inherently difficult due to a lack of physical metrics and the challenge of extracting metric-scale kinematics from uncalibrated video rollouts. To bridge this gap, we introduce CrashTwin, a physics-grounded evaluation framework designed to stress-test the physical trustworthiness of world models. CrashTwin couples a diverse dataset of multi-agent collision scenarios, comprising 25K controllable synthetic and 12K in-the-wild real-world collision sequences with a novel calibration-free reconstruction pipeline, enabling the recovery of 3D physical attributes directly from world model rollouts. We propose a diagnostic suite that systematically evaluates three dimensions: spatio-temporal consistency, momentum and kinetic energy conservation, and world-dynamics integrity. Extensive benchmarking of state-of-the-art models reveals a crucial insight: high perceptual quality frequently masks severe physical violations during complex interactions. By quantitatively exposing these failure modes, CrashTwin provides a vital diagnostic tool for developing physically grounded world models capable of reliable real-world simulation.

• The paper introduces CrashTwin, a benchmark merging synthetic and real-world crash data with physics-based metrics to assess physical plausibility in multi-agent dynamics.
• It presents a calibration-free 3D reconstruction pipeline combining tracking, segmentation, depth estimation, and SLAM to accurately recover metric-scale trajectories in collision scenarios.
• Empirical results show that while current world models achieve good visual fidelity, they often violate fundamental physical laws such as momentum and energy conservation.

A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models

Motivation for Physics-Grounded Evaluation

Conventional evaluation paradigms for generative world models, especially in the autonomous driving domain, have predominantly focused on visual fidelity and semantic alignment. However, these criteria fail to address a critical dimension—physical plausibility—necessary for safe deployment in real-world safety-critical tasks such as vehicle collision simulation. The absence of metrics that explicitly test for adherence to fundamental physical laws results in a blind spot: current models can generate visually plausible but physically inconsistent multi-agent interactions. The framework introduced in "A Physics-Grounded Benchmark for Multi-Agent Dynamics in World Models" (2606.28757) directly addresses this deficit.

Figure 1: CrashTwin Benchmark design and evaluation pipeline, demonstrating physics-derived metrics that align with human assessments of physical realism and expose deficiencies missed by visual fidelity proxies.

CrashTwin: Benchmark Design and Dataset Construction

CrashTwin is centered on a comprehensive benchmark that integrates both simulated and real-world vehicle collision data. The dataset consists of approximately 25,000 synthetic crash sequences generated with CARLA [Dosovitskiy17CoRL-CARLA] and over 12,000 real-world accidents curated from public sources. Each sequence includes annotated agent identities, kinetic state logs, and textual descriptions, providing granular control for analyzing multi-agent interactions under varying environmental and collision configurations.

The synthetic data systematically spans seven high-frequency pre-crash typologies derived from national statistics, ensuring physically interpretable coverage across urban and rural settings, diverse angles, and dynamic contexts. Real-world sequences are temporally segmented around impact events and manually annotated to guarantee identity and context fidelity.

Importantly, the dataset overcomes the lack of physical observability in uncalibrated monocular videos via a calibration-free reconstruction pipeline.

Calibration-Free Global Dynamic Reconstruction

Central to the benchmark is the ability to reconstruct 3D, metric-scale trajectories for all collision actors directly from monocular videos, without calibration data. The pipeline fuses object-centric detection and tracking (CenterTrack), robust video instance segmentation (SAM2), metric depth estimation (Metric3D V2), and SLAM-based ego-motion recovery (DROID-SLAM). Tracking fragments are relinked and refined to maintain consistent instance identities and metric scales, then mapped into a global coordinate frame with Kalman smoothing to ensure spatio-temporal coherence.

This reconstruction enables accurate attribution of physical quantities such as velocity, angular momentum, and contact geometry even in visually complex, occluded, and dynamic crash scenarios.

Figure 3: The global dynamic reconstruction pipeline, synthesizing tracking, segmentation, depth, and ego-motion to recover temporally smooth 3D trajectories suitable for physics-based evaluation.

Physics-Grounded Evaluation Metrics

CrashTwin evaluates world-model rollouts across three complementary axes:

1. Spatio-Temporal Consistency

Temporal coherence is measured using a flow-based warping error that penalizes abrupt motion discontinuities, while spatial rigidity is quantified via divergence in optical flow within the instance mask, targeting non-rigid deformations inconsistent with rigid-body dynamics.

2. Conservation of Momentum and Energy

Linear and angular momentum residuals are evaluated before and after collision impulses, normalized by pre-impact magnitudes to account for differing scale scenarios. An energy gain penalty is triggered for non-physical increases in kinetic energy, consistent with dissipative impact dynamics.

3. World-Dynamics Integrity

Instance stability is assessed using the Simpson index on tracking IDs, while appearance drift is tracked via the evolution of CLIP embeddings extracted from segmented instance crops. These metrics ensure that world models retain identity and feature consistency for each agent across the collision event.

The design of these metrics enables localization and quantification of physical law violations, as opposed to the global indirectness of standard visual metrics.

Figure 5: Visualization of typical physical law violations identified by each metric, from temporal warping and rigid distortion to angular inconsistency and identity instability.

Empirical Analysis of State-of-the-Art World Models

Extensive benchmarking of both open-source (e.g., SkyReel, Wan, Cosmos-Predict2) and proprietary (e.g., Google Veo, Hailuo, Seedance) models on CrashTwin reveals consistent patterns:

• High perceptual and spatio-temporal consistency scores are often accompanied by substantial violations of physical conservation laws, such as unbalanced momentum exchange and non-physical energy gain during collisions.
• Models frequently maintain coherent identities and consistent appearance, while still failing to generate physically plausible post-impact dynamics.

Quantitative results highlight that momentum and energy residuals are systematically higher than temporal or appearance fidelity metrics across all evaluated methods. Post-training with physically motivated objectives yields improvements, but spatio-temporal smoothness and appearance stability remain easier to optimize than correct physical law adherence.

Figure 2: Failure cases in current world models revealed by the benchmark, illustrating various classes of physical breakdowns that are visually plausible but physically non-compliant.

Figure 6: Qualitative improvement after physics-aware post-training, where lateral momentum transfer and appearance stability are restored compared to the baseline.

Human Alignment and Diagnostic Value

A two-alternative forced-choice (2AFC) human study demonstrates that the physics-grounded metrics align more strongly with human judgments of physical realism than conventional visual similarity proxies, supporting their diagnostic validity for model development. This confirms the orthogonality and necessity of physical metrics in safety-critical multi-agent simulation evaluation.

Theoretical and Practical Implications

The development of CrashTwin represents a substantive advancement in world model evaluation methodology:

• Theoretically, it provides a unified decomposition of physical integrity for generated rollouts and a foundation for physically aware generative model training—shifting the focus from perceptual realism to interpretable and testable physical law compliance.
• Practically, the benchmark and its open evaluation protocol provide actionable guidance for model improvement cycles, especially in high-risk domains such as autonomous driving testing, virtual data generation, and safety-critical human-in-the-loop planning.

Existing results suggest that further progress will require not only post-training on physically grounded data, but also tighter integration of explicit physical priors—potentially hybridizing neural and classical simulation approaches—and more sophisticated calibration-free reconstruction under severe domain transfer and sensor noise.

Figure 4: Progressive stages of the reconstruction pipeline, showing reduction of identity, depth, and ego-motion errors.

Figure 7: Examples of reconstructed trajectories, demonstrating high spatial and temporal fidelity compared to ground truth for diverse crash scenarios.

Conclusion

CrashTwin delivers a large-scale, physically diagnosable benchmark for multi-agent generative models in crisis events. Its rigorous metrics expose latent failure modes undetectable via traditional visual evaluations, supporting the development of world models that are not only visually credible but also physically trustworthy. The framework sets a new standard for evaluation in safety-critical AI, highlighting the necessity of physics-based metrics for reliable deployment, and invites future work on physically consistent generative modeling, hybrid neural-physical simulators, and improved dense reconstruction from uncalibrated real-world data.