Resonance4D: Frequency-Domain Motion Supervision for Preset-Free Physical Parameter Learning in 4D Dynamic Physical Scene Simulation

Abstract: Physics-driven 4D dynamic simulation from static 3D scenes remains constrained by an overlooked contradiction: reliable motion supervision often relies on online video diffusion or optical-flow pipelines whose computational cost exceeds that of the simulator itself. Existing methods further simplify inverse physical modeling by optimizing only partial material parameters, limiting realism in scenes with complex materials and dynamics. We present Resonance4D, a physics-driven 4D dynamic simulation framework that couples 3D Gaussian Splatting with the Material Point Method through lightweight yet physically expressive supervision. Our key insight is that dynamic consistency can be enforced without dense temporal generation by jointly constraining motion in complementary domains. To this end, we introduce Dual-domain Motion Supervision (DMS), which combines spatial structural consistency for local deformation with frequency-domain spectral consistency for oscillatory and global dynamic patterns, substantially reducing training cost and memory overhead while preserving physically meaningful motion cues. To enable stable full-parameter physical recovery, we further combine zero-shot text-prompted segmentation with simulation-guided initialization to automatically decompose Gaussians into object-part-level regions and support joint optimization of full material parameters. Experiments on both synthetic and real scenes show that Resonance4D achieves strong physical fidelity and motion consistency while reducing peak GPU memory from over 35\,GB to around 20\,GB, enabling high-fidelity physics-driven 4D simulation on a single consumer-grade GPU.

• The paper introduces dual-domain motion supervision combining spatial and frequency constraints to enable preset-free full physical parameter optimization in 4D simulations.
• It demonstrates superior physical and visual fidelity with reduced GPU memory usage compared to traditional online motion prior methods.
• Simulation-driven initialization and zero-shot segmentation ensure robust part-level optimization, achieving accurate dynamic behavior in heterogeneous scenes.

Resonance4D: Dual-Domain Frequency-Based Supervision for Physics-Driven 4D Scene Simulation

Introduction and Context

"Resonance4D: Frequency-Domain Motion Supervision for Preset-Free Physical Parameter Learning in 4D Dynamic Physical Scene Simulation" (2604.01994) addresses a critical bottleneck in physics-based dynamic scene synthesis from static reconstructions: the heavy reliance on dense online motion priors (e.g., video diffusion or optical-flow pipelines) and incomplete material parameter modeling. Existing approaches often incur significant computational overhead, limiting scalability and practical usability, and typically reduce physical parameter optimization to a subset of the full parameter space. Resonance4D proposes a new unified framework that eliminates these dependencies by (i) introducing Dual-domain Motion Supervision (DMS)—a lightweight yet expressive set of constraints combining local spatial (structural) and frequency-domain (spectral) losses, and (ii) enabling part-level full physical parameter joint optimization, including density and constitutive parameters, initialized through simulation-guided search and leveraging zero-shot segmentation for automatic part assignment.

Figure 1: Projection of temporal dynamics into $w$-$t$ spatiotemporal/frequency domains enables interpretable differentiation between deformation and oscillation, and highlights the necessity of part-level physical modeling for plausible motion.

This approach is situated at the intersection of 3D Gaussian Splatting (3DGS), differentiable Material Point Method (MPM) simulation, and automated material property inversion. The technical premise is that optimizing for both spatial structural consistency and local spectral patterns yields sufficient supervision for physically meaningful dynamic fitting, thus obviating the need for data-hungry or compute-intensive temporal supervision pipelines.

Methodology

The Resonance4D pipeline is architected as a sequence of (1) reference motion video generation, (2) automatic part-level assignment via zero-shot segmentation and 3D mapping, (3) simulation-driven optimal initialization of the full physical parameter vector space, and (4) differentiable, part-level joint parameter optimization under dual-domain supervision.

Figure 2: Full pipeline: static 3DGS input, zero-shot part assignment, simulation-driven parameter sampling, and dual-domain motion-constrained differentiable physical parameter optimization.

Dual-domain Motion Supervision (DMS)

The DMS module constitutes the cornerstone of the framework. Unlike frame-wise only alignment, DMS introduces:

• Spatial domain constraint: Frame-wise SSIM loss, enforcing local structural appearance consistency.
• Frequency domain constraint: Patch-based 3D FFT over local spatiotemporal blocks of frame-difference sequences. Spectral (magnitude and phase) alignment losses are computed for each local region, effectively penalizing both amplitude and phase mismatches in dynamic patterns.

By combining these with patch-level granularity, the supervision is robust to static backgrounds, sensitive to oscillatory features, and enforces physically plausible motion trajectories absent from static-only losses.

Simulation-driven Initialization

Since the high-dimensional and coupled physical parameter space (e.g., Young's modulus, Poisson ratio, density, plastic viscosity) is highly sensitive to initialization, Resonance4D performs Latin Hypercube Sampling in log-parameter space. For each candidate, short horizon MPM rollouts are compared against the reference video using MS-SSIM, and the best candidate initializes the learnable parameter vectors for subsequent optimization.

Part-level Full Parameter Optimization

Automatic part assignment is performed via DINOv3 and SAM3 zero-shot segmentation over multi-view images, lifted to the 3DGS scene using soft category voting and KNN smoothing. Each part is associated with a physical parameter vector, shared by all constituent Gaussians/MPM particles. Optimization proceeds in the log domain (for positivity and stability), using AdamW and gradient clipping. Parameters are regularized to fall within empirically valid physical bounds, and all updates and loss evaluations are fully differentiable and memory efficient.

Experimental Results

Resonance4D is evaluated on both real-world (PhysDreamer dataset) and synthetic (PAC-NeRF dataset) 4D dynamics benchmarks, comparing with DreamPhysics, Physics3D, PhysFlow, and PAC-NeRF. Evaluation metrics include MS-SSIM, PSNR, LPIPS, Chamfer Distance (CD), HD95, and F-score.

Key findings:

• Physical and visual fidelity: Achieves state-of-the-art or highly competitive scores in both visual (SSIM/PSNR/LPIPS) and physical (CD/HD95/F-score) domains, besting methods that utilize temporal diffusion priors, especially in scenes characterized by heterogeneous material distributions and part-specific dynamic behaviors.
• Efficiency: Peak GPU memory requirements are reduced from $>35$ GB (for diffusion/optical flow-supervised baselines) to approximately 20–22 GB, making high-fidelity physics-driven scene synthesis feasible on a single consumer-grade GPU.
• Motion consistency: DMS notably enhances recovery of local oscillatory details, producing motion traces that exhibit periodicity and dominant frequencies closely matching real-world references.

  Figure 3: Scene-wide qualitative comparison; Resonance4D preserves oscillatory and restoring motion with spectral characteristics closer to ground-truth compared to prior work.

  

  Figure 4: Scene- and material-dependent temporal evolution: in stiff/soft regimes, Resonance4D avoids both over-softening and unnatural rebound effects.

Ablation Studies: Removing frequency-domain supervision degrades oscillatory motion modeling and overall task performance; eliminating density optimization or reverting to purely object- or particle-level parameters likewise degrades dynamic accuracy or leads to unstable convergence. Simulation-driven initialization outperforms GPT-based or naïve initialization, justifying its moderate offline overhead.

Implications and Future Directions

Resonance4D demonstrates that heavy online motion priors are not strictly necessary for high-quality physics-driven 4D scene simulation given appropriately designed spatial-spectral supervision. The method scales to complex, materially heterogeneous, and visually realistic settings while keeping computational cost tractable. Part-level optimization with automated zero-shot segmentation robustly balances expressiveness and stability, pointing toward future avenues in scaling physical property recovery even in the presence of ambiguous or sparse visual cues.

Practically, this opens the path to scalable, interactive, and physically plausible world modeling for robotics, digital twin applications, and immersive virtual environments—especially where computational resources are limited or annotation is infeasible. Theoretically, the results support the premise that carefully chosen, domain-tailored, lightweight supervisory signals suffice for parameter inversion in complex physical systems, a thesis likely to influence future physics-driven scene and material modeling research.

Remaining limitations include the requirement for relatively complete 3DGS reconstructions and the absence of high-level semantic guidance beyond segmentation; extending the approach to few-shot/sparse-view settings and integrating end-to-end feedback from foundation models are promising future research directions.

Conclusion

Resonance4D (2604.01994) advances the state-of-the-art in physics-based 4D dynamic scene simulation by establishing that dual-domain spatial-spectral motion supervision, coupled with robust part-level material parameter optimization, suffices to recover physically and visually plausible scene dynamics without reliance on expensive online priors. The framework yields improved dynamic interpretability, lower memory footprint, and practical accessibility for real-world and synthetic scenes. This work lays the groundwork for further scaling and adoption of physically grounded 4D modeling methodologies in AI, vision, and graphics systems.