Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis

Abstract: Weather synthesis aims to add weather effects to input videos while preserving scene identity, structure, and motion. The key limitation of existing methods is the lack of diversity in weather appearance and effective control over weather dynamics (e.g., temporal evolution and particle motion). Most approaches rely on text prompts, which are inherently underspecified and often fail to produce detailed weather characteristics. Additionally, general-purpose video editors optimized for clean and aesthetic outputs tend to suppress heavy weather phenomena, making dense particle effects difficult to generate. To address these, we propose a Semantic-Aware, Physics-Informed, and Geometry-Grounded framework that steers an off-the-shelf video editor to synthesize diverse global appearances and detailed particle dynamics. We factorize the synthesis into three conditional signals, so that each provides a distinct and stable source of guidance: semantics specifies what the weather should look like, dynamics governs how it evolves over time, and geometry determines where it should appear in the scene. Specifically, we introduce (1) semantic-aware appearance anchoring to establish the target appearance from scene semantics and user input; (2) physics-informed dynamic simulation to generate particle effects by simulating a Gaussian-represented particle field under gravity, wind, and turbulence; and (3) geometry-grounded video synthesis to align the simulated particles with target scene geometry and synthesize the final video. Experiments demonstrate that our method produces diverse, physically and visually realistic weather effects. Furthermore, we show that our synthesized data significantly improves the robustness of autonomous driving semantic segmentation under adverse weather conditions. Project page: https://jumponthemoon.github.io/w-crafter/.

• The paper introduces a tri-prior framework combining semantic parsing, physics simulation, and geometry grounding to achieve realistic weather video synthesis.
• It employs anisotropic Gaussian particles and dynamic force-based simulation to create diverse and physically consistent weather phenomena.
• The method enables fine-grained control over weather effects, notably improving downstream tasks like semantic segmentation with up to a 14.5 percentage point mIoU gain.

Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis

Introduction

The paper "Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Video Synthesis" (2606.29020) introduces a principled framework for the explicit, controllable, and realistic synthesis of weather phenomena (snow, rain) in unconstrained natural video. Existing methods struggle with limited diversity in weather effects, poor controllability over spatial-temporal dynamics, and difficulties in eliciting fine-grained atmospheric particles using text prompts alone. This framework proposes a compositional conditioning scheme, tightly coupling scene semantics, particle-level dynamics, and explicit 3D geometry to unlock complex weather manipulation.

Method

Factorized Conditioning Interface

The approach decomposes the synthesis task into three channels: semantic-aware appearance anchoring, physics-informed particle simulation, and geometry-grounded video synthesis. This factorization is motivated by the observation that pretrained video diffusion models encode, but underutilize, strong priors related to weather effects, which can be activated by providing explicit per-frame particle cues as structured conditions.

Semantic-Aware Appearance Anchoring

A Vision-LLM (VLM) parses scene semantics from the initial frame. An LLM, guided by user intent (type, severity, duration), generates detailed editing instructions, producing refined prompts with rich scene context. An image-to-image model creates an initial frame that visually anchors the target atmospheric appearance, providing both a strong reference and a compact, captioned video prompt for subsequent video synthesis.

Figure 1: Semantic-aware appearance anchoring leverages VLM parsing, LLM-based reasoning, and I2I editing to ground weather effects in semantics.

This strategy substantially improves controllability, particularly in disentangling local/global effects, selecting lighting modes, and producing scene-consistent edits. Compared to text-only prompts, it enables reliable synthesis of both subtle and drastic weather conditions while maintaining identity (Figure 2).

Figure 2: Detailed semantic prompts lead to accurate, diverse scene-specific weather edits compared to naive prompting.

Physics-Informed Dynamic Simulation

Weather particles are instantiated as grid-sampled anisotropic Gaussians in 3D. Their states are evolved using explicit Newtonian dynamics incorporating gravity (estimated from geometry), wind, and curl-noise turbulence for local perturbations. Particle parameterization governs shape (round/elongated), severity, directionality, and temporal persistence. The simulation yields physically plausible, temporally coherent, and controllable cues for video synthesis.

Figure 3: Physically-informed simulation pipelines yield explicit dynamic priors, with anisotropic Gaussian particles evolved under wind, gravity, and turbulence.

Geometry-Grounded Video Synthesis

Robust geometry is extracted via state-of-the-art monocular depth estimation. Scene-level gravity orientation is inferred by ground plane fitting on 3D point clouds, ensuring the simulated particle field aligns with the actual gravitational direction, even in non-axis-aligned scenes. Camera intrinsics/extrinsics are used to project the evolving particle field per-frame, generating dynamic conditioning maps. The final output is generated using a pretrained video diffusion model, conditioned on the initial appearance anchor, per-frame particle dynamics, and geometry signals.

Figure 4: Geometry-grounding aligns simulation with scene layout, projecting dynamic particles to each frame and ensuring physical consistency across camera motion and scene structure.

Integration and Implementation

All presentation modules (appearance, dynamics, geometry) are modular and efficiently composable. The system leverages top-performing VLMs, LLMs, I2I, depth models, and video diffusion editors in a zero-shot regime, requiring no task-specific fine-tuning, reducing data and computational overhead.

Results

The framework achieves superior performance across both qualitative and quantitative benchmarks. On challenging real-world datasets encompassing static and dynamic scenes, it delivers atmospheric editing that is semantically consistent, physically realistic, and structurally faithful.

Figure 5: Weather synthesis comparison—prompt-only and naive conditional methods fail to produce fine-grained particles or local realism; the proposed method generates dense, well-integrated snow/rain with consistent dynamics.

Quantitative metrics reveal the highest CLIP-D (directional) scores for both snow and rain, indicating the strongest alignment with desired edit directions, while retaining competitive CLIP-S (similarity) scores, ensuring identity is preserved. VLM-based perceptual evaluation shows the highest visual quality and semantic correctness among all evaluated methods. In human studies, this method is rated the most photorealistic and physically plausible, with 41.3% (rain) to 56.5% (snow) of preferences, outperforming prior state-of-the-art by a large margin.

A series of ablations demonstrate the criticality of each factor—removing detailed semantic anchoring or geometry grounding leads to degraded visual coherence, particle hallucination, or implausible motion (Figure 6, Figure 7).

Figure 6: Without semantic parsing and captioning, results are susceptible to hallucinations or loss of scene structure; detailed semantic anchoring counteracts these shortcomings.

Figure 7: Geometry grounding is essential—without it, snow/rain fall directions and scales are inconsistent with scene structure and camera motion.

Applications

The system enables precise, interpretable, and disentangled control over both high-level appearance (weather type, duration, severity) and fine-level attributes (particle shape, wind direction, local density) (Figure 8).

Figure 8: Fine-grained, user-steerable controls of weather synthesis, including duration and wind-driven particle dynamics.

For autonomous driving perception, the synthesized videos augment mIoU in domain generalization benchmarks, with segmentation accuracy increased by up to 14.5% for models trained with this data, outperforming all baselines (Figure 9).

Figure 9: Synthetic adverse weather data significantly improves segmentation model robustness in rain and snow conditions.

Theoretical and Practical Implications

This work demonstrates that compositional, physically-grounded conditioning can bridge the gap between implicit generative modeling and explicit scene simulation. By injecting explicit, human-controllable priors at each axis (semantics, dynamics, geometry), pretrained diffusion models' latent capacity for rare, complex phenomena is accessed, bypassing the need for massive weather-specific retraining. This approach makes it feasible to synthesize corner cases (e.g., heavy snow on a desert pyramid) unsolvable via text prompts or current neural rendering approaches.

Practically, robust synthetic data generation is critical for safety validation of perception systems in domains (autonomy, robotics) where real weather data is prohibitively rare or unsafe to collect. The composability and modularity of this framework point toward scalable scene manipulation tools across not only weather, but also other physics-consistent effects (e.g., fog, smog, sandstorms) and may support further expansion to end-to-end closed-loop simulation workloads.

Limitations and Future Directions

The system is susceptible to errors in upstream modules, particularly geometry estimation and gravity alignment in scenes lacking large planar structures or with poor depth maps. Diffusion models’ expressivity places an upper bound on achievable particle effects. Joint optimization of the upstream modules and design of stronger domain-adaptive generative priors are possible avenues to improve robustness and quality. The factorized conditioning approach provides a foundation for further progress in explicit neural scene editing under complex, multi-physics phenomena.

Conclusion

This paper makes a significant methodological contribution by unifying semantic, physical, and geometric constraints for controllable weather video synthesis. The approach yields a controllable interface for precise, diverse weather manipulation, robustly transfers to downstream perception, and is readily extensible to other domains of physical scene editing. The demonstrated empirical gains and methodological generality underscore the utility of structured, simulation-grounded conditioning in video generative frameworks.