PanoWorld: Real-World Panoramic Generation

Abstract: In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.

• The paper introduces a diffusion-based framework that decouples camera rotation from translation to enhance panoramic video synthesis and reduce geometric artifacts.
• Its novel DPRC and GMA modules enable precise ray-level motion conditioning and memory-anchored synthesis for maintaining long-term spatiotemporal coherence.
• Empirical results demonstrate superior FID, PSNR, and real-time generation speeds compared to existing methods, validating the model’s robustness and efficiency.

PanoWorld: A Diffusion Framework for High-Fidelity and Controllable Real-World Panoramic Video Generation

Introduction and Problem Setting

PanoWorld introduces a panoramic world modeling framework that explicitly addresses spatiotemporal consistency and trajectory control in real-world $360^\circ$ video generation. Recognizing the rotation-equivariant nature of equirectangular panoramic representations, the framework focuses on decoupling camera translation and rotation during generative modeling. This approach overcomes limitations in existing panoramic video models that conflate rotational transformations with translation-induced parallax, causing artifacts and geometric inconsistencies, especially in complex outdoor environments. The research further formulates a large-scale benchmark (World360) to train and evaluate the robustness and physical realism of panoramic video synthesis across both real-world and high-fidelity simulated trajectories.

Figure 1: PanoWorld achieves high-fidelity and controllable panoramic video generation with precise trajectory control and physical consistency in diverse environments.

Data Pipeline and Benchmark Construction

A significant contribution is the curation of World360, a dataset constructed via a bespoke pipeline targeting physically-grounded correspondence between visual frames and camera trajectories. The pipeline operates through three key data-processing modules: Rotation Decoupling (aligning all frames to a canonical heading to isolate translational motion), Uniform Spatial Resampling (structuring video sequences by fixed spatial increments regardless of camera velocity), and Illumination Filtering (removing sequences with uneven or abnormal exposure profiles).

Figure 2: The data curation pipeline enforces geometric and radiometric consistency via rotation decoupling, spatial resampling, and illumination filtering.

World360 comprises 120,000 high-quality panoramic sequences: 70,000 from diverse UAV-captured real-world scenes and 50,000 synthetic samples from AirSim360, spanning multi-altitude trajectories with dense pose and depth annotations, providing a comprehensive evaluation ground for panoramic world modeling.

Architecture and Methodology

The PanoWorld backbone is built upon a Wan2.2 diffusion transformer, extended with two pivotal innovations: Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).

Figure 3: Network architecture integrates visual self-attention, DPRC-based action modeling, and the GMA module within a triple-stream DiT, unified on a geometric manifold.

Motion Decoupling and Geometric Adaptation

The motion decoupling principle treats camera rotation as an explicit geometric transformation independently of translation, enabling the model to focus learning capacity on translation-induced parallax and structural depth modeling. This is operationalized by fine-tuning the backbone via LoRA on panoramic datasets, enhancing adaptation to equirectangular projection properties, such as horizontal wraparound and polar distortion, and imposing a latitude-aware reconstruction loss to privilege geometric fidelity across the sphere.

Action Modeling: Dense Panoramic Ray-Conditioning (DPRC)

DPRC models motion not at the pixel level but in the panoramic ray domain, mapping each latent feature to a $\mathbb{S}^2$ ray direction. The motion manifold is defined as a function of camera translation, and explicit projective positional embeddings (PRoPE) are leveraged for conditioning the diffusion process. This geometric conditioning allows the model to align light-field evolution with physical camera movements—ensuring radiometric, geometric, and temporal consistency and accurately controlling camera trajectory across both simple and complex motion patterns.

Memory Modeling: Geometry-aware Memory Augmentation (GMA)

To maintain long-horizon consistency, the GMA module anchors synthesis to a shared geometric coordinate frame. Query and memory features are aligned in PRoPE space, enabling attention-based retrieval based on 3D ray correspondence rather than explicit spatial warping. A confidence-guided gating mechanism adaptively fuses memory content with current features, maximizing consistency for re-visited regions and suppressing hallucinations in unseen areas. This eliminates flicker and geometric drift even during extended or revisited trajectories.

Training Pipeline

PanoWorld training progresses through a staged pipeline:

Figure 4: Progressive three-stage pipeline: (1) geometric adaptation, (2) view-dependent motion control, (3) memory-anchored, long-term coherence.

• Stage 1: Panoramic fine-tuning for equirectangular geometric adaptation through LoRA and a latitude-weighted loss.
• Stage 2: View-dependent translational motion learning with frozen backbone and trajectory-centric data augmentations via DPRC.
• Stage 3: Memory-anchored synthesis via GMA to enforce long-term radiometric, geometric, and temporal stability.

Experimental Results and Analysis

Visual Quality and Scene Fidelity

Qualitative benchmarks demonstrate that PanoWorld produces sharper and more consistent videos with higher scene persistence than Matrix-3D and OmniRoam, particularly in the presence of complex trajectory variations and challenging outdoor environments.

Figure 5: Qualitative comparison on real-world outdoor sequences shows PanoWorld preserves structure and visual fidelity under diverse motion.

Quantitatively, PanoWorld yields the best FID across full, equatorial, and polar regions, as well as substantial improvements in QA and PSNR metrics for perceptual quality and trajectory control. Best-case FID scores are reported as 16.93 (720p), significantly outperforming alternatives. The framework also achieves strong aesthetic quality (QA$_{qual}$ = 4.14) and lowest structural drift per FAED and PSNR.

Trajectory Adherence

Trajectory-control is quantitatively superior: PanoWorld maintains PSNRs of $22.94 \pm 3.77$ (PSNR$_{20-25}$, 720p) across extended rollouts—reflecting high-fidelity motion-following and minimized scene drift.

Figure 6: ViPE-based evaluation of trajectory fidelity confirms tight ground-truth alignment over long horizons.

Ablation: Memory Module Impact

Ablative testing clearly demonstrates the efficacy of the GMA mechanism. Models without GMA exhibit geometric inconsistencies, broken scene structure, and increased drift, while the full system preserves long-term visual and geometric coherence.

Figure 7: Ablation results for GMA: only the full model (c) sustains consistent, artifact-free generations across sequences.

Real-Time Generation Performance

By leveraging Causal Forcing-based distillation, PanoWorld supports real-time panoramic generation, reducing inference time for a 161-frame video from 4m48s (full model) to 8 seconds (causal-forcing) with minimal quality degradation—orders of magnitude improvement versus Matrix-3D and OmniRoam.

Figure 8: Real-time generation from user-controlled trajectories demonstrates both fidelity and responsiveness.

Figure 9: Comparative generation efficiency: causal-forcing achieves ultrafast generation without significant quality loss.

Implications and Future Directions

PanoWorld expands the capabilities of video diffusion models to support high-fidelity, robust, and controllable $360^\circ$ video synthesis in physically complex environments, addressing a critical gap for real-world robotics, simulation, and immersive media. Its geometric memory mechanisms are likely to inform future research in memory-augmented video generation, especially as practical demands for spatial and temporal coherence rise in open-world and interactive applications. The modularity of DPRC and GMA suggests transferability to other panoramic representation domains; subsequent work may refine memory dynamics, address domain gaps at sequence boundaries, or extend towards fully interactive and agent-driven panoramic world models.

Conclusion

PanoWorld establishes a new standard for panoramic video generation, integrating rigorous geometric reasoning and memory-anchored synthesis within a diffusion framework. Extensive empirical results validate its superiority over prior methods in visual quality, trajectory adherence, and computational efficiency. Its proposed World360 benchmark and innovations in geometric memory modeling position it as a foundational methodology for next-generation open-world simulation and embodied AI research.

Figure 10: Further results showcase stable high-quality synthesis across diverse real-world outdoor sequences using PanoWorld.