DiT360: Diffusion Transformer for 360° Images

• DiT360 is a diffusion-based transformer framework designed to overcome geometric challenges in 360° image synthesis by leveraging both panoramic and perspective data.
• It employs a hybrid architecture with a Vision Transformer, latent diffusion process, and VAE compression, augmented by circular padding, yaw loss, and cube loss to ensure boundary and polar fidelity.
• DiT360 achieves state-of-the-art performance with improved FID scores and enhanced photorealism, enabling robust applications in AR/VR, immersive media, and panoramic scene reconstruction.

DiT360 refers to a class of diffusion-based transformer frameworks for high-fidelity panoramic (360°) image generation that systematically leverage both panoramic and perspective data through hybrid training, along with architectural and loss innovations to address domain-specific geometric constraints and challenges (Feng et al., 13 Oct 2025).

Unlike earlier approaches focused purely on model design, DiT360 advances a data-centric perspective: it integrates cross-domain supervision and regularization at both the image and token (latent) level, enabling state-of-the-art performance on tasks such as text-to-360° panorama synthesis, panoramic inpainting, and outpainting. The result is a system capable of generating seamless, perceptually realistic, and geometrically correct equirectangular images, with particular emphasis on polar fidelity and boundary continuity.

1. Motivations and Challenges in Panoramic Generation

Panoramic image synthesis introduces unique constraints not present in conventional perspective image generation. Equirectangular (ERP) projections suffer from severe polar distortions and require exact left-right (0°/360°) boundary continuity to preserve spherical topology. Scarcity of high-quality panoramic training data further limits photorealism and diversity; most available datasets exhibit pronounced blurring or rendered artifacts, particularly in polar regions (Feng et al., 13 Oct 2025).

By contrast, vast perspective image corpora are rich in natural detail and visual variety. Previous models that learn exclusively from panoramic data often fail to transfer local realism and exhibit global discontinuities at ERP boundaries. DiT360 is specifically designed to fuse the geometric fidelity inherent in 360° imagery with the high-frequency content and realism of perspective images via hybrid training pipelines.

2. Core Architecture: Hybrid VAE + Diffusion Transformer

DiT360’s architecture is anchored in a latent diffusion model based on a Vision Transformer (ViT)–conditioned DiT backbone, coupled with a pretrained VAE for equirectangular images. The pipeline can be summarized as follows:

• Input Encoding: Either text prompts (via CLIP or ViT encoders) or reference images are mapped to conditioning vectors through cross-attention.
• VAE Compression: The panoramic RGB image $I\in\mathbb{R}^{H\times W\times3}$ is compressed to latent tokens $X_0\in\mathbb{R}^{N\times d}$, with $N=\frac{H}{f}\times\frac{W}{f}$, where $f$ is the VAE downsampling factor.
• Diffusion Process: Forward noise is added to the token sequence:

$$X_t = \sqrt{\alpha_t}\,X_0 + \sqrt{1-\alpha_t}\,\epsilon,\quad \epsilon\sim\mathcal{N}(0,I)$$

The DiT backbone predicts $\epsilon_\theta(X_t, c, t)$ per standard score-matching loss.

• Circular Padding: To ensure boundary consistency, the last and first columns of $X_t$ are prepended and appended, creating a horizontally periodic latent map. Positional embeddings are similarly padded, enforcing periodicity in all attention layers.
• Denosing and Decoding: Iterative denoising reconstructs the ERP panorama, which is decoded to pixel space by the VAE.

This backbone supports multiple tasks: text-to-panorama synthesis, inpainting/circular outpainting, and allows for seamless swapping between panoramic and perspective training modalities (Feng et al., 13 Oct 2025).

3. Inter-Domain Transformations and Image-Level Regularization

At the image level (pre-VAE), DiT360 performs inter-domain transformations to address the polar blurring and photorealism deficit present in panoramic datasets.

• Panoramic Refinement: To enhance polar regions (which are often blurry in datasets like Matterport3D), panoramas are mapped to cubemaps, masked at the poles, and inpainted using perspective inpainting models. The inpainted cubemap is reprojected back to ERP for training, resulting in sharper polar content without distorting the intrinsic geometry.
• Perspective Image Guidance: High-quality, internet-sourced landscape images (perspective shots) are projected as lateral faces of the cubemap, then mapped to the ERP through masks. These perspective-guided panoramas are injected into training, using masked loss to strongly encourage photorealism and local detail.

This dual-regime regularization bridges the gap between the highly structured, artifact-prone panoramic datasets and the diverse, visually rich perspective datasets.

4. Intra-Domain Augmentation: Token-Level Supervision

In the latent space post-VAE encoding, DiT360 introduces a suite of augmentation mechanisms that enforce geometric, rotational, and distortion-aware priors:

• Position-Aware Circular Padding: By padding the latent token grid horizontally (first/last column prepended/appended), the architecture intrinsically enforces left-right continuity, preventing boundary artifacts without architectural changes.
• Rotation-Consistent Yaw Loss: To achieve global rotational invariance, ERP representations are randomly yaw-rotated; their denoising predictions and noise targets are also rotated, and a dedicated yaw loss penalizes inconsistent predictions across yaw-aligned pairs.
• Distortion-Aware Cube Loss: Both model predictions and ground-truth noisy latents are projected to six cubemap faces. A cube loss penalizes differences in predicted versus actual noise in these projections, directly regularizing the model against polar distortions and enforcing perspective consistency.

These mechanisms correspond to token-level losses that complement the pixel or perspective-level strategies, yielding improvements in both boundary and polar fidelity (Feng et al., 13 Oct 2025).

5. Supervision Strategy and Training Regime

DiT360's hybrid supervision alternates (or blends) batches from both perspective and panorama domains. The two branches use complementary objectives:

• Perspective branch: Masked denoising MSE on ERP regions reprojected from perspective crops—injecting high-quality detail.
• Panoramic branch: Joint MSE, cube loss, and yaw loss to regularize geometry, continuity, and distortion invariance.

The result is a model trained to unify the strengths of both domains, yielding sharp, photorealistic, and distortion-aware 360° images.

6. Experimental Performance and Ablation

Empirical results on Matterport3D and other benchmarks show that DiT360 outperforms prior models across numerous quantitative metrics relevant to panoramic generation:

Metric	DiT360	Next Best (e.g., SMGD, prior baselines)
FID (Inception)	42.88	46.72
FID₍pole₎	50.88	65.69
FAED	2.91	≤2.91
IS	1.60	<1.60
BRISQUE	10.25	>10.25

The integration of each module yields additive or complementary gains. For example, circular padding reduces FID by nearly 3 points with pronounced boundary artifact elimination, while cube and yaw losses specifically target polar and rotational robustness (Feng et al., 13 Oct 2025).

Qualitatively, DiT360-generated panoramas exhibit seamless boundaries, higher-frequency realism even in polar skies, and consistent global structure. In inpainting/outpainting, the framework preserves subject identity and maintains spherical geometry without requiring extra fine-tuning.

7. Broader Significance and Implications

DiT360 demonstrates that panoramic image generation is best addressed by hybrid methods that combine cross-domain training with architecture-level and loss-level innovation. The reliance on both perspective and panoramic data overcomes key limitations imposed by the scarcity and limitations of stand-alone 360° corpora. The position-aware padding and rotational/distortion-aware token-level augmentations are particularly effective in yielding panoramas with continuous seams, distortion-free poles, and enhanced photorealism.

This framework natively supports a wide array of generative tasks, including text-conditional synthesis, inpainting, and outpainting with high geometric accuracy. Immediate applications include AR/VR scene reconstruction, photo-realistic panoramic content creation, and robust scene completion pipelines for robotics and immersive media (Feng et al., 13 Oct 2025).

A plausible implication is that future panoramic generation architectures will continue to embrace cross-domain, hybridized frameworks, leveraging abundant perspective data as a persistent source of photorealistic supervision while tuning task-specific modules for the unique demands of 360° geometry.