PanoDiff: Latent Diffusion for 360° Panoramas

• PanoDiff is a latent diffusion framework that creates full equirectangular panoramas from one or several unregistered NFoV images with optional text prompts.
• It employs a two-stage angle prediction module and geometry-aware mechanisms, such as rotation-equivariance loss and circular padding, to ensure spherical consistency.
• Empirical studies show improved FID scores and robust performance on datasets like SUN360 and Laval, making it valuable for VR/AR, environment maps, and scene editing.

PanoDiff is a latent diffusion framework for generating complete $360^\circ \times 180^\circ$ panoramas from one or more unregistered narrow field-of-view (NFoV) images and an optional text prompt. It was introduced in “360-Degree Panorama Generation from Few Unregistered NFoV Images” as a pipeline that combines a two-stage angle prediction module with a Stable Diffusion–based panorama generator conditioned on an incomplete panorama, a visibility mask, and text, together with geometry-aware mechanisms for spherical consistency (Wang et al., 2023). In the paper’s formulation, the inputs are captured from the same position but at unknown orientations and with arbitrary coverage, and the central objective is to synthesize a full equirectangular panorama that preserves visible content, remains geometrically consistent on the sphere, and supports controllable generation.

1. Problem formulation and scope

PanoDiff addresses the setting in which only one or a few standard NFoV images are available, taken from the same location but with unknown viewing directions. The target is a full 360° panorama suitable for applications such as environment maps, VR/AR backgrounds, and scene editing. The method is designed to handle flexible numbers of inputs, including the single-image case and multi-image cases with arbitrary angular spacing, rather than assuming a single central crop or known camera poses (Wang et al., 2023).

The paper positions this problem as a combination of unknown pose estimation and large-hole structured outpainting. Unknown poses matter because the NFoV images are unregistered; the method must infer how each crop should be placed on the sphere. Large-hole outpainting matters because most of the panorama is typically missing, so the model must generate global structure, not merely fill local texture gaps. PanoDiff further treats controllability as a first-class objective by conditioning on text prompts in addition to image evidence, which distinguishes it from earlier panorama completion systems that were image-conditioned only (Wang et al., 2023).

A central modeling assumption is that all NFoV images share the same center of projection, so camera pose reduces to orientation. The panorama is represented in equirectangular projection, with longitude and latitude as the natural spherical coordinates. This lets the method define a projection operator from oriented NFoV views to a partial panorama and a corresponding visibility mask, which then become the conditioning signals for the latent diffusion model (Wang et al., 2023).

2. Angle prediction and partial panorama construction

PanoDiff’s first stage is a two-stage angle prediction module. Each NFoV camera is parameterized by look-at direction—longitude (yaw) $\alpha$, latitude (pitch) $\beta$—and roll $\gamma$. The method assumes upright cameras, so $\gamma_i = 0$ for all $i$, and it estimates absolute pitch together with relative yaw between pairs of images. For a set of images $\mathbf{I} = \{I_0,\dots,I_{N-1}\}$, the pairwise predictor relative to an anchor image $I_0$ is written as

$$\mathcal{R}_\theta(I_0, I_i) \to [\Delta \alpha_{i \to 0}, \beta_0, \beta_i].$$

For a single NFoV input, only $\beta$ is predicted, and the method sets $\alpha$0, $\alpha$1, placing the crop in the central longitude (Wang et al., 2023).

The two-stage design explicitly separates overlap and wide-baseline regimes. Stage 1 is an overlap classifier that predicts whether an NFoV pair overlaps in field of view or is wide baseline. Stage 2 uses separate regressors for these two regimes, each with its own prediction heads for relative yaw and the two absolute latitudes. The rationale is that a single regressor performs poorly when overlap and non-overlap cases are mixed; the classifier reduces misrouting, and the specialized regressors improve angular accuracy, particularly for wide-baseline pairs (Wang et al., 2023).

After angle estimation, the NFoV images are projected to the equirectangular canvas to form a partial panorama $\alpha$2 and a visibility mask $\alpha$3. The sphere-to-ERP mapping is expressed using longitude $\alpha$4 and latitude $\alpha$5, with pixel coordinates $\alpha$6 and panorama width $\alpha$7, height $\alpha$8 related by

$\alpha$9

The paper describes the NFoV projection as a backward equirectangular projection: for each panorama pixel, it computes which NFoV pixel would observe that direction under the predicted camera orientation (Wang et al., 2023).

This stage is operationally significant because the diffusion model does not consume raw NFoV images directly as separate views. Instead, it receives a spherical partial observation $\beta$0 and a binary support mask $\beta$1, turning view registration into a preprocessing problem and allowing the downstream generator to work entirely in panorama space.

3. Latent diffusion architecture and conditioning

PanoDiff’s second stage is a latent diffusion panorama generator built on Stable Diffusion. The method uses the standard latent diffusion decomposition into an autoencoder and a U-Net denoiser. Given a clean latent $\beta$2, the diffusion forward process is

$\beta$3

and the denoiser is trained with

$\beta$4

In PanoDiff, the conditioning $\beta$5 includes text and an image-based panorama condition derived from the partial panorama and the mask (Wang et al., 2023).

The text prompt is embedded with the standard Stable Diffusion text encoder. The image-based condition is the concatenation $\beta$6, encoded by a shallow Pano-Encoder $\beta$7 into a feature map $\beta$8. PanoDiff adopts a ControlNet-style conditioning mechanism: a trainable copy of a pretrained U-Net block is augmented with zero-initialized $\beta$9 convolutions so that the control path initially has no effect and gradually learns to inject panorama-specific information. In the paper’s notation, for a pretrained block $\gamma$0, the controlled output is

$\gamma$1

with the original $\gamma$2 frozen (Wang et al., 2023).

The overall generator is thus conditioned jointly on text, the incomplete panorama, and the visibility mask. During inference, the latent to be denoised is initialized as Gaussian noise rather than as a noised version of the partial panorama itself; the partial panorama is used as control, not as the direct optimization target. This arrangement makes the model a panorama completion system, but also a text-guided generator whose semantics can depart from the visible regions in unobserved areas while staying compatible with the observed context (Wang et al., 2023).

The architecture is notable because the model remains a standard latent diffusion model at its core. Panorama-specific behavior is introduced through conditioning and spherical-aware training and sampling schemes rather than through a bespoke spherical backbone.

4. Geometry-aware consistency mechanisms

PanoDiff introduces three mechanisms to preserve spherical structure in equirectangular space: a rotation-equivariance loss during training, a rotating schedule during denoising, and circular padding along longitude during inference (Wang et al., 2023).

The rotation-equivariance objective formalizes the idea that a global spherical rotation should commute with denoising. Let $\gamma$3 denote an image-space rotation implemented by spherical reprojection. The desired property is

$\gamma$4

PanoDiff enforces this approximately by sampling random small rotations $\gamma$5 and applying the same transform to the noisy latent and the panorama condition during training, producing the loss

$\gamma$6

This explicitly encourages the denoiser to respect the rotational symmetries of the spherical domain (Wang et al., 2023).

At inference, the rotating schedule exploits the same equivariance by rotating the latent feature and conditioning features during denoising. The latent panorama is effectively “rolled” through a sequence of spherical rotations as the reverse process proceeds. A separate circular padding step addresses the topological mismatch between ERP and standard convolutional processing: because the panorama is periodic along longitude, the method copies a thin slice from the leftmost region to the right side and a slice from the rightmost region to the left side before convolution, then crops after decoding. This removes left-right seam artifacts that would otherwise arise at the wrap boundary (Wang et al., 2023).

These design choices are central to PanoDiff’s identity within the panorama diffusion literature. The method does not merely adapt Stable Diffusion to a non-square canvas; it modifies both training and sampling to make the denoiser behave more like an operator on a cylindrical or spherical signal.

5. Empirical performance and ablation findings

PanoDiff is evaluated on SUN360 and Laval Indoor for panorama generation, and on SUN360 and Laval for angle prediction. On SUN360, the method reports FID $\gamma$7 for a single NFoV input, $\gamma$8 for paired inputs with ground-truth rotations, and $\gamma$9 for paired inputs with predicted rotations. On Laval, where the model is not trained, it reports FID $\gamma_i = 0$0 for a single input, $\gamma_i = 0$1 for pairs with ground-truth rotations, and $\gamma_i = 0$2 for pairs with predicted rotations (Wang et al., 2023).

Dataset	Setting	FID
SUN360	Single NFoV input	8.61
SUN360	Pair input with ground-truth rotations	6.94
SUN360	Pair input with predicted rotations	7.04
Laval	Single NFoV input	7.72
Laval	Pair input with ground-truth rotations	6.96
Laval	Pair input with predicted rotations	7.18

These numbers are presented against recent GAN-based panorama methods, and the paper states that PanoDiff is best among all baselines in all reported settings. The Laval results are particularly important because PanoDiff is not trained on Laval, which the paper uses to argue for cross-dataset generalization (Wang et al., 2023).

For rotation estimation, the two-stage angle predictor improves the average geodesic error on all pairs from $\gamma_i = 0$3 to $\gamma_i = 0$4 on SUN360 and from $\gamma_i = 0$5 to $\gamma_i = 0$6 on Laval. For wide-baseline pairs on SUN360, the average error improves from $\gamma_i = 0$7 to $\gamma_i = 0$8. This matters directly for the second stage because the quality of the partial panorama and visibility mask depends on pose accuracy (Wang et al., 2023).

The ablations isolate the geometry-aware components. For denoising strategies on SUN360, removing both rotation-equivariance loss and rotating schedule yields FID $\gamma_i = 0$9; adding equivariance loss alone improves this to $i$0; adding both yields $i$1, the best setting in that ablation. The paper also reports that circular padding removes visible discontinuities at the left-right boundary after a $i$2 horizontal rotation, whereas the same boundary remains visible without padding (Wang et al., 2023).

Qualitatively, PanoDiff is reported to produce smooth transitions between conditioned NFoV patches and generated content, semantically consistent completions, and realistic indoor lighting and geometry. The paper’s examples emphasize that the method handles both single-view and paired-view inputs and can produce multiple variants from the same visible region when different text prompts are supplied (Wang et al., 2023).

6. Later interpretations, term ambiguity, limitations, and applications

In subsequent literature, “PanoDiff” acquired a broader descriptive role within panorama diffusion. CamFreeDiff cites PanoDiff as a method for “360° generation from few unregistered NFoV images” and contrasts it with a camera-free formulation that predicts field of view and roll/pitch through homography rather than estimating only longitude and latitude (Yuan et al., 2024). Diffusion360 identifies PanoDiff as the most closely related work for wraparound consistency and explicitly derives its own circular blending strategy from PanoDiff’s circular padding while omitting PanoDiff’s rotating schedule (Feng et al., 2023). TanDiT later refers to PanoDiff as part of the ERP-based lineage that uses circular padding or blending to enforce loop-consistency, while shifting the main generative burden to tangent-plane grids (Çapuk et al., 26 Jun 2025).

The name is also ambiguous outside panorama synthesis. In “Panoptic Diffusion Models: co-generation of images and segmentation maps,” the exact string “PanoDiff” does not appear; the method is consistently called Panoptic Diffusion Model (PDM), and the text explicitly states that “PanoDiff” would only be an informal shorthand someone might use for panoptic diffusion in that context (Long et al., 2024). Separately, “PanoDiff-SR” uses “PanoDiff” for the low-resolution diffusion stage of a medical imaging system that synthesizes dental panoramic radiographs before super-resolution, which is a distinct domain and task from 360° scene generation (Jain et al., 12 Jul 2025).

For the panorama-generation method itself, the main limitations are explicit. Pose estimation errors can misalign NFoV projections and induce artifacts, especially in wide-baseline cases. Output resolution is $i$3, which reflects the Stable Diffusion v1-scale regime rather than very-high-resolution panorama synthesis. The model is trained only on SUN360 and evaluated zero-shot on Laval, so extreme domain shifts remain outside the reported evidence. The method is also designed for images rather than temporally coherent video, and it does not explicitly model full 3D scene structure beyond spherical projection and the partial panorama representation (Wang et al., 2023).

Within those limits, PanoDiff’s applications are broad but concrete. The paper frames it as an environment-map generator for rendering, a VR/AR background synthesizer from a few photographs, a controllable panorama editor via text prompts, and a general panorama completion model for arbitrary-shaped partial observations. A plausible implication is that its architectural pattern—pose estimation followed by latent panorama diffusion with spherical consistency mechanisms—became a template for later panoramic diffusion research, even where later systems replaced ERP-only processing with multi-view, tangent-plane, or spherical latent representations (Wang et al., 2023, Çapuk et al., 26 Jun 2025, Park et al., 19 Apr 2025).