Perspective-to-Panorama Adaptation

• Perspective-to-panorama adaptation is a framework that converts models trained on narrow field-of-view images to effectively process 360° panoramic imagery.
• The approach addresses challenges such as severe geometric distortions, non-uniform sampling, and boundary seam artifacts through projection-driven and architectural modifications.
• Advanced techniques like LoRA adaptation, cross-attention modules, and circular positional corrections enhance data efficiency and cross-domain generalization.

Perspective-to-panorama adaptation refers to the set of mathematical frameworks, architectural modifications, and training strategies that enable models—originally developed for perspective (narrow field-of-view, rectilinear) images—to process, understand, or generate omnidirectional panoramic data, notably 360° × 180° imagery in spherical or equirectangular projection. This adaptation is crucial for transfer learning, data efficiency, and cross-domain generalization in panoramic vision, including 360° generation, recognition, reconstruction, and spatial reasoning tasks.

1. Geometric Foundations and Domain Gap

The foundational obstacle in perspective-to-panorama adaptation is the geometric and statistical domain gap between perspective and panoramic imagery. Perspective images are typically modeled by a pinhole camera model with a limited FoV; panoramic images are global environment captures mapped through projections such as equirectangular (ERP) or cubemap (CMP). The equirectangular mapping from 3D to image coordinates is:

$$u = \frac{\phi + \pi}{2\pi} W, \quad v = \frac{\theta}{\pi} H$$

where $(u,v)$ are image pixels, $(\theta,\phi)$ are longitude and latitude, and $W=2H$.

Three main factors comprise the domain gap (Lin et al., 4 Sep 2025):

• Severe geometric distortions: Notably near the poles in ERP, where horizontal stretching grows as $1/\cos\lambda$.
• Non-uniform sampling: Pixel area varies with latitude; dense at equator, sparse near poles.
• Boundary continuity (periodicity): Horizontal seams wrap around, violating planar convolutional priors.

This gap impedes direct application of perspective-trained convolutional and transformer models, causing misalignment of structure, feature statistics, and spatial priors (Ni et al., 28 May 2025, Lin et al., 4 Sep 2025).

2. Theoretical Analyses and Adaptation Degrees of Freedom

The mapping between perspective and panoramic domains can be characterized by the camera calibration pipeline, involving a pinhole-to-sphere transformation (Dong et al., 14 Sep 2025):

• Intrinsic parameters ($f_x, f_y, c_x, c_y, s$): 5 DoF control camera calibration.
• Extrinsic parameters ($R\in SO(3), t\in\mathbb{R}^3$): 6 DoF determine camera pose.
• Projection alignment: For panoramic mapping, pitch, roll, and vertical shift may be held fixed, reducing the effective DoF. In practice, PanoLora finds an 8 DoF mapping sufficient to span the transformation required for perspective-to-panorama adaptation (Dong et al., 14 Sep 2025).

LoRA adaptation rank (minimal subspace sufficient for adaptation) is set to at least 8 to guarantee coverage of these DoF.

3. Architectural Strategies for Adaptation

Adaptation approaches stratify into three core classes: projection-driven, architectural, and statistical/feature-level.

3.1 Projection-Driven Adaptation

• Cubemap decomposition: CubeDiff and PaGeR project panoramas to six 90° perspective faces, enabling reuse of perspective-trained models with minimal architectural change. CubeDiff treats all faces as interchangeable perspective views and modifies the attention layer to attend jointly across faces, while preserving wrap-around continuity (Kalischek et al., 28 Jan 2025, Bozic et al., 25 May 2026).
• View stitching and trajectory scanning: SAP decomposes 4K ERP panoramas into overlapping perspective view sequences sampled along a fixed trajectory, aligning with memory-augmented video models and enabling stable instance segmentation at panoramic scale (Jiang et al., 13 Mar 2026).
• Tangent or fixed-FoV projections: Segmentation adaptation pipelines introduce tangent-plane views (local perspective slices) or fixed FoV ERP strips, facilitating knowledge transfer and prototype adaptation from perspective sources (Zheng et al., 2024).

3.2 Architectural Modifications

• Attention Inflations: CubeDiff inflates all 2D attention layers to operate over the union of six cubemap faces, supporting seamless content transfer and semantic consistency across faces (Kalischek et al., 28 Jan 2025).
• Projection-aware Cross-Attention: PanFusion introduces Equirectangular–Perspective Projection Attention (EPPA) modules that bias cross-branch attention to geometrically corresponding ERP/perspective regions using shared spherical positional encodings and projection masks, minimizing destructive resampling artifacts (Zhang et al., 2024).
• Positional Encoding Corrections: SHERPA and PanoSplatt3R both address seam discontinuity via modifications to rotary positional embeddings (RoPE); SHERPA restricts the correction to the high-frequency sub-band, enforcing horizontal periodicity (Kang et al., 10 Jun 2026), while PanoSplatt3R applies "RoPE rolling" where each attention head's horizontal coordinate is cyclically shifted to guarantee periodic adjacency (Ren et al., 29 Jul 2025).
• Residual Adapters and Self-Modulation: RePer-360 modulates intermediate activations in a frozen transformer backbone via self-conditioned AdaLN-Zero, adapting only pixel-wise LayerNorm parameters derived from geometry-aligned features, thereby preserving pretrained perspective priors while guiding toward panoramic distributions (Guan et al., 6 Mar 2026).
• Seamless Decoding/Fusion: Circular VAE encoding/decoding in SHERPA, circular padding in PanFusion, and cross-face valid padding in PaGeR eliminate edge artifacts inherent to standard planar architectures (Kang et al., 10 Jun 2026, Zhang et al., 2024, Bozic et al., 25 May 2026).

4. Model Training, Conditioning, and Loss Functions

Adaptation is typically achieved by a combination of:

• LoRA/Adapter Injection: Lightweight, low-rank adapters are inserted into selected UNet layers (typically query, key, value, output projections), with most parameters frozen to ensure data-efficient finetuning (Ni et al., 28 May 2025, Dong et al., 14 Sep 2025, Ye et al., 2024, Zhang et al., 2024).
• Classifier-free guidance and conditional conditioning: CubeDiff and PanFusion support fine-grained prompt control, per-face text embedding, and random dropout of text/image in cross-attention for regularization (Kalischek et al., 28 Jan 2025, Zhang et al., 2024).
• Circular or projection-aware augmentation: Wrap-shifting or rolling augmentation forces horizontal periodicity via vertical-strip randomization or cyclic permutation, enforcing seam closure (Ye et al., 2024, Zhang et al., 2024, Kang et al., 10 Jun 2026).
• Multi-task and projection-aligned losses: Several frameworks enforce multi-modal alignment, such as cube-projection losses, CLIP-based semantic similarity, or geometric consistency (e.g. spherical epipolar constraint in DiffPano) (Ye et al., 2024, Kang et al., 10 Jun 2026, Ren et al., 29 Jul 2025).
• Layout and depth conditioning: Layout-conditional generation (PanFusion), depth regularization (PaGeR, RePer-360), and 3D constraints ensure geometric plausibility and enable spatially aware synthesis (Zhang et al., 2024, Bozic et al., 25 May 2026, Guan et al., 6 Mar 2026).

5. Empirical Results and Benchmarks

Quantitative evaluation consistently shows that adapted models outperform both naive perspective applications and earlier “stitched” panoramic approaches, across diverse tasks:

Method	FID (Panorama)	8-view FID	mIoU (Segm)	AbsRel (Depth)	Key Properties
UniPano	46.47	17.74	--	--	LoRA/MoE, cross-attention, single-branch
PanFusion	46.47	19.88	--	--	Dual-branch, EPPA, layout-conditioning
PanoLora	--	--	--	--	LoRA rank-16, state-of-the-art video
P2PDA (segm)	--	--	44.66%	--	UDA, adversarial+attention adaptation
RePer-360	--	--	--	0.0691	Self-modulation, ERP+CP guidance (Stanford2D3D)
PaGeR	--	--	--	9.36 (AbsRel)	Cross-face padding, cubemap, 4-task output
SHERPA	27.80	--	--	--	Circular RoPE, dual supervision
SAP	--	--	75.8 (1-clk)	--	Trajectory-aligned segment-anything

• CubeDiff demonstrates superior FID and qualitative seam-free synthesis versus prior MVDiffusion and Diffusion360 (Kalischek et al., 28 Jan 2025).
• PanFusion and UniPano achieve best FAED and FID metrics for text-to-panorama, with UniPano providing efficient memory and training time usage (Ni et al., 28 May 2025, Zhang et al., 2024).
• PanoLora achieves 0.99 left–right (L–R) seam consistency and subjectively superior spatial-temporal coherence relative to all baselines, with LoRA rank ≥ 8 a necessary condition (Dong et al., 14 Sep 2025).
• In segmentation, P2PDA improves cross-domain mIoU by over 16 points versus source-only; prototype and attention adaptation outperform earlier UDA baselines (Zhang et al., 2021, Zheng et al., 2024).
• PaGeR achieves state-of-the-art panoramic depth, normal, and sky mask estimation, using a perspective-pretrained transformer with minimal cubemap adaptation (Bozic et al., 25 May 2026).
• SHERPA demonstrates lowest seam and polar FID across photorealistic and stylized prompt domains using minimal frequency-selective RoPE surgery and circular VAE wrap (Kang et al., 10 Jun 2026).
• SAP achieves +17.2 mIoU gain over the SAM2 baseline for 4K panoramic instance segmentation with a fixed-trajectory perspective video approach (Jiang et al., 13 Mar 2026).

6. Task Coverage and Application Domains

Perspective-to-panorama adaptation underpins advances across:

• Generative modeling: 360° text/image-conditioned synthesis, photorealistic and stylized panorama generation (Kalischek et al., 28 Jan 2025, Zhang et al., 2024, Ni et al., 28 May 2025, Ye et al., 2024, Kang et al., 10 Jun 2026).
• Video generation: 360° video synthesis leveraging pretrained perspective priors and dynamic spatial continuity (Dong et al., 14 Sep 2025, Fang et al., 30 Jun 2025).
• Depth and geometry estimation: Direct adaptation of foundation models for depth, normals, and sky estimation using cubemap or ERP representations (Bozic et al., 25 May 2026, Guan et al., 6 Mar 2026, Peng et al., 2022).
• Semantic segmentation and detection: Source-free and domain-adaptive pipelines using prototype and attention mechanisms, enabling robust transfer in segmentation (Zheng et al., 2024, Zhang et al., 2021).
• Multimodal and spatial reasoning: Instruction-fine-tuned large multimodal models (MLLMs) for ERP-native spatial, semantic, and 3D reasoning in navigation and VQA contexts (Wang et al., 13 May 2026).

7. Open Challenges and Ongoing Directions

Despite significant progress, perspective-to-panorama adaptation faces ongoing limitations:

• Data scarcity and distribution shift: Panoramic datasets are orders of magnitude smaller than perspective corpora, leading to overfitting and insufficient diversity (Lin et al., 4 Sep 2025).
• Polar distortion and under-sampling: Even with circular or cubemap adaptation, distortion at the poles and uneven surface-area sampling remain significant obstacles, impacting both generative fidelity and recognition accuracy (Lin et al., 4 Sep 2025, Kang et al., 10 Jun 2026).
• Boundary seam artifacts: Ensuring seamless wrap-around, especially for discriminative tasks and non-Cartesian outputs (e.g., instance masks), still requires targeted architectural modifications (Kang et al., 10 Jun 2026, Ren et al., 29 Jul 2025).
• Unified spherical backbones: There is a lack of large-scale pretraining and foundation models explicitly spherical or rotation-equivariant, limiting generalization (Lin et al., 4 Sep 2025, Bozic et al., 25 May 2026).
• Multi-modal panoramic tasks: Integration of depth, audio, language, and other modalities in panoramic settings for world modeling and embodied agents is just beginning (Wang et al., 13 May 2026, Kang et al., 10 Jun 2026).

Promising future directions include rotation-equivariant networks, differentiable projection layers, dynamic positional encodings (e.g., slope-adaptive RoPE), and large-scale spherical pretraining ("Spheroid-BERT") for omnidirectional visual intelligence (Lin et al., 4 Sep 2025).

---

References:

• "CubeDiff: Repurposing Diffusion-Based Image Models for Panorama Generation" (Kalischek et al., 28 Jan 2025)
• "PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptation" (Dong et al., 14 Sep 2025)
• "Taming Stable Diffusion for Text to 360° Panorama Image Generation" (Zhang et al., 2024)
• "What Makes for Text to 360-degree Panorama Generation with Stable Diffusion?" (Ni et al., 28 May 2025)
• "PanoSplatt3R: Leveraging Perspective Pretraining for Generalized Unposed Wide-Baseline Panorama Reconstruction" (Ren et al., 29 Jul 2025)
• "Semantics, Distortion, and Style Matter: Towards Source-free UDA for Panoramic Segmentation" (Zheng et al., 2024)
• "Unified Panoramic Geometry Estimation via Multi-View Foundation Models" (Bozic et al., 25 May 2026)
• "SHERPA: Seam-aware Harmonized ERP Adaptation for Open-Domain 360° Panorama Generation" (Kang et al., 10 Jun 2026)
• "SAP: Segment Any 4K Panorama" (Jiang et al., 13 Mar 2026)
• "PanoWorld: Towards Spatial Supersensing in 360° Panorama World" (Wang et al., 13 May 2026)
• "One Flight Over the Gap: A Survey from Perspective to Panoramic Vision" (Lin et al., 4 Sep 2025)
• "Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation" (Zhang et al., 2021)
• "High-Resolution Depth Estimation for 360-degree Panoramas through Perspective and Panoramic Depth Images Registration" (Peng et al., 2022)
• "DiffPano: Scalable and Consistent Text to Panorama Generation with Spherical Epipolar-Aware Diffusion" (Ye et al., 2024)