Depth Any Panoramas: A Foundation Model for Panoramic Depth Estimation (2512.16913v1)

Abstract: In this work, we present a panoramic metric depth foundation model that generalizes across diverse scene distances. We explore a data-in-the-loop paradigm from the view of both data construction and framework design. We collect a large-scale dataset by combining public datasets, high-quality synthetic data from our UE5 simulator and text-to-image models, and real panoramic images from the web. To reduce domain gaps between indoor/outdoor and synthetic/real data, we introduce a three-stage pseudo-label curation pipeline to generate reliable ground truth for unlabeled images. For the model, we adopt DINOv3-Large as the backbone for its strong pre-trained generalization, and introduce a plug-and-play range mask head, sharpness-centric optimization, and geometry-centric optimization to improve robustness to varying distances and enforce geometric consistency across views. Experiments on multiple benchmarks (e.g., Stanford2D3D, Matterport3D, and Deep360) demonstrate strong performance and zero-shot generalization, with particularly robust and stable metric predictions in diverse real-world scenes. The project page can be found at: \href{https://insta360-research-team.github.io/DAP_website/} {https://insta360-research-team.github.io/DAP\_website/}

• The paper introduces DAP, a novel foundation model that leverages a three-stage semi-supervised pipeline and diverse 2M-scale data to overcome limitations in panoramic depth estimation.
• The model architecture integrates a DINOv3-L backbone with specialized range mask and metric depth heads, optimized with distortion-aware loss functions for accurate metric predictions.
• Experimental results reveal significant improvements in absolute relative error and stability across varied indoor and outdoor panoramas, demonstrating strong zero-shot generalization.

Depth Any Panoramas: A Foundation Model for Panoramic Depth Estimation

Introduction

"Depth Any Panoramas" introduces DAP, an omnidirectional depth prediction foundation model designed to deliver metric depth estimates—both indoors and outdoors—with robust zero-shot generalization. The work addresses two primary challenges in panoramic depth estimation: (1) the scarcity and limited diversity of training data, and (2) the need for architecture and optimization strategies tailored to the distortions and scale diversity inherent to 360$\degree$ environments. DAP leverages a meticulously curated 2M sample panorama dataset and a progressive three-stage semi-supervised pipeline for domain-bridged pseudo-label generation, combined with a DINOv3-Large-based backbone and tailored heads and loss functions to ensure metric fidelity and geometric consistency.

Data Strategy and Three-Stage Semi-Supervised Pipeline

DAp's data engine unites high-quality labeled synthetic indoor data (Structured3D), photorealistic synthetic outdoor panoramas (90K from AirSim360), 1.9M real panoramic images curated from internet video frames, and Diffusion-based DiT360-generated indoor panoramas. This massive, heterogenous data pool ensures exposure across a vast range of indoor/outdoor and synthetic/real-world scene configurations.

DAP employs a sequential pseudo-label curation protocol:

1. Scene-Invariant Labeler: Trained exclusively on synthetic data to learn geometric and photometric invariances, providing robust pseudo-labels for the real/unlabeled panorama pool.
2. Realism-Invariant Labeler: A PatchGAN-based depth quality discriminator prioritizes high-confidence pseudo-labels (selecting 300K each for indoor and outdoor), refining the labeling model by pre-training with these examples to further mitigate synthetic-real domain gaps.
3. DAP Training: The full model trains with the union of all labeled and high-quality pseudo-labeled data, maximizing exposure and minimizing cross-domain discrepancies.

   Figure 1: Overview of the progressive three-stage pipeline, highlighting data curation and labeling strategy for robust panoramic depth modeling.

This pipeline enables efficient exploitation of the vast but noisy internet-scale data, effectively scaling supervision quality without requiring costly manual annotation, while systematically bridging scene and domain variation.

Model Architecture and Loss Design

DAP incorporates DINOv3-Large for feature extraction, after which a dual-headed architecture is employed:

• Range Mask Head: Outputs binary spatial validity masks for specified maximum distances (10m, 20m, 50m, 100m), enabling training and inference to be distance-aware and mitigating the impact of uncertain, far-field predictions. This also facilitates plug-and-play spatial scaling.
• Metric Depth Head: Outputs dense, metrically consistent depth maps.

The model is optimized using a suite of custom loss terms:

• $\mathcal{L}_{\text{SILog}}$: Scale-invariant logarithmic error for depth regression.
• $\mathcal{L}_{\text{DF}}$: Dense fidelity loss leveraging perspective patches from icosahedral projections, addressing equirectangular distortions while preserving fine structure.
• $\mathcal{L}_{\text{grad}}$: Edge-focused gradient loss improving object boundary sharpness.
• $$\mathcal{L}_{\text{normal}}, \mathcal{L}_{\text{pts}}$$: Geometry-oriented losses for normal and point cloud-level consistency, ensuring local planarity and global structure.
• Distortion Map Compensation: All losses are masked with a spherical distortion map to counteract non-uniform pixel distributions inherent to ERP.

  Figure 2: Architecture of the DAP network: DINOv3-Large backbone, distortion-aware decoder, range mask head, and geometry/sharpness-centric objectives.

Experimental Evaluation

Zero-Shot and In-Domain Performance

DAP is evaluated across standard benchmarks—Stanford2D3D and Matterport3D (indoor), Deep360 and a newly proposed DAP-Test (outdoor). It robustly outperforms all prior metric and scale-invariant baselines in zero-shot scenarios without any test set fine-tuning:

• Stanford2D3D: AbsRel $=0.0921$, $\delta_1 = 0.9135$
• Deep360: AbsRel $=0.0659$, RMSE $=5.224$, $\delta_1=0.9525$
• DAP-Test: AbsRel $=0.0781$, RMSE $=6.804$, $\delta_1=0.9370$

Compared to DAC [DepthAnyCamera] and Unik3D [piccinelli2025unik3d], DAP exhibits significant reduction in absolute relative error and RMSE, and a clear gain in the accuracy threshold $\delta_1$—demonstrating its capacity for scale-consistent predictions across domains and environments.

Qualitative Comparison

Qualitative results indicate DAP consistently delivers sharper object boundaries, less depth collapse in distant and sky regions, and increased stability in both indoor and outdoor real-world panoramas. The model also preserves fine structural features and demonstrates marked scale awareness, outperforming competitive baselines on structural fidelity and global consistency.

Figure 3: DAP generates sharper, more consistent depth across real-world indoor and outdoor scenes versus DAC and Unik3D.

Figure 4: DAP maintains fine-scale structural delineation and accurate scale in Stanford2D3D indoor panoramas.

Ablation and Range Mask Study

Ablations confirm that the distortion map, additional geometric and sharpness-centric losses, and the plug-and-play range mask head each contribute consistently to reducing AbsRel and improving $\delta_1$. Notably, the range mask head (thresholded at 100m) provides a flexible solution for stabilizing far-field predictions that typically challenge panoramic depth models.

Theoretical and Practical Implications

DAP demonstrates that foundation models for 360$\degree$ perception require a co-optimized blend of data scaling, cross-domain pseudo-labeling, and ERP-specific architectural design. Its paradigm exemplifies how synthetic data, diffusion-generated imagery, and internet curation can be systematically leveraged for deep semi-supervised learning, while addressing panoptic challenges posed by ERP distortions and the need for metric consistency.

Practically, DAP enables downstream 3D applications—robotic mapping, AR/VR, autonomous navigation, and large-scale virtual world generation—by providing accurate, metric-aware geometry across diverse real-world scenes. The architectural strategy, especially the modular range mask, facilitates easy adaptation to new operational ranges and deployment scenarios.

Theoretically, DAP provides a scalable blueprint for other vision foundation models to systematically bridge generative, synthetic, and real data sources, and demonstrates the effectiveness of synergy between modern ViTs (DINOv3-L) and pseudo-label bootstrapping for geometric prediction tasks.

Conclusion

Depth Any Panoramas establishes a new benchmark in panoramic metric depth estimation via a 2M-scale unified data corpus, a scalable three-stage pseudo-label pipeline, and a distortion-robust, range-adaptive architecture. Numerical and qualitative results confirm strong zero-shot generalization, accurate metric consistency, and structural fidelity across domains. DAP sets a precedent for panoramic perception foundation models and raises the standard for future geometry-centric 360$\degree$ vision research.