PanoImager: Geometry-Guided Novel View Synthesis and Reconstruction from Sparse Panoramic Views

Abstract: Panoramic sensing offers wide field-of-view coverage, yet 3D reconstruction from sparse panoramas remains challenging under rotation-dominant, weak-parallax motion. In such regimes, SfM/SLAM initialization is often ill-conditioned and unreliable. We present PanoImager, an SfM-free framework that combines feed-forward pose/depth priors, geometry-conditioned diffusion view completion, and depth-guided 3DGS optimization. Given only a few panoramic images, PanoImager decomposes them into local perspective views, synthesizes auxiliary observations to enrich sparse evidence, and stabilizes Gaussian optimization for improved cross-view consistency. Experiments on multiple benchmarks show improved stability under extreme sparsity, suggesting PanoImager as an offline/background component for map refinement when SfM/SLAM fails to initialize.

• The paper presents a novel pipeline for 3D reconstruction from sparse panoramic views by eliminating traditional correspondence-based initialization.
• It integrates feed-forward geometric priors, tangent-space decomposition, and diffusion-based view synthesis to enhance reconstruction coherence.
• Experimental results demonstrate improved PSNR, SSIM, and LPIPS scores, proving its robustness over conventional methods.

Geometry-Guided Novel View Synthesis and Sparse Panoramic 3D Reconstruction via PanoImager

Introduction

PanoImager addresses the persistent failure modes in 3D scene reconstruction from sparse panoramic image sets, particularly those dominated by rotational, low-parallax camera motion where conventional Structure-from-Motion (SfM) or Simultaneous Localization and Mapping (SLAM) initialization becomes degenerate. Whereas panoramic sensing increases field-of-view and scene coverage, its benefits are offset by pronounced projection distortion and lack of geometric diversity. Standard scene representations—such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS)—exhibit sensitivity to panoramic distortion and initialization, and their optimization via multi-view constraints collapses under weak-parallax inputs.

PanoImager circumvents these limitations by eschewing correspondence-based geometric initialization. The framework composes a multi-stage pipeline integrating feed-forward geometric priors, tangent-space view parameterizations, reliability-aware diffusion-based view completion, and depth-constrained 3DGS optimization. The resulting approach produces robust, spatially coherent 3D reconstructions in scenarios where baseline methods fail.

Figure 1: End-to-end pipeline: sparse panoramas are decomposed into local perspectives, geometry is initialized, new views synthesized with diffusion, and all are fused into a 3D Gaussian map.

Panoramic Geometry Initialization and Tangent-Space Decomposition

Operating directly on equirectangular projection (ERP) introduces ill-conditioning due to highly anisotropic pixel-to-angle mapping, with severe impact in high-latitude regions. PanoImager mitigates this by decomposing each panorama over uniformly sampled spherical directions into a structured set of overlapping perspective pinhole views (typically $N_v=18$), forming a well-conditioned intermediate set for geometric processing. This tangent-space parameterization aligns with the assumptions of downstream pose/depth predictors and 3DGS.

Figure 2: Example of panorama decomposition into overlapping local perspective views as tangent-space charts.

Geometric priors are estimated using a feed-forward visual foundation model, specifically Visual Geometry Grounded Transformer (VGGT), that predicts both relative camera poses and depth fields across views. This step establishes an initial geometric scaffold without relying on multi-view triangulation or feature correspondence.

Geometry-Conditioned Diffusion View Completion

Sparse panoramic coverage yields large unobserved scene regions, undermining subsequent multi-view optimization frameworks. To resolve this, PanoImager densifies the observation set by synthesizing plausible RGB and depth views, leveraging a geometry-guided diffusion model (with MVGenMaster backbone). The model is conditioned on local perspective reference images, their predicted depths, warped priors, and visibility masks, ensuring plausible synthesis that adheres to physically valid geometric anchors.

A smooth spline trajectory bridges the spatial gap between observed poses; $N=16$ additional target camera configurations are sampled uniformly along this spline, maximizing tangent-space coverage. Each target is supported by multi-reference cues, and occlusion boundaries are handled with depth discontinuity thresholds in the reprojected priors.

Figure 3: Illustration of geometry-guided sampling and interpolation of virtual camera trajectories for view synthesis.

Synthesized local completions are fused back onto the spherical panoramic domain through Gaussian-weighted blending, promoting visual coherence and reducing seam artifacts.

Figure 4: Workflow showing reference-driven warping, mask construction, geometry-conditioned diffusion, and spherical-level fusion.

Reliability-aware condition dropout is implemented during conditioning to degrade gracefully in regions with poor geometric support, modulating the effect of synthesized priors on the reconstruction.

Depth-Constrained 3D Gaussian Splatting with Reliability Priors

The optimization phase integrates both the real and synthesized observations to fit a 3D Gaussian scene representation, subject to several regularization terms:

• The photometric term matches rendered appearance to input and auxiliary views, with soft weights $\omega_t$ on synthesized views based on reprojection and depth agreement errors.
• The depth term explicitly anchors reconstructed depth to the predicted/warped priors, again reliability-weighted.
• The multi-view geometric consistency term enforces cross-view depth alignment via differentiable reprojection, filtering by a Huber loss and weighting by pointwise residuals.
• The anti-floater regularization penalizes high-opacity “floater” Gaussians at depths contradicted by geometric measurements, suppressing artifacts prevalent in sparse settings.

The overall loss combines these components, with hyperparameters held fixed across all scene types to substantiate generalizability.

Experimental Evaluation

PanoImager is validated on three benchmarks: Replica (ground truth geometry), OmniScenes, and 360Roam (real-world indoor scanning). Comparisons encompass direct 3DGS (OmniSplat, ODGS, SPaGS) and iterative feed-forward Splatting variants.

In the dense (visibility) regime, conventional methods degrade under sparse panoramic input, exhibiting blur, scene fragmentation, and geometric hallucination. PanoImager demonstrates superior spatial alignment and cross-view consistency, as shown both qualitatively and through direct metrics (e.g., PSNR/SSIM/LPIPS):

Figure 5: Qualitative comparison: PanoImager vs. recent baselines on real and synthetic sparse panorama datasets.

For novel view synthesis and geometry consistency, PanoImager exhibits improved radiance fidelity and robust geometric reconstruction compared with geometry-blind or naive diffusion-only approaches. Notably, synthesized views contribute essential regularization for 3DGS, confirmed by ablation studies and performance in non-visible (out-of-coverage) view scenarios.

Figure 6: Novel view synthesis: diffusion-based completions vs. ground truth under extreme sparse-view settings.

Numerical results confirm these qualitative outcomes. For example, in 360Roam, PanoImager attains a PSNR of 25.27, SSIM of 0.813, and LPIPS of 0.213—substantially outperforming both perspective and panoramic 3DGS variants under identical conditions. On challenging Replica scenes with less than 35% warp coverage, PanoImager maintains stable performance and achieves a multi-view reprojection error of 0.32m and IoU of 0.76, indicating high geometric consistency even in extrapolated regions.

Implications and Future Directions

Practically, PanoImager enables reliable offline/background map refinement in robotic applications where motion constraints or scene structure preclude parallax-rich capture, and where standard SLAM/SfM are expected to fail. Theoretically, the pipeline underscores the value of integrating generative geometric priors and reliability-aware supervision for robust 3D scene representation in extreme observation sparsity regimes.

Three limitations warrant emphasis: (i) Only visually plausible, not metrically guaranteed, completions are possible under substantial coverage gaps; (ii) The tangent-space partitioning may introduce subtle stitching artifacts; (iii) The computational footprint currently limits deployment to asynchronous or non-realtime contexts. Future progress hinges on incorporating explicit uncertainty quantification for mission-critical tasks and advancing model distillation for scalable, real-time onboard operation.

Conclusion

PanoImager advances SfM-free 3D scene reconstruction in low-parallax, sparse panoramic capture settings via a principled integration of feed-forward geometric priors, geometry-guided diffusion observation augmentation, and reliability-constrained 3D Gaussian optimization. Its robust performance under degenerate geometric regimes supports both theoretical and applied advances in scene representation, particularly for safety-critical or resource-constrained robotic mapping.