PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting (2412.12096v2)

Abstract: With the advent of portable 360{\deg} cameras, panorama has gained significant attention in applications like virtual reality (VR), virtual tours, robotics, and autonomous driving. As a result, wide-baseline panorama view synthesis has emerged as a vital task, where high resolution, fast inference, and memory efficiency are essential. Nevertheless, existing methods are typically constrained to lower resolutions (512 $\times$ 1024) due to demanding memory and computational requirements. In this paper, we present PanSplat, a generalizable, feed-forward approach that efficiently supports resolution up to 4K (2048 $\times$ 4096). Our approach features a tailored spherical 3D Gaussian pyramid with a Fibonacci lattice arrangement, enhancing image quality while reducing information redundancy. To accommodate the demands of high resolution, we propose a pipeline that integrates a hierarchical spherical cost volume and Gaussian heads with local operations, enabling two-step deferred backpropagation for memory-efficient training on a single A100 GPU. Experiments demonstrate that PanSplat achieves state-of-the-art results with superior efficiency and image quality across both synthetic and real-world datasets. Code is available at https://github.com/chengzhag/PanSplat.

• The paper introduces a feed-forward Gaussian splatting model that synthesizes novel 4K panoramic views from wide-baseline 360° images.
• It leverages a spherical 3D Gaussian pyramid and Fibonacci lattice to optimize memory and computational efficiency while maintaining high image quality.
• Experiments demonstrate up to 70x faster rendering and state-of-the-art WS-PSNR, SSIM, and LPIPS scores, advancing VR and autonomous applications.

Overview of PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting

The paper "PanSplat: 4K Panorama Synthesis with Feed-Forward Gaussian Splatting" presents a novel framework for synthesizing novel panoramic views from wide-baseline 360° photographs. A significant advancement in this work is the ability to operate efficiently at 4K resolution, addressing ongoing challenges in computational efficiency and memory requirements traditionally encountered in wide-baseline panorama view synthesis.

Methodological Innovations

PanSplat proposes a feed-forward model that leverages a spherical 3D Gaussian pyramid combined with a Fibonacci lattice arrangement for more efficient panoramic image processing. This innovation tackles the inherent redundancy and overlapping of Gaussians near the poles due to spherical geometry. The proposed Fibonacci lattice dramatically reduces Gaussian redundancy without sacrificing image quality, thereby optimizing memory and computational resources.

To address the challenge of maintaining image quality at high resolutions, the authors introduce a hierarchical spherical cost volume and Gaussian heads. These components allow for an efficient depth estimation and prediction of Gaussian parameters, respectively. A key technical contribution is the two-step deferred backpropagation technique that reduces the memory footprint during training, enabling high-resolution processing on a standard A100 GPU.

Numerical Performance

The efficacy of PanSplat is demonstrated through experiments on several datasets, showing state-of-the-art performance concerning image quality and computational efficiency. For instance, on synthetic datasets such as Matterport3D and Replica, PanSplat not only achieved superior WS-PSNR, SSIM, and LPIPS scores but also demonstrated improved generalization to unseen datasets. Notably, the inference speed exhibits a substantial improvement, achieving up to 70 times faster rendering compared to prior methods like PanoGRF.

Impact and Future Directions

The presented work has significant implications for applications in virtual reality (VR), robotics, and autonomous driving, where 360° panoramic content is increasingly valuable. By lifting the constraints associated with high-resolution processing, PanSplat fosters a more immersive and seamless visual experience, especially pertinent for VR applications. Furthermore, the methodology lays a solid foundation for extending synthesis techniques to environments with dynamic elements, although PanSplat currently assumes static scenes.

In summary, PanSplat represents a meaningful step towards efficient and high-quality novel view synthesis in panoramic formats. The introduced framework enables scalable, high-resolution panoramic synthesis, marking an essential advancement for capturing real-world scenes in virtual environments. As future research explores dynamic scene elements and further optimizations, PanSplat's approach is poised to impact the development of more robust and flexible panoramic synthesis systems, enhancing real-time VR experiences and autonomous perception tasks.