Coca-Splat: Collaborative Optimization for Camera Parameters and 3D Gaussians (2504.00639v1)

Abstract: In this work, we introduce Coca-Splat, a novel approach to addressing the challenges of sparse view pose-free scene reconstruction and novel view synthesis (NVS) by jointly optimizing camera parameters with 3D Gaussians. Inspired by deformable DEtection TRansformer, we design separate queries for 3D Gaussians and camera parameters and update them layer by layer through deformable Transformer layers, enabling joint optimization in a single network. This design demonstrates better performance because to accurately render views that closely approximate ground-truth images relies on precise estimation of both 3D Gaussians and camera parameters. In such a design, the centers of 3D Gaussians are projected onto each view by camera parameters to get projected points, which are regarded as 2D reference points in deformable cross-attention. With camera-aware multi-view deformable cross-attention (CaMDFA), 3D Gaussians and camera parameters are intrinsically connected by sharing the 2D reference points. Additionally, 2D reference point determined rays (RayRef) defined from camera centers to the reference points assist in modeling relationship between 3D Gaussians and camera parameters through RQ-decomposition on an overdetermined system of equations derived from the rays, enhancing the relationship between 3D Gaussians and camera parameters. Extensive evaluation shows that our approach outperforms previous methods, both pose-required and pose-free, on RealEstate10K and ACID within the same pose-free setting.

Collaborative Optimization for Camera Parameters and 3D Gaussians

The paper under discussion introduces Coca-Splat, a method that integrates 3D Gaussian scene reconstruction and novel view synthesis (NVS) without relying on pre-defined poses. This framework addresses the limitations in pose-free scene reconstruction by aligning optimization of both 3D Gaussians and camera parameters within a single network structure.

The traditional approach in novel view synthesis often bifurcates the tasks of 3D scene reconstruction and pose estimation, frequently leading to inaccuracies and inefficiencies. Previous schisms in these tasks have resulted in compounding errors, especially in pose estimation, when only a limited number of views are available. Coca-Splat circumvents this issue by utilizing a deformable Transformer-based architecture to jointly optimize these parameters.

Coca-Splat's architecture positions itself distinctively from existing methodologies by implementing differentiated queries for 3D Gaussians and camera parameters. This is pursued through a carefully structured pipeline that employs Deformable DETR, leveraging its capacity for both precision and speed. The design ensures that 3D Gaussians center projections onto each view are utilized as 2D reference points, effectively connecting camera parameters through shared multi-view deformable cross-attention (CaMDFA). The approach further enhances the relationship between 3D Gaussians and camera parameters through defined rays (RefRay), extending from camera centers to these points and enabling joint optimization in the network.

Performance evaluation is conducted rigorously on RealEstate10K and ACID datasets where Coca-Splat exhibits superior results compared to prior works, including both pose-required and pose-free methods. The numerical evidence highlights its efficacy in conditions with varying overlap among input views, with substantial improvements in both qualitative and quantitative metrics such as PSNR, SSIM, and LPIPS. Additionally, the novel network design allows Coca-Splat to refine 3D Gaussian and camera pose estimation at a computation time faster than many existing techniques.

In terms of future implications, Coca-Splat offers a significant step towards efficient and autonomous novel view synthesis. By simplifying the pipeline to eschew pre-processing and post-processing steps while handling complete intrinsic and extrinsic optimizations internally, Coca-Splat emphasizes model adaptability in more varied and real-world scenarios than many conventional methods.

The research fosters advancements in augmented reality (AR), virtual reality (VR), and robotics, where precise scene reconstructions are pivotal. However, while showing promise, Coca-Splat's reliance on training datasets and limited handling of full 360-degree scenes presents an avenue for further exploration and enhancement within this domain.

Concluding, Coca-Splat emerges as a proficient methodology aligning joint camera parameter estimation with 3D Gaussian rendering, contributing valuable insights and practical tools to the evolving landscape of computer vision. By refining the intersection of geometry and photometric streams into a singular optimization process, Coca-Splat heralds progress that could significantly influence future developments in the field.