Extending 3D Gaussian Splatting gradients beyond pinhole cameras

Develop an efficient gradient formulation for 3D Gaussian Splatting that supports non-pinhole camera models, including fisheye and other complex lens types, to enable training and optimization from arbitrary camera lens geometries.

Background

The current 3D Gaussian Splatting (3DGS) optimization assumes a pinhole camera model, which limits applicability to datasets captured with more complex lens systems such as fisheye. In contrast, Neural Radiance Fields can flexibly cast rays for arbitrary lens models, allowing training with diverse camera types.

An efficient extension of the 3DGS gradient formulation to non-pinhole lens models is not yet established, preventing native training of 3DGS from fisheye or other complex cameras without intermediary supervision.