Mathematical Supplement for the $\texttt{gsplat}$ Library (2312.02121v1)

Abstract: This report provides the mathematical details of the gsplat library, a modular toolbox for efficient differentiable Gaussian splatting, as proposed by Kerbl et al. It provides a self-contained reference for the computations involved in the forward and backward passes of differentiable Gaussian splatting. To facilitate practical usage and development, we provide a user friendly Python API that exposes each component of the forward and backward passes in rasterization at github.com/nerfstudio-project/gsplat .

• The paper presents a detailed mathematical framework for projecting and rasterizing 3D Gaussians onto 2D images.
• It derives comprehensive gradient computations, including those for color, opacity, and affine transformations during back-propagation.
• The methodology enhances differentiable rendering, offering a modular approach for optimizing and advancing visual computing techniques.

Mathematical Supplement for the gsplat Library: An Analytical Overview

The paper under review provides a detailed mathematical exposition supporting the gsplat library, which offers a modular toolbox for efficient differentiable Gaussian splatting as initially proposed by Kerbl et al. This comprehensive document targets researchers who require an in-depth understanding of the mathematical underpinnings necessary for performing the forward and backward passes of differentiable Gaussian splatting. This discussion will traverse the key sections of the original paper, encapsulating its core contributions and mathematical rigor.

Forward Pass of Gaussian Splatting

The forward pass in Gaussian splatting is heavily reliant on projecting and rasterizing 3D Gaussians into a 2D output image. A 3D Gaussian is parameterized by its mean, covariance, color, and opacity. To provide context, the mean $\mu$ is a 3D vector, the covariance $\Sigma$ is a $3 \times 3$ matrix, the color $c$ is a 3D vector, and the opacity $o$ is a scalar value.

The initial stage involves projecting these 3D Gaussians onto the 2D camera plane. This requires transforming the Gaussian parameters from the camera space to the normalized clip space using a projection matrix $P$ and the extrinsics matrix $T_{\textrm{cw}}$. The mean $\mu$ undergoes a series of transformations to convert it into the pixel space coordinates $\mu'$.

However, the projection of a 3D Gaussian does not yield a 2D Gaussian directly. The authors approximate this projection with a first-order Taylor expansion at the camera frame, leading to the computation of an affine transform $J$, used to derive the 2D covariance matrix $\Sigma'$.

Thereafter, each Gaussian is depth composited following the tile sorting method introduced by Kerbl et al. This involves binning the Gaussians into tiles based on depth, rasterizing them accordingly, and utilizing their opacities and colors to compute the final pixel values. The depth compositing effectively handles the overlap and visibility of multiple Gaussians, ensuring accurate rendering of the 3D scene.

Computation of Gradients

An integral part of the gsplat library is the backward pass, where gradients of a loss function, defined on the rendered images, are propagated back to the Gaussian parameters. This backward propagation is imperative for optimizing the splatting process through techniques such as gradient descent.

Utilizing the Frobenius inner product, the paper elucidates how the gradients are propagated from the rendered image back to the original 3D Gaussian parameters. This includes detailed derivations for gradients with respect to the color, opacity, 2D means, and covariances of the Gaussians affecting each pixel.

Depth compositing gradients are first computed by considering the contribution of each Gaussian to the color and opacity of each pixel. These gradients are then back-propagated through the projection transformations using the Jacobians of the respective transforms.

For scale and rotation gradients, the covariance matrix $\Sigma$ is decomposed into its constituent scale $S$ and rotation $R$ matrices. The Jacobians of these transformations with respect to their parameters are employed to propagate the gradients to the original scale and rotation parameters.

Practical Implications and Future Directions

The mathematical elucidation provided in the paper strengthens the utility of the gsplat library for researchers and practitioners aiming to advance differentiable rendering techniques. The modularity and accuracy of the library, augmented by the detailed API, facilitate modifications and enhancements to specific components of the computational graph, fostering innovation and development in the field.

The theoretical rigor and practical implementation outlined can lead to enhanced Gaussian splatting techniques applicable in various domains such as computer vision, graphics, and augmented reality. Future research could potentially explore optimizing the performance and scalability of these techniques, deeper integration with machine learning frameworks, and extending the mathematical models to handle more complex scenes and rendering challenges.

Conclusion

The paper presents an exhaustive mathematical supplement that underpins the gsplat library, ensuring both forward and backward passes in differentiable Gaussian splatting are comprehensively understood and accurately implemented. This work stands as an essential reference for researchers aiming for an in-depth grasp of the mathematics involved in Gaussian splatting and its implications for efficient, differentiable rendering applications. With its lucid derivations and practical focus, the gsplat library is poised to be a valuable tool in pushing the boundaries of visual computing.