Second-order Optimization of Gaussian Splats with Importance Sampling (2504.12905v1)

Abstract: 3D Gaussian Splatting (3DGS) is widely used for novel view synthesis due to its high rendering quality and fast inference time. However, 3DGS predominantly relies on first-order optimizers such as Adam, which leads to long training times. To address this limitation, we propose a novel second-order optimization strategy based on Levenberg-Marquardt (LM) and Conjugate Gradient (CG), which we specifically tailor towards Gaussian Splatting. Our key insight is that the Jacobian in 3DGS exhibits significant sparsity since each Gaussian affects only a limited number of pixels. We exploit this sparsity by proposing a matrix-free and GPU-parallelized LM optimization. To further improve its efficiency, we propose sampling strategies for both the camera views and loss function and, consequently, the normal equation, significantly reducing the computational complexity. In addition, we increase the convergence rate of the second-order approximation by introducing an effective heuristic to determine the learning rate that avoids the expensive computation cost of line search methods. As a result, our method achieves a $3\times$ speedup over standard LM and outperforms Adam by $~6\times$ when the Gaussian count is low while remaining competitive for moderate counts. Project Page: https://vcai.mpi-inf.mpg.de/projects/LM-IS

Second-order Optimization of Gaussian Splats with Importance Sampling

This paper introduces an innovative second-order optimization framework tailored for 3D Gaussian Splatting (3DGS), specifically targeting the challenges and inefficiencies of current first-order methods such as Adam. By proposing a novel approach grounded in the Levenberg-Marquardt (LM) and Conjugate Gradient (CG) algorithms, the authors aim to enhance the training efficiency of 3DGS models, which are highly valued for their fast inference times and high rendering quality in novel view synthesis.

3D Gaussian Splatting is a technique that replaces the conventional coordinate-based representations for 3D rendering, leveraging a set of Gaussians to synthesize images effectively. Despite recent advancements, the optimization of parameters within 3DGS remains computationally demanding and time-intensive. Common approaches like neural radiance fields (NeRFs) are often hindered by substantial training durations. First-order optimizers, although extensively utilized, offer limited scope in accelerating convergence.

The paper identifies the inherent sparsity in the Jacobian matrix associated with 3DGS, a key insight that forms the basis of their second-order optimization strategy. Each Gaussian component affects only a limited number of image pixels, which results in a sparse Jacobian. Capitalizing on this, the authors propose a matrix-free and GPU-optimized LM framework that eschews the need to store large Jacobians explicitly. This is achieved by employing a GPU-parallelized CG solver that iteratively solves the matrix inverse, eliminating direct computation of the full Jacobian.

Significantly, the authors also introduce innovative sampling strategies to further enhance computational efficiency. The paper advocates for strategic sampling of camera views and pixels, approximating the loss functions to decrease complexity without sacrificing convergence. By effectively sampling important views and pixels, the optimizer achieves faster and more reliable convergence, particularly when the Gaussian count is low, evidencing a sixfold improvement in speed over the Adam optimizer.

The implications of this research are substantial, suggesting that second-order methods, typically underexplored in this domain, hold promise for expediting 3DGS training processes. Practical benefits are evident in applications requiring rapid high-fidelity novel view synthesis, such as virtual reality, gaming, and the film industry. Theoretically, this work paves the way for future research on second-order methods in graphics rendering, particularly in cases where first-order methodologies stagnate.

Looking ahead, the exploration of second-order optimizers, especially in scenarios involving larger Gaussian counts, could further optimize 3DGS performance. Additionally, the strategies adopted here might inform applications in other computational settings that benefit from Jacobian sparsity.

In conclusion, this paper successfully demonstrates a method to reinforce 3D Gaussian Splatting’s efficiency through advanced optimization techniques, suggesting a viable pathway to address the computational bottlenecks associated with current first-order approaches. Given these results, second-order optimization presents a promising avenue for further advancement in the field of novel view synthesis and beyond.