GPU-Based LM Solvers for Scene Reconstruction

• GPU-based Levenberg–Marquardt solvers are optimization methods that use custom CUDA kernels and a matrix-free approach to efficiently solve large-scale nonlinear least-squares problems.
• They leverage a cache-based memory strategy and batch processing to compute Jacobian–vector products, yielding approximately 30% faster convergence compared to ADAM-based methods.
• Empirical results demonstrate that these solvers maintain high output quality in 3D scene reconstruction, making them vital for advanced computer vision and graphics applications.

GPU-based Levenberg–Marquardt solvers are high-performance optimization methods for large-scale nonlinear least-squares problems routinely encountered in computer vision and graphics pipelines, such as 3D Gaussian Splatting (3DGS). The 3DGS-LM approach replaces classical first-order optimizers (e.g., ADAM) with a matrix-free, GPU-accelerated implementation of Levenberg–Marquardt (LM), achieving significantly faster convergence for scene reconstruction while maintaining the same output quality (Höllein et al., 2024). This solver is characterized by a sum-of-squares objective, direct GPU implementation via custom CUDA kernels, a cache-based memory strategy for efficient Jacobian-vector computation, and tight integration with batch-based processing for scalability across large datasets.

1. Mathematical Framework and Problem Setup

The optimization target is a scene parameterized by $M$ three-dimensional Gaussians. Each Gaussian is described by a 59-dimensional vector: 11 parameters represent position, orientation, scaling, and opacity, and 48 parameters correspond to spherical-harmonic color coefficients. A dataset of $P$ registered RGB images (each $H \times W$) defines the observation space. For every pixel $i$, two per-pixel losses are measured post-rendering:

• $\ell_{1i} = L_{1}$(rendered color, ground-truth color)
• $\ell_{2i} = 1 - SSIM$(rendered patch, ground-truth patch)

The global energy is a sum of weighted squared residuals:

$$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$

where $N = 3 \cdot H \cdot W \cdot P$, $\lambda_1 = 0.2$, and $\lambda_2 = 0.8$. The residual vector $P$0 is defined entrywise as either $P$1 or $P$2. The goal is $P$3.

LM proceeds via the Gauss–Newton step with Tikhonov-style damping. The update equation for the $P$4-th iteration is:

$P$5

where $P$6 is the Jacobian of $P$7 with respect to $P$8, and $P$9 is a regularization coefficient adapted per iteration via the standard $H \times W$0-test.

2. GPU-based Solver Architecture

The implementation is fully matrix-free, leveraging GPU parallelism and explicit CUDA kernels rather than materializing the massive sparse Jacobian matrix. All matrix computations are performed implicitly through custom operations over a “gradient cache”:

• buildCache: For each pixel, forward rasterization is followed by a per-pixel backward pass to store all necessary “pixel-to-splat” ($H \times W$1) gradients as flat cache entries, along with $H \times W$2 and the gradient payload.
• diagJTJ: Sums squared gradients per parameter to accumulate diagonal elements of $H \times W$3.
• applyJ: Applies $H \times W$4 to a vector, streaming over the cache (sorted by Gaussian).
• applyJT: Applies $H \times W$5, streaming over the cache (sorted by splat).

Cache construction ensures efficient computation and memory coalescing, using sort operations to optimize data layout before each step.

3. Batching, Partial Update Aggregation, and Memory Management

To scale to datasets where the aggregate cache size exceeds GPU memory, the images are divided into $H \times W$6 batches. For each batch $H \times W$7:

• Construct the gradient cache for the batch.
• Accumulate batch-specific $H \times W$8 and diagonal preconditioner $H \times W$9.
• Solve the regularized normal equations using Preconditioned Conjugate Gradient (PCG, see Section 4).

Partial update directions $i$0 are combined as a weighted mean across batches, weighted by $i$1, reflecting each batch’s constraint strength on each Gaussian:

$i$2

A line search is performed on a subset of images to choose step size $i$3 for the update $i$4. The damping coefficient $i$5 is then adjusted by the $i$6-test.

4. Preconditioned Conjugate Gradient (PCG) in CUDA

Each LM update solves the regularized normal system with PCG, requiring only implicit matrix–vector multiplies. The workflow is as follows:

• Initialize residual $i$7, preconditioned vector $i$8, search direction $i$9.
• For $\ell_{1i} = L_{1}$0 iterations, compute:
  • $\ell_{1i} = L_{1}$1 via applyJ
  • $\ell_{1i} = L_{1}$2 via applyJT and diagonal scaling
  • $\ell_{1i} = L_{1}$3, $\ell_{1i} = L_{1}$4 and updates to $\ell_{1i} = L_{1}$5, $\ell_{1i} = L_{1}$6, $\ell_{1i} = L_{1}$7, $\ell_{1i} = L_{1}$8 accordingly

All these steps are implemented as specialized GPU kernels, using the prebuilt gradient cache.

5. Computational Complexity and Efficiency

The dominant cost per LM iteration is in the PCG loop:

• Rasterization + cache build: $\ell_{1i} = L_{1}$9 for a batch of $\ell_{2i} = 1 - SSIM$0 images, with $\ell_{2i} = 1 - SSIM$1 average splats per pixel.
• PCG matvecs: Each iteration costs $\ell_{2i} = 1 - SSIM$2, with three such passes per batch.
• Batch combination: $\ell_{2i} = 1 - SSIM$3 to aggregate updates.

In practice: $\ell_{2i} = 1 - SSIM$4, $\ell_{2i} = 1 - SSIM$5–$\ell_{2i} = 1 - SSIM$6, $\ell_{2i} = 1 - SSIM$7 PCG iterations; LM requires $\ell_{2i} = 1 - SSIM$8 iterations beyond an initial ADAM warmup, whereas ADAM alone requires $\ell_{2i} = 1 - SSIM$9 iterations. This yields approximately $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$0 faster end-to-end optimization over the original ADAM-only 3DGS pipeline (Höllein et al., 2024).

6. Empirical Benchmarks and Output Quality

On NVIDIA A100 (Tanks & Temples “Train” scene):

• 3DGS + ADAM: $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$1736 s, SSIM 0.844, PSNR 23.68, LPIPS 0.178
• 3DGS + LM: $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$2663 s ($$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$3), identical SSIM, PSNR, LPIPS

Combining LM with additional system-level speedups (DISTWAR, gsplat, Taming-3DGS) maintains a $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$430% total speed increase relative to vanilla 3DGS. Output quality remains unchanged according to standard perceptual quality metrics.

7. Key Equations and GPU Code Constructs

The core update equation is the regularized normal equation:

$$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$5

PCG matrix-vector steps are realized as:

• applyJ: For each Gaussian $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$6, $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$7gaussianParamIndex$$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$8
• applyJT: For each ray $$E(x) = \sum_{i=1}^{N} \left[ (\sqrt{\lambda_1 \ell_{1i}})^2 + (\sqrt{\lambda_2 \ell_{2i}})^2 \right]$$9, $N = 3 \cdot H \cdot W \cdot P$0pixelIndex$N = 3 \cdot H \cdot W \cdot P$1
• diagJTJ: For parameter $N = 3 \cdot H \cdot W \cdot P$2, $N = 3 \cdot H \cdot W \cdot P$3

CUDA pseudocode for per-pixel cache build: $N = 3 \cdot H \cdot W \cdot P$4

Efficient implementation follows this workflow for matrix-free LM updates, leveraging full GPU parallelism by explicit kernel launches and gradient caching.

---

For further details on implementation specifics and the complete pipeline, see (Höllein et al., 2024).