Gaussian Splatting SLAM

• Gaussian Splatting SLAM is a dense mapping system that represents scenes with parameterized anisotropic 3D Gaussians, enabling real-time, photo-realistic reconstruction.
• It employs fully differentiable rendering and joint optimization of pose and Gaussian parameters to minimize photometric and geometric losses for robust tracking.
• The approach achieves sub-centimeter tracking accuracy and state-of-the-art photometric metrics on standard benchmarks, supporting multi-modal sensor fusion and scalable mapping.

Gaussian Splatting SLAM encompasses a class of dense SLAM (Simultaneous Localization and Mapping) systems that employ clouds of parameterized 3D Gaussians as the sole scene representation. Modern implementations leverage fully differentiable, anisotropic Gaussians, whose positions, shapes, colors, and opacities are optimized online together with camera poses. This approach enables metrically accurate pose estimation and high-fidelity, real-time photo-realistic reconstruction by directly minimizing comparably rendered images and depth maps against sensor observations. Gaussian Splatting SLAM systems have been demonstrated on monocular and RGB-D inputs, and more recently extended to fused IMU, stereo, or even LiDAR modalities and dynamic scene modeling. State-of-the-art systems achieve sub-centimeter tracking accuracy and leading photometric metrics on standard benchmarks (Sun et al., 2024, Yugay et al., 2023, Xu et al., 2024, Zhu et al., 2 Dec 2025), outperforming both classical feature-based and neural implicit SLAM approaches in fidelity, robustness, and efficiency.

1. Scene Representation and Differentiable Rendering

Each scene is encoded as a collection of $N$ spatially localized, anisotropic 3D Gaussians. Each Gaussian $i$ is parameterized by a 3D mean $\mu_i\in\mathbb R^3$, a covariance matrix $\Sigma_i\in\mathbb R^{3\times3}$ (factored into scale and rotation), RGB color $c_i\in\mathbb R^3$, and scalar opacity $o_i\in[0,1]$. Rendering proceeds by projecting each 3D Gaussian to a 2D elliptical footprint in the image plane under the current camera pose and intrinsics, yielding a 2D mean $\mu'_i$ and covariance $\Sigma'_i$. For each pixel, overlapping Gaussians are depth-sorted and blended via front-to-back alpha compositing: $$\alpha_i(x) = o_i \exp\left[-\frac{1}{2}(x-\mu'_i)^\top (\Sigma'_i)^{-1} (x-\mu'_i)\right]$$

$$\hat C(x) = \sum_i c_i\, \alpha_i(x)\, \prod_{j<i}(1-\alpha_j(x)), \qquad \hat D(x) = \sum_i z_i\, \alpha_i(x)\, \prod_{j<i}(1-\alpha_j(x))$$

(Here $i$0 is the depth of the $i$1-th Gaussian center in camera coordinates.)

This explicit, rasterizable model supports highly parallelized tile-based rendering at real-time rates and is fully differentiable, enabling direct backpropagation of photometric and geometric losses (Sun et al., 2024, Yugay et al., 2023).

2. Pose Estimation and Tracking

Camera pose tracking is performed by minimizing residuals between rendered color/depth and observed frames. For each incoming RGB-D frame, the system renders synthetic color $i$2 and depth $i$3 under the candidate pose, and the pose is refined by minimizing a differentiable loss: $i$4 Weight masks may be imposed to down-weight unmodeled regions (e.g., by $i$5) and ignore large outlier pixels. Some systems convert color to LAB and drop luminance for robustness to illumination (Sun et al., 2024). The process alternates tracking and mapping in a single optimization loop per frame or per sliding window (Matsuki et al., 2023, Sun et al., 2024).

For purely monocular setups, pose tracking is performed using photometric alignment only; for RGB-D or multi-modal systems, geometric losses are included (Matsuki et al., 2023, Yugay et al., 2023, Liu et al., 24 Mar 2025).

3. Map Expansion and Densification

To avoid under-modeling unexplored or revisited regions, modern systems implement geometry-guided densification. Pixels with low rendered opacity (holes) or high re-rendering error (color or depth) are detected in the current frame: $i$6 New Gaussians are spawned for pixels where $i$7 or $i$8 exceed thresholds. The spatial initialization is from back-projected depth with local scale and rotation guessed from neighborhood structure. By refining re-observed regions in an error-aware manner, the system improves geometric and photometric completeness (Sun et al., 2024, Yugay et al., 2023).

In large-scale or online settings, the map is partitioned into submaps. Each submap is optimized only while active, controlling memory footprint and maintaining real-time performance (Yugay et al., 2023, Xu et al., 2024).

4. Map Regularization and Forgetting Prevention

Continuous online updates risk catastrophic forgetting: parameters can rapidly overfit to the most recent frames, degrading rendering on previously seen views. To address this, statistics are maintained for each Gaussian—count of frame “seens” and accumulated absolute gradients—with importance weights defined as: $i$9 A regularization loss penalizes sudden deviations from prior Gaussian parameters, weighted by these importance factors: $\mu_i\in\mathbb R^3$0 The mapping loss combines photometric/depth terms and $\mu_i\in\mathbb R^3$1 to ensure global map stability and coverage (Sun et al., 2024).

Regularization is typically augmented by an isotropy penalty that discourages degenerate, highly elongated Gaussians: $\mu_i\in\mathbb R^3$2 (Matsuki et al., 2023, Yugay et al., 2023, Bai et al., 2024).

5. Joint Optimization and System Workflow

The main system loop alternates camera pose tracking and mapping: for each RGB-D frame, tracking is performed by optimizing pose to minimize the render-and-compare loss; mapping then executes densification/error-aware refinement and joint optimization of all active Gaussian parameters: $\mu_i\in\mathbb R^3$3 Gradient-based optimizers (e.g., Adam) are run for dozens to hundreds of iterations per frame for both pose and map parameter updates (Sun et al., 2024). Per-frame timings are typically $\mu_i\in\mathbb R^3$450 iterations for tracking and 100–200 for mapping, with full-resolution GPU-based rendering achieving 100+ fps in pure rasterization (Sun et al., 2024, Yugay et al., 2023).

Implementation is hardware-accelerated using C++/CUDA and PyTorch autograd for differentiable loss computation (Sun et al., 2024, Yugay et al., 2023).

6. Quantitative Performance and Benchmarks

Gaussian Splatting SLAM achieves competitive or state-of-the-art results on standard synthetic and real benchmarks. For example, on Replica (synthetic RGB-D), typical numbers are:

• PSNR: 36–42 dB (best among dense SLAMs)
• SSIM: 0.98–0.99
• LPIPS: 0.05–0.018 (lower is better)
• ATE RMSE: 0.23–0.31 cm (Sun et al., 2024, Yugay et al., 2023, Xu et al., 2024)

On real-world TUM-RGBD:

• PSNR: 20–25 dB (first place)
• SSIM: 0.85–0.91
• LPIPS: 0.15–0.32
• ATE RMSE: 2.2–23 cm (competitive; dynamic scenes and motion blur remain challenging) (Sun et al., 2024, Xu et al., 2024)

Gaussian Splatting SLAM methods generally surpass prior neural SLAMs (e.g., NICE-SLAM, ESLAM) in rendering fidelity and often in tracking. Notable is the absence of neural decoders, as the Gaussian map encodes geometry and color in explicit parameters (Sun et al., 2024).

7. Key Contributions and Impact

Recent advances in Gaussian Splatting SLAM include:

• Rendering-guided densification for robust filling of unobserved and error-prone regions (Sun et al., 2024).
• Online regularization strategies to counteract forgetting in continual mapping (Sun et al., 2024, Yugay et al., 2023, Bai et al., 2024).
• Unified mapping and tracking in the Gaussian parameter space, obviating reliance on offline SfM (Sun et al., 2024).
• Demonstration of state-of-the-art photorealistic reconstruction and competitive tracking at real-time rates (Yugay et al., 2023, Xu et al., 2024).
• Hardware-accelerated implementations enabling scalable, interactive, or fully real-time SLAM on commodity GPUs (Yugay et al., 2023, Sun et al., 2024, Bai et al., 2024).

This body of work establishes Gaussian Splatting as a unifying, explicit, and highly efficient representation for real-time dense SLAM, with empirically validated robustness and accuracy on large synthetic and real-world benchmarks. It provides a foundation for ongoing extensions to dynamic scenes, multi-modal sensor fusion, and large-scale deployment (Sun et al., 2024, Yugay et al., 2023, Xu et al., 2024, Zhu et al., 2 Dec 2025).