3D Gaussian-Based SLAM

• 3D Gaussian-Based SLAM is a dense mapping and localization technique that uses explicit 3D Gaussian primitives for differentiable rendering and efficient camera tracking.
• It employs hybrid optimization strategies ranging from direct frame-to-model alignment to feature-based bundle adjustment to balance speed with reconstruction quality.
• Advanced systems integrate semantic embeddings, dynamic scene handling, and multi-sensor fusion to achieve robust performance across indoor, outdoor, and large-scale environments.

3D Gaussian-Based SLAM denotes a family of dense SLAM systems in which the scene is modeled as an explicit set of 3D Gaussian primitives and optimized jointly or alternately with camera motion. In these systems, the map is no longer a voxel grid, point cloud, or implicit radiance field alone; it is a splatted collection of Gaussian ellipsoids or blobs whose differentiable rasterization supports camera tracking, dense reconstruction, and real-time or near-real-time novel-view rendering. The formulation emerged in dense RGB-D SLAM and was rapidly extended to monocular, stereo, semantic, dynamic-scene, visual-inertial, and large-scale outdoor settings, with recurrent emphasis on compactness, robustness, and scalable optimization (Yugay et al., 2023, Deng et al., 2024).

1. Representation and differentiable rendering

At the core of 3D Gaussian-Based SLAM is an explicit map $\mathcal{G}=\{G_i\}$ in which each primitive usually carries a mean $\mu_i\in\mathbb{R}^3$, a covariance $\Sigma_i\in\mathbb{R}^{3\times3}$, an opacity or weight, and a color descriptor. In anisotropic formulations, the covariance is commonly factorized as $\Sigma_i = R_i S_i S_i^\top R_i^\top$ or $\Sigma_i = R_i S_i^2 R_i^\top$, with $R_i\in SO(3)$ and diagonal scale $S_i$; isotropic variants replace this with a scalar radius. Gaussian-SLAM, Compact 3D Gaussian Splatting For Dense Visual SLAM, MM3DGS, GI-SLAM, BGS-SLAM, and related systems all adopt this explicit parameterization, while differing in whether appearance is stored as direct RGB, spherical harmonics coefficients, semantic descriptors, or explicit labels (Yugay et al., 2023, Deng et al., 2024, Sun et al., 2024, Liu et al., 24 Mar 2025, Li et al., 31 Jul 2025).

Rendering proceeds by projecting each 3D Gaussian to the image plane, approximating it as a 2D Gaussian, sorting contributions front-to-back, and alpha-compositing color and depth. A representative formulation writes the density of Gaussian $n$ as

$$f_n(x)=o_n \exp\!\left[-\tfrac12(x-\mu_n)^\top\Sigma_n^{-1}(x-\mu_n)\right],$$

projects covariance by

$\Sigma_n' = J W \Sigma_n W^\top J^\top,$

and composites

$\mu_i\in\mathbb{R}^3$0

Equivalent front-to-back formulations appear across Compact 3D Gaussian Splatting, RP-SLAM, GARAD-SLAM, Dy3DGS-SLAM, GI-SLAM, and MM3DGS, differing mainly in notation and attribute choice (Deng et al., 2024, Bai et al., 2024, Li et al., 5 Feb 2025, Li et al., 6 Jun 2025, Liu et al., 24 Mar 2025, Sun et al., 2024).

The representation is not purely geometric. SemGauss-SLAM appends a learnable semantic feature vector $\mu_i\in\mathbb{R}^3$1 to each Gaussian and renders a semantic embedding map by the same over-compositing weights, while NEDS-SLAM stores a low-dimensional semantic descriptor after encoder-decoder compression, and OpenGS-SLAM associates each Gaussian with an explicit non-differentiable semantic label $\mu_i\in\mathbb{R}^3$2 for object-level scene understanding (Zhu et al., 2024, Ji et al., 2024, Yang et al., 3 Mar 2025). FGO-SLAM further reinterprets the Gaussian set as a globally consistent opacity radiance field and defines the field value at a 3D point by a minimum across views, then uses tetrahedral level sets for direct surface extraction (Zhu et al., 1 Sep 2025).

2. Pose estimation and optimization architectures

Tracking in 3D Gaussian-Based SLAM spans a spectrum from direct frame-to-model optimization to feature-guided bundle adjustment and feed-forward inference. Gaussian-SLAM estimates each new pose by aligning rendered color and depth to the live RGB-D frame with a constant-velocity initialization, an inlier mask, and Gauss–Newton updates on a 6D twist; Compact 3D Gaussian Splatting adds a reprojection loss built from SIFT matches and performs global bundle adjustment over keyframes and the Gaussian map; MM3DGS alternates a pose block and a map block, using photometric and depth losses while masking pixels with low rendered opacity (Yugay et al., 2023, Deng et al., 2024, Sun et al., 2024).

A second design line couples the Gaussian map to classical geometric front ends. GSORB-SLAM combines ORB-feature PnP initialization with a joint frame-to-model objective using ORB reprojection error and dense transmittance or surface-depth cues from 3DGS, while maintaining an independent sparse back-end bundle adjustment. FGO-SLAM begins with feature-based pose estimation, then optimizes all keyframe poses and sparse points in a factor graph with reprojection and loop factors, and only afterward freezes poses and optimizes the Gaussian field. MMD-SLAM extends this line further by introducing point-line fusion, with 2D and 3D residuals for both points and line segments inside a Levenberg–Marquardt bundle-adjustment objective (Zheng et al., 2024, Zhu et al., 1 Sep 2025, Zhu et al., 18 Jun 2026).

A third line decouples or amortizes pose estimation. RP-SLAM explicitly decouples camera pose estimation from Gaussian primitive optimization and uses ORB-SLAM3 to provide keyframes, poses, and sparse point clouds. “Pseudo Depth Meets Gaussian” replaces iterative test-time pose optimization with a recurrent optical-flow network and a differentiable DBA layer, augmented by local graph rendering from neighboring synthesized views. This directly targets the latency of optimization-heavy trackers and is reported as reducing tracking time by more than $\mu_i\in\mathbb{R}^3$3 relative to SplaTAM while keeping reconstruction performance on par (Bai et al., 2024, Zhao et al., 6 Aug 2025).

The literature therefore does not define a single tracking template. Some systems are map-centric and differentiable end-to-end; others are tightly coupled hybrids; others retain a conventional sparse SLAM skeleton and use Gaussians primarily for mapping and rendering. A plausible implication is that 3DGS functions as a rendering-geometry substrate rather than a commitment to one front-end estimator.

3. Map growth, redundancy control, and regularization

Most systems must solve three recurrent mapping problems: how to initialize Gaussians, how to densify newly observed regions, and how to prevent uncontrolled growth. Gaussian-SLAM seeds new Gaussians from RGB-D keyframes only in under-covered regions, using an alpha mask from the current submap and a radius based on nearest-neighbor distance. MM3DGS spawns new Gaussians where rendered opacity $\mu_i\in\mathbb{R}^3$4 and the depth error is large, initializing color from the pixel RGB, mean from back-projected depth, opacity at $\mu_i\in\mathbb{R}^3$5, and covariance isotropically at roughly one-pixel footprint at depth $\mu_i\in\mathbb{R}^3$6 (Yugay et al., 2023, Sun et al., 2024).

More elaborate insertion policies appear in later systems. RP-SLAM uses quadtree-based adaptive image sampling and KNN-based primitive filtering, with thresholds scaled by $\mu_i\in\mathbb{R}^3$7, so that candidate Gaussians are sampled where image variation is high but redundant insertions are rejected. GSORB-SLAM performs adaptive densification in regions flagged by low accumulated transmittance or combined depth and appearance error, and adds regularization terms to suppress redundant primitives and excessively elongated splats. VPGS-SLAM replaces per-point insertion with voxel anchors and spawns multiple Gaussians from anchor features and offsets, optimizing anchor attributes rather than isolated splats (Bai et al., 2024, Zheng et al., 2024, Deng et al., 25 May 2025).

Redundancy control is especially explicit in Compact 3D Gaussian Splatting For Dense Visual SLAM. There, each Gaussian acquires a learnable scalar mask optimized only within a sliding window of recent keyframe frusta, with a regularizer that pushes masks toward zero; masked Gaussians have scale and opacity zeroed out. The same system also compresses geometry with a residual vector quantization codebook after observing that the covariance matrix of most Gaussians is highly repetitive, reporting that the active $\mu_i\in\mathbb{R}^3$8 drops by $\mu_i\in\mathbb{R}^3$9 and per-Gaussian attribute size drops by $\Sigma_i\in\mathbb{R}^{3\times3}$0–$\Sigma_i\in\mathbb{R}^{3\times3}$1 (Deng et al., 2024).

Regularization and pruning remain central because 3DGS maps otherwise accumulate unstable or spurious structure. RP-SLAM uses an isotropy penalty to avoid long “needle” Gaussians; FGO-SLAM introduces depth distortion and normal consistency terms; VPGS-SLAM includes a volume regularizer $\Sigma_i\in\mathbb{R}^{3\times3}$2; NEDS-SLAM renders from four virtual nearby views and degrades or prunes Gaussians that are never visible in any virtual view; OpenGS-SLAM prunes semantically inconsistent Gaussians whose principal axis length exceeds a threshold after label disagreement; and KiloGS-SLAM adds chunk-based densification, pruning, and same-voxel Gaussian merging through moment matching (Bai et al., 2024, Zhu et al., 1 Sep 2025, Deng et al., 25 May 2025, Ji et al., 2024, Yang et al., 3 Mar 2025, Yu et al., 29 Jun 2026).

4. Semantic and dynamic-scene formulations

One major branch of 3D Gaussian-Based SLAM adds semantics directly to the map. SemGauss-SLAM stores a $\Sigma_i\in\mathbb{R}^{3\times3}$3-dimensional semantic feature vector in each Gaussian, renders a semantic embedding map, and optimizes a feature-level loss together with photometric and geometric terms. It also introduces semantic-informed bundle adjustment that jointly refines camera poses and Gaussians by multi-view semantic, RGB, and depth consistency, reporting $\Sigma_i\in\mathbb{R}^{3\times3}$4 cm ATE RMSE on Replica and approximately $\Sigma_i\in\mathbb{R}^{3\times3}$5 mIoU on input views (Zhu et al., 2024).

NEDS-SLAM and OpenGS-SLAM pursue two different semantic encodings. NEDS-SLAM fuses high-dimensional semantic features with appearance or spatial features through a Spatially Consistent Feature Fusion model, compresses them to about $\Sigma_i\in\mathbb{R}^{3\times3}$6 dimensions per Gaussian with a lightweight encoder, and uses a Virtual Camera View Pruning method to remove outliers; on Replica it reports $\Sigma_i\in\mathbb{R}^{3\times3}$7 semantic mIoU and a $\Sigma_i\in\mathbb{R}^{3\times3}$8 PSNR boost from virtual-view pruning (Ji et al., 2024). OpenGS-SLAM instead assigns each Gaussian an explicit semantic label and introduces Gaussian Voting Splatting, Confidence-based 2D Label Consensus, and Segmentation Counter Pruning; on Replica it reports semantic rendering at $\Sigma_i\in\mathbb{R}^{3\times3}$9 Hz versus $\Sigma_i = R_i S_i S_i^\top R_i^\top$0 Hz for GS-Grouping and learnable parameters of $\Sigma_i = R_i S_i S_i^\top R_i^\top$1 MB versus $\Sigma_i = R_i S_i S_i^\top R_i^\top$2 MB (Yang et al., 3 Mar 2025).

Dynamic-scene handling forms another active branch. GARAD-SLAM performs dynamic segmentation directly on Gaussians with a Gaussian pyramid network and a CRF whose unary cues are reprojection error, depth variation, observation count, and epipolar distance, then maps labels back to front-end features and validates missed cases by sparse optical flow. Its mapping stage does not simply prune dynamic Gaussians; it imposes a rendering penalty $\Sigma_i = R_i S_i S_i^\top R_i^\top$3 and a time-window retention rule to avoid irreversible erroneous removal (Li et al., 5 Feb 2025).

Dy3DGS-SLAM addresses the same problem under monocular RGB input. It is presented as the first 3DGS SLAM method for dynamic scenes using monocular RGB input, and fuses optical flow masks with monocular depth masks through a probabilistic model to obtain a fused dynamic mask thresholded at $\Sigma_i = R_i S_i S_i^\top R_i^\top$4. Tracking is constrained by a motion loss defined only on static pixels, and mapping reweights color and depth losses differently on dynamic and static regions. The reported runtime is approximately $\Sigma_i = R_i S_i S_i^\top R_i^\top$5 FPS for tracking and $\Sigma_i = R_i S_i S_i^\top R_i^\top$6 ms per keyframe for mapping, but the stated limitations are also clear: dynamic objects are only masked out, not explicitly reconstructed or tracked (Li et al., 6 Jun 2025).

These systems rebut a common misconception that 3D Gaussian-Based SLAM is only a photorealistic mapper. In the surveyed literature it also serves as a substrate for dense semantic mapping, open-set labeling, and dynamic-scene robustness, although the mechanisms differ substantially between feature embeddings, explicit labels, CRF-based Gaussian segmentation, and mask-based motion suppression.

5. Sensor fusion, outdoor operation, and large-scale mapping

3D Gaussian-Based SLAM was initially concentrated in indoor RGB-D settings, but later work broadens the sensor envelope considerably. MM3DGS combines vision, depth, and inertial measurements, uses IMU pre-integration only as an open-loop pose seed, and introduces a depth loss based on linear correlation for monocular scale handling; on UT-MM it reports an average ATE RMSE improvement from $\Sigma_i = R_i S_i S_i^\top R_i^\top$7 cm to $\Sigma_i = R_i S_i S_i^\top R_i^\top$8 cm and a PSNR increase from $\Sigma_i = R_i S_i S_i^\top R_i^\top$9 dB to $\Sigma_i = R_i S_i^2 R_i^\top$0 dB relative to SplaTAM, with end-to-end mapping and tracking at approximately $\Sigma_i = R_i S_i^2 R_i^\top$1 fps on a single RTX A5000 (Sun et al., 2024). GI-SLAM goes further toward tight coupling by adding an IMU loss directly to the tracking objective and supporting monocular, stereo, and RGB-D cameras both with and without IMU integration, reporting about $\Sigma_i = R_i S_i^2 R_i^\top$2–$\Sigma_i = R_i S_i^2 R_i^\top$3 Hz tracking and mapping on an RTX 4090 (Liu et al., 24 Mar 2025).

Gaussian-LIC2 extends the modality set to LiDAR-Inertial-Camera SLAM. It uses a continuous-time B-spline trajectory, IMU pre-integration, LiDAR point-to-plane residuals, camera factors, and a zero-shot depth completion network to initialize Gaussians in LiDAR-blind regions. The total cost includes inertial, LiDAR, camera, depth, and color terms, while CUDA-accelerated tile culling, per-Gaussian backpropagation, sparse Adam, and efficient SSIM make the mapping thread run within bag duration on all tested LiDARs; the front end tracks at $\Sigma_i = R_i S_i^2 R_i^\top$4 Hz (Lang et al., 5 Jul 2025).

Outdoor and large-scale operation required further architectural changes.

Focus	Representative systems	Reported mechanism
Stereo outdoor SLAM	BGS-SLAM (Li et al., 31 Jul 2025)	Deep stereo depth guidance, multi-loss Gaussian optimization
Large-scale RGB-D SLAM	VPGS-SLAM (Deng et al., 25 May 2025)	Voxel-based progressive mapping, multiple submaps, 2D-3D tracking, Gaussian loop closure
Kilometer-scale monocular SLAM	KiloGS-SLAM (Yu et al., 29 Jun 2026)	Motion-adaptive hybrid tracking, lifecycle-managed Gaussian mapping
Visual-inertial Gaussian SLAM	VIGS-SLAM (Zhu et al., 2 Dec 2025)	Sliding-window nonlinear optimization, time-varying IMU biases, loop closure

BGS-SLAM is presented as the first binocular 3D Gaussian Splatting SLAM system designed for outdoor scenarios, using only RGB stereo pairs and a pre-trained deep stereo network; on KITTI it reports an average ATE of $\Sigma_i = R_i S_i^2 R_i^\top$5 m and on KITTI-360 an ATE of $\Sigma_i = R_i S_i^2 R_i^\top$6 m, together with outdoor rendering metrics and depth errors against LiDAR (Li et al., 31 Jul 2025). VPGS-SLAM is presented as the first 3DGS-based large-scale RGBD SLAM framework for both indoor and outdoor scenarios, and combines voxel anchors, sequential submaps, 2D-3D fusion tracking, 2D-3D Gaussian loop closure, and submap fusion with online distillation; it reports Replica ATE RMSE of $\Sigma_i = R_i S_i^2 R_i^\top$7 cm, ScanNet mean ATE RMSE of about $\Sigma_i = R_i S_i^2 R_i^\top$8 cm, and KITTI ATE RMSE of about $\Sigma_i = R_i S_i^2 R_i^\top$9 m with peak GPU usage of only $R_i\in SO(3)$0 GB (Deng et al., 25 May 2025). KiloGS-SLAM targets sequences beyond $R_i\in SO(3)$1 frames on a single GPU and reports a peak count of approximately $R_i\in SO(3)$2 M anchors and a $R_i\in SO(3)$3 MB map size on KITTI-360, while VIGS-SLAM introduces time-varying IMU bias modeling and consistent Gaussian updates after loop closure, though with a stated memory footprint of about $R_i\in SO(3)$4 M Gaussians for a $R_i\in SO(3)$5-frame sequence (Yu et al., 29 Jun 2026, Zhu et al., 2 Dec 2025).

6. Reported performance, recurrent limitations, and open technical issues

The empirical profile of 3D Gaussian-Based SLAM is defined by unusually strong rendering performance combined with dense mapping and, in many settings, competitive or state-of-the-art trajectory accuracy. Gaussian-SLAM reported mapping speed of $R_i\in SO(3)$6 ms per iteration, tracking at $R_i\in SO(3)$7 ms, and rendering at $R_i\in SO(3)$8 FPS on an A6000, with better PSNR and ATE than several neural dense SLAM baselines on Replica and real-world datasets (Yugay et al., 2023). Compact 3D Gaussian Splatting reduced the number of Gaussians on Replica Room0 by $R_i\in SO(3)$9 from approximately $S_i$0 M to approximately $S_i$1 M, reduced GPU memory from $S_i$2 MB to $S_i$3 MB, increased rendering from $S_i$4 FPS to $S_i$5 FPS, and improved ATE RMSE from $S_i$6 cm to $S_i$7 cm while keeping PSNR around $S_i$8 dB (Deng et al., 2024). RP-SLAM reported $S_i$9 FPS in monocular mode and $n$0 FPS in RGB-D mode on Replica, with model sizes of $n$1 MB and $n$2 MB respectively (Bai et al., 2024).

At the same time, the literature identifies a stable set of unresolved problems. Gaussian redundancy and memory explosion recur in Compact 3D Gaussian Splatting, RP-SLAM, VPGS-SLAM, and KiloGS-SLAM. Forgetting during continuous optimization is explicitly raised by RP-SLAM. Sensitivity to artifacts, noise, and training-view selection appears in GSORB-SLAM. Dynamic objects remain difficult: GARAD-SLAM and Dy3DGS-SLAM mitigate them through segmentation, penalties, and mask fusion, but the latter explicitly states that moving objects are only masked out, not reconstructed (Deng et al., 2024, Bai et al., 2024, Zheng et al., 2024, Li et al., 5 Feb 2025, Li et al., 6 Jun 2025). Visual-inertial systems still note incomplete IMU handling: MM3DGS uses the IMU only open-loop, GI-SLAM does not explicitly model IMU noise online, and VIGS-SLAM reports CPU-side IMU preintegration as a throughput bottleneck (Sun et al., 2024, Liu et al., 24 Mar 2025, Zhu et al., 2 Dec 2025).

A further misconception is that the field has converged on a single notion of “real time.” The reported operating points vary widely: MM3DGS states approximately $n$3 fps on a single RTX A5000, GI-SLAM reports about $n$4–$n$5 Hz, GARAD-SLAM reports $n$6 tracking FPS and $n$7 mapping FPS on fr3/wk/xyz, while some outdoor and large-scale systems retain slower mapping threads or high memory footprints despite fast tracking (Sun et al., 2024, Liu et al., 24 Mar 2025, Li et al., 5 Feb 2025, Zhu et al., 2 Dec 2025). This suggests that “real-time” in the 3DGS-SLAM literature may refer variously to tracking, rendering, the combined pipeline, or amortized back-end optimization.

Taken together, the field has moved from small-room RGB-D dense SLAM toward a broader research program: compact Gaussian maps, feature-augmented or label-augmented semantics, dynamic-scene suppression, visual-inertial and LiDAR-inertial-camera fusion, and outdoor long-horizon operation. What remains unsettled is not whether 3D Gaussian splatting can support SLAM, but which combination of front-end estimation, map lifecycle management, and multi-sensor constraints yields the most stable trade-off between fidelity, robustness, and scalability.