Open-Vocabulary 3D Semantic SLAM

• Open-vocabulary 3D Semantic SLAM is an innovative paradigm that combines photorealistic 3D reconstruction with dynamic language grounding for query-driven mapping.
• It integrates vision-language models with compact, adaptive representations to improve real-time tracking, semantic segmentation, and efficient map storage.
• Benchmark results demonstrate significant gains in accuracy, memory reduction, and robust loop closure through advanced feature fusion and online encoder adaptation.

Open-vocabulary 3D Semantic SLAM is an emerging paradigm in simultaneous localization and mapping that augments photorealistic and metric 3D reconstruction with open-set semantic understanding. Leveraging vision-language foundation models to embed text-driven semantics directly into the 3D map, current systems achieve real-time, query-driven mapping that is robust to novel categories and adapts online to new environments. Recent advances focus on overcoming storage, adaptability, and scalability limitations by introducing compact representations, dynamic semantic fusion, and unified architectures that tightly couple geometry, appearance, and language features.

1. Principles and Motivations

Traditional SLAM approaches generate metric maps using geometric or photometric data, typically lacking semantic richness or confining semantics to closed sets of labels. Open-vocabulary 3D Semantic SLAM addresses the challenge of endowing robots and embodied agents with open-set language grounding, enabling the retrieval, segmentation, and interaction with arbitrary objects or spatial concepts specified in natural language.

Central design principles include:

• Vision-language embedding: Integration of foundation models (e.g., CLIP, LSeg, DINO) enables per-point, per-Gaussian, per-segment language features.
• Compactness and efficiency: High-dimensional features (e.g., 512D CLIP vectors) are distilled into compact, reusable latent spaces (e.g., 16D in LEGO-SLAM (Lee et al., 20 Nov 2025)).
• Adaptivity: Scene-adaptive encoders support online learning to handle novel or dynamically evolving environments.
• Unification: Language representations are reused across mapping, pruning, loop closure, and semantic querying modules, yielding a tightly integrated and computationally efficient SLAM backbone.

2. Representative Architectures

Advanced systems in open-vocabulary semantic SLAM typically fuse geometric and semantic data in real-time, realizing joint estimation pipelines. A comprehensive template is LEGO-SLAM (Lee et al., 20 Nov 2025), which is organized into four key modules:

Module	Role in Pipeline	Technical Implementation
Tracking	6-DOF pose estimation, aligning new frames to global map	RGB-D input, G-ICP alignment
Mapping	Photorealistic and semantic feature optimization	Joint photometric/geometric/semantic loss
Language-guided pruning	Map sparsification via semantic redundancy detection	Cosine similarity + local 3D proximity
Loop closure (semantic)	Relocalization via semantic histogram matching	Feature clustering + cosine scoring

In LEGO-SLAM, each Gaussian is parameterized as $G_i = \{p_i, q_i, s_i, \alpha_i, c_i, f_i\}$, where $f_i\in\mathbb{R}^{16}$ is the compact language feature distilled from a frozen high-dimensional foundation model. Mapping and rendering reuse the same feature, and downstream modules perform pruning and semantic loop closure using cosine similarity of these embeddings. Encoder adaptation is performed online, alternating optimization of scene-specific features and regularization through joint rendering/feature matching (Lee et al., 20 Nov 2025).

Other leading designs include:

• KM-ViPE (Nasser et al., 1 Dec 2025): Monocular SLAM with real-time online fusion of DINO visual features, language grounding via CLIP, and dynamic scene robustness through adaptive robust kernel (ARK) bundle adjustment.
• OpenMonoGS-SLAM (Yoo et al., 9 Dec 2025): Monocular Gaussian splatting, leveraging foundation models MASt3R, SAM, and CLIP, with a high-dimensional memory bank for open-set semantic inference.
• OpenFusion++ (Jin et al., 27 Apr 2025): TSDF-based geometric mapping with adaptive semantic caching and dual-path language-context alignment for nested and environment-grounded queries.

3. Mathematical Formulations

Map representations in open-vocabulary SLAM are enriched with semantic features and governed by multi-objective losses.

In LEGO-SLAM (Lee et al., 20 Nov 2025), rendering along a ray $r$ combines color and language features:

• Photometric rendering:

$$\hat{C}(r)=\frac{\sum_{i}w_i(r)\,c_i}{\sum_{i}w_i(r)},\quad w_i(r)=\alpha_i\,\mathcal{N}(x; p_i, \Sigma_i)$$

• Semantic feature rendering:

$$F_{\mathrm{render}}(r)=\frac{\sum_i w_i(r)\,f_i}{\sum_i w_i(r)}$$

• Loss function:

$$\mathcal{L}_{\mathrm{total}} = \mathcal{L}_{\mathrm{rgb}} + w_{\mathrm{depth}}\mathcal{L}_{\mathrm{depth}} + w_{\mathrm{feat}}\mathcal{L}_{\mathrm{feat}}$$

where - $\mathcal{L}_{\mathrm{rgb}}$ is photometric (L1+SSIM), - $\mathcal{L}_{\mathrm{depth}}$ is geometric (L1), - $\mathcal{L}_{\mathrm{feat}}$ is feature distillation (decoder-based L1).

Language-guided pruning relies on 3D distance and cosine similarity between features:

$$d_{ij}=\|p_i-p_j\|_2,\quad \mathrm{sim}_{ij}=\frac{f_i\cdot f_j}{\|f_i\|\|f_j\|}$$

Redundant Gaussians are removed if $$(d_{ij}<\tau_{\mathrm{dist}})\;\wedge\;(\mathrm{sim}_{ij}>\tau_{\mathrm{sim}})$$.

Semantic loop closure builds feature histograms via k-means clustering and matches by cosine similarity:

$$s = \frac{h_{\mathrm{cur}}\cdot h_{p}}{\|h_{\mathrm{cur}}\|\|h_{p}\|}$$

4. Scene Adaptivity and Feature Management

A distinguishing aspect of modern frameworks is on-the-fly adaptation to unseen environments and control of feature dimension for efficiency.

• Scene-adaptive embedding (LEGO-SLAM): The encoder $E_\phi$ is periodically trained online, first frozen during map optimization; later adapted by minimizing the encoder consistency loss:

$$\mathcal{L}_{\rm enc} = \bigl\Vert E_\phi(F_{gt})-F_{\mathrm{render}}\bigr\Vert_1$$

This ensures that compact features reflect newly observed semantics without static constraints.

• Feature compression and memory (OpenMonoGS-SLAM): A memory bank $M$ of representative high-dimensional CLIP vectors enables retrieval of semantic context for newly mapped regions, without storing full dimensions per-Gaussian (Yoo et al., 9 Dec 2025).
• Adaptive semantic cache (OpenFusion++): Instance-level caches of embeddings, weighted by area, dynamically update semantic labels and stabilize descriptors against drift, enabling sharp, up-to-date instance semantics in the 3D map (Jin et al., 27 Apr 2025).

5. Semantic Querying and Open-vocabulary Capabilities

Real-time open-vocabulary querying is a principal application, realized via efficient feature matching pipelines.

• In LEGO-SLAM (Lee et al., 20 Nov 2025), user queries are encoded by a frozen 2D LLM to high-dimensional features, then projected to compact 16-D vectors. Semantic relevancy maps are rendered by splatting cosine similarity scores; full segmentation or label retrieval is enabled by reconstructing high-D features via the decoder and querying a label dictionary.
• In OpenFusion++ (Jin et al., 27 Apr 2025), a dual-path query engine parses textual input to extract fine-grained object/noun embeddings (SEEM text encoder) for coarse matching and leverages environmental context (Alpha-CLIP) for fine re-ranking, supporting arbitrarily nested queries (e.g., “blue vase on the shelf”).
• In FindAnything (Laina et al., 11 Apr 2025), pixel-wise vision-language features are aggregated per object-centric segment; queries are matched via cosine similarity between stored segment embeddings and the query’s CLIP text vector, supporting language-guided exploration even on resource-constrained platforms.

6. Performance Metrics and Experimental Results

Benchmarks focus on mapping quality, semantic accuracy, memory footprint, tracking robustness, and latency.

Selected metrics from LEGO-SLAM (Lee et al., 20 Nov 2025):

• Photorealistic map quality: PSNR up to 36.38 dB (Replica), SSIM up to 0.758 (ScanNet)
• Semantic segmentation: mIoU 0.674 (Replica), 0.650 (TUM), 0.519 (ScanNet) with compact 16-D features
• Tracking: ATE RMSE 0.20 cm (Replica), 2.30 cm (TUM), 8.68 cm (ScanNet)
• Speed: 15 FPS for mapping and rendering (baselines <1 FPS)
• Memory: ∼80 MB for 16-D; >120 MB for 32-D; infeasible for 128-D

Pruning eliminates over 60% of map Gaussians with minimal quality drop (PSNR −0.9 dB, mIoU −0.01), while scene-adaptive encoder ensures resiliency to novel objects and layouts (Lee et al., 20 Nov 2025).

In OpenMonoGS-SLAM (Yoo et al., 9 Dec 2025), open-set mIoU reaches 0.845 on Replica, exceeding prior methods (Feature-3DGS 0.571, Gaussian-Grouping 0.690). Camera-tracking RMSE achieves 1.60 cm, the best among monocular SLAM baselines.

OpenFusion++ (Jin et al., 27 Apr 2025) reports consistent accuracy gains (+4.2pp mAcc) and 15 Hz real-time end-to-end performance. Memory footprint is dramatically reduced via instance-level embeddings.

7. Limitations and Ongoing Challenges

While current systems deliver strong performance and enable real-time open-vocabulary 3D semantic SLAM, several open issues remain:

• Dimensionality bottlenecks: Extremely fine-grained linguistic concepts may exceed the representational capacity of compressed features (e.g., 16-D in LEGO-SLAM). Dynamic adjustment or hierarchical methods could further improve fidelity (Lee et al., 20 Nov 2025).
• Dynamic scene modeling: Most systems optimize for static environments; handling dynamic/moving objects at the per-instance level remains ongoing (necessitating motion modeling and non-rigid scene updates) (Lee et al., 20 Nov 2025, Nasser et al., 1 Dec 2025).
• Semantic label retrieval: Reconstructing high-dimensional features suffices for open-vocabulary segmentation but direct label grounding or promptable interaction with unseen objects requires further advances in memory, label dictionary integration, and LLM-driven planning (Lee et al., 20 Nov 2025, Laina et al., 11 Apr 2025, Jin et al., 27 Apr 2025).
• Resource constraints: Efficient scaling to embedded or robotic hardware, especially for foundation models and real-time inference, is an active research area.

These limitations delineate future research directions aimed at extending semantic SLAM to dynamic, open-world, embodied intelligence scenarios with lifelong semantic adaptation and rich language interaction.