Open-Vocabulary Semantic SLAM

• Open-vocabulary semantic SLAM is a mapping paradigm that fuses traditional SLAM with language-driven semantic understanding using large-scale vision-language models.
• Its architectures leverage techniques like 3D Gaussian splatting, voxel grids, and dynamic memory banks to achieve robust online mapping and semantic querying.
• The approach enables zero-shot segmentation and language-guided navigation, paving the way for advanced robotics, AR/VR, and autonomous spatial intelligence.

Open-vocabulary semantic SLAM is a paradigm that fuses Simultaneous Localization and Mapping (SLAM) with the capacity for open-ended, language-driven semantic understanding. Unlike traditional closed-set approaches, open-vocabulary semantic SLAM leverages large-scale vision-LLMs to enable real-time mapping, retrieval, and interaction with any object or concept described in natural language, without requiring task-specific retraining or pre-defined label sets. Modern instantiations integrate geometric mapping, foundation models for perception and language, and learned memory mechanisms to produce scalable 3D representations supporting zero-shot semantic querying, instance segmentation, and downstream embodied intelligence.

1. Foundational Principles and Formalization

Open-vocabulary semantic SLAM generalizes classic SLAM by introducing a joint geometry–language feature space over the environment. The core components are:

• Scene Representation: Each 3D point or primitive (e.g., Gaussian splat, voxel, segment) carries, in addition to its spatial, appearance, and physical attributes, a semantic embedding $f_i\in\mathbb{R}^d$ that aligns with a LLM’s concept space.
• Observation Model: Incoming RGB (or RGB-D) frames are processed via foundation models (e.g., CLIP, DINO, SAM) to segment the scene and extract open-vocabulary features. These features supervise both geometric consistency (for tracking and mapping) and semantic feature learning.
• Memory Mechanism: Dynamic memory banks or compact encoders/decoders manage high-dimensional semantic information, supporting real-time inference and scalable storage.
• Querying: Language queries are encoded (e.g., via CLIP text encoder) and matched against the semantic representations in the map using similarity metrics (cosine, attention), yielding semantic segmentation or retrieval over any concept (Yoo et al., 9 Dec 2025).

Mathematically, suppose a set of 3D primitives $\{G_i\}$, where $G_i=(x_i, q_i, s_i, \alpha_i, c_i, f_i)$. The semantic content of a pixel $p$ is rendered as a feature $F_p$ by convex combination of $f_i$ along the ray: $$F_p = \sum_{i=1}^N f_i\,\alpha_i\prod_{j<i}(1-\alpha_j)$$ This is directly compatible with fast differentiable renderers (e.g., Gaussian splatting) and enables unified self-supervised learning of geometry, photometry, and semantics.

2. Architectures and Algorithms

Advanced open-vocabulary semantic SLAM systems build on diverse architectural choices, unified by the integration of vision-language foundation models:

• 3D Gaussian Splatting Pipelines: OpenMonoGS-SLAM and LEGO-SLAM instantiate 3DGS-based SLAM with semantic codes at each Gaussian, optimized end-to-end via combined photometric, contrastive, and language-alignment losses. LEGO-SLAM further compresses language features to 16-D via a scene-adaptive encoder, enabling real-time inference (Yoo et al., 9 Dec 2025, Lee et al., 20 Nov 2025).
• Probabilistic Instance Voxel Representations: OpenVox maintains Dirichlet-distributed instance probabilities at each voxel and a codebook of open-vocabulary instance features. Instance association and live map evolution are solved via Bayesian inference, supporting robust instance-level querying (Deng et al., 23 Feb 2025).
• Neural Implicit Open-Vocabulary Fields: O2V-Mapping fuses object-centric SAM+CLIP language embeddings into a voxel grid supporting local updates, adaptive splitting, and hierarchical semantic segmentation at various scales (Tie et al., 2024).
• Topological and Scene-Graph Approaches: LEXIS and osmAG-LLM employ graph-based maps with semantic annotations on nodes, enabling high-level open-vocabulary spatial reasoning and room-centric retrieval via language-guided graph traversals (Kassab et al., 2023, Xie et al., 17 Jul 2025).
• Monocular and Calibration-Free SLAM: KM-ViPE demonstrates online, self-supervised monocular SLAM with tightly coupled metric geometry and high-level DINO+CLIP features, employing robust loss kernels to handle dynamic objects (Nasser et al., 1 Dec 2025).

Table 1 summarizes representative architectures:

System	Representation	Foundation Models	Semantic Storage	Main Application
OpenMonoGS	3D Gaussians	MASt3R, SAM, CLIP	Memory bank (CLIP)	Monocular mapping & segmentation
LEGO-SLAM	3D Gaussians	CLIP, LSeg	16-D encoder	Real-time RGB-D SLAM
OpenVox	Voxel grid	Yolo-World, SBERT	Dirichlet, codebook	Instance-level querying
OVO	Point cloud, segments	SAM, SigLIP	Neural CLIP merging	Online 3D mapping, loop closure
FindAnything	Vol. submaps	eSAM, CLIP	Obj.-centric slots	Exploration on MAVs
KM-ViPE	Point cloud	DINO, CLIP	PCA+MLP alignment	Monocular, uncalibrated video

3. Integration of Visual Foundation Models

Modern systems exploit the complementarity of foundation models:

• Geometric Backbone: MASt3R, DINO, and Stereo/IMU-based models provide robust 3D geometry estimation, even under monocular setups.
• Open-vocabulary Semantics: CLIP (ViT-B, ViT-L, etc.), SBERT, and similar models generate aligned vision-and-language features that allow semantic supervision without class restrictions.
• Segmentation: Class-agnostic mask generators such as SAM or eSAM enable fine-grained segmentation, facilitating multi-scale object and part reasoning.
• Feature Optimization: End-to-end fine-tuning incorporates CLIP-aligned losses (e.g., language regression, contrastive losses), feature distillation, and, in some cases, online adaptation of encoders for unseen scenes (Yoo et al., 9 Dec 2025, Lee et al., 20 Nov 2025, Tie et al., 2024, Deng et al., 23 Feb 2025).

Attention-based memory retrieval, language-guided feature pruning, and learning-based CLIP merger modules efficiently bridge the gap between high-dimensional foundation model features and the constrained representations needed for real-time mapping (Yoo et al., 9 Dec 2025, Lee et al., 20 Nov 2025, Martins et al., 2024).

4. Semantic Memory and Efficient Retrieval

Semantic memory is critical for open-vocabulary operation under bounded compute and memory:

• Memory Banks: A dynamic bank stores representative concept embeddings (e.g., CLIP, SBERT), augmented only when unique concepts are observed, and pruned based on similarity thresholds or usage (Yoo et al., 9 Dec 2025).
• Compact Feature Encodings: LEGO-SLAM’s online encoder compresses to 16-dimensional semantic codes, enabling fast rendering and low memory per Gaussian (Lee et al., 20 Nov 2025).
• Codebooks and Object Slots: Instance and object-level features are maintained in fused codebooks (OpenVox) or submap slots (FindAnything), associating each region/instance with a canonical feature for query-time similarity computation (Deng et al., 23 Feb 2025, Laina et al., 11 Apr 2025).
• Neural Merging: OVO leverages a small Transformer-MLP to optimally merge multi-view and multi-crop CLIP embeddings per segment (Martins et al., 2024).

Ablations and system analyses demonstrate substantial drops in open-vocabulary segmentation, mIoU, and retrieval accuracy upon disabling these mechanisms, confirming their central role (Yoo et al., 9 Dec 2025, Lee et al., 20 Nov 2025).

5. Performance and Evaluation

Evaluation protocols converge on both geometric and semantic metrics:

• Geometric tracking: Absolute trajectory error (ATE-RMSE) on benchmarks such as Replica and TUM RGB-D. State-of-the-art error of 1.60 cm reported by OpenMonoGS-SLAM for monocular tracking (Yoo et al., 9 Dec 2025).
• Photometric/novel-view synthesis: PSNR, SSIM, and LPIPS on rendered views.
• Semantic segmentation: Mean Intersection over Union (mIoU), frequency-weighted IoU (FWIoU), prompt-based open-set IoU; OpenMonoGS achieves prompt-IoU = 0.845 and closed-set mIoU = 0.896 (Yoo et al., 9 Dec 2025).
• Zero-shot instance segmentation/semantic retrieval: Quantified by AP, recall@k, and qualitative analyses (e.g. for never-seen classes such as “ceiling fan” or attribute/functional queries).
• Runtime: Current best systems deliver 8–15 FPS on high-end GPUs (Lee et al., 20 Nov 2025, Nasser et al., 1 Dec 2025), with embedded versions (MAVs, real robots) achieving 1–3 Hz (Laina et al., 11 Apr 2025).

Ablation studies reveal loss terms (e.g., contrastive, language regression), memory mechanisms, and multi-view fusion to be necessary for maximal semantic accuracy.

6. Applications, Capabilities, and Open Challenges

Open-vocabulary semantic SLAM opens the door to a spectrum of advanced capabilities:

• Zero-shot segmentation and localization: Semantic maps supporting arbitrary linguistic queries, including attributes and function-level references.
• Dynamic and unstructured environments: Robust handling of moving or never-seen objects, achieved via language-level priors, active querying, and online adaptation (Nasser et al., 1 Dec 2025, Xie et al., 17 Jul 2025).
• Navigation and exploration: Language-guided navigation and goal finding in complex, multi-room environments, leveraging semantic utility in planners (Laina et al., 11 Apr 2025, Busch et al., 2024).
• Planning and reasoning: Scene graphs and room-level reasoning enable planners to operate at semantic granularity (e.g., “navigate to the kitchen, then to the sofa”), integrating with high-level task execution (Kassab et al., 2023, Xie et al., 17 Jul 2025).

However, several limitations persist:

• Semantic segmentation accuracy (mIoU) in point-level or monocular-only systems remains modest, especially on rare or fine-grained classes (Nasser et al., 1 Dec 2025).
• Scalability and memory for open-vocabulary labels are nontrivial, though pruning, codebooks, and compact encoders address these partly (Lee et al., 20 Nov 2025, Deng et al., 23 Feb 2025).
• Real-time operation in cluttered or large outdoor scenes, lifelong map updating, and explicit modeling of occlusion or hidden objects remain open research questions.

7. Significance and Future Directions

Open-vocabulary semantic SLAM fundamentally enhances the capability of embodied agents to interact with and understand their environments in a language-agnostic, generalizable manner. By blending geometric SLAM with web-scale foundation models and efficient adaptive memory, these systems can localize, segment, and retrieve arbitrary semantic concepts online, without closed-set limitations or retraining. Advances in latent space alignment, online learning, and robust feature fusion are likely to further bridge the gap between geometric mapping and natural-language-grounded scene understanding, with implications for robotics, AR/VR, and autonomous spatial intelligence (Yoo et al., 9 Dec 2025, Lee et al., 20 Nov 2025, Deng et al., 23 Feb 2025, Nasser et al., 1 Dec 2025).