Open-Vocabulary 3D Semantic SLAM

• Open-vocabulary 3D Semantic SLAM is a framework that integrates sensor data with vision-language models to enable real-time 3D mapping and semantic labeling.
• It utilizes robust representations such as TSDF grids, occupancy maps, or Gaussian splats combined with foundation model descriptors like CLIP for effective feature fusion.
• By incorporating semantic fusion and loop closure, the system supports dynamic robotic navigation, enhanced scene understanding, and responsive language-guided queries.

Open-vocabulary 3D Semantic SLAM unifies metric scene reconstruction and language-grounded scene understanding, enabling robotic agents to map, localize, and interact with their environments via arbitrary semantic queries over geometry. Unlike closed-set pipelines that require predefined class taxonomies, open-vocabulary approaches utilize vision-language representations (e.g., CLIP or similar) to support semantic mapping, navigation, and interaction with objects or regions described in natural language, often in a zero-shot regime. State-of-the-art systems integrate foundation models for mask and descriptor extraction, semantic fusion, and efficient map representations to achieve memory-efficient, real-time, and loop-closure-capable simultaneous localization and mapping.

1. Core Principles and System Architectures

At their core, open-vocabulary 3D Semantic SLAM frameworks combine several modules: metric SLAM (e.g., bundle adjustment, TSDF/occupancy/Gaussian map fusion), vision–language feature extraction (2D or multi-modal descriptors), online or offline semantic label fusion, and query mechanisms mapping language embeddings to 3D map items. The input is typically a stream of RGB-D or stereo images with pose priors from classical SLAM. The front-end tracks camera pose via photometric, geometric, and sometimes semantic residual minimization. Map representations are incrementally maintained, ranging from dense TSDF/occupancy grids (Jiang et al., 18 Mar 2024, Laina et al., 11 Apr 2025, Jin et al., 27 Apr 2025), 3D Gaussian splat maps (Lee et al., 20 Nov 2025), to hierarchical segment graphs or submaps with independent local coordinate frames (Laina et al., 11 Apr 2025).

The semantic mapping front-end aggregates per-frame open-vocabulary features using foundation models, such as CLIP, SigLIP, or LSeg for descriptor extraction, and Segment Anything (SAM/eSAM) or SEEM for instance segmentation. Loop closure and backend global optimization reconcile changes in geometry and semantics, leveraging semantic features for robust place recognition and semantic consistency constraints (Martins et al., 22 Nov 2024, Lee et al., 20 Nov 2025).

Systems such as OpenOcc (Jiang et al., 18 Mar 2024), OVO (Martins et al., 22 Nov 2024), FindAnything (Laina et al., 11 Apr 2025), OpenFusion++ (Jin et al., 27 Apr 2025), and LEGO-SLAM (Lee et al., 20 Nov 2025) exemplify these principles:

System	Scene Geometry	Semantic Representation	Mapping Loop Closure
OpenOcc	Occupancy NeRF	CLIP-distilled 3D feature field	Semantic descriptor matching
OVO	Dense cloud/segment	Learned CLIP fusion per segment	Loop closure via SLAM backend, segment merge
FindAnything	TSDF submaps	Per-segment CLIP aggregation	Submap deformation/re-projection
OpenFusion++	TSDF volume	SEEM/CLIP, cache and dual-path	Pose graph optimization
LEGO-SLAM	3DGS splats	Scene-adaptive compact language embedding	Language-based loop detection

2. Geometric and Semantic Map Representations

Open-vocabulary 3D SLAM pipelines utilize several volumetric or surface-based representations. Occupancy grids and TSDFs remain prevalent for metric geometry, supporting ray casting and efficient updates (Jiang et al., 18 Mar 2024, Jin et al., 27 Apr 2025, Laina et al., 11 Apr 2025). LEGO-SLAM implements a 3D Gaussian Splatting backbone for photorealistic mapping, integrating compact, scene-adaptive per-Gaussian language embeddings via an encoder-decoder framework that distills high-dimensional features (e.g., LSeg) into a 16D space, enabling low-overhead per-Gaussian storage and rendering (Lee et al., 20 Nov 2025).

Semantic features are aggregated either per voxel, segment, or Gaussian, depending on the system. Volume rendering and ray-based fusion (Jiang et al., 18 Mar 2024, Lee et al., 20 Nov 2025), block-level aggregation (OpenFusion++), and per-object segment-level fusion (OVO, FindAnything) are established strategies. Memory efficiency is addressed via active submap partitioning, hashed voxel grids, compact segment-level storage, or feature compression methods (e.g., PCA, quantization).

Submap approaches (FindAnything) afford pose-correctable local volumes, assisting global drift correction and supporting parallel fusion, while segment-centric fusions mitigate memory overhead inherent in per-voxel semantic storage (Laina et al., 11 Apr 2025).

3. Foundation Model Integration and Feature Fusion

Foundation models underpin open-vocabulary semantic capability. The predominant approach is to extract vision-language descriptors using CLIP, SigLIP, or LSeg. Semantic segmentation masks are generated through SAM/eSAM/SEEM or similar, then used to aggregate multi-view feature embeddings for map assignment.

Several systems introduce algorithmic or learned fusion of features:

• OpenOcc distills CLIP-aligned features into a 3D semantic field via volume rendering, supervised with a cosine loss and a semantic-aware confidence propagation (SCP) mechanism that maintains class log-odds to suppress noisy labels (Jiang et al., 18 Mar 2024).
• OVO merges three modes of CLIP embeddings (full image, masked, bounding box crop) through learned per-dimension weights from a transformer+MLP, yielding a representative descriptor per segment. The system caches the best-K views (by mask area) for computational efficiency (Martins et al., 22 Nov 2024).
• OpenFusion++ employs confidence-guided, block-level Bayesian selection and maintains area-weighted adaptive caches per instance to refine global embeddings. Relational or contextual queries are answered with a dual-path encoding (object keyphrase via SEEM, context via Alpha-CLIP) (Jin et al., 27 Apr 2025).
• FindAnything uses eSAM and pixelwise CLIP features tracked through segment IDs, incrementally fusing CLIP descriptors per object segment or voxel by running average or visibility weighting (Laina et al., 11 Apr 2025).
• LEGO-SLAM reduces semantic memory cost by distilling 512D language descriptors to 16D features, adapting the encoder per-scene, and enforcing distillation and SLAM losses in rendering, enabling real-time semantic mapping at 15 FPS (Lee et al., 20 Nov 2025).

4. Semantic Fusion, Loop Closure, and Querying

Semantic SLAM pipelines maintain map consistency and semantic refinement through periodic or event-driven updates, loop closure, and re-projection. Loop closure is performed based on geometric, photometric, and—distinctively—semantic cues. Matching of codebook histograms over per-keyframe semantic features (LEGO-SLAM), global segment descriptors (OpenOcc, OVO), or adaptive cache entries (OpenFusion++) supports robust loop candidate selection, especially in visually ambiguous environments (Jiang et al., 18 Mar 2024, Martins et al., 22 Nov 2024, Lee et al., 20 Nov 2025).

Semantic fusion mechanisms aggregate features over time and views, using confidence weighting, log-odds propagation, or cache-based multi-view selection to address outlier measurements and improve label stability. Efficient masking and depth-alignment corrections mitigate label “bleeding” and support high-precision object association (Martins et al., 22 Nov 2024, Jin et al., 27 Apr 2025).

Arbitrary language queries are mapped to text-encoded features, and similarity metrics (usually cosine) are used to retrieve relevant points, segments, or Gaussians from the map. Recent systems support complex, relational, and context-enhanced queries (e.g., “the chair near the window”), employing dual-path encoders and staged filtering for high precision (Jin et al., 27 Apr 2025). This enables targeted exploration, region-centric navigation, or mobile manipulation based on user-specified goals or natural instructions (Qiu et al., 26 Jun 2024).

5. Memory, Computational Efficiency, and Online Operation

Real-time operation with open-vocabulary semantics requires careful management of memory, compute, and update strategies. Systems report TSDF submap sizes on the order of 80 MB for 50 m³ at 5 cm resolution, with segment-level semantic storage at ~300 KB per submap (Laina et al., 11 Apr 2025). LEGO-SLAM achieves map memory of 82–257 MB for major datasets at 16D descriptors, contrasting with order-of-magnitude higher usage for non-compressed baselines (Lee et al., 20 Nov 2025).

Online variants (e.g., OVO, FindAnything, OpenFusion++) minimize CLIP compute via best-view selection, asynchronous processing queues, and active segment culling. Computational strategies include mini-batch updates, sparse voxel hashing, map pruning, and hybrid CPU/GPU execution. FindAnything demonstrates real-time mapping (7–8 min/scene on RTX 3060) and embedded capability on Jetson Orin NX at 3 Hz, with full mapping loops running at up to 15 FPS for LEGO-SLAM (Laina et al., 11 Apr 2025, Lee et al., 20 Nov 2025).

Key elements enabling online, scalable operation include submap partitioning, dynamic pruning for semantic redundancy, segment-centric data models, and efficient foundation model inference pipelines.

6. Quantitative Performance and Experimental Findings

Systems demonstrate robust open-vocabulary mapping and segmentation on standard datasets, typically in a zero-shot setting. Representative results include:

Dataset	Method	mIoU (%)	mAcc (%)	Notable Results
Replica	OVO–ORB-SLAM2	25.6	39.0	Real-time, with loop closure (Martins et al., 22 Nov 2024)
Replica	OpenFusion++	44.5	62.97	+6.29 mAcc over baseline (Jin et al., 27 Apr 2025)
Replica	FindAnything	62.9*	48.8*	Outperforms Octree-Graph (55.3*) (Laina et al., 11 Apr 2025)
ScanNet	OpenFusion++	64.4	67.62	Query accuracy >93% (Jin et al., 27 Apr 2025)
ScanNet-200	OpenOcc	17.5	26.8	Long-tail object gains (e.g., fan: 29.9%) (Jiang et al., 18 Mar 2024)
Replica	LEGO-SLAM	67.4	88.2	Sub-cm ATE, 15 FPS (Lee et al., 20 Nov 2025)

(*FindAnything uses class-mean recall mAcc and freq.-weighted mIoU.)

Experimental ablations demonstrate that learned fusion methods (e.g., per-dimension CLIP weighting in OVO, area-weighted caches in OpenFusion++) outperform hand-tuned or naive aggregation. Pruning based on language and geometry preserves reconstruction quality while reducing resource footprint.

7. Application Domains, Extensions, and Limitations

Open-vocabulary 3D Semantic SLAM is foundational for applications in autonomous navigation, mobile manipulation, embodied intelligence, and AR/VR. Systems have been demonstrated on mobile robots and MAVs in real-world indoor scenes, supporting tasks such as natural language object retrieval, room-aware exploration, and online replanning under dynamic environments (Qiu et al., 26 Jun 2024, Laina et al., 11 Apr 2025).

Plug-in compatibility with a range of SLAM back-ends (e.g., ORB-SLAM2/3, VINS-Mono, Gaussian-SLAM, OKVIS2) allows rapid integration and mobility platform deployment (Martins et al., 22 Nov 2024, Laina et al., 11 Apr 2025). Extensions include higher-level region or room semantics, hierarchical mapping, dynamic scene filtering, model compression for embedded deployment, and adaptation to multimodal foundation models (e.g., LSeg, OpenCLIP, RegionCLIP, LASER-based segmenters).

Key limitations concern memory/computation trade-offs for dense volumetric representations, real-time handling of long sequences or massive spaces, and the semantic noise/ambiguity introduced by vision-LLMs. Ongoing research addresses these via submap partitioning, language embedding compression (e.g., 16D in LEGO-SLAM), cache- or heap-based best view selection, dynamic pruning, and map deformation on pose correction (Lee et al., 20 Nov 2025, Laina et al., 11 Apr 2025).

A plausible implication is that as foundation models improve in multimodal understanding and efficiency, open-vocabulary semantic SLAM performance, robustness, and generalization will further advance, deepening integration with embodied and interactive AI systems.