Open-Vocabulary Semantic Maps

• Open-Vocabulary Semantic Maps are structured representations that integrate spatial geometry with semantic information to enable context-aware navigation and reasoning.
• They utilize diverse architectures, including spatial grids, topological graphs, dense point clouds, and hybrid models, each balancing detail and computational cost.
• Fusion methods, such as neural accumulators, Bayesian updates, and semantic histograms, combine vision, sensor, and language data to achieve robust, real-time mapping.

Semantic mapping in indoor embodied AI encompasses techniques for constructing structured representations that jointly encode the geometry and semantics of an environment, supporting embodied agents (such as robots) in tasks that require higher-level reasoning, context-aware planning, and robust interaction with physical spaces. Semantic maps serve as fundamental knowledge substrates, enabling spatial understanding (what is where), object interaction, instruction following, visual navigation, and embodied question answering. These maps integrate observations across time and space, fusing data from RGB-D sensors, semantic detectors, prior scene knowledge, and language-guided modules to build multi-layered topological, metric, and semantic representations of the environment.

1. Foundations and Formal Definitions

A semantic map $M$ is a mathematical structure combining spatial and semantic information. A general formalism expresses a semantic map as

$M = (S,\, E,\, \mathcal{F})$

where $S$ is the spatial substrate (e.g., grid, point cloud, or graph), $E$ encodes topological or relational links, and $\mathcal{F}: S \rightarrow \mathcal{X}$ assigns semantic information to each spatial element. For spatial grid maps, $M$ may be represented as a tensor

$M(x,y,z) = \{ O(x,y,z),\, S(x,y,z) \}$

with $O(x,y,z)\in\{0,1\}$ as occupancy and $S(x,y,z)\in\mathbb{R}^C$ as a semantic feature vector or class probabilities.

For topological maps,

$M = (V,\,E,\,F)$

with $M = (S,\, E,\, \mathcal{F})$0 nodes (places, objects), $M = (S,\, E,\, \mathcal{F})$1 edges (spatial relations), and $M = (S,\, E,\, \mathcal{F})$2 assigning semantic labels or learned embeddings (Raychaudhuri et al., 10 Jan 2025).

These representations support various reasoning operations, including metric localization, semantic querying, navigation to objects of specified type, interaction with scene elements based on affordances, and spatial relationship questions.

2. Representation Architectures

Semantic mapping approaches in indoor embodied AI manifest in a range of structural choices:

2.1 Spatial Grid Maps

Spatial grids discretize the environment into regular cells, each storing occupancy and semantic channels. High-resolution grid maps are widely used for detailed spatial reasoning, but incur quadratic memory growth with increased coverage/resolution. They are suitable for convolutional processing and are especially effective for fine-grained object geometry and obstacle avoidance (Raychaudhuri et al., 10 Jan 2025, Wang et al., 2023). Grid-based semantic mapping pipelines commonly use neural feature accumulators (e.g., GRU/ConvGRU) to fuse projected per-frame observations into global memory tensors, as in SemanticMapNet (Cartillier et al., 2020) and downstream map decoders.

2.2 Topological and Scene Graphs

Topological approaches represent the environment as graphs $M = (S,\, E,\, \mathcal{F})$3 where nodes correspond to locations (room centers, objects, waypoints, landmarks) and edges encode traversability or higher-level relations (adjacency, reachability, semantic affordances). SENT-Map (Kathirvel et al., 5 Nov 2025) introduces semantically enhanced topological graphs where nodes carry structured JSON-encoded semantic fields (object lists, affordances, ownership), facilitating both FM-driven planning and editability.

2.3 Dense Point Clouds

Dense point-cloud maps store millions of 3D points annotated with semantic or feature descriptors (e.g., CLIP embeddings or per-class scores). This paradigm provides high-fidelity 3D geometry with per-point semantics, compatible with SLAM-generated geometric fusion, but is challenged by redundancy, memory footprint, and costly query/fusion operations at large scale (Wang et al., 2023). Projects such as EmbodiedScan provide canonical examples.

2.4 Hybrid Representations

Recent systems combine multiple structures: grid/graph hybrids (StructNav (Chen et al., 2023)), occupancy maps with embedded scene graphs, or spatial grids augmented by parametric region representations ($M = (S,\, E,\, \mathcal{F})$4, with units, types, and relations (Liu et al., 2020)), capturing both metric and relational structure.

Map Type	Local Detail	Global Structure	Memory Cost
Spatial grid	High	Limited	$M = (S,\, E,\, \mathcal{F})$5
Topological graph	Low	High	$M = (S,\, E,\, \mathcal{F})$6
Point cloud	High	None	$M = (S,\, E,\, \mathcal{F})$7
Hybrid	High	High	Mixed

3. Integrating Semantics: Data Sources and Fusion

Semantic information is introduced through modalities including per-pixel segmentation (supervised CNNs, Mask R-CNN), 3D object detection, vision-LLMs (CLIP, BERT), scene context/priors, and human-in-the-loop annotation during mapping. Notable fusion strategies include:

• Feature-level projection: Egocentric visual features are projected into the global grid via known camera geometry. Neural accumulators such as GRUs handle repeated or partial observations, refining uncertain areas (Cartillier et al., 2020).
• Bayesian fusion: Cell-wise class probabilities are recursively updated by combining prior distribution estimates with likelihoods from new observations (Seichter et al., 2022).
• Semantic histograms: Histogram-based fusion accumulates discrete or probabilistic class counts, robust to segmentation noise (Seichter et al., 2022).
• Markov Logic Networks: Contextual prior knowledge is encoded as weighted first-order logic rules, generating soft constraints to regularize mapping via MLN-informed priors (e.g., adjacency, shared wall length, plausible room/corridor relations) (Liu et al., 2020).
• Vision-language alignment: Embedding-level alignment (e.g., CLIP or KB-derived), supports open-vocabulary queries, downstream natural language tasks, and embedding-based semantic reasoning (Bigazzi et al., 2024, Jain et al., 2021, Kathirvel et al., 5 Nov 2025).

4. Inference, Learning, and Efficient Generation

Semantic map construction leverages a spectrum of algorithmic and learning approaches:

• Data-driven learning (CNN/Transformer mappers): End-to-end models such as MapBERT (Deng et al., 9 Jun 2025) and SMNet (Cartillier et al., 2020) learn to directly infer semantic grids from partial or egocentric observations, employing masked modeling (MLM), binary VAEs for efficient bit-level encodings, and Transformer architectures for long-range spatial reasoning. Object-aware masking enhances distributional learning for rare categories.
• Rule-based and probabilistic inference: Systems such as (Liu et al., 2020) combine MCMC sampling, parametric abstraction (rectangular units, room types, adjacency matrices), and semantic observation models, guided by weighted logic rules and context priors.
• Real-time pipelines: S-NDT (Seichter et al., 2022) achieves sub-voxel accuracy and high map rates (>3 Hz) by integrating geometric evidence (NDT cell mean/covariance) with semantic class histograms, suitable for online deployment with modest compute.
• Foundation model integration: SENT-Map (Kathirvel et al., 5 Nov 2025) employs FMs for both semantic extraction (vision FM yielding editable JSON node records) and language-driven planning, allowing closed-loop FM-in-the-loop reasoning.

Performance metrics include per-class and average mean Intersection-over-Union (IoU), boundary-based metrics (mean BF1), task success (SR), SPL, and efficiency measures (frame or map generation rates, memory/compute usage).

5. Role in Downstream Reasoning and Control

Semantic maps underpin numerous high-level embodied tasks:

• Object-Goal and Room-Goal Navigation: Spatial semantics, either by explicit region labeling (Bigazzi et al., 2024, Narasimhan et al., 2020) or latent representations aligned to map features (Hong et al., 2023), guide agents to search for, and navigate toward, specified target categories or rooms. Incorporating semantic priors or region belief maps reduces search complexity and improves SPL/Success compared to geometric or RL-only navigation (Jain et al., 2021, Chen et al., 2023).
• Affordance-based planning and manipulation: Topological graph and SENT-Map representations annotate nodes with action-relevant attributes (affordances, object lists, ownership), enabling FMs or planners to generate feasible multi-step plans, and ensuring grounded execution in the physical environment (Kathirvel et al., 5 Nov 2025).
• Situated question answering: Memory tensors and occupancy grids can be queried for spatial or semantic information, supporting tasks such as counting objects of a type, describing layouts, or finding affordance-relevant objects (Cartillier et al., 2020, Bigazzi et al., 2024).
• Region/room-level mapping: Beyond object-level labels, approaches directly infer region-level semantics (e.g. kitchen, hallway) using vision-LLMs fine-tuned for spatial context, outperforming object-centric heuristics and traditional classifiers (Bigazzi et al., 2024).

6. Challenges, Limitations, and Future Directions

Key challenges persist across representation, learning, and map utilization:

• Memory and Scalability: Grid maps incur a quadratic memory burden with increased physical coverage or resolution; point clouds scale linearly with number of points, and hybrid maps inherit both costs (Raychaudhuri et al., 10 Jan 2025).
• Real-time Consistency: Achieving low-latency updates at sufficient fidelity is non-trivial; S-NDT (Seichter et al., 2022) and MapBERT (Deng et al., 9 Jun 2025) exemplify recent improvements, but dense neural mappers can require extensive GPU resources (Wang et al., 2023).
• Robust semantic fusion: Noisy segmentation, partial observations, and dynamic scene changes challenge the consistency of semantic estimates. Fusion strategies (histograms, Bayesian updates, moving averages) and object-aware reasoning via masking or MLN-guided priors improve map stability but may underperform in open-set or rapidly changing settings (Seichter et al., 2022, Deng et al., 9 Jun 2025).
• Open-vocabulary and continual learning: Integrating foundation models and vision-language embeddings is advancing open-set reasoning (Bigazzi et al., 2024, Kathirvel et al., 5 Nov 2025), but adapting to new categories and online updating of semantic encoders remain open research areas (Raychaudhuri et al., 10 Jan 2025).
• Intrinsic map evaluation: Most work prioritizes downstream task metrics (navigation SR/SPL); standardizing intrinsic map quality metrics (IoU, coverage, region consistency) and community benchmarks is an emerging priority (Raychaudhuri et al., 10 Jan 2025).
• Multi-modal, multi-scale reasoning: Unifying geometric, semantic, and language data in a queryable, task-agnostic map (including foundation model embeddings) is an open area, as is hierarchical map structure for lifelong mapping (Bigazzi et al., 2024, Kathirvel et al., 5 Nov 2025).

7. Benchmark Datasets and Experimental Realizations

Multiple datasets support semantic mapping development:

• Matterport3D, Replica, and EmbodiedScan deliver high-fidelity, per-pixel or per-primitive semantic annotations, with Replica emphasizing photo-realistic mesh/texture and semantic instance structure (Straub et al., 2019), and EmbodiedScan adding multi-modal text-object alignment and dense occupancy (Wang et al., 2023).
• Domain-specific benchmarks (360BEV-Matterport, 360BEV-Stanford) enable evaluation of panoramic→BEV semantic transfer (Teng et al., 2023).
• Evaluation metrics and protocols consistently report mean/classwise IoU, map consistency, and downstream navigation/QA success.

The field is advancing toward compressed, open-vocabulary, foundation model-compatible, queryable map representations, with improved efficiency, robustness, and explainability in unknown or long-term dynamic environments (Deng et al., 9 Jun 2025, Kathirvel et al., 5 Nov 2025, Raychaudhuri et al., 10 Jan 2025).