---
title: Open-Vocabulary Semantic Maps
url: https://www.emergentmind.com/topics/open-vocabulary-semantic-maps
type: topic
---

# Open-Vocabulary Semantic Maps

Semantic mapping in indoor embodied AI encompasses techniques for constructing structured representations that jointly encode the geometry and semantics of an environment, supporting embodied agents (such as robots) in tasks that require higher-level reasoning, context-aware planning, and robust interaction with physical spaces. Semantic maps serve as fundamental knowledge substrates, enabling spatial understanding (what is where), object interaction, instruction following, visual navigation, and embodied question answering. These maps integrate observations across time and space, fusing data from RGB-D sensors, semantic detectors, prior scene knowledge, and language-guided modules to build multi-layered topological, metric, and semantic representations of the environment.

## 1. Foundations and Formal Definitions

A semantic map $M$ is a mathematical structure combining spatial and semantic information. A general formalism expresses a semantic map as
$$
M = (S,\, E,\, \mathcal{F})
$$
where $S$ is the spatial substrate (e.g., grid, point cloud, or graph), $E$ encodes topological or relational links, and $\mathcal{F}: S \rightarrow \mathcal{X}$ assigns semantic information to each spatial element. For spatial grid maps, $M$ may be represented as a tensor
$$
M(x,y,z) = \{ O(x,y,z),\, S(x,y,z) \}
$$
with $O(x,y,z)\in\{0,1\}$ as occupancy and $S(x,y,z)\in\mathbb{R}^C$ as a semantic feature vector or class probabilities.

For topological maps,
$$
M = (V,\,E,\,F)
$$
with $V = \{v_i\}$ nodes (places, objects), $E$ edges (spatial relations), and $F:V\cup E\rightarrow \mathcal{X}$ assigning semantic labels or learned embeddings [2501.05750].

These representations support various reasoning operations, including metric localization, semantic querying, navigation to objects of specified type, interaction with scene elements based on affordances, and spatial relationship questions.

## 2. Representation Architectures

Semantic mapping approaches in indoor embodied AI manifest in a range of structural choices:

### 2.1 Spatial Grid Maps

Spatial grids discretize the environment into regular cells, each storing occupancy and semantic channels. High-resolution grid maps are widely used for detailed spatial reasoning, but incur quadratic memory growth with increased coverage/resolution. They are suitable for convolutional processing and are especially effective for fine-grained object geometry and obstacle avoidance [2501.05750, 2312.16170]. Grid-based semantic mapping pipelines commonly use neural feature accumulators (e.g., GRU/ConvGRU) to fuse projected per-frame observations into global memory tensors, as in SemanticMapNet [2010.01191] and downstream map decoders.

### 2.2 Topological and Scene Graphs

Topological approaches represent the environment as graphs $(V,E)$ where nodes correspond to locations (room centers, objects, waypoints, landmarks) and edges encode traversability or higher-level relations (adjacency, reachability, semantic affordances). SENT-Map [2511.03165] introduces semantically enhanced topological graphs where nodes carry structured JSON-encoded semantic fields (object lists, affordances, ownership), facilitating both FM-driven planning and editability.

### 2.3 Dense Point Clouds

Dense point-cloud maps store millions of 3D points annotated with semantic or feature descriptors (e.g., CLIP embeddings or per-class scores). This paradigm provides high-fidelity 3D geometry with per-point semantics, compatible with SLAM-generated geometric fusion, but is challenged by redundancy, memory footprint, and costly query/fusion operations at large scale [2312.16170]. Projects such as EmbodiedScan provide canonical examples.

### 2.4 Hybrid Representations

Recent systems combine multiple structures: grid/graph hybrids (StructNav [2305.16925]), occupancy maps with embedded scene graphs, or spatial grids augmented by parametric region representations ($W=\{U,T,R\}$, with units, types, and relations [2002.10938]), capturing both metric and relational structure.

| Map Type           | Local Detail | Global Structure | Memory Cost      |
|--------------------|--------------|------------------|------------------|
| Spatial grid       | High         | Limited          | $O(N^2 \cdot C)$ |
| Topological graph  | Low          | High             | $O(|V|+|E|)$     |
| Point cloud        | High         | None             | $O(N\cdot d)$    |
| Hybrid             | High         | High             | Mixed            |

## 3. Integrating Semantics: Data Sources and Fusion

Semantic information is introduced through modalities including per-pixel segmentation (supervised CNNs, Mask R-CNN), 3D object detection, vision-language models (CLIP, BERT), scene context/priors, and human-in-the-loop annotation during mapping. Notable fusion strategies include:

- Feature-level projection: Egocentric visual features are projected into the global grid via known camera geometry. Neural accumulators such as GRUs handle repeated or partial observations, refining uncertain areas [2010.01191].
- Bayesian fusion: Cell-wise class probabilities are recursively updated by combining prior distribution estimates with likelihoods from new observations [2203.05836].
- Semantic histograms: Histogram-based fusion accumulates discrete or probabilistic class counts, robust to segmentation noise [2203.05836].
- Markov Logic Networks: Contextual prior knowledge is encoded as weighted first-order logic rules, generating soft constraints to regularize mapping via MLN-informed priors (e.g., adjacency, shared wall length, plausible room/corridor relations) [2002.10938].
- Vision-language alignment: Embedding-level alignment (e.g., CLIP or KB-derived), supports open-vocabulary queries, downstream natural language tasks, and embedding-based semantic reasoning [2403.07076, 2108.00159, 2511.03165].

## 4. Inference, Learning, and Efficient Generation

Semantic map construction leverages a spectrum of algorithmic and learning approaches:

- Data-driven learning (CNN/Transformer mappers): End-to-end models such as MapBERT [2506.07350] and SMNet [2010.01191] learn to directly infer semantic grids from partial or egocentric observations, employing masked modeling (MLM), binary VAEs for efficient bit-level encodings, and Transformer architectures for long-range spatial reasoning. Object-aware masking enhances distributional learning for rare categories.
- Rule-based and probabilistic inference: Systems such as [2002.10938] combine MCMC sampling, parametric abstraction (rectangular units, room types, adjacency matrices), and semantic observation models, guided by weighted logic rules and context priors.
- Real-time pipelines: S-NDT [2203.05836] achieves sub-voxel accuracy and high map rates (>3 Hz) by integrating geometric evidence (NDT cell mean/covariance) with semantic class histograms, suitable for online deployment with modest compute.
- Foundation model integration: SENT-Map [2511.03165] employs FMs for both semantic extraction (vision FM yielding editable JSON node records) and language-driven planning, allowing closed-loop FM-in-the-loop reasoning.

Performance metrics include per-class and average mean Intersection-over-Union (IoU), boundary-based metrics (mean BF1), task success (SR), SPL, and efficiency measures (frame or map generation rates, memory/compute usage).

## 5. Role in Downstream Reasoning and Control

Semantic maps underpin numerous high-level embodied tasks:

- Object-Goal and Room-Goal Navigation: Spatial semantics, either by explicit region labeling [2403.07076, 2007.09841] or latent representations aligned to map features [2307.12335], guide agents to search for, and navigate toward, specified target categories or rooms. Incorporating semantic priors or region belief maps reduces search complexity and improves SPL/Success compared to geometric or RL-only navigation [2108.00159, 2305.16925].
- Affordance-based planning and manipulation: Topological graph and SENT-Map representations annotate nodes with action-relevant attributes (affordances, object lists, ownership), enabling FMs or planners to generate feasible multi-step plans, and ensuring grounded execution in the physical environment [2511.03165].
- Situated question answering: Memory tensors and occupancy grids can be queried for spatial or semantic information, supporting tasks such as counting objects of a type, describing layouts, or finding affordance-relevant objects [2010.01191, 2403.07076].
- Region/room-level mapping: Beyond object-level labels, approaches directly infer region-level semantics (e.g. kitchen, hallway) using vision-language models fine-tuned for spatial context, outperforming object-centric heuristics and traditional classifiers [2403.07076].

## 6. Challenges, Limitations, and Future Directions

Key challenges persist across representation, learning, and map utilization:

- Memory and Scalability: Grid maps incur a quadratic memory burden with increased physical coverage or resolution; point clouds scale linearly with number of points, and hybrid maps inherit both costs [2501.05750].
- Real-time Consistency: Achieving low-latency updates at sufficient fidelity is non-trivial; S-NDT [2203.05836] and MapBERT [2506.07350] exemplify recent improvements, but dense neural mappers can require extensive GPU resources [2312.16170].
- Robust semantic fusion: Noisy segmentation, partial observations, and dynamic scene changes challenge the consistency of semantic estimates. Fusion strategies (histograms, Bayesian updates, moving averages) and object-aware reasoning via masking or MLN-guided priors improve map stability but may underperform in open-set or rapidly changing settings [2203.05836, 2506.07350].
- Open-vocabulary and continual learning: Integrating foundation models and vision-language embeddings is advancing open-set reasoning [2403.07076, 2511.03165], but adapting to new categories and online updating of semantic encoders remain open research areas [2501.05750].
- Intrinsic map evaluation: Most work prioritizes downstream task metrics (navigation SR/SPL); standardizing intrinsic map quality metrics (IoU, coverage, region consistency) and community benchmarks is an emerging priority [2501.05750].
- Multi-modal, multi-scale reasoning: Unifying geometric, semantic, and language data in a queryable, task-agnostic map (including foundation model embeddings) is an open area, as is hierarchical map structure for lifelong mapping [2403.07076, 2511.03165].

## 7. Benchmark Datasets and Experimental Realizations

Multiple datasets support semantic mapping development:

- Matterport3D, Replica, and EmbodiedScan deliver high-fidelity, per-pixel or per-primitive semantic annotations, with Replica emphasizing photo-realistic mesh/texture and semantic instance structure [1906.05797], and EmbodiedScan adding multi-modal text-object alignment and dense occupancy [2312.16170].
- Domain-specific benchmarks (360BEV-Matterport, 360BEV-Stanford) enable evaluation of panoramic→BEV semantic transfer [2303.11910].
- Evaluation metrics and protocols consistently report mean/classwise IoU, map consistency, and downstream navigation/QA success.

The field is advancing toward compressed, open-vocabulary, foundation model-compatible, queryable map representations, with improved efficiency, robustness, and explainability in unknown or long-term dynamic environments [2506.07350, 2511.03165, 2501.05750].

Source: https://www.emergentmind.com/topics/open-vocabulary-semantic-maps