Semantic Mapping in Indoor Embodied AI

• Semantic mapping in indoor embodied AI is a method that encodes both spatial structures and semantic labels to support localization, navigation, and task planning.
• It integrates diverse representations such as spatial grid maps, topological graphs, and point-cloud fusions to provide detailed scene understanding.
• Recent advances leverage foundation models and transformer-based methods to achieve real-time, open-vocabulary mapping with enhanced inference and scalability.

Semantic mapping is a foundational capability in indoor embodied AI, enabling agents to localize, navigate, interact, and reason about their environments through representations that unify geometric and semantic information. This article provides a technical overview of the principles, representations, inference methods, and empirical results central to semantic mapping in the context of embodied indoor agents, integrating perspectives from recent research literature.

1. Formal Definitions and Objectives

A semantic map in the indoor embodied AI context is a data structure that encodes both the geometric configuration of the environment and semantic information about its regions, objects, and spatial relations. Canonically, such a map can be defined as $M = (S, E, \mathcal{F})$ where:

• $S$ is the spatial substrate—grid, graph, or set of 3D points.
• $E$ are topological or spatial relations (e.g., adjacency, connectivity).
• $\mathcal{F}: S \to \mathcal{X}$ assigns a semantic label or feature vector to each spatial element.

For grid maps: $$M = \{\,O(x,y,z),\; S(x,y,z)\,\mid\; (x,y,z)\in\mathbb{Z}^3\}$$ with $O(x,y,z)$ as occupancy indicators and $S(x,y,z)$ as semantic class vectors.

For topological maps: $M = (V,\,E,\,F)$ where $V$ are spatial nodes, $E$ connectivity edges, and $S$0 semantic information per node or edge (Raychaudhuri et al., 10 Jan 2025).

The overarching goal is to support robust long-horizon reasoning, efficient navigation, and semantic querying by providing a representation that captures "what is where" (and, increasingly, "what can be done where") in the environment.

2. Structural Map Representations

2.1 Spatial Grid Maps

Spatial grids discretize the environment into regular cells or voxels, storing per-cell occupancy and semantic information. For 2D mapping, grids of size $S$1 (with $S$2 semantic classes or features) are standard. Semantic labels (e.g., object or region classes) are usually assigned through maximum likelihood, Bayesian fusion, or temporal filtering as new observations arrive (Cartillier et al., 2020, Seichter et al., 2022, Wang et al., 2023).

2.2 Topological Graphs & Scene Graphs

Topological maps abstract space as navigational or semantic nodes (rooms, landmarks, objects), with connectivity described by a graph $S$3. Each node may hold attributes—pose, object inventory, affordances (Kathirvel et al., 5 Nov 2025). Edges encode adjacency or reachability (including after manipulation actions). Topological approaches support low-memory, abstracted planning and efficient grounding of semantic queries but lack dense geometric resolution.

2.3 Point-Cloud and Hybrid Maps

Dense point-clouds fuse 3D observations (from SLAM or RGB-D) into sets $S$4, where $S$5 may be a semantic label or learned feature vector. Hybrid maps combine grid, graph, and point representations to interleave local metric detail with global semantic and topological structure (Chen et al., 2023).

A survey of these structures, their memory and inference tradeoffs is summarized below (Raychaudhuri et al., 10 Jan 2025):

Structure	Memory	Strengths / Limitations
Grid	$S$6	High local detail, poor scalability
Topological graph	$S$7	Abstraction, efficient for planning
Point-cloud	$S$8	3D fidelity, per-point semantics, memory intensive
Hybrid	$S$9(sum)	Combines strengths, increased system complexity

3. Semantic Fusion and Uncertainty Modeling

Semantic mapping requires robust data association and uncertainty handling due to noise in perception, dynamic changes, and partial observability. Approaches include:

3.1 Bayesian and Histogram Fusion

Maps accumulate semantic beliefs per cell/node through direct accumulation (semantic histogram), Bayesian updates, or moving-average blending (Seichter et al., 2022, Bigazzi et al., 2024). For example, in Occupancy-NDT grids, each cell maintains a class histogram $E$0 updated per new observation, with final class probabilities $E$1.

3.2 Projective Geometry and Memory Accumulation

State-of-the-art grid mappers project egocentric semantic segmentation features (from CNNs or Transformers) onto global maps using camera intrinsics and odometry, followed by memory units (GRUs or similar) that integrate observations over time (Cartillier et al., 2020, Hong et al., 2023). This enables feature-level map refinement and temporal consistency.

3.3 Structured Priors: Language, Context, and Logic

Semantic reasoning can be augmented by structural priors derived from:

• Pretrained language or knowledge-base embeddings guiding search (e.g., cosine similarity for subgoal selection (Jain et al., 2021)).
• Markov Logic Networks encoding architectural constraints (e.g., room adjacency, shared-wall rules) to sharply prune invalid map candidates during inference (Liu et al., 2020).
• Scene co-location statistics and language priors to score unexplored map frontiers (Chen et al., 2023).

4. Advancements in End-to-End and Open-Vocabulary Mapping

Recent approaches leverage foundation models and transformers for high-level, expressive, and scalable mapping:

4.1 Bitwise & Transformer-based Generation

MapBERT introduces "bitwise masked modeling," using a BitVAE encoder to map one-hot semantic maps into compact binary tokens, which a masked transformer then completes in real-time, inferring plausible layouts for unobserved regions (Deng et al., 9 Jun 2025). Object-aware masking strategies further enhance object-centric spatial reasoning, outperforming prior GAN-based or convolutional imputation methods on the Gibson benchmark.

4.2 Region Mapping without Explicit Object Detection

New pipelines bypass object recognition, mapping directly to high-level semantic regions (e.g., "kitchen," "corridor"). These systems fuse vision-language features, projected via global pose, into a per-cell region-label belief, updating maps with either moving averages or Bayesian fusion. Injecting CLIP-derived features and finetuning with supervised contrastive loss yields state-of-the-art masked accuracy and IoU for region mapping (Bigazzi et al., 2024).

4.3 Foundation Model Integration for Task Planning

Systems like SENT-Map encode semantic maps as editable JSON scene graphs, with nodes carrying object, affordance, and ownership metadata, enabling downstream planners (LLMs) to ground and validate high-level action sequences before execution (Kathirvel et al., 5 Nov 2025).

5. Evaluation Protocols and Empirical Performance

Evaluation of semantic mapping pipelines encompasses both intrinsic map quality and downstream embodied reasoning performance.

5.1 Map Quality Metrics

• Mean Intersection-over-Union (mIoU): Standard for segmentation and occupancy maps, reflecting per-class agreement (e.g., 45.78% mIoU, 360Mapper on 360BEV-Stanford (Teng et al., 2023); 20.8% mIoU, EmbodiedScan on 80-way semantic occupancy (Wang et al., 2023)).
• Masked/Explored Pixel Accuracy: Fraction of semantic predictions over explored regions matching ground-truth (Bigazzi et al., 2024).
• Boundary-F1 Score: Used to quantify boundary adherence in semantic segmentations (Cartillier et al., 2020).

5.2 Downstream Navigation and Reasoning

Agents exploiting semantic maps for navigation (ObjectNav, RoomNav) and spatial QA show significant gains over sensor-only or RL-only baselines, with supervised or contrastive map learning improving both success rates and SPL (Success weighted by Path Length) (Hong et al., 2023, Narasimhan et al., 2020).

Ablation studies highlight the dependence of navigation performance on both object semantics (SPL drops ∼4% without object cues) and map completeness (oracle maps nearly double navigation SPL versus learned maps) (Hong et al., 2023, Narasimhan et al., 2020).

6. Computational Efficiency, Scalability, and Open Problems

Efficient real-time mapping is critical for deployment on resource-constrained robots:

• Semantic NDT (Normal Distribution Transform) grids achieve up to 5× speedup over kernel-based methods on 10 cm voxel grids (3.5 Hz vs 0.9 Hz on Hypersim), with higher IoU and sub-voxel accuracy (Seichter et al., 2022).
• MapBERT achieves real-time inference at 90 FPS for semantic map generation, with negligible computational overhead compared to prior methods (Deng et al., 9 Jun 2025).
• End-to-end systems must balance memory—scalability tradeoffs, especially with high-resolution (O($E$2)) grid maps or point-cloud fusion (O($E$3) with $E$4 up to $E$5) (Raychaudhuri et al., 10 Jan 2025).

Persistent challenges include: robust mapping in lifelong/dynamic environments, open-vocabulary semantic labeling, transfer to real-world visual distributions, and multi-modal fusion (vision, language, affordance, etc.).

7. Future Directions

Open research avenues identified in the recent survey literature include:

• Development of open-vocabulary, queryable semantic maps via joint visual-language embedding.
• Creation of general-purpose, task-agnostic maps reusable across planning, QA, and manipulation.
• Advancements in dynamic & lifelong mapping—online updates, sparsification, and continual learning.
• Standardization of intrinsic map quality metrics (map-IoU, completeness, confidence) to facilitate benchmarking.
• Integration of foundation models for semantic extraction, mapping, and high-level planning with explicit map grounding (Raychaudhuri et al., 10 Jan 2025, Kathirvel et al., 5 Nov 2025).

Collectively, semantic mapping in indoor embodied AI is progressing toward scalable, robust, and richly annotated representations that empower agents with deep scene understanding, effective task execution, and human-compatible reasoning.