OpenMap: Zero-Shot 3D Nav Mapping

• OpenMap is a zero-shot, open-vocabulary framework that constructs 3D instance maps by aggregating RGB-D observations using a structural-semantic consensus.
• It combines a two-stage pipeline that performs multi-view instance formation followed by LLM-assisted grounding of free-form instructions.
• Empirical evaluations on ScanNet200 and Matterport3D show significant improvements in semantic accuracy and retrieval success compared to prior methods.

OpenMap is a zero-shot open-vocabulary visual-language mapping framework for embodied navigation that grounds free-form natural-language instructions to concrete 3D scene instances. It was introduced as a response to two coupled deficiencies in prior systems: weak instance-level semantic consistency during multi-view 3D mapping, and overly restrictive instruction grounding pipelines that collapse free-form commands into fixed label vocabularies. In its intended formulation, OpenMap constructs a 3D instance map from RGB-D observations and then performs LLM-assisted instruction-to-instance grounding, so that commands such as object-goal, demand-driven, and language-guided instructions can be resolved to specific targets in a scene (Li et al., 3 Aug 2025).

1. Scope and nomenclature

In the embodied-navigation literature, OpenMap denotes the framework introduced in “OpenMap: Instruction Grounding via Open-Vocabulary Visual-Language Mapping,” not a generic open map platform and not a synonym for OpenStreetMap. The term is used differently in other domains: “OpenMap” can refer to OpenEarthMap, a benchmark for global high-resolution land-cover semantic segmentation (Xia et al., 2022), and “OpenMAP” also appears in OpenMAP-BrainAge, a transformer-based brain-age predictor for T1 MRI (Kan et al., 21 Jun 2025). A precise reading therefore requires domain context.

Within embodied AI, OpenMap addresses the problem of grounding instructions such as “go to the lounge chairs,” “I want to eat,” or “prepare the chair, I want to eat” to a specific 3D scene instance. The framework assumes that open-world navigation failure is driven less by the absence of broad visual-language knowledge than by two systems-level bottlenecks: multi-view instance aggregation that either over-merges or fragments objects, and language grounding that reduces rich instructions to closed-vocabulary label retrieval. OpenMap is therefore defined by the joint treatment of 3D instance formation and instruction grounding, rather than by either component in isolation (Li et al., 3 Aug 2025).

2. Input representation and overall pipeline

OpenMap takes as input RGB-D observations collected during exploration, written as

$$\mathcal{I}=\{I_1, I_2, \dots, I_T\}, \qquad \mathcal{D}=\{D_1, D_2, \dots, D_T\}.$$

For each frame $I_t$, an off-the-shelf 2D segmentation model predicts masks $\{m_i^t\}$, and a vision-LLM extracts open-vocabulary mask features $\{f_i^t\}$. Each mask is back-projected through the depth map into a 3D point cloud, so the pipeline begins from a sequence of 2D region proposals with associated 3D geometry and semantic embeddings.

The architecture is explicitly split into two stages. The first stage constructs an open-vocabulary 3D instance map. It uses CropFormer for complete object masks, back-projects masks to 3D, uses OVSAM features for efficient merge-time semantic comparison, and then applies an iterative cross-view merging process. After instance formation, the framework performs completeness-guided semantic aggregation with CLIP ViT-H to produce final instance embeddings. The second stage performs instruction-to-instance grounding. An LLM first rewrites the navigation instruction into a concise target description; the system then retrieves semantically similar candidate instances and invokes a second LLM pass that uses spatial context and nearby-object information to choose the intended target (Li et al., 3 Aug 2025).

This division is operationally important. OpenMap is not merely a scene-graph retriever over static labels, and it is not merely a 3D segmentation system. Its map is a set of 3D instances endowed with geometry, spatial location, and open-vocabulary semantic features suitable for downstream grounding.

3. Structural-Semantic Consensus and 3D instance formation

The central mapping contribution is the Structural-Semantic Consensus constraint. OpenMap begins from the observation that geometry-only aggregation can merge nearby but distinct objects, while semantics-only aggregation can confuse visually related but physically separate instances. It therefore requires agreement in both structure and semantics before merging masks across views.

For each mask $m_k$, OpenMap defines the set of images in which it is observable, $\mathcal{I}(m_k)$, using an overlap threshold $\tau_{obs}$. For a pair of masks $m_i$ and $m_j$, the common observer set is

$$O(m_i,m_j) = \mathcal{I}(m_i)\cap \mathcal{I}(m_j).$$

Among these observers, OpenMap identifies supporter images whose masks spatially encompass both point clouds, using a containment threshold $I_t$0:

$I_t$1

The structural consensus rate is then

$I_t$2

Semantic agreement is defined through cosine similarity of VLM features:

$I_t$3

The merge criterion is multiplicative:

$I_t$4

This gating rule is the essential mechanism by which OpenMap reduces both over-merging and under-merging. Structural support across multiple views is used to preserve geometric consistency, while semantic similarity prevents the fusion of physically adjacent but semantically distinct instances.

Merging is iterative and robust-first. A mask pair is considered only if it satisfies both the consensus threshold and an observer-count condition $I_t$5. The threshold $I_t$6 starts at the top 5% of all mask pairs by observer count and is reduced by 5% each iteration, so merges with stronger multi-view support are accepted earlier. After each merge, structural statistics are recomputed, and the semantic feature of the merged instance is updated by selecting the mask that most completely captures the new point cloud. Final instance features are not taken from a single observation; OpenMap selects the top-$I_t$7 most complete masks, extracts $I_t$8 multi-level crops per mask, encodes them with CLIP ViT-H, and average-pools the resulting $I_t$9 features into a holistic instance descriptor. The reported implementation uses $\{m_i^t\}$0, $\{m_i^t\}$1, and $\{m_i^t\}$2 (Li et al., 3 Aug 2025).

4. Instruction-to-instance grounding

OpenMap’s second contribution is an LLM-assisted grounding module designed for instructions that are indirect, functional, or context-dependent. The framework does not force the instruction into a fixed object dictionary. Instead, it uses two LLM rounds.

In the first round, the LLM rewrites the navigation command into a concise but fine-grained target description. The prompting criteria are explicitly designed to preserve descriptive content from the original instruction while inferring additional characteristics needed for disambiguation, but also to keep the output short enough for VLM text encoding. If the resulting text embedding is $\{m_i^t\}$3 and mapped instance embeddings are $\{m_i^t\}$4, OpenMap ranks instances by semantic similarity

$\{m_i^t\}$5

The top $\{m_i^t\}$6 candidates are retained; the implementation uses $\{m_i^t\}$7.

In the second round, OpenMap adds map-aware spatial context. For each candidate instance $\{m_i^t\}$8, a KD-tree retrieves the $\{m_i^t\}$9 nearest surrounding instances within a 2-meter radius; the implementation uses $\{f_i^t\}$0. These nearby objects are annotated with labels, for example from LVIS, specifically to support LLM reasoning about context. The prompt supplied to the second LLM includes the candidate’s location $\{f_i^t\}$1, its semantic similarity score $\{f_i^t\}$2, and the labels and coordinates of neighboring instances. The LLM then selects the intended instance in a multiple-choice setting.

This design allows OpenMap to handle cases where the target is defined relationally rather than categorically. A chair “for eating” can be selected by its proximity to a dining table; a demand-driven instruction such as “I am thirsty” can be reframed toward an instance consistent with the implied task. The grounding module is therefore not just text-to-embedding retrieval; it is retrieval followed by explicit spatial-context reasoning over the 3D map (Li et al., 3 Aug 2025).

5. Benchmarks, evaluation protocol, and empirical results

OpenMap is evaluated on two tasks. For open-vocabulary semantic mapping, it uses the ScanNet200 validation split, which contains 312 indoor scans and 200 categories with head, common, and tail subsets. For instruction-to-target retrieval, it uses Matterport3D Semantics over 20 scenes, comprising 11 scenes from R2R-CE val-unseen and 9 scenes from the VLMap evaluation dataset, with instructions drawn from R2R-CE, VLMap, and ALFRED. The retrieval task spans object-goal, demand-driven, and language-guided instructions (Li et al., 3 Aug 2025).

For mapping, OpenMap reports semantic AP, $\{f_i^t\}$3, $\{f_i^t\}$4, head/common/tail AP, and class-agnostic AP variants. On ScanNet200, OpenMap achieves semantic AP $\{f_i^t\}$5, $\{f_i^t\}$6, $\{f_i^t\}$7, class-agnostic AP $\{f_i^t\}$8, class-agnostic $\{f_i^t\}$9, and class-agnostic $m_k$0. Relative to MaskClustering, this corresponds to improvements from 12.0 to 14.3 in semantic AP and from 19.0 to 19.8 in class-agnostic AP. Relative to OVIR-3D, semantic AP improves from 9.3 to 14.3 and class-agnostic AP from 14.4 to 19.8. OpenMap also reports head/common/tail AP values of 14.5, 13.8, and 14.7 respectively, indicating a narrow spread across frequent and infrequent categories (Li et al., 3 Aug 2025).

For retrieval, a prediction is counted as correct if the selected target lies within 1 meter of the ground-truth center. OpenMap reports first-shot success rate $m_k$1 and top-$m_k$2 variants $m_k$3, $m_k$4, and $m_k$5. Its reported performance is $m_k$6, $m_k$7, $m_k$8, and $m_k$9. The corresponding numbers for NLMap are 25.1, 28.4, 31.5, and 37.1; for VLMap, 27.2, 29.7, 32.1, and 37.5; and for ConceptGraphs, 40.9, 43.4, 50.6, and 54.9. The strongest practical gain is in first-attempt retrieval, where OpenMap improves over ConceptGraphs by 8.7 points in $\mathcal{I}(m_k)$0 (Li et al., 3 Aug 2025).

The ablation studies are tightly aligned with the method’s two-part claim. For mapping, using structure only gives AP $\mathcal{I}(m_k)$1, using semantics only gives AP $\mathcal{I}(m_k)$2, and using both yields AP $\mathcal{I}(m_k)$3. For grounding, removing both instruction parsing and context-aware instance selection reduces $\mathcal{I}(m_k)$4 from 49.6 to 38.1. Keeping parsing but removing selection yields 47.2, while keeping selection but removing parsing yields 44.7. These results support the interpretation that OpenMap’s gains come from both better 3D instance formation and more expressive instruction interpretation (Li et al., 3 Aug 2025).

6. Relation to adjacent map paradigms, limitations, and significance

OpenMap belongs to a broader research shift in which maps are no longer treated as closed-vocabulary geometric substrates alone. Related work illustrates distinct but neighboring directions. Chatmap uses synthetic supervision to turn OpenStreetMap neighborhood summaries into a natural-language interface over local urban context (Unlu, 2023). P-MapNet uses OpenStreetMap as a weakly aligned SDMap prior for online HD map generation in autonomous driving, emphasizing long-range structure and attention-based fusion (Jiang et al., 2024). Map It Anywhere uses Mapillary and OpenStreetMap to create 1.2 million FPV–BEV pairs for generalizable BEV semantic map prediction from public data (Ho et al., 2024). OpenMap differs from these systems in its target problem: its map is explicitly an open-vocabulary 3D instance map designed for embodied instruction grounding rather than land-cover labeling, urban QA, BEV supervision, or HD lane-map refinement.

Its limitations are equally specific. The mapping stage depends on the quality of the initial 2D masks; poor segmentation propagates downstream. Despite strong zero-shot semantic results, OpenMap still trails fully supervised systems such as Mask3D in class-agnostic mask quality. The grounding stage is prompt-sensitive because both rounds depend on LLM behavior. The second-stage contextual reasoning uses fixed labels, such as LVIS labels for surrounding objects, even though the query side remains open-vocabulary. The method also lacks an explicit trainable probabilistic model for final candidate fusion, delegating that decision to prompted LLM selection. Finally, incomplete scans remain a practical constraint: if a target instance or its relational context is weakly observed, retrieval reliability degrades (Li et al., 3 Aug 2025).

The significance of OpenMap lies in the way it binds open-vocabulary semantics, 3D instance formation, and free-form instruction grounding into one zero-shot system. A common misconception is that embodied instruction grounding is solved once a strong VLM or LLM is available. OpenMap shows that this is insufficient without a map that preserves instance identity across views and without a grounding mechanism that can reinterpret language relationally rather than as a label lookup. In that sense, OpenMap is best understood as a systems-level synthesis: a visual-language map whose utility emerges from the joint design of cross-view instance consensus and context-aware instruction grounding.