Open-Vocabulary Object-Goal Navigation

• The paper shifts from closed-set navigation to free-form language grounding, enabling agents to interpret complex object descriptions for contextual search.
• It employs multimodal embeddings to fuse visual and textual cues, supporting detailed queries at scene, room, region, and instance levels.
• Practical insights include diverse mapping, memory, and exploration strategies that tackle semantic generalization and efficiency challenges.

Open-vocabulary object-goal navigation is an embodied navigation setting in which an agent must locate an object specified through free-form language in an unseen environment, rather than search over a fixed closed set of benchmark categories. In the literature, this setting is often abbreviated as OVON, while some papers also describe it as open-world or zero-shot ObjectNav when the system must generalize to arbitrary language goals without task-specific training on those categories (Yokoyama et al., 2024, Majumdar et al., 2022). The topic has expanded from category-level targets such as “sink” or “sofa” toward finer-grained language grounding, including room-qualified goals, relational descriptions, sequential multi-goal search, interactive personalization, and lifelong reuse of semantic memory (Miao et al., 2 Feb 2026).

1. Task definition and semantic scope

In its standard form, Object Goal Navigation asks an embodied agent to navigate through an unknown environment and stop near an instance of a target object. HM3D-OVON makes the defining shift from a closed set of 6–20 categories toward free-form language at test time, with goals represented as a goal object category string $G$ and encoded with a frozen SigLIP text encoder inside the benchmark’s main policy architecture (Yokoyama et al., 2024). ZSON sharpened the distinction between open-world and zero-shot: open-world denotes support for arbitrary language descriptions at deployment, while zero-shot denotes the absence of ObjectNav rewards or demonstrations during training (Majumdar et al., 2022).

The semantic range of the task has broadened substantially. ZSON showed that a policy trained on ImageNav can be transferred to semantic navigation by embedding goal images and text in the same CLIP space, allowing queries such as “sink,” “bathroom sink,” “kitchen sink,” and compound instructions like “Find a sink and a stove” (Majumdar et al., 2022). HieraNav extends this further by defining a four-level hierarchy of scene, room, region, and instance targets. In that formulation, a scene-level goal means finding any object instance of a category in the scene, a room-level goal adds the room type, a region-level goal disambiguates a room instance with contextual cues, and an instance-level goal identifies a unique object through attributes or relations such as “white bed with teal runner” or “armchair beside the bed and balcony” (Miao et al., 2 Feb 2026). The paper explicitly states that this instance level corresponds most directly to open-vocabulary object-goal navigation.

Input and action specifications vary across systems, but the dominant benchmark protocol uses RGB or RGB-D observations, pose or odometry when available, and a discrete action set that includes forward motion, turns, and stop. HM3D-OVON uses a Stretch-like simulated agent with RGB-D images, relative pose $P_t=(\Delta x,\Delta y,\Delta\theta)$, and success defined by issuing STOP within 1 m of a goal object within 500 time steps (Yokoyama et al., 2024). Other methods intentionally remove privileged modalities: Nav-$R^2$ uses only first-person RGB and no depth or map (Xiang et al., 2 Dec 2025), while OVSegDT reports RGB-only open-vocabulary navigation without depth, odometry, or large vision-LLMs (Zemskova et al., 15 Aug 2025). This divergence reflects a central axis of the field: some approaches emphasize explicit geometry and mapping, whereas others emphasize end-to-end semantic control.

2. Benchmarks and evaluation regimes

HM3D-OVON established the first large-scale benchmark explicitly centered on open-vocabulary ObjectNav in photorealistic real-world scans. It incorporates over 15k annotated instances across 379 distinct categories, generates 50k episodes per scene for 145 training scenes and 3k episodes per scene for 36 validation scenes, and separates evaluation into $v_{seen}$, $v_{ue}$, and $v_{uh}$, where $v_{ue}$ contains unseen but semantically similar categories with SentenceBERT similarity in $[0.68,0.96]$ and $v_{uh}$ contains semantically distant unseen categories with similarity in $[0.45,0.68]$ (Yokoyama et al., 2024). This design evaluates semantic generalization directly rather than only scene generalization.

LangMap and HieraNav broaden the benchmark problem beyond category retrieval. LangMap is built on all 36 HM3D-Sem validation scenes and contains over 18K navigation tasks, including about 15K non-redundant single-goal tasks and 720 multi-goal episodes comprising 3.6K individual tasks. It adds 12 room categories, 926 discriminative region descriptions, and 7,510 instance descriptions spanning 414 object categories, with both concise and detailed target descriptions for each task (Miao et al., 2 Feb 2026). Its annotation protocol is explicitly contrastive and human-verified, and the paper reports that LangMap’s instance descriptions reach 79.7% discriminative accuracy versus 55.9% for GOAT-Bench in a one-to-many text-to-view matching setup while using 5.2 average words instead of 21.1, together with an exclusive win rate of 29.1% (Miao et al., 2 Feb 2026).

Benchmarking has also diversified along multi-goal and multi-modal axes. LagMemo introduces GOAT-Core, a curated subset of GOAT-Bench with 4 representative scenes, all tasks on a single floor, 20 average subtasks per episode, 13.37 unique categories per episode, 6.89 m inter-subtask distance, and 480 subtasks total, covering 163 image goals, 158 object goals, and 159 text goals (Zhou et al., 28 Oct 2025). OneMap introduces a zero-shot multi-object navigation benchmark on top of Habitat ObjectNav / HM3D with 236 episodes, 20 scenes, 6 object categories, and 3 goals per episode, explicitly preventing environment reset between queries so that semantic memory can be reused (Busch et al., 2024). These changes shift evaluation away from isolated single-query search toward repeated search within a persistent environment.

3. Map construction, semantic memory, and representation learning

A large fraction of OVON research is organized around the question of how semantic information should be stored and queried. ZSON represents goals in a common multimodal embedding space built from CLIP image and text encoders, trains a navigation policy on ImageNav, and then replaces image-goal embeddings with language-goal embeddings at test time (Majumdar et al., 2022). OVExp instead treats exploration as prediction over a top-down map in CLIP space: during training it uses low-cost categorical semantic maps from 92 HM3DSem object categories and transforms them into CLIP space through the text encoder, while at inference it uses vision-based maps from dense pixel embeddings and a transformer-based decoder to predict long-term goal locations (Wei et al., 2024).

Subsequent work has emphasized explicit persistent memory. OneMap stores a reusable probabilistic semantic feature map $P_t=(\Delta x,\Delta y,\Delta\theta)$0, where each cell holds a CLIP-aligned semantic feature vector and an uncertainty term, updated through patch-level SED features, depth projection, spatially varying uncertainty modeling, and recursive Bayesian fusion (Busch et al., 2024). LagMemo replaces 2D feature maps with a language 3D Gaussian Splatting memory, combining geometry from RGB-D exploration with language features from SAM and CLIP, discretized through a two-level codebook that supports stable open-vocabulary retrieval and 3D waypoint prediction (Zhou et al., 28 Oct 2025). OVAL formalizes lifelong memory as structured entries $P_t=(\Delta x,\Delta y,\Delta\theta)$1 containing open-semantic labels, image buffers, positional descriptors, HSV scene descriptors, and confidence, then uses descriptor matching and optional SuperGlue refinement to merge repeated observations across long horizons (Pei et al., 14 Apr 2026).

Several systems reduce memory to more abstract or dynamic structures. DualMap explicitly separates a detailed concrete map $P_t=(\Delta x,\Delta y,\Delta\theta)$2 from an abstract map $P_t=(\Delta x,\Delta y,\Delta\theta)$3, classifying objects into anchors and volatile objects so that global navigation is performed over sparse anchor structure while local goal confirmation uses the concrete map (Jiang et al., 2 Jun 2025). The system avoids expensive 3D inter-object merging by combining a hybrid segmentation front-end, stability checks, and split detection on an object’s observation history (Jiang et al., 2 Jun 2025). osmAG-LLM makes an even stronger compression choice: instead of dense 3D object maps, it extends osmAG with sparse object-nodes, viewpoint-nodes, and room descriptions so that an LLM can infer likely search nodes from room semantics and then rely on online detection for verification, particularly when the queried object has moved or was never mapped (Xie et al., 17 Jul 2025). This suggests a broader methodological split between high-fidelity semantic maps and context-centric semantic maps, both of which remain active in the literature.

4. Exploration, grounding, and decision policies

Open-vocabulary search requires not only a semantic representation of the goal, but also a strategy for deciding where to go when the goal is not visible. Early zero-shot work such as ZSON concentrated on representation transfer through CLIP embeddings (Majumdar et al., 2022). More recent systems intervene directly in exploration. DRIVE-Nav argues that frontier-based OVON is inefficient because many frontier points correspond to the same physical exit, then replaces point-level reasoning with persistent directions extracted from weighted Fast Marching Method paths, tracks those directions over time, restricts semantic inspection to a forward $P_t=(\Delta x,\Delta y,\Delta\theta)$4 range, and uses Qwen3-VL plus SAM3 for prompt enrichment and cross-frame verification (Gao et al., 30 Mar 2026). EffiNav uses depth thresholds to generate candidate exploration regions, asks a VLM to select among them in the egocentric image, projects the selected region into a top-down map for a second global-wise checking query, prunes history-internal regions, and falls back to the nearest frontier only when all candidates are rejected (Yin et al., 17 Jun 2026).

Uncertainty in semantic perception has become a second major theme. “Uncertainty-Informed Active Perception for Open Vocabulary Object Goal Navigation” models BLIP-2 semantic relevance under prompt variation as a Gaussian random variable estimated from a GPT-4-generated prompt ensemble, fuses the resulting mean and variance into a probabilistic geometric-semantic map, and scores frontier arms with expected improvement or GP-UCB rather than a single deterministic similarity score (Bajpai et al., 16 Jun 2025). OneMap likewise models uncertainty in feature extraction, leakage at depth discontinuities, and depth-projection noise inside a Bayesian feature map (Busch et al., 2024). In these systems, uncertainty is not treated as a by-product of perception but as a planning signal.

Other lines of work push semantic reasoning into the policy itself. Nav-$P_t=(\Delta x,\Delta y,\Delta\theta)$5 decomposes OVON into explicit target-environment and environment-action relations, teaches these relations through a 300K-sample Nav$P_t=(\Delta x,\Delta y,\Delta\theta)$6-CoT dataset generated from HM3D-OVON expert trajectories, and maintains compressed historical context with a non-parametric Similarity-Aware Memory that preserves instruction-relevant and current-observation-relevant tokens (Xiang et al., 2 Dec 2025). OVSegDT introduces a lightweight decoder-only transformer policy with a semantic branch for target binary masks, an auxiliary segmentation loss $P_t=(\Delta x,\Delta y,\Delta\theta)$7, and Entropy-Adaptive Loss Modulation that mixes behavior cloning and PPO according to policy entropy, replacing hand-designed switching schedules (Zemskova et al., 15 Aug 2025). GoalVLM inserts a VLM into the exploration loop itself: SAM3 provides text-prompted detections and masks, a Goal Projector back-projects those detections into BEV coordinates, and SpaceOM scores frontiers through a structured prompt chain with room-type classification and multi-frontier ranking (James et al., 18 Mar 2026). Across these approaches, the field has moved from simple open-vocabulary detection toward joint semantic grounding, exploration control, and temporal verification.

5. Multi-goal, lifelong, interactive, and multi-agent generalizations

The single-goal episode is no longer the only dominant formulation. LagMemo targets multi-modal open-vocabulary multi-goal visual navigation, supporting object, image, and text queries within a single explored environment. It constructs a 3D language memory once during exploration, retrieves candidate goal locations for later queries, and verifies them through SEEM, CLIP similarity, or LightGlue depending on the goal modality (Zhou et al., 28 Oct 2025). OVAL extends ObjectNav into lifelong open-vocabulary navigation, where a sequence of target-object queries is issued over extended periods and the system must retain map and semantic memory across related tasks, merging repeated observations with structured descriptors and exploring with a probability-based frontier score (Pei et al., 14 Apr 2026). OneMap similarly reformulates zero-shot navigation as reusable semantic memory across multiple object searches rather than one fresh search per query (Busch et al., 2024).

Interactive and user-centered formulations add another layer of complexity. ORION introduces Zero-shot Interactive Personalized Object Navigation, or ZIPON, in which the goal is not merely a class label but a personalized object identity such as “Alice’s computer,” and the agent may take a Talk action to request information or confirmation from a user while executing the task (Dai et al., 2023). OpenGuide frames open-vocabulary multi-object retrieval for blind or visually impaired users as a POMDP-based multi-object search problem, combining BLIP-2, Grounding DINO, YOLOv7, SAM, frontier exploration, a value map, and value decay to recover from missed detections and avoid endlessly revisiting misleading high-value regions (Xu et al., 2 Sep 2025). LOVON targets long-horizon missions on legged robots by using an LLM to decompose a long natural-language task into subtasks, then pairing an instruction object extractor with real-time detection and a transformer-based Language-to-Motion Model that explicitly enters search states when the target is lost (Peng et al., 9 Jul 2025).

OVON has also expanded beyond single ground robots and indoor settings. GoalSwarm formulates decentralized multi-UAV open-vocabulary ObjectNav with SAM3-based grounding, a Bayesian Value Map, UCB frontier scoring, and cost-utility bidding under geodesic path costs and separation penalties (James et al., 13 Mar 2026). GoalVLM studies cooperative multi-agent zero-shot object navigation on GOAT-Bench with two agents, shared BEV semantic maps, and sequential frontier allocation (James et al., 18 Mar 2026). WildOS extends open-vocabulary object search to long-range outdoor settings, building a sparse navigation graph, using ExploRFM to score frontier nodes by traversability, frontier confidence, and object similarity, and triangulating coarse goal estimates beyond the local depth horizon through a particle-filter-style procedure (Shah et al., 22 Feb 2026). These variants indicate that open-vocabulary object-goal navigation has become a general framework for language-conditioned search, rather than a narrowly indoor benchmark problem.

6. Metrics, empirical patterns, and unresolved difficulties

The canonical metrics remain Success Rate and Success weighted by Path Length. LangMap states them as

$P_t=(\Delta x,\Delta y,\Delta\theta)$8

where $P_t=(\Delta x,\Delta y,\Delta\theta)$9 indicates success, $R^2$0 is the executed path length, and $R^2$1 is the optimal path length (Miao et al., 2 Feb 2026). For multi-goal episodes, LangMap adds

$R^2$2

with $R^2$3, so that SeqSR measures full ordered completion and SeqSR-4 measures near-complete performance (Miao et al., 2 Feb 2026). EffiNav additionally reports EoS, defined as $R^2$4, to isolate efficiency on successful episodes (Yin et al., 17 Jun 2026). The emphasis on SPL, SeqSR, and EoS reflects a recurring finding: raw success can obscure inefficient or unreliable search.

Empirical results show clear progress but also persistent generalization gaps. ZSON reported absolute improvements of 4.2%–20.0% in success over prior zero-shot methods across Gibson, HM3D, and MP3D (Majumdar et al., 2022). HM3D-OVON showed that pure end-to-end learning remains semantically brittle: DAggerRL achieved 41.3% SR on $R^2$5 but only 18.3% on $R^2$6, whereas VLFM was more stable across semantic splits and DAggerDetect improved further to 38.5%, 39.0%, and 37.1% SR on $R^2$7, $R^2$8, and $R^2$9 respectively (Yokoyama et al., 2024). OVExp-ZS reported 59.7 success and 28.8 SPL on HM3D zero-shot ObjectNav (Wei et al., 2024); OneMap reported 55.8% SR and 37.4% SPL on single-object navigation and 54.24% SR with 27.77% SPL on its multi-object benchmark (Busch et al., 2024); DRIVE-Nav achieved 50.2% SR and 32.6% SPL on HM3D-OVON (Gao et al., 30 Mar 2026); EffiNav reported 63.0 SR and 43.7 SPL on OVON unseen_val (Yin et al., 17 Jun 2026); Nav-$v_{seen}$0 reported 44.0 SR and 18.0 SPL on HM3D-OVON val-unseen (Xiang et al., 2 Dec 2025); and OVSegDT reported 40.1% SR and 20.9% SPL on HM3D-OVON val unseen, while claiming 33% lower sample complexity and twofold collision reduction relative to the compared baseline (Zemskova et al., 15 Aug 2025). Because evaluation setups differ, these numbers function more reliably as benchmark-specific reference points than as a single unified leaderboard.

Several difficulties recur across nearly all benchmarks and system designs. LangMap shows that performance is better at scene, room, and region levels than at the instance level, and that richer context and memory improve success, while long-tailed categories, small objects, context-dependent objects, distant targets, and multi-goal completion remain challenging (Miao et al., 2 Feb 2026). EffiNav’s failure analysis reports that OVON has a larger share of failures due to semantic understanding and category mismatch than HM3D, even though OVON is often easier geometrically because many targets are closer to the start position (Yin et al., 17 Jun 2026). HM3D-OVON’s robustness analysis further shows that map-dependent modular methods can be more sensitive to localization and actuation noise than policies relying mainly on RGB history, whereas detector-augmented learned explorers can be more robust under noise (Yokoyama et al., 2024). A common misconception is therefore that open-vocabulary navigation is only a detection problem. The surveyed work indicates that it is simultaneously a problem of semantic generalization, exploration efficiency, memory management, stopping reliability, and—in multi-goal or lifelong settings—ordered task completion.

Open-vocabulary object-goal navigation has consequently developed into a composite research area linking visuo-semantic representation learning, active exploration, memory systems, VLM- or LLM-mediated reasoning, and embodied evaluation. The present trajectory, as documented by recent benchmarks and systems, moves from “find a chair” toward navigation grounded in scene context, relational language, persistent memory, and repeated interaction with the same world (Miao et al., 2 Feb 2026).