OSG Navigator: Modular Open-World Navigation

• OSG Navigator is a modular system for open-world Object-Goal Navigation that couples foundation models with a hierarchical Open Scene Graph.
• It organizes semantic observations into a persistent spatial memory using automatically generated schemas for zero-shot adaptation across diverse environments and robotic embodiments.
• The system integrates mapping, reasoning, and control modules—including probabilistic extensions—to enhance navigation performance under partial observability.

OSG Navigator is a modular system for open-world Object-Goal Navigation in which foundation models are coupled to a persistent external memory called the Open Scene Graph. Its central premise is that open-vocabulary semantic navigation is not only a perception problem or a control problem, but also a spatial-memory problem: LLMs and vision foundation models contribute semantic knowledge and open-set recognition, yet require a structured representation that can organize observations hierarchically over time. In OSG Navigator, that role is played by the Open Scene Graph, whose schema can be generated automatically from simple environment labels such as “home” or “supermarket,” enabling zero-shot adaptation across environment classes, goals, and robot embodiments (Loo et al., 6 Aug 2025).

1. Task formulation and motivation

OSG Navigator addresses open-world Object-Goal Navigation in unmapped indoor environments, with the goal specified in natural language and potentially outside a fixed category vocabulary. The system is intended to generalize across novel goals, diverse environments such as homes, offices, supermarkets, and hospitals, and different embodiments, including wheeled and legged robots. The task is formalized as an object-and-region-centric POMDP in which the state includes the agent state $s_a$, spatial regions $s_r$, objects $s_o$, and relations $s_e$; observations are RGB images; actions include MoveToObject(o) and MoveToRegion(r); and the goal is an open-vocabulary object description (Loo et al., 6 Aug 2025).

The motivation for this formulation is that open-world navigation requires both semantic and spatial reasoning under partial observability. Foundation models supply broad semantic priors, object co-occurrence knowledge, VQA-style description, and robust goal-conditioned local navigation, but they do not by themselves organize and maintain spatial information effectively at scale. OSG Navigator therefore treats the Open Scene Graph as an approximate belief or memory state, accumulating scene structure over time so that reasoning can proceed over objects, places, and region abstractions rather than over isolated image observations (Loo et al., 6 Aug 2025).

This design also reflects a specific critique of flat scene representations. A robot searching for an open-vocabulary target may need to reason that a target object is likely inside a particular room type, that the room is likely reachable via a connector such as a doorway, and that the current observation matches or does not match a previously visited place. Such reasoning depends on persistent hierarchical organization rather than short-term visual context alone. OSG Navigator is thus explicitly a memory-centric navigation architecture (Loo et al., 6 Aug 2025).

2. Open Scene Graphs as hierarchical spatial memory

The Open Scene Graph is a heterogeneous directed graph organized into layers. It contains four abstract node types: Objects, Places, Connectors, and Region Abstractions. Objects are localized semantic entities such as beds, sinks, and chairs; Places are the most fine-grained semantically meaningful regions, such as rooms, aisles, or corridors; Connectors are boundary-spanning elements such as doors, entrances, or stairs; and Region Abstractions are coarser groupings such as floors or supermarket sections. The graph uses four abstract edge types: Proximity (is near), Connectivity (connects to), Inclusion (has), and Hierarchy (contains) (Loo et al., 6 Aug 2025).

The representation is layered. Layer 1 contains leaf Object nodes, each storing a label, desc, id, and img. Layer 2 contains Places and Connectors. Places store a schema class such as room or aisle, a semantic label, and an id; they connect to objects through contains edges and to other Places or Connectors through connects to edges. Higher layers contain Region Abstractions, each a coarser partition of the layer below. This produces a hierarchy of objects $\rightarrow$ places $\rightarrow$ regions that supports both recognition and topological planning (Loo et al., 6 Aug 2025).

A defining feature of OSG Navigator is the OSG schema. A schema is an environment-type template that specifies which node types exist, how they connect, and what hierarchy is expected. Homes are described by structures such as rooms, corridors, floors, and entrances; supermarkets by aisles, sections, and walkways; offices by offices, hallways, floors, and stairs. The schema gives the LLM a structured vocabulary and constrains graph construction. This is the mechanism by which the system adapts to new environment classes without redesigning the core Mapper, Reasoner, or Controller (Loo et al., 6 Aug 2025).

The graph is therefore not merely a list of objects or a flat metric map. It is a topo-semantic memory in which semantic content is arranged according to connectivity and containment. This structure is what allows OSG Navigator to reason over likely search regions, connectors to unseen areas, and local object context while preserving zero-shot flexibility (Loo et al., 6 Aug 2025).

3. System architecture and operating pipeline

OSG Navigator is composed of foundation-model modules arranged into high-level mapping and reasoning plus low-level control. The perception stack uses GroundingDINO for open-set object detection and BLIP-2 for image-to-text or VQA-style description. The main experiments use GPT-3.5 as the LLM, and ViNT serves as the low-level goal-conditioned navigation model. The overall system has three major modules: Mapper, Reasoner, and Controller (Loo et al., 6 Aug 2025).

The Mapper performs perception, state estimation, and graph update. It first parses each RGB observation into semantic elements, identifying Places, Objects, and Connectors. It then estimates the current Place by retrieving semantically similar Place nodes from the graph, sorting them by distance from the previous state, and matching observed nearby objects against stored object features. If a match is found, the current observation is localized to an existing Place; otherwise, the state estimator returns None, signaling a new Place (Loo et al., 6 Aug 2025).

The OSG Updater then integrates the observation into memory. If the current Place is new, it adds the Place, adds observed Object and Connector leaf nodes, and updates successively higher region layers using LLM-based abstraction. If the Place has been visited before, it associates current observations with existing leaf nodes through data association and updates node and edge attributes. In this way, the graph supports both growth into unseen space and refinement under revisitation (Loo et al., 6 Aug 2025).

The Reasoner chooses where to search next. If no target region is active, it invokes RegionProposer_LLM, which performs a coarse-to-fine greedy search down the OSG hierarchy to select the most promising region for the goal object. If the robot has not yet reached that region, Pathfinder extracts the Place/Connector connectivity graph and uses Dijkstra to compute a path. ObjectProposer_LLM then chooses an Object or Connector along the route and returns an image-goal subgoal for the low-level controller. Once the target region is reached, ObjectProposer_LLM instead chooses an object around which to search locally (Loo et al., 6 Aug 2025).

The Controller executes MoveToObject. In simulation it uses an FMM controller; in the real world it uses ViNT. It consumes RGB input and a crop of the goal object and outputs velocity commands. This preserves a clean division of labor: the OSG stores memory and search structure, the Reasoner selects semantically promising graph targets, and the Controller handles embodiment-specific local motion (Loo et al., 6 Aug 2025).

The paper also proposes a probabilistic extension inspired by topological mapping with particle-filter-like map hypotheses. In that extension, each particle is a possible topological map, observation likelihoods are computed from LLM-based semantic similarity, and particle weights are updated and resampled. This is presented as an optional robustness improvement rather than the primary implementation (Loo et al., 6 Aug 2025).

4. Schema generation and zero-shot adaptation

A major contribution of OSG Navigator is automatic OSG schema generation from a simple environment label such as “home,” “supermarket,” “hospital,” or “airport.” The process is a three-stage iterative LLM pipeline. In Description generation, the LLM produces a long-form natural-language account of the environment type, including common spatial abstractions and their relations. In Canonicalisation, the description is converted into a canonical graph by extracting triplets, mapping entities to OSG node types, mapping relations to allowed OSG relations such as contains, connects to, and is near, and assembling a JSON schema. In Verification, the generated schema is checked for structural correctness; if errors are found, feedback is returned and the LLM regenerates the schema (Loo et al., 6 Aug 2025).

The appendix reports generated schemas for Apartment home, Hospital, Airport, Supermarket, Office, and Mall. This schema machinery is the basis of zero-shot adaptation across environment classes. Changing the environment label changes the schema, while leaving the Mapper and Reasoner unchanged. The resulting system can therefore reconfigure its topo-semantic prior without retraining or task-specific engineering (Loo et al., 6 Aug 2025).

The evaluation includes a direct comparison of schema construction methods. Manual schemas achieve SR 0.7. Direct LLM prompting often yields invalid schemas and produces 0.0 SR. By contrast, the Generate–Canonicalise–Verify pipeline achieves SR 0.6 with valid schema rate 1.0. These results indicate that schema quality is not a cosmetic detail but a structural requirement for robust OSG-based reasoning (Loo et al., 6 Aug 2025).

A plausible implication is that OSG Navigator treats environment type as a high-level inductive bias rather than as a fixed prior baked into the codebase. The graph ontology remains stable, but its concrete layering and connectivity constraints are specialized on demand. This is what allows the same system to navigate homes, supermarkets, hospitals, offices, and malls under a shared reasoning interface (Loo et al., 6 Aug 2025).

5. Evaluation, performance, and generalization

OSG Navigator is evaluated in simulation on Habitat using Gibson and HM3D-Semantics v0.1, with SR, SPL, and DTG as primary metrics. The SPL metric is reported in the standard form

$$\frac{1}{N}\sum\limits_{i=1}^{N}S_i\frac{l_i}{\text{max}(p_i, l_i)}.$$

Baselines include SemExp, PONI, FBE, SemUtil, LGX, and LFG, as well as OSG-Nav-GT, which uses ground-truth semantic perception with the same OSG representation (Loo et al., 6 Aug 2025).

On HM3D, OSG-Nav-GT achieves SR 0.775, SPL 0.380, and DTG 1.702, while the full OSG-Nav system with vision foundation models achieves SR 0.693, SPL 0.283, and DTG 2.338. On Gibson, OSG-Nav achieves SR 0.734, SPL 0.386, and DTG 1.722. The paper interprets the progression from LGX-GT to LFG-GT to OSG-Nav-GT as evidence that persistent global scene memory helps, and that adding regions and topology improves navigation beyond frontier-based memory alone (Loo et al., 6 Aug 2025).

The ablations reinforce the central role of structured memory. Representation ablation shows that maintaining an OSG is better than flat or local prompting. The paper reports that OSG-Nav-GT uses room and connector semantics to search efficiently, whereas LFG-GT often defaults to less informed exploration. Data-association analysis further indicates that Places are recognized quite reliably using object context, while connectors and objects are distinguishable reasonably well, although aliasing can occur when multiple similar objects are clustered (Loo et al., 6 Aug 2025).

The system is explicitly evaluated for zero-shot generalization across goals, environments, and embodiments. Across goals, it handles standard object categories, rare nouns such as “settee” and “nightstand,” compositional queries such as “red floral bed,” and real-world open-vocabulary goals such as “guitar,” “dish washer,” and “computer.” Across environments, it is evaluated in homes, supermarkets, hospitals, offices, malls, and real-world apartments. Across embodiments, the same system runs on Spot and Fetch despite different locomotion, camera models, mounting heights, and fields of view, and the real-world deployments are reported to succeed with no parameter retuning (Loo et al., 6 Aug 2025).

The real-world experiments use Fetch, a wheeled mobile manipulator with a Realsense camera, and Spot, a quadruped with a fisheye camera. The deployments run locally on a Jetson AGX Orin except for cloud LLM queries. Tested environments include an open-plan studio apartment, a nursing home mock-up, and office or mall-like settings. The paper presents these experiments as evidence that the OSG abstraction transfers across embodiments while preserving the same topo-semantic reasoning interface (Loo et al., 6 Aug 2025).

6. Relation to adjacent work, limitations, and significance

OSG Navigator belongs to a broader line of work that treats structured scene memory as central to open-world navigation. An earlier system, OpenSearch, already used Open Scene Graphs as a configurable topo-semantic representation for open-world object-goal navigation, along with VQA, open-set detection, LLM-based reasoning, and ViNT-based control. This suggests that OSG Navigator consolidates that research direction around the Open Scene Graph itself as the defining systems concept (Loo et al., 2024).

Subsequent work explores alternative memory substrates and richer scene-graph designs. MSGNav replaces text-only relational edges with dynamically assigned images in a Multi-modal 3D Scene Graph, adds Key Subgraph Selection, Adaptive Vocabulary Update, Closed-Loop Reasoning, and Visibility-based Viewpoint Decision, and reports state-of-the-art performance on GOAT-Bench and HM3D-OVON (Huang et al., 13 Nov 2025). 3DGSNav shifts from topo-semantic graphs to persistent 3D Gaussian Splatting memory, using frontier-aware free-viewpoint rendering and VLM-driven target re-verification to improve zero-shot object navigation (Zheng et al., 12 Feb 2026). WildOS transfers the general semantic-memory idea outdoors by combining a sparse navigation graph with ExploRFM for traversability, visual frontier, and object-similarity prediction in long-range open-vocabulary object search (Shah et al., 22 Feb 2026). GraphMapper, although not an open-world ObjectNav system, provides a related example in which a 3D scene graph is learned online as an intermediate representation for navigation rather than relying on raw visual features alone (Seymour et al., 2022).

OSG Navigator also has explicit limitations. It currently relies heavily on topo-semantic information and does not model metric geometry explicitly. It has been evaluated only on indoor environments. The main system depends on large cloud LLMs. The schema generation pipeline is not fully adaptive online. General Navigation Models remain a bottleneck in cluttered real-world scenes, and VFM or VQA errors can affect place labeling and object description. These are not incidental implementation issues: they mark the current boundary of what structured semantic memory can deliver without a stronger integration of geometry, perception robustness, and local control (Loo et al., 6 Aug 2025).

Its significance lies in making spatial memory a first-class object in open-world navigation. OSG Navigator shows that hierarchical scene organization, schema-conditioned environment priors, and graph-based reasoning can be combined with foundation models into a zero-shot embodied system that searches for open-vocabulary goals across diverse environments and embodiments. The broader implication is that long-horizon semantic navigation benefits when perception, memory, reasoning, and local control are separated but coupled through a structured representation rather than through a flat latent state (Loo et al., 6 Aug 2025).