- The paper introduces a three-level navigation system that limits VLM decisions to room selection while using scene graphs, classical coverage planning, and embodiment-specific controllers for fine-grained movement.
- SysNav achieves 100.0%, 97.5%, and 98.3% success rates on easy, medium, and hard real-world tasks, outperforming baselines across 112 quantitative episodes.
- The system reaches state-of-the-art success on four simulation benchmarks, including a 14.1% success-rate and 6.5% SPL improvement on HM3D-OVON, while accepting conservative over-coverage for robustness.
Motivation and problem framing
SysNav (2603.06914) addresses object navigation (ObjectNav) in real-world indoor environments, where an agent must locate and reach an instance of a target category—or, in the general setting, an instance satisfying both a category constraint and a set of semantic constraints Φ (attributes such as color, or spatial relations such as "near the refrigerator")—in an unknown environment. The authors' central position is that real-world ObjectNav should be treated as a system-level problem rather than a single-policy learning task. They argue that end-to-end policies, which dominate simulation benchmarks, cannot simultaneously handle complex spatial structure, long-horizon planning, and semantic understanding, and that real-robot data scarcity further undermines learned approaches. A second argument concerns Vision-LLMs (VLMs): while VLMs possess strong commonsense reasoning, they lack precise 3D spatial grounding and long-term spatial consistency, so systems that delegate fine-grained exploration decisions to VLMs (e.g., frontier scoring as in ApexNav or InstructNav) exhibit inefficient behaviors such as frequent backtracking. SysNav's design principle is therefore to invoke VLM reasoning only at the spatial granularity where it is reliable—room-level decisions—and to delegate everything finer to classical planning.
System architecture
The system decouples into three levels. The high-level semantic reasoning module builds a structured three-layer scene graph and queries a VLM (Gemini-2.5-flash) over it. The mid-level room-based navigation module performs hierarchical exploration: rooms are the minimal decision units for the VLM, while coverage within a room uses classical two-level local/global planning adapted from TARE, with stochastic pose-candidate sampling scored by uncovered surface points, TSP ordering, and a rolling-window handoff between local and global horizons. The low-level base autonomy module executes waypoints with waypoint following, collision avoidance, and traversability analysis, using embodiment-specific controllers (built-in locomotion policies on the Unitree Go2 and G1), which is what enables cross-embodiment deployment without retraining.
Scene representation
The scene representation R contains three node types and five edge types. Room nodes are obtained by detecting wall structures via planar fitting and vertical point-cloud distribution analysis (following Hydra-style construction), dilating wall regions to partition space into connected components; each stores a 2D mask, a predicted room category, and a representative image maximizing visible voxels. Viewpoint nodes discretize explored areas: a new viewpoint is added when novel voxel coverage exceeds a threshold ϵ, so RGB imagery is stored sparsely at representative poses rather than along the full trajectory. Object nodes come from open-vocabulary detection (YOLO-World/YOLOE) and segmentation (SAM2), with 3D point clouds reconstructed from masks, LiDAR, and robot pose, and instance merging across observations. Edges encode doorway connectivity (room–room), affiliation (room–viewpoint), containment (room–object), visibility (viewpoint–object), and—added lazily, only when a task requires it—object–object spatial relations verified by the VLM over co-visible viewpoints. Object self-attributes are likewise inferred on demand by prompting the VLM with representative images of retrieved nodes, avoiding redundant attribute computation for all objects.
Room-level VLM decision-making
Cross-room navigation operates in two modes. In early-stop mode, when a new room is discovered mid-exploration, the VLM receives attributes and object sets of both rooms plus the goal, and decides whether to abandon the current room—an interruption mechanism that avoids exhaustively covering semantically irrelevant spaces. In room-query mode, when in-room exploration completes without finding the target, the VLM reassesses all uncovered rooms given their attributes, the trajectory, and the goal, selecting the most promising next room. This restriction of VLM responsibility to room selection is the paper's key architectural claim: it exploits semantic priors (e.g., refrigerators in kitchens) where VLMs are strong, while avoiding their documented weaknesses in 3D spatial reasoning at frontier granularity.
Real-world evaluation
Deployment spans three embodiments—a Mecanum-wheeled robot, Unitree Go2 quadruped, and Unitree G1 humanoid—all carrying a Livox Mid-360 LiDAR and Ricoh Theta Z1 panoramic camera, with the entire stack running on a laptop-class GPU (RTX 4090). A total of 190 real-world episodes were conducted: 78 qualitative (including building-scale runs) and 112 quantitative on the wheeled platform across three difficulty tiers, with baselines sharing identical inputs and low-level autonomy.
| Method |
Easy SR / SPT / AT |
Medium SR / SPT / AT |
Hard SR / SPT / AT |
| VLFM |
58.3 / 40.3 / 52.1 |
47.5 / 34.4 / 75.3 |
25.6 / 12.9 / 97.4 |
| InstructNav |
45.8 / 29.5 / 67.4 |
55.0 / 27.8 / 84.6 |
37.2 / 20.7 / 103.4 |
| SysNav |
100.0 / 83.8 / 29.1 |
97.5 / 67.9 / 72.8 |
98.3 / 71.8 / 67.6 |
In the hard (cross-room) setting, SysNav improves success rate by 61.1% and SPT by 51.1% over the best baseline while reducing average time by 29.8 seconds. Near-ceiling success rates (100/97.5/98.3) across all difficulty levels are a strong claim of reliability, and the authors state that SysNav is, to their knowledge, the first system to reliably complete building-scale long-range ObjectNav in complex real environments. Notably, hard scenes yield slightly better SPT than medium scenes; the authors attribute this to denser obstacles in the medium environment slowing the robot rather than to any property of the navigation logic—an honest caveat about confounds in the difficulty stratification. Qualitative results demonstrate instruction-conditioned behavior such as prioritizing a classroom identified from scene context and verifying "near" relations between a microwave and refrigerator before declaring success.
Simulation benchmarks
Across 8,195 episodes on HM3D-v1, HM3D-v2, MP3D, and HM3D-OVON (val-unseen), SysNav achieves state-of-the-art success rates under zero-shot, open-set conditions:
| Benchmark |
Best prior SR/SPL |
SysNav SR/SPL |
| HM3D-v1 |
59.6 / 33.0 (ApexNav) |
63.7 / 30.5 |
| HM3D-v2 |
76.2 / 38.0 (ApexNav) |
80.8 / 37.2 |
| MP3D |
40.2 / 17.8 (SG-Nav/ApexNav) |
50.7 / 18.1 |
| HM3D-OVON |
40.8 / 19.6 (MTU3D/VLFM) |
54.9 / 26.1 |
On HM3D-OVON—the open-vocabulary benchmark most aligned with the constrained-goal formulation—SR improves by 14.1% and SPL by 6.5%. SPL gains are consistently smaller than SR gains; the authors attribute this to the real-world-oriented design adopting stricter coverage thresholds, causing slight over-coverage in clean simulation. This is a fair concession: the system is tuned for robustness under sensor noise and environmental complexity, at some cost to path optimality in idealized settings.
Limitations and open questions
Several limitations are acknowledged or evident. First, the quantitative real-world comparison covers only the wheeled embodiment; quadruped and humanoid results are presented qualitatively, so cross-embodiment reliability claims rest on demonstration rather than controlled statistics. Second, the system depends on a proprietary VLM (Gemini-2.5-flash); latency, cost, and sensitivity to prompt design are not analyzed, and the early-stop mechanism's failure modes (e.g., premature abandonment of a room containing the target) are not quantified. Third, the difficulty-tier confound noted above means the real-world efficiency comparisons are not fully isolated from environmental factors such as obstacle density. Fourth, the SPL gap in simulation indicates the coverage strategy is conservative; whether room-granularity decisions remain optimal in environments whose layouts violate the assumed functional decomposition into semantically distinct rooms (e.g., open-plan studios) remains untested. Finally, the general instruction-following setting with attribute and spatial constraints is evaluated mainly through OVON and qualitative episodes; a dedicated large-scale benchmark for constraint-rich goals in the real world is left open.
Conclusion
SysNav demonstrates that a deliberately hierarchical decomposition—VLM reasoning confined to room-level decisions over a structured three-layer scene graph, classical coverage planning within rooms, and modular base autonomy below—yields substantial gains in both reliability and efficiency for real-world ObjectNav, with 190 real-world episodes across three embodiments and state-of-the-art results on four simulation benchmarks. Its principal contribution is less any individual component than the disciplined allocation of semantic reasoning to the granularity where it is dependable, supported by a scene representation designed for grounded, on-demand VLM querying.
