SuperNav: Agentic Navigation

• SuperNav is an agentic navigation system utilizing a language model, facilitating tasks such as object search and ordered multi-step requests, in unfamiliar and open-vocabulary environments
• It leverages specialized navigational tools ensuring task generality, command interpretation and execution
• The architecture harnesses a visual point interface for destination selection, a progress manager, and two motion back-ends improving task generalizability and flexibility

SuperNav is an agentic navigation system for general-purpose service robots that uses a pretrained multimodal LLM (MLLM) for request interpretation, scene understanding, destination selection, task-progress reasoning, and completion decisions, while delegating physical motion to specialized navigation tools. Its central design principle is to avoid navigation-specific fine-tuning of the MLLM and instead provide an agent harness containing Navigation Skills, agent-oriented Tools, task-progress and context management, execution feedback, and a unified visual-point interface. The system targets task generality—instance search, ordered multi-object navigation, demand-driven requests, and category-level navigation—and scene generality across unfamiliar environments (Zhang et al., 8 Oct 2026).

1. Problem formulation and design rationale

SuperNav addresses navigation requests whose semantic and spatial requirements may not be explicitly specified. These include locating a particular object, visiting multiple objects in a prescribed order, navigating to an open-vocabulary category, or satisfying a demand-driven request involving several inferred objects and spatial relations. Examples include finding a particular sofa or washing machine, visiting a printer followed by a trash bin, or preparing a workspace by identifying a table, a vase to move, and a suitable shelf.

The system assigns semantic reasoning to the MLLM. It must infer what satisfies the request, identify relevant visual evidence, select plausible destinations, determine whether a candidate has been sufficiently verified, update which goals remain, recover from blocked motion, and decide whether execution should terminate. Unlike a fixed navigation policy, SuperNav permits the MLLM to determine at runtime how a request should be interpreted, what should be searched, when a candidate should be rechecked, and which goals remain incomplete.

The architecture is motivated by a distinction between two limitations of existing systems. End-to-end vision-language-action models may depend on navigation tasks and environments represented in their training data, whereas conventional modular systems may use fixed workflows for exploration, target confirmation, and termination. SuperNav instead preserves the pretrained MLLM’s general-purpose visual and linguistic capabilities and exposes movement as a callable capability.

For instance-level evaluation, a target $g$ has an approved set of viewpoints $V_g$. Arrival occurs when the shortest geodesic distance from the agent position $p$ to an approved viewpoint is less than $1$ meter:

$$\min_{v\in V_g} d_{\mathrm{nav}}(p,v) < 1\ \mathrm{m}.$$

For episode $i$, viewpoint-reference SPL is

$\mathrm{SPL}_i=\frac{S_iL_i}{\max(L_i,P_i)},$

where $S_i$ indicates success, $L_i$ is the reference shortest-path length, and $P_i$ is the executed path length. Multi-object reference paths are computed in fixed target order using dynamic programming over approved viewpoints. Failed episodes receive zero SPL.

SuperNav’s progress records are agent reports rather than independently verified semantic predicates. The termination Tool records whether the MLLM believes the task was achieved or blocked, while benchmark scoring separately evaluates the resulting trajectory.

2. Agent harness and execution loop

SuperNav consists of a pretrained MLLM, an agent harness, agent-oriented physical-interaction Tools, optional Navigation Skills, task-progress management, context management, a unified visual-point interface, and one of two motion backends: the Geo-based Executor or the Learned Executor.

The MLLM interprets requests, understands scenes, reasons about objects and relations, selects image points, chooses Tools, judges progress, and decides when to reobserve or terminate. The harness maintains the interaction loop, exposes capabilities, preserves relevant state, loads Skills, and controls execution context. Tools perform observation, turning, movement, progress updates, and session closure. Skills provide procedural guidance for search, verification, recovery, and completion. Motion backends convert image-space destinations into physical movement.

A typical interaction cycle is:

1. The harness initializes a session and obtains observations.
2. The MLLM interprets the request and infers possible goals and completion conditions.
3. It selects an observation, turning, movement, progress, Skill, or termination Tool.
4. A movement Tool navigates toward a selected visual point.
5. The Tool returns fresh images, execution status, and diagnostics.
6. The MLLM reassesses the candidate, scene, and remaining goals.
7. It records progress, searches elsewhere, invokes recovery, or terminates.

Movement therefore does not terminate reasoning. A possible target may be approached, visually rechecked, rejected, or accepted after execution feedback.

The implementation uses the Codex CLI as the agent runtime and exposes environment operations through the Model Context Protocol (MCP). Concrete Tools include hab_init_scene, hab_panorama, hab_turn, hab_visual_point_navigate, hab_local_navigate, hab_localnav_status, hab_localnav_stop, and hab_close_session. The exact Tool set depends on the motion backend.

Navigation Skills are Markdown instruction packages loaded by the MLLM on demand. They specify when they apply, what evidence to inspect, recommended procedural steps, and linked references. The bundle includes a main navigation Skill, backend-specific Tool-use guidance, exploration guidance, recovery guidance, entrance-search guidance, and multi-goal guidance. Skills encourage the MLLM to record inspected areas, visible openings, candidate objects, uncertain branches, and failed or incomplete searches. They influence behavior through context; they do not independently enforce action sequences or verify semantics.

Task-progress Tools maintain pending goals, completed or found goals, the order of multi-object targets, and remaining task stages. Context management retains textual interaction history, task-state records, descriptions of inspected regions, candidate clues, failed attempts, and image paths. Once an image has been processed and superseded, its media payload may be removed while its textual reference and workspace path remain available.

3. Unified visual-point navigation

SuperNav’s principal interface between semantic decision-making and physical motion is:

$V_g$0

where $V_g$1 identifies a directional view and $V_g$2 is a normalized image coordinate measured from the top-left corner. The robot receives four RGB images labeled front, right, back, and left relative to its current heading.

The MLLM can select a point on an object, a nearby floor region, a doorway, a passage, or an exploratory location. This separates destination selection from motion execution: the MLLM specifies where in an image the robot should move toward, while the selected backend determines how to realize that destination.

The same interface is used by both motion backends. The geometric backend grounds the point using depth and navigation geometry. The learned backend treats the marked point as a visual goal for a learned controller. Consequently, the high-level agent loop does not need to change when the motion backend changes.

The paper compares direct visual-point selection with a grounding-based alternative in which the MLLM produces a referring expression, an external grounding model localizes the object, and the system may present candidates for confirmation. In the reported ablation, external grounding did not improve performance over direct pointing.

This interface also supports exploration. The selected point need not be the target’s exact center: it may indicate a reachable approach region, an entrance, or a viewpoint from which further scene information can be obtained. This suggests that the visual-point representation functions as a general destination primitive rather than merely an object-localization output.

4. Motion backends

Geo-based Executor

The Geo-based Executor uses RGB observations, depth, camera calibration, camera pose, a navigation mesh or mapped geometry, path planning, and path following. In simulation, the selected point is back-projected using rendered depth, nearby candidate destinations are checked against the navigation mesh, and a reachable goal is selected.

The default standoff distance is $V_g$3 m, the candidate snap-distance limit is $V_g$4 m, the default follower goal radius is $V_g$5 m, and the follower can execute a maximum of 200 steps. Recast/Detour’s PathFinder provides path queries, and a greedy geodesic follower executes discrete turning and forward actions.

On the Unitree Go2 quadruped, the geometric executor combines four directional RGB cameras, LiDAR, odometry, camera intrinsics and extrinsics, a voxel-grid map, an A* path planner, and a lookahead follower. The reported voxel size and grid resolution are $V_g$6 m. The follower uses proportional heading control and produces linear and angular velocity commands at $V_g$7 Hz. The deployment relies on robot odometry rather than an additional loop-closing SLAM system.

The geometric backend generally provides better path efficiency because it has explicit geometric information, although the comparison also changes the resources available to the two backends.

Learned Executor

The Learned Executor uses RGB observations and learned prediction to convert a point in a reference image into local motion. It is inspired by NoMaD-style goal-conditioned generative policies but uses a marked image point instead of a destination image.

A local request consists of a reference image $V_g$8, a normalized point $V_g$9, and a changing current image $p$0. The point is marked with a red ring and central dot:

$p$1

Images are resized to $p$2 and encoded by a shared DINOv2 ViT-S/14 encoder with global feature dimension 384. Observation and goal representations are

$p$3

$p$4

The goal multilayer perceptron has dimensions $p$5. Because the target may leave the current field of view, the policy uses a variable-length history containing at most 12 observations, while retaining the initial and current frames.

Historical observations are combined with projected sinusoidal age embeddings and observation/goal token identifiers. A four-layer, four-head Transformer with width 256 and feed-forward width 1024 fuses the history and goal representation.

The policy predicts eight two-dimensional displacement increments in the robot coordinate frame, with $p$6 forward and $p$7 left. The normalized expert target increment is clipped after division by $p$8 m. The flow-matching objective is

$p$9

where $1$0 is Gaussian noise, $1$1 is uniformly sampled, $1$2 is the normalized action sequence, and $1$3 is the fused visual representation. The total objective combines flow, stopping, and remaining-step losses:

$1$4

Training uses simulator expert trajectories, 45,000 updates, global batch size 256, AdamW, learning rates of $1$5 for policy layers and $1$6 for the pretrained visual backbone, weight decay $1$7, 500 warm-up updates, cosine decay, bfloat16 training, gradient clipping at 1, and exponential moving average decay $1$8.

At inference, four Euler steps are used for flow sampling and 16 sequences are considered. One sequence is selected using endpoint-direction agreement and path straightness. The executor normally executes a prefix of three predicted increments before obtaining new observations and replanning. A local stop is confirmed when the stopping probability is at least $1$9 on two consecutive replanning calls and a motion-progress condition is satisfied. This local stop ends the movement request, not necessarily the semantic task.

5. Evaluation across task types

SuperNav is evaluated with GPT-5.6 Terra and either the Geo-based or Learned Executor. The main metrics are success rate and SPL.

Instance-level and multi-object navigation

A Habitat-GS benchmark constructed from InteriorGS scenes contains 150 single-object tasks and 150 ordered multi-object tasks, with two to five targets in each multi-object instruction. Object annotations were generated with a vision-LLM and manually reviewed.

Method	Single SR	Single SPL	Multi SR	Multi SPL
NaVid	24.67	0.1599	2.67	0.0216
UniNaVid	34.00	0.1833	1.33	0.0131
StreamVLN	13.33	0.1036	0.00	0.0000
OmniNav Action Former	27.33	0.2133	4.00	0.0275
SuperNav + Learned Executor	68.00	0.2059	34.00	0.0862
SuperNav + Geo-based Executor	78.00	0.4127	34.00	0.1388

The Geo-based Executor achieves the highest single-object success and SPL among the listed methods. The two SuperNav backends obtain the same reported multi-object success rate, while the geometric backend achieves higher multi-object SPL.

With the Geo-based Executor, multi-object stage success declines across stages one through five:

$$\min_{v\in V_g} d_{\mathrm{nav}}(p,v) < 1\ \mathrm{m}.$$0

This pattern indicates that ordered multi-stage navigation remains substantially more difficult than single-object navigation.

Demand-driven navigation

The demand-driven evaluation uses 200 AI2-THOR tasks from Demand-Bench under an adapted protocol. Tasks contain one to eight ordered stages. The evaluation measures ordered arrival at admissible objects within a $$\min_{v\in V_g} d_{\mathrm{nav}}(p,v) < 1\ \mathrm{m}.$$1 m bounding-box region rather than completion of the underlying activity.

Method	SR	SPL
NaVid	17.00	0.0920
UniNaVid	25.00	0.0943
StreamVLN	25.50	0.0775
OmniNav Action Former	37.50	0.1213
SuperNav + Learned Executor	45.00	0.1603
SuperNav + Geo-based Executor	59.50	0.1801

The Geo-based system leads the first five demand-driven stages with success rates of 72.00%, 55.38%, 45.12%, 24.32%, and 9.09%. No method completes six or more stages in this evaluation. The benchmark therefore tests ordered navigation to relevant objects but not whether the underlying demand—such as physically preparing a workspace—has been fulfilled.

Category-level navigation

SuperNav is evaluated on a shared 120-episode HM3D-OVON validation-unseen subset and on 1,000 HM3Dv2 validation episodes without navigation-specific adaptation.

On HM3D-OVON:

Method	SR at 0.25 m	SPL at 0.25 m	SR at 1 m	SPL at 1 m
MTU3D	40.80	0.1210	—	—
SoftNav	66.70	0.2570	—	—
AstraNav-Memory	—	—	62.50	0.3490
OmniNav slow + CoT	—	—	59.20	0.3320
SuperNav + Geo	68.33	0.3837	73.33	0.4105
SuperNav + Learned	65.00	0.1355	70.83	0.1443

On HM3Dv2:

Method	SR at 0.2 m	SPL at 0.2 m	SR at 1 m	SPL at 1 m
L3MVN	36.30	0.1570	—	—
VLFM	63.60	0.3250	—	—
SG-Nav	49.60	0.2550	—	—
ApexNav	76.20	0.3800	—	—
WMNav	—	—	72.20	0.3330
MSGNav	—	—	74.10	0.3340
SuperNav + Geo	80.30	0.3338	86.50	0.3638
SuperNav + Learned	79.00	0.1288	85.40	0.1383

The paper cautions that published HM3D results use different subsets, thresholds, action budgets, path-following implementations, and stopping conventions. The comparisons are therefore indicative rather than perfectly controlled.

6. Ablations, deployment, and limitations

Ablations on the same 120 HM3D-OVON episodes use the Geo-based Executor.

Variant	SR 0.25 m	SPL 0.25 m	SR 1 m	SPL 1 m
Without Navigation Skills	35.00	0.2460	54.17	0.3894
Without agent-oriented Tool design	52.50	0.2230	65.83	0.2855
Without Skills and Tool design	31.67	0.1870	54.17	0.3417
Grounding Motion Tool	55.83	0.3201	62.50	0.3624
SuperNav with GPT-6 Astra medium	71.67	0.4579	79.17	0.4974
SuperNav with GPT-5.6 Terra high	68.33	0.3837	73.33	0.4105

Removing Skills substantially reduces success, especially at the stricter threshold, indicating the importance of procedural guidance for search, recovery, and completion. Removing agent-oriented Tool design produces more explicit turning and lower efficiency. Removing both Skills and Tool design is particularly damaging. The direct visual-point interface performs better than the tested grounding-and-confirmation alternative. GPT-6 Astra medium performs better than GPT-5.6 Terra high in this comparison.

The reported interaction statistics are:

Variant	Tool calls	Navigation calls	Turns	Path length
Without Skills	7.43	5.33	0.10	7.56 m
Without agent-oriented Tool design	30.28	10.72	17.56	23.60 m
Without Skills and Tool design	15.37	5.61	7.72	10.47 m
Full SuperNav	11.17	8.76	0.41	16.44 m

These measurements support the interpretation that Tool design affects interaction efficiency and that Skills affect how capabilities are used, although they do not independently establish causality.

SuperNav was also deployed on a Unitree Go2 quadruped equipped with four-view RGB cameras, LiDAR, odometry, and geometric path following. Demonstrations included searching for a basketball, searching for another Go2, and visiting a printer followed by a trash bin. The robot revised routes at blocked passages, verified candidate targets, transitioned between multiple goals, and used LiDAR-based geometry for execution. This is qualitative evidence of simulation-to-real transfer rather than a large-scale statistical real-world evaluation.

The principal limitations are methodological and architectural. SuperNav depends on the MLLM for object recognition, interpretation of abstract needs, candidate verification, progress updates, and completion judgments. The harness does not independently correct semantic errors. The Geo-based Executor requires depth, maps, navigation meshes, or equivalent geometry, while the Learned Executor has lower path efficiency in the reported comparisons. MLLM inference introduces high and variable latency. Long-horizon performance decreases as the number of ordered goals grows.

The demand-driven evaluation does not verify completion of the underlying activity. A trajectory reaching a shelf after locating a vase does not establish that the vase was moved or that a workspace was prepared. Agent-declared completion is likewise not semantic verification. The termination Tool records the MLLM’s judgment but does not independently determine whether the task succeeded.

The evaluation comparisons also have protocol limitations. HM3D results use different thresholds, subsets, action budgets, path-following implementations, and stopping conventions. The paper therefore does not establish universal superiority across all navigation settings. More broadly, the results depend on the MLLM’s semantic competence, perception and geometry reliability, Skill quality, and the ability to preserve progress over long sequences.

SuperNav’s principal contribution is an agent-harness architecture rather than a new standalone navigation policy. It demonstrates that a pretrained MLLM can support diverse navigation tasks when given callable physical Tools, a consistent visual-point destination interface, procedural Skills, task-progress records, manageable context, and execution feedback. The geometric and learned backends show that high-level semantic reasoning can remain invariant while motion realization changes.

The resulting architecture can be summarized as:

$$\min_{v\in V_g} d_{\mathrm{nav}}(p,v) < 1\ \mathrm{m}.$$2

The empirical evidence supports narrower conclusions: SuperNav outperforms the evaluated action-prediction baselines on instance-level, multi-object, and demand-driven tests; achieves strong results on the reported category-level evaluations; benefits from Navigation Skills and agent-oriented Tools; supports two motion backends; and transfers qualitatively to a real quadruped. It does not establish that arbitrary requests can be solved in arbitrary environments. Its generality remains bounded by MLLM reasoning, visual perception, geometric reliability, long-horizon degradation, and the absence of an independent semantic completion verifier.