Navigation-guided Vision–Language–Action Models

• Navigation-guided vision-language-action models are embodied systems that couple spatial cues with visual perception and language to synthesize feasible navigation actions.
• They employ diverse architectures such as mapless view selection, modular decomposition, and world modeling to enhance robustness and adaptability across tasks like VLN, ObjectNav, and UAV control.
• These models leverage varied supervision regimes and latency-handling techniques to optimize planning, perception, and control in both simulated and real-world environments.

Searching arXiv for papers on navigation-guided vision-language-action models and related embodied navigation VLA frameworks. Navigation-guided vision-language-action models are embodied policies in which navigation structure governs the coupling of visual perception, language grounding, and action generation. In recent work, this coupling has been instantiated through mapless imagined-view selection, obstacle-aware waypoint prediction, dual-view continuous UAV control, delayed semantic-control interfaces, topology-aware node reasoning, and unified video-to-action policies spanning VLN, ObjectNav, Embodied Question Answering, and human following (Zhao et al., 2024, Zhang et al., 2024, Xu et al., 15 Mar 2026, Huang et al., 2 Feb 2026, Liu et al., 3 Mar 2026, Zhang et al., 2024). The resulting systems differ substantially in embodiment, action space, and supervision regime, but they share a common objective: to use language-conditioned visual reasoning to select or synthesize navigation actions that remain geometrically feasible, semantically grounded, and temporally coherent.

1. Task regimes and problem formulations

The literature covers several distinct navigation settings. In open-vocabulary ObjectNav, ImagineNav operates in Habitat v3.0 on HM3D and HSSD, using only on-board camera captured RGB/RGB-D stream inputs in a mapless manner; at each time step the agent captures a 360° panorama $I_t$, split into six egocentric RGB + depth views $\{I_{t,i},D_{t,i}\}_{i=1}^6$, and selects among six imagined next views before invoking a Point-Goal policy (Zhao et al., 2024). In VLN-CE, the agent commonly observes 12 RGB views and 12 depth maps together with an instruction, and navigation is posed as either high-level candidate-view selection, low-level atomic action generation, or both (Zhang et al., 2024).

Aerial navigation broadens both the state space and the control space. AerialVLA recasts UAV-based VLN as an end-to-end VLA problem with forward-looking and downward-looking RGB feeds, a fuzzy directional prompt derived from IMU/GPS, and a unified control space $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$ with intrinsic landing (Xu et al., 15 Mar 2026). ImagineUAV instead treats UAV VLN as cascaded world-action modeling: a latent video diffusion module generates instruction-conditioned future observations, an action extractor infers relative $6$-DoF poses $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$, and a kinodynamic planner refines them into collision-free trajectories (Liu et al., 31 May 2026). AutoFly likewise targets autonomous UAV navigation in unknown outdoor environments, but conditions on coarse positional or directional guidance and uses a pseudo-depth encoder derived from RGB (Sun et al., 10 Feb 2026). UAV-VLN uses a fine-tuned TinyLlama-1.1B, Grounding DINO, a cross-modal grounding module, and a ROS 2-based low-level planner over a discrete action set such as ascend, descend, move forward, yaw left, hover, and land (Saxena et al., 30 Apr 2025).

Several frameworks emphasize task unification rather than a single benchmark. Uni-NaVid harmonizes input and output configurations for four sub-tasks—VLN, ObjectNav, EQA, and human following—by casting them as next-token prediction over RGB video and language (Zhang et al., 2024). Uni-LaViRA formalizes embodied navigation as Language–Vision–Robot Actions Translation across VLN-CE, ObjectNav, EQA, and Aerial-VLN, and deploys the same agentic structure on wheeled, quadruped, humanoid, and UAV platforms in a zero-shot manner (Ding et al., 26 May 2026). This suggests that “navigation-guided” is not tied to one embodiment or one action parameterization; rather, it denotes a family of multimodal decision systems whose intermediate structure is explicitly navigation-centric.

2. Recurrent architectural patterns

A recurrent pattern is the decomposition of navigation into semantically interpretable subproblems instead of direct monolithic action regression. ImagineNav makes this explicit through an “imagine → reason → move” loop: Where2Imagine regresses six candidate relative poses $\{P_{t+1}^{(i)}\}_{i=1}^6$, a pretrained Novel View Synthesis model produces imagined future RGBs $\{I_{t+1}^{(i)}\}_{i=1}^6$, a VLM scores the imagined views given the textual goal $g$, and a PointNav policy drives the robot to the selected pose (Zhao et al., 2024). The high-level planning problem is thereby translated into best-view image selection.

Other systems formalize the decomposition differently. Uni-LaViRA factorizes each step into a Language Action $f_{\mathrm{lang}}$, which emits a semantic direction token such as $\mathtt{front}$, $\{I_{t,i},D_{t,i}\}_{i=1}^6$0, $\{I_{t,i},D_{t,i}\}_{i=1}^6$1, $\{I_{t,i},D_{t,i}\}_{i=1}^6$2, or $\{I_{t,i},D_{t,i}\}_{i=1}^6$3; a Vision Action $\{I_{t,i},D_{t,i}\}_{i=1}^6$4, which grounds the selected directional frame to a 2-D bounding box or pixel coordinate; and a Robot Action $\{I_{t,i},D_{t,i}\}_{i=1}^6$5, which back-projects the target into 3-D, plans a short-horizon path on the current occupancy map, and issues platform-specific low-level commands (Ding et al., 26 May 2026). The architecture is agentic rather than end-to-end in the conventional gradient-flow sense, but it is still navigation-guided because the decomposition is defined directly in navigational terms.

Hierarchical prediction-feedback is another recurrent design. UNeMo sits atop a topology-based VLN policy and introduces a Multimodal World Model that predicts the next visual state before the agent moves; a first policy layer proposes a coarse action $\{I_{t,i},D_{t,i}\}_{i=1}^6$6, the world model predicts $\{I_{t,i},D_{t,i}\}_{i=1}^6$7, and a second layer refines the action to $\{I_{t,i},D_{t,i}\}_{i=1}^6$8 after cross-attending the imagined future into all node embeddings (Huang et al., 24 Nov 2025). NavForesee similarly unifies hierarchical language planning and predictive foresight: the model generates the next sub-goal text $\{I_{t,i},D_{t,i}\}_{i=1}^6$9, predicts short-term and long-term future depth and semantic features via dual-horizon dream queries, and decodes a waypoint sequence $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$0 (Liu et al., 1 Dec 2025).

At the opposite end of the design spectrum are minimalist end-to-end systems. AerialVLA removes dense oracle hints and external detectors, using only dual-view perception, fuzzy onboard prompting, and autoregressive prediction of three numerical control tokens per frame (Xu et al., 15 Mar 2026). The “Navigation Framework Utilizing Vision-LLMs” is even more modular: Qwen2.5-VL-7B-Instruct remains frozen, the planner is prompt engineering plus JSON parsing, and execution is limited to $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$1 (Duan et al., 11 Jun 2025). The coexistence of these paradigms is a central characteristic of the field.

3. Navigation guidance as representation engineering

The defining technical question is how navigation guidance is represented inside the model. One line of work uses imagined or predicted future observations. In ImagineNav, the VLM is asked to compare six imagined images labeled A–F and return a JSON object with a “Reason” field and a “Choice” field; internally, the paper describes the selection as

$\mathcal A=(\Delta x,\Delta z,\Delta\psi)$2

This turns long-horizon object search into image ranking over plausible next views (Zhao et al., 2024). ImagineUAV generalizes this principle from single-view imagination to latent video diffusion, while NavForesee predicts both short-term and milestone-conditioned future features (Liu et al., 31 May 2026, Liu et al., 1 Dec 2025). FutureNav does not generate explicit images, but it still embeds world evolution through forward dynamics, inverse dynamics, and future generation heads operating over spatial latents (Zhang et al., 29 Jun 2026).

A second line encodes navigability through explicit geometric or semantic priors. “Narrowing the Gap between Vision and Action in Navigation” augments the waypoint predictor with CLIP ViT-B/16 RGB features, PPO-trained ResNet-50 depth features, and semantic obstacle masking that zeros out regions containing impassable objects such as sofa and table while retaining an open-area vocabulary including floor, stairs, and door (Zhang et al., 2024). “Think, Remember, Navigate” renders a top-down obstacle map $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$3 as a second image, serializes recent action history $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$4 as text, and uses LLaVA-1.6 to produce action scores $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$5 that are fused into a semantic value map for frontier ranking (Habibpour et al., 12 Nov 2025). Ground-then-Navigate in CARLA-NAV uses segmentation masks for the navigable region and short-term trajectory shape, and converts the largest connected component’s centroid into continuous steering, throttle, and brake via inverse projection and a local planner (Jain et al., 2022).

A third line represents navigation through topology or memory. TagaVLM constructs an online graph $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$6, interleaves node-specific visual embeddings with segmented instruction text through an Interleaved Navigation Prompt, and injects the node distance matrix into every self-attention layer through Spatial Topology Aware Residual Attention (Liu et al., 3 Mar 2026). FutureNav fuses Qwen3-VL visual tokens with frozen spatial-encoder features by residual addition $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$7 with $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$8, thereby placing spatial-awareness tokens directly inside the LLM sequence (Zhang et al., 29 Jun 2026). Uni-NaVid addresses long horizons with online visual token merging into current, short-term, and long-term buckets, using grid pooling factors $\mathcal A=(\Delta x,\Delta z,\Delta\psi)$9, $6$0, and $6$1 (Zhang et al., 2024). Uni-LaViRA externalizes memory into TODO List Memory and error recovery into Second Chance Backtrack (Ding et al., 26 May 2026).

Latency itself has also been treated as a guidance variable. TIC-VLA defines an effective latency $6$2, a delayed semantic state $6$3 obtained from lagged frames and the instruction, and an ego-motion offset $6$4. The control policy is then conditioned as

$6$5

so the controller reasons explicitly about stale semantics rather than treating latency as noise (Huang et al., 2 Feb 2026). AerialVLA uses a related but simpler strategy: it discretizes the relative bearing $6$6 into coarse prompts such as “straight ahead” or “forward-left,” and prepends these tokens to the navigation instruction (Xu et al., 15 Mar 2026).

4. Supervision, optimization, and training regimes

The supervision regimes range from pure prompting to multi-objective world-model learning. ImagineNav trains Where2Imagine as a 6-way relative-pose regressor distilled from human trajectories in Habitat-Web using mean-squared error on $6$7, with nearly-flat observations filtered by $6$8, angular shifts restricted to $6$9, and $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$0 frames apart reported as best (Zhao et al., 2024). In VLN-CE, the dual-action navigator combines a cross-entropy heatmap loss for high-level waypoint prediction,

$$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$1

with a sequence cross-entropy loss for low-level atomic actions,

$$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$2

and uses the joint objective $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$3 or its weighted variant (Zhang et al., 2024).

Several systems explicitly couple policy learning with world modeling. FutureNav adds inverse dynamics, forward dynamics, and future generation objectives to the policy head and optimizes

$$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$4

with typical weights $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$5 (Zhang et al., 29 Jun 2026). UNeMo trains the navigation policy and the CVAE-based Multimodal World Model jointly via

$$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$6

with $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$7, $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$8, and $$(\delta x,\delta y,\delta z,\delta\alpha,\delta\beta,\delta\gamma)$$9 in the reported setup (Huang et al., 24 Nov 2025). NavForesee combines planning loss, prediction loss, and action loss over $\{P_{t+1}^{(i)}\}_{i=1}^6$0M waypoint-annotated examples from R2R-CE and RxR-CE (Liu et al., 1 Dec 2025).

Aerial and real-time systems often combine autoregressive imitation with careful control parameterization. AerialVLA builds on OpenVLA-7B, fine-tunes the visual projector, adds LoRA with $\{P_{t+1}^{(i)}\}_{i=1}^6$1, discretizes each of $\{P_{t+1}^{(i)}\}_{i=1}^6$2 into $\{P_{t+1}^{(i)}\}_{i=1}^6$3 bins, and minimizes negative log-likelihood over expert action tokens across $\{P_{t+1}^{(i)}\}_{i=1}^6$4K frames and $\{P_{t+1}^{(i)}\}_{i=1}^6$5 trajectories (Xu et al., 15 Mar 2026). TIC-VLA uses a three-stage pipeline: VLM supervised fine-tuning on reasoning traces and GPT-5 waypoints, imitation learning with injected delay $\{P_{t+1}^{(i)}\}_{i=1}^6$6 and Smooth $\{P_{t+1}^{(i)}\}_{i=1}^6$7 loss on predicted poses, and online PPO reinforcement learning with asynchronously injected delays (Huang et al., 2 Feb 2026). BiliVLA first performs grounding-enhanced supervised fine-tuning with category, box, and action losses on $\{P_{t+1}^{(i)}\}_{i=1}^6$8 annotated frames, then refines the policy with Group Relative Policy Optimization, sampling groups of hypotheses and normalizing structured rewards defined over IoU, action correctness, and output schema validity (Lin et al., 22 Jun 2026).

Not all systems rely on gradient-based adaptation. Uni-LaViRA is explicitly training-free and argues that navigation can be recast into outputs already lying inside the natural output manifold of pretrained multimodal LLMs (Ding et al., 26 May 2026). LangNav likewise shows that navigation can be trained in low-data regimes using language as the perceptual representation, augmented by $\{P_{t+1}^{(i)}\}_{i=1}^6$9 GPT-4 synthetic trajectories generated from just $\{I_{t+1}^{(i)}\}_{i=1}^6$0 human-annotated R2R examples (Pan et al., 2023).

5. Benchmarks and empirical findings

Reported performance spans distinct benchmarks and metrics, so the results below are not a single leaderboard but a cross-section of representative outcomes.

Model	Benchmark	Reported result
ImagineNav	HM3D / HSSD	HM3D: SR $\{I_{t+1}^{(i)}\}_{i=1}^6$1, SPL $\{I_{t+1}^{(i)}\}_{i=1}^6$2; HSSD: SR $\{I_{t+1}^{(i)}\}_{i=1}^6$3, SPL $\{I_{t+1}^{(i)}\}_{i=1}^6$4
AerialVLA	TravelUAV	Seen: SR $\{I_{t+1}^{(i)}\}_{i=1}^6$5, SPL $\{I_{t+1}^{(i)}\}_{i=1}^6$6; Unseen-Map: SR $\{I_{t+1}^{(i)}\}_{i=1}^6$7, SPL $\{I_{t+1}^{(i)}\}_{i=1}^6$8
TIC-VLA (full)	DynaNav	NE $\{I_{t+1}^{(i)}\}_{i=1}^6$9, SR $g$0, SPL $g$1, CR $g$2
TagaVLM	R2R val-unseen	SR $g$3, SPL $g$4, NE $g$5
FutureNav-4B	R2R-CE / RxR-CE	R2R SR $g$6 in 0K; full-data R2R SR $g$7, SPL $g$8; full-data RxR SR $g$9, nDTW $f_{\mathrm{lang}}$0
Uni-NaVid	R2R / HM3D / MP3D-EQA / HM3D Following	R2R SR $f_{\mathrm{lang}}$1, SPL $f_{\mathrm{lang}}$2; HM3D ObjectNav SR $f_{\mathrm{lang}}$3, SPL $f_{\mathrm{lang}}$4; MP3D-EQA ACC $f_{\mathrm{lang}}$5; Following SR $f_{\mathrm{lang}}$6

These results are accompanied by extensive ablations. ImagineNav reports that feeding only current views to the VLM yields $f_{\mathrm{lang}}$7 SR on HM3D, adding imagined views with uniform sampling yields $f_{\mathrm{lang}}$8, learning Where2Imagine proposals with real images yields $f_{\mathrm{lang}}$9, and using NVS drops the full variant to $\mathtt{front}$0, which the paper interprets as evidence that both imagination and learned proposal generation are beneficial (Zhao et al., 2024). AerialVLA reports Unseen-Map performance of $\mathtt{front}$1 SR/SPL for raw training without geometry filtering, $\mathtt{front}$2 for a 5-view setup, $\mathtt{front}$3 for custom action tokens, and $\mathtt{front}$4 for the full model, attributing robustness to minimalist dual views, fuzzy prompting, and numerical tokenization (Xu et al., 15 Mar 2026).

TIC-VLA reports that blocking control during inference (“sync”) produces only $\mathtt{front}$5 SR and $\mathtt{front}$6 SPL, while the full asynchronous model reaches $\mathtt{front}$7 SR and $\mathtt{front}$8 SPL; its ablations further attribute $\mathtt{front}$9 SR to KV-cache versus a sparse waypoint interface, $\{I_{t,i},D_{t,i}\}_{i=1}^6$00 SR to latency injection during training, and $\{I_{t,i},D_{t,i}\}_{i=1}^6$01 SR to ego-motion offset modeling (Huang et al., 2 Feb 2026). TagaVLM reports a staged rise on R2R val-unseen from roughly $\{I_{t,i},D_{t,i}\}_{i=1}^6$02 SR for the base model without STAR-Att, INP, GA, or augmented data, to roughly $\{I_{t,i},D_{t,i}\}_{i=1}^6$03 with STAR-Att only, $\{I_{t,i},D_{t,i}\}_{i=1}^6$04 with STAR-Att + INP, $\{I_{t,i},D_{t,i}\}_{i=1}^6$05 with STAR-Att + INP + GA, and $\{I_{t,i},D_{t,i}\}_{i=1}^6$06 with all components plus data augmentation (Liu et al., 3 Mar 2026). “Think, Remember, Navigate” reports that removing Chain-of-Thought, action history, or the obstacle map reduces HM3D performance from $\{I_{t,i},D_{t,i}\}_{i=1}^6$07 SR/SPL to $\{I_{t,i},D_{t,i}\}_{i=1}^6$08, $\{I_{t,i},D_{t,i}\}_{i=1}^6$09, and $\{I_{t,i},D_{t,i}\}_{i=1}^6$10, respectively, and that progressively richer CoT prompts improve performance across HM3D, MP3D, and Gibson (Habibpour et al., 12 Nov 2025).

Zero-shot and real-world results form a separate strand of evidence. Uni-LaViRA reports zero-shot SR values of $\{I_{t,i},D_{t,i}\}_{i=1}^6$11 on VLN-CE R2R, $\{I_{t,i},D_{t,i}\}_{i=1}^6$12 on VLN-CE RxR, $\{I_{t,i},D_{t,i}\}_{i=1}^6$13 on HM3D-v2, $\{I_{t,i},D_{t,i}\}_{i=1}^6$14 on HM3D-OVON, $\{I_{t,i},D_{t,i}\}_{i=1}^6$15 ACC on MP3D-EQA, and $\{I_{t,i},D_{t,i}\}_{i=1}^6$16 on OpenUAV, while remaining training-free (Ding et al., 26 May 2026). Uni-NaVid reports zero-shot real-robot VLN on a Unitree GO2 with $\{I_{t,i},D_{t,i}\}_{i=1}^6$17 SR on $\{I_{t,i},D_{t,i}\}_{i=1}^6$18 simple instructions and $\{I_{t,i},D_{t,i}\}_{i=1}^6$19 SR on $\{I_{t,i},D_{t,i}\}_{i=1}^6$20 complex instructions, compared with $\{I_{t,i},D_{t,i}\}_{i=1}^6$21 and $\{I_{t,i},D_{t,i}\}_{i=1}^6$22 for NaVid (Zhang et al., 2024). BiliVLA reports an average action precision of $\{I_{t,i},D_{t,i}\}_{i=1}^6$23 and an overall success rate of $\{I_{t,i},D_{t,i}\}_{i=1}^6$24 in real-world phantom ERCP subtasks (Lin et al., 22 Jun 2026).

6. Limitations, misconceptions, and research directions

A common misconception is that stronger LLMs alone solve embodied navigation. Multiple papers argue otherwise, but in different ways. ImagineNav states that planning limited to text struggles to represent spatial occupancy and geometry layout, motivating imagined future images instead of text-only planning (Zhao et al., 2024). TagaVLM argues that static, disembodied VLM pretraining clashes with the dynamic and topologically structured nature of navigation, and therefore injects topology directly into self-attention rather than relying on textual map descriptions (Liu et al., 3 Mar 2026). FutureNav similarly contends that direct action generation without explicit world modeling can suffer from weaker long-range consistency and cumulative drift, and responds with auxiliary objectives for state transitions and future spatial states while keeping inference cost unchanged because only the policy head runs at test time (Zhang et al., 29 Jun 2026).

Another misconception is that mapless systems are geometry-free. The evidence points in the opposite direction. ImagineNav is mapless but still depends on NVS, relative pose proposals, and a PointNav controller (Zhao et al., 2024). AutoFly is end-to-end but introduces pseudo-depth from RGB to enhance spatial reasoning (Sun et al., 10 Feb 2026). “Think, Remember, Navigate” improves zero-shot ObjectNav by supplying a top-down obstacle map as a second image (Habibpour et al., 12 Nov 2025). Uni-LaViRA uses deterministic back-projection, local occupancy mapping, TODO-list memory, and backtracking despite being training-free (Ding et al., 26 May 2026). This suggests that the debate is not “geometry versus language,” but how geometry is represented and where it enters the action-selection loop.

Real-time deployment remains a persistent constraint. TIC-VLA is built around the observation that semantic inference is delayed relative to control, and shows that policies trained without asynchronous delay handling degrade sharply when VLM latency grows from $\{I_{t,i},D_{t,i}\}_{i=1}^6$25 to $\{I_{t,i},D_{t,i}\}_{i=1}^6$26 (Huang et al., 2 Feb 2026). ImagineUAV introduces step-distilled inference precisely to remove diffusion rollout and MPC at runtime (Liu et al., 31 May 2026). Uni-NaVid’s real-robot deployment uses a remote A100 server, approximately $\{I_{t,i},D_{t,i}\}_{i=1}^6$27 round-trip communication, and approximately $\{I_{t,i},D_{t,i}\}_{i=1}^6$28 server-side token-merge plus LLM inference (Zhang et al., 2024). By contrast, the frozen-Qwen modular framework reports poor generalization under a $\{I_{t,i},D_{t,i}\}_{i=1}^6$29-step budget and higher-resolution images, with only $\{I_{t,i},D_{t,i}\}_{i=1}^6$30 SR and $\{I_{t,i},D_{t,i}\}_{i=1}^6$31 SPL on the first $\{I_{t,i},D_{t,i}\}_{i=1}^6$32 R2R val-unseen trajectories (Duan et al., 11 Jun 2025).

The future directions identified in the literature are correspondingly concrete. AutoFly lists full-surround sensing via 360° LiDAR or multiple cameras, on-policy reinforcement learning, and higher-level search/exploration planners for long-range target discovery (Sun et al., 10 Feb 2026). Uni-LaViRA identifies knowledge distillation into open models, integration of SAM or Grounding DINO when large-area grounding confidence is low, stronger support for long instructions, and explicit pedestrian-intent reasoning for dynamic obstacles (Ding et al., 26 May 2026). “Think, Remember, Navigate” points to automated prompt search, better spatial training for map interpretation, and improved benchmarks due to annotation incompleteness (Habibpour et al., 12 Nov 2025). Across these proposals, a plausible implication is that navigation-guided VLA research is converging on hybrid designs in which semantic reasoning, spatial abstraction, and action execution remain tightly coupled, but not necessarily collapsed into a single undifferentiated predictor.