GoViG: Goal-Conditioned Navigation Instruction Generation

• The paper introduces a novel task formulation (GoViG) that generates natural language navigation instructions from egocentric start and goal visual observations without relying on full-route supervision.
• It decomposes the problem into visual forecasting and instruction generation subtasks, leveraging an autoregressive multimodal model with techniques like token discrepancy loss and LoRA adapters.
• Experiments on the R2R-Goal benchmark reveal that interleaved reasoning improves BLEU-4 and SSIM scores compared to traditional approaches, highlighting both strengths and current limitations.

Goal-Conditioned Visual Navigation Instruction Generation (GoViG) is a task formulation in embodied AI that aims to generate natural-language navigation instructions from raw egocentric visual observations of an initial state and a goal state, without relying on semantic annotations, environmental maps, panoramic route traces, action labels, GPS, or other privileged structure (Wu et al., 13 Aug 2025). In its explicit formulation, the system receives an initial visual sequence and a goal observation, then produces a route description that is expected to be spatially accurate, linguistically coherent, and grounded in both observed and anticipated visual evidence. Within the broader navigation-language literature, GoViG occupies a specific position: it is more constrained than general path-conditioned “speaker” models because it avoids full-route supervision at inference time, yet more structured than generic image captioning because its output must encode navigational intent, path progression, and goal attainment.

1. Task definition and scope

The formal GoViG input consists of an initial egocentric visual sequence

$\mathcal{O}=\{o_1,o_2,\dots,o_n\}$

and a goal observation

$o_g,$

where each frame is an RGB image

$o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$

The output is a navigation instruction $I$ describing how to reach the goal from the start (Wu et al., 13 Aug 2025).

This formulation is explicitly presented as the inverse of the more common instruction-following setting. Instead of mapping language and perception to actions, GoViG maps start/goal visual evidence to language. The formulation is also intentionally austere. The task is defined under five constraints: it is egocentric only; it requires no privileged structured inputs; it does not require an explicit intermediate symbolic representation; it is goal-conditioned by the target visual observation; and it treats visual forecasting as part of the reasoning process rather than as an optional add-on (Wu et al., 13 Aug 2025).

In the benchmark instantiation R2R-Goal, each trajectory retains an initial sequence of six egocentric observations and a final goal observation. This makes GoViG distinct from route-conditioned generators that receive the entire traversed path during generation. A common misconception is that any navigation speaker is already a GoViG model. That is not the case. Many earlier systems are path-conditioned or trajectory-conditioned, whereas GoViG is defined around initial-state and goal-state visual conditioning without explicit access to the full route at inference time (Wu et al., 13 Aug 2025).

2. Core methodology: visual forecasting and multimodal instruction generation

The defining methodological move in GoViG is to decompose instruction generation into two coupled subtasks: Navigation Visualization and Instruction Generation with Visual Cues. The first predicts intermediate visual observations that bridge the start and goal; the second generates language conditioned on observed and forecasted visual states. Both are implemented within a single autoregressive multimodal LLM based on the Chameleon architecture and instantiated with GAIR Anole-7B (Wu et al., 13 Aug 2025).

In Navigation Visualization, the model takes a partial visual sequence and the goal observation, then predicts the next visual observation autoregressively. In Instruction Generation with Visual Cues, the model takes observed frames, predicted intermediate frames, the goal frame, and optionally a previous instruction, then generates a route description. This decomposition is motivated by the claim that direct start/goal-to-text mapping is underconstrained, whereas visual forecasting provides explicit intermediate structure more closely aligned with human route planning (Wu et al., 13 Aug 2025).

The model uses a VQ image tokenizer and a BPE text tokenizer. Input images are resized to $256\times256$, and each image is discretized into 784 visual tokens; in larger-context ablations, this is reduced to 400 tokens per frame. The model context length is 4096 tokens. The tokenizers are frozen, and training updates only LoRA adapters inserted into the $qkv$ projections of the Transformer, with LoRA rank 16 (Wu et al., 13 Aug 2025).

Two inference-time reasoning schemes are central. In One-Pass Multimodal Reasoning, the model first forecasts a route and only then generates the final instruction: $$I = F_\Theta\bigl(\{o_1, \hat{o}_{i_1}, \dots, \hat{o}_{i_{m-1}}, o_g\}\bigr).$$ In Interleaved Multimodal Reasoning, the model alternates between next-frame prediction and instruction refinement: $$I_t = F_\Theta\big(\hat{\mathcal{O}}^{(t+1)} \cup \{o_g, I_{t-1}\}\big).$$ Both procedures stop visual rollout when the predicted frame becomes sufficiently similar to the goal: $$\mathrm{SSIM}(\hat{o}_{k+t}, o_g) > \tau,\qquad \tau = 0.7.$$ One-Pass emphasizes global route summarization, whereas Interleaved emphasizes incremental revision and local grounding. The reported results show that Interleaved is the stronger of the two on the main instruction metrics (Wu et al., 13 Aug 2025).

3. Objectives, optimization, and representation learning

GoViG is trained jointly over interleaved batches from the two subtasks. For visual forecasting, the paper defines a Token Discrepancy Loss $\mathcal{L}_{\text{vis}}$, which weights the predicted visual-token distribution by embedding-level discrepancy to the VQ codebook rather than relying only on hard-token cross-entropy. For instruction generation, it uses a label-smoothed cross-entropy objective $o_g,$0 over text tokens (Wu et al., 13 Aug 2025).

The paper does not give a single weighted combined loss equation, but it explicitly states that samples from both subtasks are interleaved within batches and optimized jointly. Training uses input-label concatenation with $o_g,$1 masking so that loss is computed only on target tokens. The optimizer is AdamW with learning rate $o_g,$2, trained for 20 epochs on 4 NVIDIA A100 80GB GPUs with global batch size 8, per-GPU batch size 1, and gradient accumulation 2 (Wu et al., 13 Aug 2025).

The visual forecasting loss is empirically important. With $o_g,$3, the model reaches SSIM $o_g,$4, PSNR $o_g,$5, LPIPS $o_g,$6, and DreamSIM $o_g,$7; without it, the corresponding values are SSIM $o_g,$8, PSNR $o_g,$9, LPIPS $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$0, and DreamSIM $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$1. This indicates that the loss is not merely auxiliary but materially affects the perceptual and structural quality of predicted intermediate observations (Wu et al., 13 Aug 2025).

A second ablation concerns context and visual token budget. In the visualization task, context size 2 with token length 784 yields SSIM $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$2, while context size 4 with token length 400 drops to SSIM $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$3. In instruction generation, context size 3 with token length 784 yields BLEU-4 $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$4, whereas context size 5 with token length 400 drops to BLEU-4 $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$5. This suggests that moderate temporal expansion helps, but aggressive image-token compression harms both forecasting and language generation (Wu et al., 13 Aug 2025).

4. Benchmarking and empirical performance

The principal benchmark is R2R-Goal, introduced specifically for GoViG. It combines simulated trajectories derived from R2R-CE and HA-R2R with a real-world subset from GO Stanford, ReCon, and HuRoN. The simulated portion contains 74,737 trajectories: 48,490 train, 3,573 validation_seen, 8,361 validation_unseen, and 14,313 test. Trajectory lengths range from 8 to 29 steps. The real-world subset contains 150 manually annotated trajectories (Wu et al., 13 Aug 2025).

The benchmark uses standard text-generation metrics for instruction quality—BLEU-4, CIDEr, METEOR, and ROUGE-L—and SSIM, PSNR, LPIPS, and DreamSIM for the visual forecasting subtask. The model is evaluated against Speaker-Follower, LANA, GPT-4o, GPT-4o + CoT, C-Instructor, Anole-7B (Direct), and Anole-7B + CoT (Wu et al., 13 Aug 2025).

On validation_unseen, the strongest GoViG variant is Interleaved reasoning, with BLEU-4 $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$6, CIDEr $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$7, METEOR $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$8, and ROUGE-L $o_i, o_g \in \mathbb{R}^{H\times W\times 3}.$9. On test, Interleaved achieves BLEU-4 $I$0 and CIDEr $I$1. On the real-world zero-shot subset, it reaches BLEU-4 $I$2, compared with $I$3 for One-Pass and substantially lower values for prior baselines (Wu et al., 13 Aug 2025).

Model	validation_unseen	real-world zero-shot
C-Instructor	BLEU-4 $I$4, CIDEr $I$5	BLEU-4 $I$6
Anole-7B + One-pass	BLEU-4 $I$7, CIDEr $I$8	BLEU-4 $I$9
Anole-7B + Interleaved	BLEU-4 $256\times256$0, CIDEr $256\times256$1	BLEU-4 $256\times256$2

The visual forecasting results are similarly strong. On R2R-Goal validation_unseen, GPT-4o + DALL·E records SSIM $256\times256$3, PSNR $256\times256$4, LPIPS $256\times256$5, and DreamSIM $256\times256$6; Anole-7B (Direct) reaches SSIM $256\times256$7, PSNR $256\times256$8, LPIPS $256\times256$9, and DreamSIM $qkv$0; the GoViG model reaches SSIM $qkv$1, PSNR $qkv$2, LPIPS $qkv$3, and DreamSIM $qkv$4. The paper highlights LPIPS reduction by $qkv$5 and DreamSIM reduction by $qkv$6 over direct Anole prompting (Wu et al., 13 Aug 2025).

The benchmark evidence also clarifies current limitations. Despite strong text-overlap performance, the paper does not report follower-based execution success for generated instructions, and the real-world subset remains small at 150 annotated trajectories. This suggests that current empirical validation is strong on textual and perceptual fidelity but weaker on downstream executability (Wu et al., 13 Aug 2025).

5. Relation to adjacent research paradigms

GoViG emerges from a broader landscape of navigation-language research in which conditioning signals, scene representations, and outputs vary substantially. The most important distinction is between systems that generate language from a complete realized path and systems that generate language from initial/goal visual context alone.

Paradigm	Representative works	Relation to GoViG
Trajectory-conditioned instruction generation	BEVInstructor (Fan et al., 2024), AIGeN (Rawal et al., 2024), InstruGen (Yan et al., 2024), FCA-NIG (Cui et al., 10 Jun 2025)	Strongly path-conditioned; often uses full observation-action route, BEV, or sub-trajectory annotations
Goal-conditioned assistive or single-image generation	BLV contextual instruction generation (Merchant et al., 2024)	Explicit goal input, but short-horizon and single-image rather than embodied route generation
Map-conditioned navigation language	Semantic map-based generation (Li et al., 2024)	Uses top-down semantic maps with route overlays; relies on structured scene abstraction rather than raw egocentric start/goal views
Goal-conditioned navigation substrates without language generation	Visual goal prediction (Misra et al., 2018), ViNG (Shah et al., 2020), image-goal Transformer policy (Pelluri, 2024), VISTA (Huang et al., 9 May 2025), PiJEPA (Chahe et al., 26 Mar 2026), NavWAM (Azuma et al., 11 Jun 2026)	Provide latent goal, waypoint, imagination, or world-model structure that could support future GoViG systems

Among route-conditioned generators, BEVInstructor is especially important because it argues that 2D perspective observations alone underrepresent 3D geometry and object semantics, and therefore fuses perspective features with Bird’s Eye View features before prompting an MLLM. Its results on REVERIE are particularly relevant because they “underscore the versatility and generality of our architectural design for goal-based tasks,” but the formulation remains primarily trajectory-conditioned rather than explicitly goal-conditioned (Fan et al., 2024).

AIGeN, InstruGen, and FCA-NIG represent a different branch: large-scale synthetic instruction generation for VLN training. AIGeN generates instructions from path image sequences with endpoint object cues through a GPT-2/BERT adversarial framework (Rawal et al., 2024). InstruGen constructs path-instruction pairs from YouTube house-tour videos using ChatGPT-4V, room labels, transition actions, and a multi-stage verification mechanism (Yan et al., 2024). FCA-NIG goes further by automatically constructing sub-instruction–sub-trajectory and entity–landmark alignments, producing the FCA-R2R dataset with 178,270 trajectories, 534,810 instructions, 2,519,919 sub-pairs, and 5,240,981 entity-landmark pairs (Cui et al., 10 Jun 2025). These works show that path-grounded language can be synthesized at scale, but they presuppose a known route.

Other neighboring lines focus not on instruction generation but on latent goal representation. Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction factorizes execution into goal prediction and action generation, introducing a visual goal distribution in observation space (Misra et al., 2018). ViNG uses goal images, graph search, and waypoint proposal for open-world navigation (Shah et al., 2020). Transformers for Image-Goal Navigation models image goals, observation history, and past actions autoregressively to predict actions (Pelluri, 2024). VISTA introduces “imagine-and-align” navigation with generated future visual states and perceptual alignment (Huang et al., 9 May 2025). PiJEPA warm-starts latent world-model planning from a policy prior for instruction-conditioned visual navigation (Chahe et al., 26 Mar 2026). NavWAM models future observations, goal-progress values, and action chunks in a shared latent sequence for closed-loop goal-conditioned control (Azuma et al., 11 Jun 2026). A plausible implication is that future GoViG systems may use such latent predictive structures as pre-linguistic planning backbones, with language generation layered on top.

6. Limitations, misconceptions, and future directions

A central misconception in the area is to equate GoViG with any navigation speaker. The literature shows several non-equivalent formulations. Route-conditioned speakers such as BEVInstructor or AIGeN assume access to the full observation-action trajectory during generation (Fan et al., 2024, Rawal et al., 2024). Single-image assistive systems for blind and low-vision users use one egocentric image and a natural-language goal, not a temporally extended embodied route (Merchant et al., 2024). Semantic-map generators rely on structured top-down abstractions and explicit route overlays (Li et al., 2024). GoViG, by contrast, is defined by start-sequence and goal-observation conditioning under raw egocentric input (Wu et al., 13 Aug 2025).

The current GoViG formulation also carries technical limitations. Training still depends on trajectory data with aligned instructions and frame sequences. Visual forecasting can accumulate errors, especially because the method uses repeated one-step prediction. The SSIM termination rule

$qkv$7

is heuristic rather than task-optimal. The benchmark’s real-world subset is small. Evaluation is dominated by text-overlap metrics rather than embodied execution, and the paper explicitly does not report follower-based success for generated instructions (Wu et al., 13 Aug 2025).

Related work points to additional concerns. The BLV study emphasizes that useful navigation language must communicate uncertainty, choose accessible landmarks, vary granularity by environment, and avoid unsupported metric precision (Merchant et al., 2024). Semantic-map-based generation shows that abstraction can improve efficiency, but also that missing room labels and object properties lead to hallucination and directional errors (Li et al., 2024). FCA-NIG shows that fine-grained alignment supervision improves downstream navigation performance, implying that future GoViG benchmarks may benefit from explicit sub-instruction and entity grounding rather than full-instruction text alone (Cui et al., 10 Jun 2025).

Several future directions are already visible in the literature. The GoViG paper itself identifies interactive navigation scenarios and integration of real-time environmental feedback as natural extensions (Wu et al., 13 Aug 2025). A plausible implication from neighboring work is that stronger future systems will combine raw egocentric goal conditioning with richer spatial abstractions such as BEV grounding, semantic-map memory, or waypoint-based world models, while preserving the task’s commitment to minimal privileged input (Fan et al., 2024, Azuma et al., 11 Jun 2026). Another plausible implication is that evaluation will need to move toward execution-aware criteria: whether a generated instruction is not only similar to a reference, but followable, safe, and robust under partial observability.

Taken together, the literature positions GoViG as a distinct research program rather than a minor variation of VLN speaker models. Its core problem is the generation of route language from raw egocentric start/goal evidence, and its current solutions increasingly rely on explicit multimodal reasoning, latent visual forecasting, and tighter grounding between predicted intermediate states and verbal descriptions (Wu et al., 13 Aug 2025).