Visual Prompt Navigation (VPN)

• Visual Prompt Navigation (VPN) is a paradigm that uses concise visual or text prompts to bridge high-level intent and low-level control in navigation tasks.
• VPN employs varied formulations—from language-conditioned reactive navigation to language-free route overlays—to address challenges in perception, memory, and action selection.
• Empirical studies demonstrate that optimized prompt designs and fine-tuned memory aggregators significantly enhance navigation success, safety, and domain adaptation.

Searching arXiv for the specified VPN papers and closely related work to ground the article in current literature. {"query": "\"Visual Prompt Navigation\" arXiv", "max_results": 10} {"query": "(Tsai et al., 2023) Multimodal LLM for Visual Navigation", "max_results": 5} {"query": "(Feng et al., 3 Aug 2025) VPN: Visual Prompt Navigation", "max_results": 5} Visual Prompt Navigation (VPN) denotes a family of prompt-conditioned navigation paradigms in which prompts are used to mediate between task intent, perception, memory, and action selection. The term is not used uniformly. In one formulation, VPN is realized as a multimodal LLM for object-goal navigation conditioned on a short text prompt, current observation, and a learned history representation (Tsai et al., 2023). In another, VPN refers to language-free guidance through compact visual prompts drawn over 2D top-view maps (Feng et al., 3 Aug 2025). Related work extends the same general idea to prompt-tuned VLN encoders, landmark-image instructions, dual-view action overlays, prompt-conditioned safe navigation, and prompt-based interfaces for vision-language-action systems (Liu et al., 2023, Liu et al., 2023, Hong et al., 2024, Wang et al., 2 Dec 2025, Wang et al., 2024, Wang et al., 23 Mar 2026).

1. Terminological scope and core formulations

The literature uses “Visual Prompt Navigation” in several technically distinct senses. Some works treat the prompt as a high-level navigation intent, some as an explicit route drawn in map space, some as a learnable token sequence inserted into a visual encoder, and some as a structured overlay that constrains downstream control. A further extension uses “navigation” metaphorically to denote search over a pool of visual prompts for black-box LVLM steering rather than embodied movement (Woo et al., 30 Apr 2025).

Formulation	Prompt form	Representative papers
Prompt-conditioned reactive navigation	Short text prompt, current RGB observation, history tokens	(Tsai et al., 2023)
Language-free route guidance	Marked trajectory on 2D top-view map	(Feng et al., 3 Aug 2025)
Prompt-based VLN adaptation	Soft/deep visual prompts, action prompts, landmark images	(Liu et al., 2023, Liu et al., 2023, Lin et al., 2022, Hong et al., 2024)
Prompt-conditioned embodied control	Dual-view overlays, drivable-mask prompts, crosshairs, boxes	(Wang et al., 2 Dec 2025, Nasser et al., 6 Apr 2026, Wang et al., 23 Mar 2026)
Prompt-space routing for LVLMs	Per-image selection of visual cue overlays	(Woo et al., 30 Apr 2025)

This plurality is not merely terminological. It reflects different assumptions about where the bottleneck lies: temporal aggregation, geometric ambiguity, indoor domain shift, action-level alignment, spatial grounding, or black-box controllability. A common denominator is that prompt design becomes an explicit interface between high-level semantics and low-level decision making.

2. Prompt-minimal multimodal LLM navigation

The 2023 formulation in "Multimodal LLM for Visual Navigation" defines VPN as a navigation paradigm in which a low-level control policy is conditioned by a concise textual prompt and multimodal observations, with the prompt serving as a high-level intent or goal signal while the agent reacts to the scene dynamically (Tsai et al., 2023). The instantiated model consumes three inputs: a short text prompt describing the object-goal, the current visual observation, and a learned history representation summarizing past observations and actions. The output is a probability distribution over the discrete action set $$A = \{\text{stop}, \text{forward 25 cm}, \text{turn left/right 30}^\circ, \text{look up/down 30}^\circ\}$$.

Its architecture separates temporal aggregation from prompt construction. BLIP-2’s ViT plus Q-Former encodes the current RGB image; a history collector uses ResNet-50, linear embeddings for GPS, compass, previous action, and object category, and a GRU to maintain $h_t$ over time; and a frozen LLaMA-13B ingests prompt tokens together with projected observation and history tokens. Only the projection layers are trained during LLM fine-tuning, while the observation encoder, history collector, and LLaMA-13B remain frozen. The policy is written as

$$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$

A key technical claim of this line of work is that prompt bloat should be avoided. Prior prompt-heavy systems are described as concatenating instructions, observations, map or memory summaries, semantic frontiers, entropy estimates, and long histories into large text prompts while keeping the LLM frozen. The VPN alternative uses a simple text prompt plus current observation plus history collector, and moves temporal compression into a learned state aggregator. The history is projected into 32 tokens, the current observation into 32 tokens, and both are fused with the prompt before action prediction.

Training uses HM3D human demonstrations and collision signals. The target distribution mixes a strong behavior cloning policy with one-hot human actions,

$$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$

followed by collision masking and renormalization. The loss is imitation learning cross-entropy against the constructed target, with the collision mask functioning as an explicit safety bias. On HM3D-Semantics validation, the method reports Success $0.6790$ and SoftSPL $0.3723$, exceeding IL at Success $0.5980$ and SoftSPL $0.3051$, and RL_Ft at Success $0.6615$ and SoftSPL $0.3604$. The ablations are particularly informative: zero-shot MiniGPT4 fails with Success $h_t$0, a 15-frame consecutive-observation variant reaches only Success $h_t$1, and removing collision masking increases Collision Count from $h_t$2 to $h_t$3. This establishes a strong empirical point within the embodied setting: fine-tuning and compact learned memory are more consequential than elaborate prompt concatenation in unseen 3D environments.

3. Visual prompts as top-view route specifications

The 2025 paper "VPN: Visual Prompt Navigation" redefines VPN as a language-free embodied navigation paradigm in which compact visual prompts over 2D top-view maps replace natural-language instructions (Feng et al., 3 Aug 2025). Here the prompt is a marked route, typically a sequence of waypoints connected by arrows, overlaid on a floor plan. The policy conditions on egocentric observations, a dynamically built topological graph, and the prompt:

$h_t$4

This formulation emphasizes geometric precision and reduced ambiguity. The input map is resized to $h_t$5, center-cropped around the trajectory with side length equal to $h_t$6 pixels, tight-cropped to remove black borders, and optionally concatenated across floors via Order-Aware Floor Concatenation. The prompt features are encoded with ViT-B/16, while node representations are built from panoramic observations and graph structure. VPNet then aligns prompt tokens and node embeddings through cross-attention and graph-aware self-attention, and predicts among navigable neighbors and stop.

The work constructs two datasets, R2R-VP and R2R-CE-VP, by extending R2R and R2R-CE with corresponding visual prompts. It also introduces two augmentation strategies. View-level augmentation rotates prompt maps by $h_t$7, $h_t$8, $h_t$9, or $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$0 and randomizes initial heading in $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$1; trajectory-level augmentation expands training with PREVALENT and ScaleVLN episodes. The training objective combines behavior cloning and DAgger with $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$2.

The reported gains are large in the discrete setting. VPNet trained on R2R+PRE+SCA reaches on R2R test unseen NE $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$3, OSR $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$4, SR $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$5, and SPL $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$6. In continuous VPN-CE, VPNet trained on R2R+PRE yields val unseen NE $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$7, SR $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$8, and SPL $$p(a_t \mid o_t, h_t, x_t) = \pi_\theta(a_t \mid s_t), \quad s_t = (o_t, h_t, x_t).$$9. Prompt-form ablations show that route-marking lines matter more than decorative elaboration: a no-trajectory variant gives val unseen SR $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$0, cropped maps without trajectory give $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$1, lines plus text give $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$2, and lines only give $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$3. The noise studies are also central. On val unseen, clean prompts yield SR $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$4 and SPL $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$5; with $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$6 salt-and-pepper noise, SR drops to $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$7; removing the first-step cue yields SR $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$8. These results place top-view visual prompts as a direct substitute for language in tasks where route geometry is available and map alignment is reliable.

4. Prompt-based VLN as encoder adaptation, action alignment, and multi-modal instruction design

A substantial adjacent literature does not always use the term VPN explicitly, but it operationalizes prompt-based navigation through three recurring mechanisms: prompt-tuning of the visual encoder, retrieval of action-relevant prompt exemplars, and insertion of external images into the instruction stream.

DAP, described as a domain-aware prompt learning framework for VLN, learns 10 soft visual prompt tokens in the input space of a frozen visual encoder and tunes them on about 1,000 in-domain image–text pairs drawn from Matterport3D (Liu et al., 2023). CLIP supplies pseudo-labels via the template “A photo of a {object},” and only the prompt tokens and a small MLP head are updated during prompt tuning. The prompt-tuned encoder is then integrated into RecBERT. On R2R Validation-Unseen, DAP improves RecBERT initialized from PREVALENT from SR $$\tilde{P}_t = \alpha P_{\mathrm{SOTA},t} + (1-\alpha) P_{\mathrm{gt},t}, \quad \alpha = 0.8,$$9 to $0.6790$0, SPL $0.6790$1 to $0.6790$2, and NE $0.6790$3 to $0.6790$4. The reported gains are larger when the initialization is more out-of-domain, which is presented as evidence for the domain-gap hypothesis.

PANDA extends this direction with deep visual prompts and hard context prompts for sequential alignment (Liu et al., 2023). Deep prompt tokens are inserted after the image embedding layer and inter-layer in the visual encoder, with $0.6790$5 prompt tokens, while a second stage introduces count, sequential, individual, and overall context prompts aligned to sub-instructions and sub-paths through KL-based contrastive learning. On R2R Val-Unseen, the baseline RecBERT initialized from PREVALENT reports NE $0.6790$6, SR $0.6790$7, SPL $0.6790$8; adding deep visual prompts gives NE $0.6790$9, SR $0.3723$0, SPL $0.3723$1; adding hard context prompts gives NE $0.3723$2, SR $0.3723$3, SPL $0.3723$4; and combining both in PANDA yields NE $0.3723$5, SR $0.3723$6, SPL $0.3723$7.

ADAPT approaches prompting at the action level rather than the encoder level (Lin et al., 2022). It retrieves instruction-related action prompts from a pre-built prompt base, where each prompt is a modality-aligned pair of a single-view image sub-prompt and a text sub-prompt such as “walk past the chair.” These prompt features are concatenated with the instruction encoding and optimized with imitation learning, reinforcement learning, a modality alignment loss, and a sequential consistency loss. On R2R Test-Unseen with ResNet-152 features, ADAPT reports NE $0.3723$8, SR $0.3723$9, SPL $0.5980$0, compared with VLN⟳BERT at NE $0.5980$1, SR $0.5980$2, SPL $0.5980$3.

VLN-MP generalizes prompting to explicit multi-modal instructions by interleaving landmark images with text (Hong et al., 2024). It defines aligned, related, and terminal prompt settings, provides a training-free pipeline using GPT-4 plus GroundingDINO or GLIP to transform text-only instructions into multi-modal ones, and introduces a Multi-modal Prompts Fusion module compatible with HAMT and DUET. On RxR with HAMT, text-only evaluation gives seen SR $0.5980$4 and unseen SR $0.5980$5; with aligned visual prompts derived by the pipeline, seen SR rises to $0.5980$6 and unseen SR to $0.5980$7; with an alternative aligned set, seen SR reaches $0.5980$8 and unseen SR $0.5980$9. The same work also introduces a pre-explore regime in which a first pass gathers images for a more informed second pass.

A distinct minimalist line freezes a large VLM and relies on prompt engineering plus structured history management rather than prompt-tuning. "A Navigation Framework Utilizing Vision-LLMs" uses Qwen2.5-VL-7B-Instruct with two-frame visual input and strict JSON output parsing, with state $0.3051$0 and action space $0.3051$1 (Duan et al., 11 Jun 2025). Under a strict 50-step cap on the first 20 val-unseen R2R-VLN-CE trajectories, it reports DTG $0.3051$2 m, SR $0.3051$3, SPL $0.3051$4, against a zero-movement baseline DTG $0.3051$5 m. This is explicitly presented as a challenging generalization result rather than a competitive SOTA claim.

SeeNav-Agent adds a further prompt-centric synthesis by projecting action candidates and target cues directly into the image space (Wang et al., 2 Dec 2025). Its dual-view Visual Prompt combines a bird’s-eye view and first-person view with bounding boxes, navigation lines, agent markers, action projections, and view alignment. Zero-shot GPT-4.1 with the full visual prompt reaches success $0.3051$6 on EmbodiedBench Navigation, compared with $0.3051$7 for a first-person-only baseline. With post-training via Step Reward Group Policy Optimization, Qwen2.5-VL-3B reaches success $0.3051$8, compared with $0.3051$9 for VP + GRPO and $0.6615$0 for VP + GiGPO with the proposed verifiable process reward. The ablations show that dual-view alone can reduce performance to $0.6615$1, indicating that view multiplicity without carefully designed prompt structure can be counterproductive.

5. Beyond classical VLN: safe navigation, off-road traversability, robotic control, and prompt-space routing

Several papers extend the VPN idea beyond indoor graph navigation. In "VisionGPT," prompts dynamically steer open-vocabulary detection, spatial semantics, and anomaly reporting for safe visual navigation (Wang et al., 2024). The system partitions each frame into Left, Right, Front, and Ground regions, combines YOLO-World detections with Instruction, Sensitivity, Location, and Motion prompts, and uses GPT-3.5 Turbo and GPT-4 to produce hazard judgments and audio guidance. On its anomaly-detection benchmark, the full prompt stack with low sensitivity, Location, and Instruction yields AP $0.6615$2 and AUC $0.6615$3; removing Instruction lowers AP to $0.6615$4, removing Location lowers AP to $0.6615$5, and high sensitivity lowers AP to $0.6615$6.

In off-road autonomy, "Visual Prompt Based Reasoning for Offroad Mapping using Multimodal LLMs" treats the visual prompt as a segmented image annotated with numeric region labels and asks the VLM to output the numbers corresponding to drivable areas (Nasser et al., 6 Apr 2026). The prompt is a side-by-side collage of the original image and the annotated segmentation. SAM2 generates masks, the VLM selects drivable regions, and the selected mask is projected into a costmap for D* Lite, Hybrid A*, and Stanley/PID control. On the simulation dataset, GPT-5-Mini obtains mIoU $0.6615$7, ChatGPT-4o-latest $0.6615$8, and MiniCPM-V-2_6-int4 $0.6615$9. In Isaac Sim reachability tests, Start S1 to Goal A and Start S1 to Goal B each achieve $0.3604$0 success, while Start S2 to Goal C achieves $0.3604$1 success. On real off-road datasets, the method reports average IoU $0.3604$2 across ORFD, RUGD, and O2DTD, outperforming PathFormer on ORFD and RUGD but underperforming on O2DTD.

VP-VLA reinterprets prompting as an explicit interface between a high-level planner and a low-level controller (Wang et al., 23 Mar 2026). System 2 identifies sub-tasks, target objects, and goal regions; SAM3 segments them; and structured prompts, specifically crosshairs and bounding boxes, are rendered into a dedicated prompt image. System 1 then conditions on language, raw observations, and the prompt image to output continuous action chunks. The training objective combines an $0.3604$3 action loss with an auxiliary grounding loss over discretized prompt coordinates. On RoboCasa-GR1-Tabletop, VP-VLA reaches $0.3604$4 versus QwenOFT at $0.3604$5; on SimplerEnv it reports $0.3604$6 versus $0.3604$7. The ablations show that removing the grounding loss drops RoboCasa performance to $0.3604$8, replacing a crosshair with a single point yields $0.3604$9, and directly overlaying prompts on the primary RGB image yields $h_t$00.

A conceptually different but terminologically related use appears in "Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision LLMs" (Woo et al., 30 Apr 2025). There, VPN is defined as the process of searching and selecting, per input image, the most effective visual cue overlay from a candidate prompt pool to steer an LVLM away from object hallucination. The prompt pool includes bounding box, circle, arrow, center point, reverse blur, crop, and no prompt; a router built from a frozen CLIP ViT-L/14@336px encoder and an MLP predicts which prompt should be applied. On POPE Random, LLaVA-1.5 improves from Acc $h_t$01 and F1 $h_t$02 to Acc $h_t$03 and F1 $h_t$04; on CHAIR, GPT-4o improves from CH_S $h_t$05 to $h_t$06 and CH_I $h_t$07 to $h_t$08. Although this is not embodied navigation, it broadens the acronym into prompt-space policy selection under black-box constraints.

6. Recurring limitations, misconceptions, and research directions

A persistent misconception is that VPN names a single task or architecture. The cited literature does not support such a unified reading. Some VPN systems are language-conditioned and prompt-minimal, some are explicitly language-free, some are parameter-efficient prompt-tuning schemes, and some transform control into prompt-conditioned visual grounding. A second misconception is that stronger prompt engineering alone is sufficient. The HM3D results of the multimodal LLM navigation paper show zero-shot MiniGPT4 at Success $h_t$09, supporting the narrower claim that embodied visual navigation in unseen 3D environments required fine-tuning in that setting (Tsai et al., 2023). SeeNav-Agent likewise reports that dual-view input without the full prompt recipe can reduce success to $h_t$10, showing that additional prompt channels are not automatically beneficial (Wang et al., 2 Dec 2025).

Several limitations recur across the literature. Top-view VPN relies on accurate map generation, floor alignment, and viewpoint projection; poor reconstructions and missing first-step cues measurably degrade performance (Feng et al., 3 Aug 2025). Prompt-tuning approaches such as DAP and PANDA depend on CLIP-derived pseudo-labels and small in-domain indoor datasets, so label noise and taxonomy coverage can cap performance (Liu et al., 2023, Liu et al., 2023). Multi-modal instruction systems depend on prompt relevance: VLN-MP improves as prompts become more aligned and more numerous, but irrelevant or conflicting prompts can misdirect attention (Hong et al., 2024). Off-road and safe-navigation variants remain sensitive to segmentation quality, ambiguity in prompt design, and environmental factors such as unseen obstacles, extreme slopes, low-resolution imagery, or scene-dependent sensitivity calibration (Nasser et al., 6 Apr 2026, Wang et al., 2024). Control-oriented variants such as VP-VLA assume reliable target naming and segmentation; prompt concurrency and inaccurate masks directly reduce performance (Wang et al., 23 Mar 2026).

The future directions named in the papers are correspondingly diverse. For prompt-minimal multimodal LLM navigation, proposed directions include transformer-based memories, spatial memory, richer visual modalities, uncertainty-aware losses, and lightweight LLM adapters (Tsai et al., 2023). For map-based VPN, the main priorities are better domain adaptation, automated prompt generation, and integration with language or interactive correction (Feng et al., 3 Aug 2025). For prompt-tuned VLN, richer multi-granularity alignment, joint prompt-policy optimization, and broader transfer beyond current indoor benchmarks are explicitly proposed (Liu et al., 2023, Liu et al., 2023). For dual-view and VLA-oriented systems, the obvious open issue is how to retain the benefits of prompt-conditioned grounding when BEV, exact object annotations, or calibrated overlays are unavailable in real deployments (Wang et al., 2 Dec 2025, Wang et al., 23 Mar 2026).

Taken together, the literature presents VPN less as a single algorithm than as a design principle: prompts can be made concise rather than verbose, visual rather than textual, learnable rather than handcrafted, route-level rather than object-level, or pixel-precise rather than latent. What remains stable across these variants is the attempt to externalize intent and spatial structure into an explicit prompt interface so that navigation policies need not infer all task structure implicitly from raw inputs alone.