---
title: 'R2R-VP: Visual Prompt Navigation Benchmark'
url: https://www.emergentmind.com/topics/r2r-vp
type: topic
---

# R2R-VP: Visual Prompt Navigation Benchmark

R2R-VP denotes **Room-to-Room with Visual Prompts**, a **discrete-environment** benchmark for **Visual Prompt Navigation (VPN)** that transfers the original **R2R** task from language-conditioned route following to **visual-prompt-guided** route following. In this formulation, the natural-language instruction is removed and replaced by a **2D top-view map** containing a rendered trajectory prompt, while the underlying Matterport3D viewpoint-graph navigation problem, panoramic observations, and goal-reaching objective are retained. The benchmark therefore studies whether route intent can be conveyed through a visually marked path rather than through text, and it serves as the discrete counterpart to **R2R-CE-VP** in continuous environments [2508.01766].

## 1. Definition and nomenclature

R2R-VP is built directly from **Room-to-Room (R2R)** episodes. In the original R2R setting, an agent is given a start viewpoint, a natural-language route instruction, and must navigate over a scene’s viewpoint graph to the target. In R2R-VP, the episode structure is preserved, but the language instruction is replaced by a **top-down map image containing a rendered visual trajectory prompt**. The resulting guidance is visual, spatially grounded, and map-based rather than symbolic and linguistic [2508.01766].

The benchmark is explicitly **discrete**. The paper separately defines **R2R-CE-VP** as the continuous-environment counterpart derived from **R2R-CE**, but R2R-VP itself remains a graph-navigation task with node selection rather than low-level motor control [2508.01766].

The term is also subject to a genuine nomenclatural ambiguity in the VLN literature. A later graph-based VLN-CE paper states that **“R2R-VP usually refers to this classic setting in practice”**, namely navigation over **panoramic viewpoints / graph nodes** with viewpoint transitions or panoramic decisions. In contrast, the VPN paper introduces **R2R-VP** as **Room-to-Room with Visual Prompts**. Accordingly, in recent work the same string can denote either the classic viewpoint-graph regime in a loose practical sense or the specific visual-prompt benchmark introduced for VPN [2512.20940].

## 2. Formal problem setting

R2R-VP is formulated as navigation on an undirected graph
$$
\mathcal{G} = (\mathcal{V}, \mathcal{E}),
$$
where $\mathcal{V}$ denotes navigable nodes and $\mathcal{E}$ denotes connectivity edges. Starting from an initial node and guided by visual prompts, the agent explores $\mathcal{G}$ and attempts to reach the target node [2508.01766].

The agent’s inputs comprise three principal components. First, it receives the visual prompt map or maps, encoded after processing as prompt embeddings. Second, at step $t$ it observes panoramic views
$$
V_t=\{v_{t,i}\}_{i=1}^{n}.
$$
Third, it identifies a navigable subset
$$
\mathcal{N}(V_t)\subseteq V_t,
$$
whose elements correspond to neighboring nodes and their spatial coordinates. The paper further specifies relative-orientation information through a direction embedding $W_{ij}^d$ and navigability-type information through a navigability embedding $W_{ij}^n$ [2508.01766].

The action space is high-level and graph-based. Candidate actions are represented as
$$
\hat{v}=(\text{stop}, nav_1, nav_2, \dots, nav_{v'}),
$$
and the model scores them by
$$
s^g_i = \text{FFN}(\hat{v}_i).
$$
After masking the current node and revisits, the agent selects the highest-scoring candidate and then **navigates to it via the shortest path in the topological graph $\mathcal{G}_t$**. This is a node-selection regime rather than a continuous-control regime [2508.01766].

A central structural property is that the agent **does not know the full graph in advance**. Instead, it incrementally builds a partial graph
$$
\mathcal{G}_t = \{\mathcal{V}_t, \mathcal{E}_t \mid \mathcal{V}_t \subseteq \mathcal{V}, \mathcal{E}_t \subseteq \mathcal{E} \},
$$
whose node types include the current node, visited nodes, and navigable nodes [2508.01766].

## 3. Dataset construction and prompt representation

The R2R-VP dataset is produced by a four-step transfer pipeline from R2R. First, for each episode, a **2D top-view map** is generated from the **90 MP3D environments**, “following a procedure similar to LED.” Second, the pixel coordinates of all viewpoints are computed on the top-view map and connected **sequentially with arrows** to represent the trajectory. Third, a **square crop centered on the trajectory** is extracted, with side length equal to the greater of trajectory width or height **plus a 60-pixel margin**. Fourth, black borders are removed by detecting the bounding box of all non-zero pixels [2508.01766].

After preprocessing, all prompt maps are resized to **$224 \times 224$** before encoding. The prompt itself is intended to mark the **trajectory path**, **waypoints / key points**, and the **direction of traversal** using arrows. The paper does not provide explicit 3D-to-2D projection formulas or a formal world-to-pixel transformation equation, so the geometric projection procedure is operationally described rather than analytically specified [2508.01766].

Multi-floor episodes are handled through **Order-Aware Floor Concatenation (OAFC)**. For the $i$-th prompt map,
$$
\mathcal{P}_i^o = \text{ViT}(\mathcal{P}_i)+b_i
$$
and the sequence is concatenated as
$$
\mathcal{P} = [\mathcal{P}_1^o,\mathcal{P}_2^o, \dots,\mathcal{P}_k^o],
$$
where $b_i$ is an order embedding that preserves floor traversal order. This mechanism is motivated by the multi-floor structure of MP3D scenes; among the 90 MP3D scenes, **32 have 1 floor, 32 have 2 floors, 17 have 3 floors, 8 have 4 floors, and 1 has 5 floors** [2508.01766].

The discrete R2R-VP training set contains **61 scenes** and **4,638 episodes**. For comparison, the original discrete R2R training split had **61 scenes** and **4,675 episodes**, implying that some episodes were removed during transfer. The paper also constructs visual-prompt training data from **PREVALENT**, yielding **60 scenes** and **177,134 episodes**, and from **ScaleVLN**, yielding **523 scenes** and **1,600,945 episodes** after retaining HM3D scenes or floors with acceptable top-view map quality [2508.01766].

## 4. VPNet architecture and training protocol

The benchmark paper introduces **VPNet** as the dedicated baseline model for VPN. In the discrete R2R-VP setting, VPNet is based on **DUET** and comprises a **ViT-based Visual Prompt Encoder**, a **Node Embedding** module, and a **Graph-aware Cross-modal Encoder**; the implementation details also state that a **local cross-modal encoder** is used in the discrete setting [2508.01766].

Prompt maps are encoded with **ViT-B/16**. Panoramic scene views are likewise represented with visual features $W_{ij}^v$, which are combined with direction and navigability signals:
$$
W_{ij}' = \text{LN}(W_{ij}^v + W_{ij}^d + W_{ij}^n).
$$
A two-layer panoramic self-attention encoder then contextualizes the views:
$$
W_{i} = \text{SelfAttn}(W_{i}').
$$
The node embedding at time $t$ is formed by adding step embedding, position embedding, and the averaged contextualized view features:
$$
\mathcal{V}_{ti} = \mathcal{V}_{ti}^s + \mathcal{V}_{ti}^p + \frac{1}{n} \sum_{j=1}^{n} W_{ij}.
$$
A special stop node is appended, and prompt embeddings and node embeddings are fused through cross-modal and graph-aware reasoning before the policy head scores actions by $s_i^g = \text{FFN}(\hat{v}_i)$ [2508.01766].

Training uses a mixture of **behavior cloning** and **DAgger**, with the shortest-path next node on the on-policy partial graph serving as the pseudo-demonstrator action in DAgger. For R2R-VP, the loss mixing coefficient is **$\lambda = 0.5$**. The reported implementation settings are **400k iterations**, **single NVIDIA RTX A5000 GPU**, **batch size 10**, and **learning rate $1.5\mathrm{e}{-5}$**. Architectural sizes are also specified: **12 transformer layers** for the visual prompt encoder, **2 layers** for the panoramic view encoder, **4 layers** for the graph-aware cross-modal encoder, **4 layers** for the local cross-modal encoder, and **hidden size 768** [2508.01766].

Two augmentation families are central to the reported performance. **Prompt-view augmentation** rotates the 2D top-view prompt map by one of
$$
0,\ \frac{\pi}{2},\ \pi,\ \frac{3\pi}{2},
$$
while **agent-view augmentation** randomly samples the initial heading from $[0,2\pi)$. The paper emphasizes that, unlike in VLN, the initial heading in VPN is independent of the prompt, making such augmentation particularly suitable. At the trajectory level, training is further expanded with **PREVALENT-derived** and **ScaleVLN-derived** VPN episodes [2508.01766].

## 5. Prompt design, quantitative results, and ablations

The paper evaluates several prompt forms and concludes that **trajectory-centered cropping** and **explicit trajectory rendering** are decisive. A full-scene top-view map without trajectory-centered cropping performs poorly because many episodes from the same scene share the same map image while differing only in route. A cropped top-view map without the visual trajectory already works “surprisingly well,” which the authors relate to rough destination-area information, but the best-performing prompt form is **“lines only”**: a clean trajectory rendering without text labels. The paper’s interpretation is that lines already provide sufficient spatiotemporal guidance, whereas text may obscure details around waypoints [2508.01766].

Quantitatively, the prompt-type ablation on **Val Unseen** reports the following progression: full-map prompt **NE 7.07, SR 33.94, SPL 29.36**; cropped-map-only prompt **NE 5.51, SR 45.83, SPL 36.78**; a stronger cropped-map variant **NE 5.13, SR 50.18, SPL 40.54**; **lines + text** prompt **NE 3.74, SR 65.36, SPL 54.81**; and **lines only** prompt **NE 2.18, SR 65.92, SPL 56.17**. On **Val Seen**, the **lines only** prompt reaches **NE 0.05, SR 100.00, SPL 99.77** [2508.01766].

The main benchmark results show three training regimes. With **R2R only**, VPNet obtains **Val Seen: NE 3.37, OSR 72.41, SR 66.69, SPL 62.13; Val Unseen: NE 5.43, OSR 59.53, SR 51.23, SPL 43.47; Test Unseen: NE 5.11, OSR 62.33, SR 52.40, SPL 42.87**. With **R2R + PRE**, the numbers improve to **Val Seen: NE 0.05, OSR 100, SR 100, SPL 99.77; Val Unseen: NE 2.18, OSR 76.76, SR 65.92, SPL 56.17; Test Unseen: NE 1.94, OSR 78.21, SR 66.38, SPL 56.26**. With **R2R + PRE + SCA**, the reported results are **Val Seen: NE 0.14, OSR 99.41, SR 99.41, SPL 99.08; Val Unseen: NE 0.48, OSR 97.45, SR 96.68, SPL 94.84; Test Unseen: NE 0.31, OSR 98.56, SR 97.56, SPL 94.60** [2508.01766].

The augmentation ablation identifies the best view-level configuration as **both prompt-view and agent-view augmentation**, yielding **TL 9.92, NE 0.48, OSR 97.45, SR 96.68, SPL 94.84**. Prompt-view augmentation alone is also strong, with **NE 0.62, SR 96.41, SPL 94.37**, and the paper notes that prompt rotation provides larger gains than randomizing the initial heading [2508.01766].

Robustness tests inject **20% salt-and-pepper noise** into the prompt or remove the first-step visual cue. Relative to the clean-prompt result **TL 9.92, NE 0.48, OSR 97.45, SR 96.68, SPL 94.84**, salt-and-pepper noise yields **TL 11.73, NE 1.24, OSR 93.53, SR 90.34, SPL 85.40**, while removing the first-step cue yields **TL 12.48, NE 2.13, OSR 89.31, SR 85.97, SPL 74.98**. The larger degradation from removing the first cue indicates that early trajectory information is especially important [2508.01766].

The paper also frames the benchmark in relation to standard VLN baselines. It states that **VPNet in VPN outperforms DUET in VLN while using only 1,600,945 ScaleVLN trajectories, compared to DUET’s 4,941,710**, presenting this as evidence of **data efficiency** in the VPN framework [2508.01766].

## 6. Significance, adjacent research, and limitations

R2R-VP is positioned as a hybrid between **VLN** and **ImageNav**. Like VLN, it can encode an intended **route**, not merely a destination. Like ImageNav, it presents information in a **direct visual form** rather than in language. The main empirical lesson drawn by the paper is that **visual prompts can successfully replace language in R2R-style navigation**, provided that the prompt clearly encodes route geometry and traversal direction [2508.01766].

Several specific findings follow from the reported ablations. First, **route structure matters more than text labels**: the best prompt is a clean trajectory rendering rather than a heavier symbolic overlay. Second, **cropping and prompt-specific map design are crucial**: if the map is too broad, episode-specific route identity becomes weaker. Third, **prompt diversity matters**: orientation-invariant learning from rotated prompts substantially improves generalization. Fourth, **scaling trajectory data is highly effective**: PREVALENT and ScaleVLN augmentation drive the largest gains, to the point that, as the paper states, **with large-scale augmented training, R2R-VP becomes nearly solved under this benchmark** [2508.01766].

The benchmark also has explicit constraints. It depends on the availability and quality of a **usable 2D top-view map**, and some scenes or floors are excluded because the top-view reconstructions are poor. Prompt generation in the benchmark is based on **ground-truth trajectories**, whereas real human-drawn prompts may be noisier or less precise. The paper further notes that formal world-to-map projection equations are under-specified, which may hinder exact replication without the released code [2508.01766].

In adjacent VLN research, graph-based continuous-environment methods remain conceptually close to R2R-VP even when their evaluation benchmark differs. **ETP-R1**, for example, is evaluated on **R2R-CE** and **RxR-CE**, not on the original discrete benchmark, but its high-level planner selects **graph nodes / waypoints / topological subgoals** while a low-level controller handles execution. The paper explicitly argues that this planning interface is very close to the viewpoint/node-level decision process that R2R-VP researchers care about. A plausible implication is that the topological abstractions emphasized by R2R-VP remain central even as VLN systems move toward continuous embodied execution and online reinforcement fine-tuning [2512.20940].

Source: https://www.emergentmind.com/topics/r2r-vp