Room-to-Room (R2R) VLN Benchmark

• Room-to-Room (R2R) is an indoor VLN dataset with sparse instruction-path pairs that defines language-conditioned navigation in Matterport3D environments.
• The benchmark highlighted key limitations such as restricted scale, monolingual data, and path bias, which later datasets like RxR aimed to address.
• Empirical evaluations using metrics like SPL and NDTW show that multitask training on R2R and newer datasets improves overall navigation performance.

Searching arXiv for the original Room-to-Room paper and related VLN context. Room-to-Room (R2R) is an earlier indoor Vision-and-Language Navigation (VLN) dataset in which guides create instructions for navigating between locations in Matterport3D environments. In later VLN work, especially "Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding" (Ku et al., 2020), R2R is treated as the canonical predecessor benchmark: it defines the same general navigation setting later reused by RxR, while also serving as the baseline against which subsequent advances in multilinguality, scale, path design, and dense grounding are formulated. Within that lineage, R2R occupies a foundational position in embodied language research, but it is also associated with identifiable limitations that shaped later dataset construction and evaluation practice.

1. Task formulation and benchmark role

R2R is described as an indoor VLN dataset where guides create instructions for navigating between locations in Matterport3D environments (Ku et al., 2020). The task couples natural-language route descriptions with embodied navigation in indoor reconstructions, making language-conditioned action selection the central modeling problem.

The same later account places R2R in direct continuity with RxR: both use the same Matterport3D indoor reconstructions and the same general navigation setting, but RxR is explicitly designed as a successor that fixes several known limitations of R2R. This positioning is significant because it identifies R2R not merely as an isolated benchmark, but as the dataset that set the default experimental regime for simulated indoor VLN and thereby exposed which properties of that regime were methodologically restrictive.

A common misconception is that success on R2R necessarily reflects full language grounding. Later work explicitly challenges that interpretation, arguing that R2R can permit agents to perform well by exploiting path priors and “goal seeking” rather than truly following the instruction (Ku et al., 2020). This does not negate R2R’s importance; rather, it specifies the conditions under which its results should be interpreted.

2. Data schema and annotation structure

R2R is English-only, contains about 22K instructions and 7K paths, and uses one instruction per path (Ku et al., 2020). The annotation protocol is comparatively simple: the instruction is written after viewing the path, and the dataset pairs text instructions with paths.

That structure established the standard sparse supervision format for VLN: instruction-path pairs without dense temporal alignment to human behavior. Later work contrasts this with richer multi-stage pipelines in which a guide traverses a path and speaks the instruction, the speech is transcribed and time-aligned, and a follower then attempts to execute the route. In that comparison, R2R appears as the simpler, earlier form of supervision rather than as a dataset with dense behavioral traces.

The practical consequence is that R2R primarily supports learning from route-level correspondence between text and navigation outcome. This suggests that models trained on R2R can learn useful instruction-following policies, but it also leaves unresolved which words should align to which viewpoints, actions, or visual entities at intermediate steps.

3. Scale, language coverage, and path geometry

Later work identifies three basic structural properties of R2R that became targets for revision: limited scale, English-only coverage, and a path distribution dominated by shortest paths (Ku et al., 2020). R2R paths are characterized as typically the shortest paths between start and goal, usually only 4–6 edges long. The paper further quantifies the contrast with RxR by reporting R2R paths at 5 edges and 9.4m on average, versus 8 edges and 14.9m on average for RxR.

The following comparison summarizes the properties explicitly contrasted in later VLN work:

Aspect	R2R	RxR
Language	English only	English, Hindi, Telugu
Scale	About 22K instructions, 7K paths	126K instructions, 16.5K paths
Path structure	Typically shortest paths; 5 edges and 9.4m on average	8 edges and 14.9m on average; 44.5% are not shortest paths
Grounding	Text instructions paired with paths	Dense spatiotemporal grounding with guide and follower traces
Annotation	One instruction per path, written after viewing the path	Multi-stage guide/follower collection with time alignment

These contrasts matter because they specify the regime in which R2R is easiest to optimize against. Shortest-path regularity reduces variance in route structure, English-only data constrains cross-linguistic generalization, and limited scale restricts data-hungry learning strategies such as multilingual training or pretraining (Ku et al., 2020).

4. Grounding limitations and path bias

The most frequently cited methodological criticism of R2R concerns path bias. Later work states that agents can exploit strong priors over R2R path structure and can do too well by “goal seeking” rather than truly following language (Ku et al., 2020). Because R2R routes are typically shortest paths, models may infer effective action patterns from the geometry of the benchmark itself rather than from detailed instruction semantics.

A second limitation is shallow grounding. R2R provides text instructions paired with paths, but not dense word-to-pose or word-to-visual grounding. Later work explicitly notes that RxR gives word-level alignments to specific pixels in panoramas, whereas earlier work on R2R only retroactively refined annotations into sub-instructions and panoramas (Ku et al., 2020). In this comparison, R2R lacks direct supervision linking individual words to the visual scene and to the navigator’s embodied trajectory.

A third limitation is scarcity. Later work states that embodied language tasks suffer from data scarcity and that R2R is not large enough to support richer multilingual and grounded learning (Ku et al., 2020). This suggests that R2R is well suited to establishing the basic VLN task, but less well suited to large-scale studies of cross-lingual transfer, dense grounding, or behaviorally aligned attention supervision.

5. Evaluation behavior and empirical comparisons

R2R remains central in comparative evaluation because later work reports direct multitask and transfer experiments involving R2R and RxR (Ku et al., 2020). The paper evaluates training on R2R only, training on RxR only, and multitask training on both, using standard VLN metrics including PL, NE, SR, SPL, NDTW, and SDTW. The central empirical statement is explicit: “A multitask model performs best on both datasets, but domain differences thwart simple transfer learning.”

The directional asymmetry is also informative. R2R-only training gives strong R2R performance but poor RxR performance; RxR-only training gives reasonable RxR performance but much weaker R2R transfer; multitask training improves both (Ku et al., 2020). Although the two datasets share environments, their path distributions and language differ enough that transfer is not trivial.

Simple nonlinguistic baselines further clarify the structure of R2R. The paper compares random walk, random heading then go straight, and given correct first step then go straight. These baselines show that RxR is more difficult than R2R, especially for strategies that rely on going straight, and baseline 3 achieves much higher success on R2R than on RxR (Ku et al., 2020). Interpreted conservatively, this supports the claim that R2R’s path structure is more exploitable by heuristic navigation policies.

6. Legacy, reinterpretation, and continued relevance

Later VLN research does not discard R2R; it reinterprets it. The practical message is that R2R is no longer enough on its own if the goal is robust, grounded VLN research, but it remains a complementary source of supervision whose combination with newer datasets improves performance on both benchmarks (Ku et al., 2020). In that sense, R2R retains methodological value even when its limitations are foregrounded.

R2R’s enduring role is therefore twofold. First, it remains the historical benchmark that fixed the Matterport3D indoor navigation setting and popularized instruction-conditioned embodied navigation. Second, it functions as the control case against which later datasets demonstrate what changes when language is made more central, path priors are weakened, and grounding is densified.

This later reinterpretation also clarifies what R2R does and does not test. It does test instruction-conditioned navigation in indoor reconstructions. It does not, by itself, provide multilingual coverage, dense word-level grounding, or follower demonstration traces. It supports conventional success-oriented VLN evaluation, but later work emphasizes metrics such as NDTW and SDTW because path adherence becomes more important when routes are indirect (Ku et al., 2020). A plausible implication is that R2R results are most informative when read together with benchmarks that deliberately reduce shortest-path bias and add richer alignment signals.

7. Position within the evolution of VLN datasets

In the developmental arc of VLN datasets, R2R is best understood as the benchmark that established the field’s basic problem formulation and exposed the need for more demanding supervision regimes. Later work presents RxR as multilingual, larger, more richly grounded, less biased in its paths, and annotated with human follower demonstrations precisely because those properties were absent or limited in R2R (Ku et al., 2020).

That contrast has broader methodological significance. It identifies a transition from sparse instruction-path supervision to densely aligned multimodal supervision; from monolingual to multilingual collection; and from shortest-path-centric route design to more variable and indirect path distributions. The fact that multitask training across R2R and RxR performs best on both datasets indicates that R2R remains part of the effective supervision pool even after its weaknesses are acknowledged (Ku et al., 2020).

R2R thus occupies a stable place in the VLN literature: foundational, still useful, but no longer sufficient as a standalone proxy for robust grounded language navigation. Its importance lies not only in the benchmark itself, but also in the way later datasets were explicitly designed in response to its structure.