Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2010.07954v1)

Abstract: We introduce Room-Across-Room (RxR), a new Vision-and-Language Navigation (VLN) dataset. RxR is multilingual (English, Hindi, and Telugu) and larger (more paths and instructions) than other VLN datasets. It emphasizes the role of language in VLN by addressing known biases in paths and eliciting more references to visible entities. Furthermore, each word in an instruction is time-aligned to the virtual poses of instruction creators and validators. We establish baseline scores for monolingual and multilingual settings and multitask learning when including Room-to-Room annotations. We also provide results for a model that learns from synchronized pose traces by focusing only on portions of the panorama attended to in human demonstrations. The size, scope and detail of RxR dramatically expands the frontier for research on embodied language agents in simulated, photo-realistic environments.

• The paper introduces RxR, a multilingual VLN dataset with 126K instructions over 16.5K paths, addressing language grounding and path bias challenges.
• By leveraging dual annotations from both Guide and Follower, the study demonstrates improved model performance and reveals challenges in multilingual training.
• Dense spatiotemporal grounding aligns instructions with human visual and navigational cues, paving the way for AI agents with human-like navigation skills.

Overview of Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding

The paper introduces Room-Across-Room (RxR), a substantial advancement in the domain of Vision-and-Language Navigation (VLN) datasets. Notably, RxR is characterized by its multilingual nature, encompassing English, Hindi, and Telugu, and its considerable expansion in terms of path diversity and instruction volume compared to existing VLN datasets. The dataset addresses existing limitations related to path biases and enhances the tangible grounding of language within VLN by incorporating comprehensive spatiotemporal alignment between instructions and human visual and navigational traces.

Novel Contributions

1. Multilingual Path Instructions: RxR departs from the dominance of English-centric datasets by incorporating instructions generated natively in English, Hindi, and Telugu. This design choice seeks to probe cross-linguistic variances in spatial reasoning and enhance the applicability of models across linguistically diverse contexts.
2. Enhanced Path and Instruction Diversity: With 126K instructions over 16.5K sampled paths, RxR provides a rich corpus for embodied agents. Unlike its predecessors, it includes paths with high variability in length and direction, counteracting the biases that simplify agent learning in other datasets.
3. Dense Spatiotemporal Grounding: The dataset includes dense semantic grounding through detailed alignment between words in instructions and human poses, captured in virtual environments. This alignment enhances the potential for training agents capable of nuanced navigation by grounding semantics in observed scenes.
4. Dual Path Annotations: Each path is annotated by both a Guide, who generates the instructions, and a Follower, who attempts to execute them. This dual annotation provides empirical insights into the interpretability and fidelity of instructions, alongside alternative but valid path interpretations.

Experimental Insights

The authors present baseline experiments on RxR using a variant of the Reinforced Cross-Modal Matching agent. Key observations include:

• Training with both Guide and Follower paths yields enhanced model performance.
• Monolingual training outperforms multilingual strategies under the evaluation metrics, emphasizing the challenges of multilingual model robustness.
• Initial exploration of visual attention supervision using human pose alignment shows mixed outcomes, indicating further potential for refining grounding methodologies.

Implications and Future Directions

The introduction of RxR emerges as a pivotal resource for the VLN community. By engendering models that not only interpret but adhere to complex language constructs in variable environments, RxR fosters a move towards more generalized and versatile AI agents. Future explorations could explore leveraging the spatiotemporal annotations for developing agents with close-to-human pragmatic understanding of navigation instructions. Furthermore, the multilingual facet opens research pathways in multilingual VLN systems, addressing the gap in cross-linguistic model deployment.

In summary, RxR stands out by pushing the methodological boundaries of grounding natural language in navigational tasks and directly addressing key limitations in prior datasets. It serves as a comprehensive resource for advancing VLN research and development, facilitating the pursuit of generalizable and linguistically adaptable navigation agents in simulated environments.