Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Volumetric Environment Representation for Vision-Language Navigation (2403.14158v1)

Published 21 Mar 2024 in cs.CV

Abstract: Vision-language navigation (VLN) requires an agent to navigate through an 3D environment based on visual observations and natural language instructions. It is clear that the pivotal factor for successful navigation lies in the comprehensive scene understanding. Previous VLN agents employ monocular frameworks to extract 2D features of perspective views directly. Though straightforward, they struggle for capturing 3D geometry and semantics, leading to a partial and incomplete environment representation. To achieve a comprehensive 3D representation with fine-grained details, we introduce a Volumetric Environment Representation (VER), which voxelizes the physical world into structured 3D cells. For each cell, VER aggregates multi-view 2D features into such a unified 3D space via 2D-3D sampling. Through coarse-to-fine feature extraction and multi-task learning for VER, our agent predicts 3D occupancy, 3D room layout, and 3D bounding boxes jointly. Based on online collected VERs, our agent performs volume state estimation and builds episodic memory for predicting the next step. Experimental results show our environment representations from multi-task learning lead to evident performance gains on VLN. Our model achieves state-of-the-art performance across VLN benchmarks (R2R, REVERIE, and R4R).

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (106)
  1. Bevbert: Multimodal map pre-training for language-guided navigation. In ICCV, 2023a.
  2. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. arXiv:2304.03047, 2023b.
  3. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In CVPR, 2018.
  4. Chasing ghosts: Instruction following as bayesian state tracking. In NeurIPS, 2019.
  5. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv:2303.12712, 2023.
  6. Semantic mapnet: Building allocentric semantic maps and representations from egocentric views. In AAAI, 2021.
  7. Matterport3d: Learning from rgb-d data in indoor environments. In 3DV, 2017.
  8. Learning to explore using active neural slam. In ICLR, 2019.
  9. Neural topological slam for visual navigation. In CVPR, 2020.
  10. Omnidirectional information gathering for knowledge transfer-based audio-visual navigation. In ICCV, 2023a.
  11. Topological planning with transformers for vision-and-language navigation. In CVPR, 2021a.
  12. Weakly-supervised multi-granularity map learning for vision-and-language navigation. In NeurIPS, 2022a.
  13. a2superscriptđť‘Ž2a^{2}italic_a start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT nav: Action-aware zero-shot robot navigation by exploiting vision-and-language ability of foundation models. In NeurIPS Workshop, 2023b.
  14. History aware multimodal transformer for vision-and-language navigation. In NeurIPS, 2021b.
  15. Learning from unlabeled 3d environments for vision-and-language navigation. In ECCV, 2022b.
  16. Think global, act local: Dual-scale graph transformer for vision-and-language navigation. In CVPR, 2022c.
  17. Learning priors for semantic 3d reconstruction. In ECCV, 2018.
  18. Manhattan world: Compass direction from a single image by bayesian inference. In ICCV, 1999.
  19. Evolving graphical planner: Contextual global planning for vision-and-language navigation. In NeurIPS, 2020.
  20. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020.
  21. Simultaneous localization and mapping: part i. IEEE robotics & automation magazine, 13(2):99–110, 2006.
  22. Alberto Elfes. Occupancy grids: A stochastic spatial representation for active robot perception. arXiv:1304.1098, 2013.
  23. Speaker-follower models for vision-and-language navigation. In NeurIPS, 2018.
  24. Adaptive zone-aware hierarchical planner for vision-language navigation. In CVPR, 2023.
  25. Semantics for robotic mapping, perception and interaction: A survey. Foundations and Trends® in Robotics, 8(1–2):1–224, 2020.
  26. Cross-modal map learning for vision and language navigation. In CVPR, 2022.
  27. Vision-and-language navigation: A survey of tasks, methods, and future directions. In ACL, 2022.
  28. Airbert: In-domain pretraining for vision-and-language navigation. In ICCV, 2021.
  29. Recurrent world models facilitate policy evolution. In NeurIPS, 2018.
  30. Towards learning a generic agent for vision-and-language navigation via pre-training. In CVPR, 2020.
  31. Mapnet: An allocentric spatial memory for mapping environments. In CVPR, 2018.
  32. Language and visual entity relationship graph for agent navigation. In NeurIPS, 2020.
  33. Vln bert: A recurrent vision-and-language bert for navigation. In CVPR, 2021.
  34. Learning navigational visual representations with semantic map supervision. In CVPR, 2023.
  35. Scenenn: A scene meshes dataset with annotations. In 3DV, 2016.
  36. Tri-perspective view for vision-based 3d semantic occupancy prediction. In CVPR, 2023.
  37. Geovln: Learning geometry-enhanced visual representation with slot attention for vision-and-language navigation. In CVPR, 2023.
  38. Semantically-aware spatio-temporal reasoning agent for vision-and-language navigation in continuous environments. In ICPR, 2022.
  39. Stay on the path: Instruction fidelity in vision-and-language navigation. In ACL, 2019.
  40. Local implicit grid representations for 3d scenes. In CVPR, 2020.
  41. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. In CVPR, 2023.
  42. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
  43. Adam: A method for stochastic optimization. In ICLR, 2015.
  44. Semi-supervised classification with graph convolutional networks. In ICLR, 2016.
  45. Pathdreamer: A world model for indoor navigation. In ICCV, 2021.
  46. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In ECCV, 2020.
  47. Renderable neural radiance map for visual navigation. In CVPR, 2023.
  48. Delving into the devils of bird’s-eye-view perception: A review, evaluation and recipe. IEEE TPAMI, 2023.
  49. Panogen: Text-conditioned panoramic environment generation for vision-and-language navigation. In NeurIPS, 2023.
  50. Envedit: Environment editing for vision-and-language navigation. In CVPR, 2022a.
  51. 3d neural scene representations for visuomotor control. In CoRL, 2022b.
  52. Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022c.
  53. Focal loss for dense object detection. In ICCV, 2017.
  54. Scene-intuitive agent for remote embodied visual grounding. In CVPR, 2021.
  55. Vision-language navigation with random environmental mixup. In ICCV, 2021.
  56. Bird’s-eye-view scene graph for vision-language navigation. In ICCV, 2023a.
  57. See and think: Disentangling semantic scene completion. In NeurIPS, 2018.
  58. Aerialvln: Vision-and-language navigation for uavs. In ICCV, 2023b.
  59. Improving vision-and-language navigation with image-text pairs from the web. In ECCV, 2020.
  60. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  61. Soat: A scene-and object-aware transformer for vision-and-language navigation. In NeurIPS, 2021.
  62. Neural map: Structured memory for deep reinforcement learning. In ICLR, 2018.
  63. Episodic transformer for vision-and-language navigation. In ICCV, 2021.
  64. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
  65. Reverie: Remote embodied visual referring expression in real indoor environments. In CVPR, 2020.
  66. Hop: history-and-order aware pre-training for vision-and-language navigation. In CVPR, 2022.
  67. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, 2019.
  68. 3d semantic scene completion: A survey. IJCV, 130(8):1978–2005, 2022.
  69. A reduction of imitation learning and structured prediction to no-regret online learning. In AISTATS, 2011.
  70. Semantic scene completion from a single depth image. In CVPR, 2017.
  71. Sequence to sequence learning with neural networks. In NeurIPS, 2014.
  72. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, 2019.
  73. Learning to navigate unseen environments: Back translation with environmental dropout. In NAACL, 2019.
  74. Self-supervised 3d semantic representation learning for vision-and-language navigation. arXiv:2201.10788, 2022.
  75. Sebastian Thrun. Probabilistic robotics. Communications of the ACM, 45(3):52–57, 2002.
  76. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. arXiv:2304.14365, 2023.
  77. Scene as occupancy. In ICCV, 2023.
  78. Attention is all you need. In NeurIPS, 2017.
  79. Learning 3d semantic scene graphs from 3d indoor reconstructions. In CVPR, 2020.
  80. Active visual information gathering for vision-language navigation. In ECCV, 2020a.
  81. Structured scene memory for vision-language navigation. In CVPR, 2021.
  82. Towards versatile embodied navigation. In NeurIPS, 2022a.
  83. Counterfactual cycle-consistent learning for instruction following and generation in vision-language navigation. In CVPR, 2022b.
  84. Dreamwalker: Mental planning for continuous vision-language navigation. In ICCV, 2023a.
  85. Active perception for visual-language navigation. IJCV, 131(3):607–625, 2023b.
  86. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In CVPR, 2024a.
  87. Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. In CVPR, 2019.
  88. Lana: A language-capable navigator for instruction following and generation. In CVPR, 2023c.
  89. Learning to follow and generate instructions for language-capable navigation. IEEE TPAMI, 2023d.
  90. Pillar-based object detection for autonomous driving. In ECCV, 2020b.
  91. Panoocc: Unified occupancy representation for camera-based 3d panoptic segmentation. In CVPR, 2024b.
  92. Scaling data generation in vision-and-language navigation. In ICCV, 2023e.
  93. Gridmm: Grid memory map for vision-and-language navigation. In ICCV, 2023f.
  94. Surroundocc: Multi-camera 3d occupancy prediction for autonomous driving. In ICCV, 2023.
  95. Vector-decomposed disentanglement for domain-invariant object detection. In ICCV, 2021.
  96. Heightformer: Explicit height modeling without extra data for camera-only 3d object detection in bird’s eye view. arXiv:2307.13510, 2023.
  97. Learning 3d dynamic scene representations for robot manipulation. In CoRL, 2020.
  98. Habitat challenge 2023. https://aihabitat.org/challenge/2023/, 2023.
  99. Efficient semantic scene completion network with spatial group convolution. In ECCV, 2018.
  100. Target-driven structured transformer planner for vision-language navigation. In ACM MM, 2022.
  101. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In AAAI, 2024.
  102. Objects as points. arXiv:1904.07850, 2019.
  103. Vision-language navigation with self-supervised auxiliary reasoning tasks. In CVPR, 2020a.
  104. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020b.
  105. Layoutnet: Reconstructing the 3d room layout from a single rgb image. In CVPR, 2018.
  106. Manhattan room layout reconstruction from a single 360 image: A comparative study of state-of-the-art methods. IJCV, 129:1410–1431, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (3)
  1. Rui Liu (320 papers)
  2. Wenguan Wang (103 papers)
  3. Yi Yang (855 papers)
Citations (10)
View on Semantic Scholar