Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
125 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DAP: Domain-aware Prompt Learning for Vision-and-Language Navigation (2311.17812v4)

Published 29 Nov 2023 in cs.CV

Abstract: Following language instructions to navigate in unseen environments is a challenging task for autonomous embodied agents. With strong representation capabilities, pretrained vision-and-LLMs are widely used in VLN. However, most of them are trained on web-crawled general-purpose datasets, which incurs a considerable domain gap when used for VLN tasks. To address the problem, we propose a novel and model-agnostic domain-aware prompt learning (DAP) framework. For equipping the pretrained models with specific object-level and scene-level cross-modal alignment in VLN tasks, DAP applies a low-cost prompt tuning paradigm to learn soft visual prompts for extracting in-domain image semantics. Specifically, we first generate a set of in-domain image-text pairs with the help of the CLIP model. Then we introduce soft visual prompts in the input space of the visual encoder in a pretrained model. DAP injects in-domain visual knowledge into the visual encoder of the pretrained model in an efficient way. Experimental results on both R2R and REVERIE show the superiority of DAP compared to existing state-of-the-art methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (27)
  1. “Multi-speaker pitch tracking via embodied self-supervised learning,” in Proc. ICASSP, 2022, pp. 8257–8261.
  2. “Reverie: Remote embodied visual referring expression in real indoor environments,” in Proc. CVPR, 2020, pp. 9982–9991.
  3. “Improving vision-and-language navigation with image-text pairs from the web,” in ECCV, 2020, pp. 259–274.
  4. “Ndc-scene: Boost monocular 3d semantic scene completion in normalized device coordinates space,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 9455–9465.
  5. “Depthssc: Depth-spatial alignment and dynamic voxel resolution for monocular 3d semantic scene completion,” arXiv preprint arXiv:2311.17084, 2023.
  6. “Towards learning a generic agent for vision-and-language navigation via pre-training,” in CVPR, 2020, pp. 13134–13143.
  7. “VGDiffZero: Text-to-image diffusion models can be zero-shot visual grounders,” arXiv preprint arXiv:2309.01141, 2023.
  8. “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  9. “Learning transferable visual models from natural language supervision,” in ICML, 2021, pp. 8748–8763.
  10. “CPT: colorful prompt tuning for pre-trained vision-language models,” CoRR, vol. abs/2109.11797, 2021.
  11. “ADAPT: vision-language navigation with modality-aligned action prompts,” in CVPR, 2022, pp. 15375–15385.
  12. “Visual-language navigation pretraining via prompt-based environmental self-exploration,” in ACL (1), 2022, pp. 4837–4851.
  13. “On evaluation of embodied navigation agents,” CoRR, vol. abs/1807.06757, 2018.
  14. “Towards learning a generic agent for vision-and-language navigation via pre-training,” in Proc. CVPR, 2020, pp. 13137–13146.
  15. “VLN BERT: A recurrent vision-and-language BERT for navigation,” in CVPR, 2021, pp. 1643–1653.
  16. “Oscar: Object-semantics aligned pre-training for vision-language tasks,” in ECCV, 2020, pp. 121–137.
  17. “Reinforced vision-and-language navigation based on historical BERT,” in ICSI, 2023, pp. 427–438.
  18. “Robust navigation with language pretraining and stochastic sampling,” in EMNLP, 2019, pp. 1494–1499.
  19. “Learning to navigate unseen environments: Back translation with environmental dropout,” in NAACL, 2019, pp. 2610–2621.
  20. “Vision-language navigation with random environmental mixup,” in ICCV, 2021, pp. 1624–1634.
  21. “Vision-language navigation with self-supervised auxiliary reasoning tasks,” in Proc. IEEE CVPR, 2020, pp. 10012–10022.
  22. “The road to know-where: An object-and-room informed sequential BERT for indoor vision-language navigation,” in ICCV. 2021, pp. 1635–1644, IEEE.
  23. “Neighbor-view enhanced model for vision and language navigation,” in ACM MM, 2021, pp. 5101–5109.
  24. “Reinforced structured state-evolution for vision-language navigation,” in CVPR, 2022, pp. 15429–15438.
  25. “Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation,” in Proc. CVPR, 2019, pp. 6629–6638.
  26. “Self-monitoring navigation agent via auxiliary progress estimation,” in ICLR, 2019.
  27. “Tactical rewind: Self-correction via backtracking in vision-and-language navigation,” in Proc. CVPR, 2019, pp. 6741–6749.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now