Dual-View Visual Contextualization for Web Navigation (2402.04476v2)
Abstract: Automatic web navigation aims to build a web agent that can follow language instructions to execute complex and diverse tasks on real-world websites. Existing work primarily takes HTML documents as input, which define the contents and action spaces (i.e., actionable elements and operations) of webpages. Nevertheless, HTML documents may not provide a clear task-related context for each element, making it hard to select the right (sequence of) actions. In this paper, we propose to contextualize HTML elements through their "dual views" in webpage screenshots: each HTML element has its corresponding bounding box and visual content in the screenshot. We build upon the insight -- web developers tend to arrange task-related elements nearby on webpages to enhance user experiences -- and propose to contextualize each element with its neighbor elements, using both textual and visual features. The resulting representations of HTML elements are more informative for the agent to take action. We validate our method on the recently released Mind2Web dataset, which features diverse navigation domains and tasks on real-world websites. Our method consistently outperforms the baseline in all the scenarios, including cross-task, cross-website, and cross-domain ones.
- Language models are few-shot learners. In NeurIPS, 2020.
- A dataset for interactive vision-language navigation with unknown command feasibility. In ECCV, 2022.
- Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024.
- Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://lmsys.org/blog/2023-03-30-vicuna/, 2023.
- Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
- Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
- Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
- Mind2web: Towards a generalist agent for the web. In NeurIPS, 2023.
- Multimodal web navigation with instruction-finetuned foundation models. In ICLR, 2024.
- A real-world webagent with planning, long context understanding, and program synthesis. In ICLR, 2024.
- Webvoyager: Building an end-to-end web agent with large multimodal models. arXiv preprint arXiv:2401.13919, 2024.
- Mask r-cnn. In ICCV, 2017.
- Deberta: Decoding-enhanced bert with disentangled attention. In ICLR, 2021.
- Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312.08914, 2023.
- Lora: Low-rank adaptation of large language models. In ICLR, 2022.
- A data-driven approach for learning to control computers. In ICML, 2022.
- Do berts learn to use browser user interface? exploring multi-step tasks with unified vision-and-language berts. arXiv preprint arXiv:2203.07828, 2022.
- Dom-q-net: Grounded rl on structured language. In ICLR, 2019.
- Language models can solve computer tasks. In NeurIPS, 2023.
- Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICML, 2023.
- Spotlight: Mobile ui understanding using vision-language models with a focus. In ICLR, 2023.
- Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
- Mapping natural language instructions to mobile ui action sequences. In ACL, 2020.
- Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
- Microsoft coco: Common objects in context. In ECCV, 2014.
- Reinforcement learning on web interfaces using workflow-guided exploration. In ICLR, 2018.
- Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
- Visual instruction tuning. In NeurIPS, 2023b.
- OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
- From pixels to ui actions: Learning to follow instructions via graphical user interfaces. In NeurIPS, 2023.
- Hierarchical prompting assists large language model on web navigation. In EMNLP, 2023.
- Meta-gui: Towards multi-modal conversational agents on mobile gui. In EMNLP, 2022.
- Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
- Benchmarking generalization via in-context instructions on 1,600+ language tasks. In EMNLP, 2022.
- Webshop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS, 2022.
- Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024.
- Jihyung Kil (10 papers)
- Chan Hee Song (10 papers)
- Boyuan Zheng (27 papers)
- Xiang Deng (43 papers)
- Yu Su (138 papers)
- Wei-Lun Chao (92 papers)