Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (2410.13232v1)

Abstract: LLMs have recently gained much attention in building autonomous agents. However, the performance of current LLM-based web agents in long-horizon tasks is far from optimal, often yielding errors such as repeatedly buying a non-refundable flight ticket. By contrast, humans can avoid such an irreversible mistake, as we have an awareness of the potential outcomes (e.g., losing money) of our actions, also known as the "world model". Motivated by this, our study first starts with preliminary analyses, confirming the absence of world models in current LLMs (e.g., GPT-4o, Claude-3.5-Sonnet, etc.). Then, we present a World-model-augmented (WMA) web agent, which simulates the outcomes of its actions for better decision-making. To overcome the challenges in training LLMs as world models predicting next observations, such as repeated elements across observations and long HTML inputs, we propose a transition-focused observation abstraction, where the prediction objectives are free-form natural language descriptions exclusively highlighting important state differences between time steps. Experiments on WebArena and Mind2Web show that our world models improve agents' policy selection without training and demonstrate our agents' cost- and time-efficiency compared to recent tree-search-based agents.

World Models in Web Navigation for LLMs

This paper explores the utilization of world models in enhancing the decision-making capabilities of LLM-based web agents. The researchers focus on addressing a key limitation observed in current LLMs: the inability to predict future states resulting from their actions, an understanding referred to as "environment dynamics" or "world models". This fundamental shortcoming leads to sub-optimal performance in long-horizon web navigation tasks.

Key Contributions

1. Preliminary Analyses: The researchers begin by examining the extent to which state-of-the-art (SOTA) LLMs, including GPT-4o and Claude-3.5-Sonnet, can predict the outcomes of their actions. The studies reveal that these models struggle with accurate next-state predictions and are unable to foresee the impacts of their actions without additional information.
2. World-Model-Augmented (WMA) Web Agent: The core proposal involves augmenting LLM-based web agents with a world model to simulate the outcomes of actions before execution. This enhancement aims to improve decision-making by enabling the agent to anticipate the consequences of different action sequences.
3. Transition-Focused Observation Abstraction: To facilitate effective training of world models, the authors introduce a method that emphasizes significant state transitions between observations. This approach alleviates the inefficiencies of processing full HTML inputs and focuses the learning on meaningful changes, thereby improving model performance and information gain.
4. Policy Optimization: During inference, the world model simulates potential future states for different action candidates. A value function then estimates rewards for these predicted states to select the most promising action. This method allows for improved policy decisions without the need for costly retraining of the policy models.

Experimental Evaluation

The paper evaluates the WMA web agent in two significant benchmarks: WebArena and Mind2Web. The WMA agent demonstrates superior performance, indicating its ability to enhance action selection in long-horizon tasks. The results show substantial improvements in success rates, with notable efficiency in terms of time and computational cost compared to tree-search-based methods like the Tree search agent. The paper also achieves a new SOTA performance in Mind2Web, showcasing its generalizability and practical utility.

Implications and Future Directions

The introduction of world models to LLM-based web agents marks a significant step forward in the deployment of more autonomous and reliable agents in web navigation contexts. The findings suggest that simulation of environmental dynamics is a critical component for improving long-term decision-making capabilities of LLMs.

Further research could expand upon multi-step planning and incorporation of multi-modal inputs, including visual data, to enhance the depth and accuracy of state predictions. Additionally, exploring the integration of the world model approach with other emerging techniques in AI and reinforcement learning, such as hierarchical reinforcement learning, could offer novel improvements in agent performance across more complex environments.

Overall, the paper provides a compelling argument for the importance of world models in autonomous agent design, setting a robust foundation for future explorations in real-world web interaction tasks.