CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation (2501.16609v3)

Abstract: While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world contexts and modeling user preference. This presents an opportunity for humans to collaborate with the agent and leverage the agent's capabilities effectively. We propose CowPilot, a framework supporting autonomous as well as human-agent collaborative web navigation, and evaluation across task success and task efficiency. CowPilot reduces the number of steps humans need to perform by allowing agents to propose next steps, while users are able to pause, reject, or take alternative actions. During execution, users can interleave their actions with the agent by overriding suggestions or resuming agent control when needed. We conducted case studies on five common websites and found that the human-agent collaborative mode achieves the highest success rate of 95% while requiring humans to perform only 15.2% of the total steps. Even with human interventions during task execution, the agent successfully drives up to half of task success on its own. CowPilot can serve as a useful tool for data collection and agent evaluation across websites, which we believe will enable research in how users and agents can work together. Video demonstrations are available at https://oaishi.github.io/cowpilot.html

• The paper presents CowPilot, a novel framework that combines LLM-driven automation with human oversight in web navigation tasks.
• It achieves a 95% task success rate, with human intervention limited to only 15.2% of steps and autonomous completions reaching 52% in some cases.
• The dual-agent model embedded in a lightweight Chrome extension sets a new benchmark for efficient, real-world web automation and interactive task management.

An Examination of CowPilot: A Framework for Human-Agent Collaborative Web Navigation

The paper introduces CowPilot, a novel framework designed to integrate both autonomous and human-agent collaborative web navigation capabilities. This paper primarily focuses on blending the capabilities of LLM-based agents with human intervention strategies to accomplish complex web-based tasks more efficiently.

Overview of CowPilot

CowPilot is engineered as a lightweight Chrome extension that enables seamless operation within live web environments. The framework proposes an interactive approach whereby an LLM agent suggests actions to a human collaborator throughout the web navigation task. The human can then either allow the agent to proceed autonomously, reject suggestions, or intervene to ensure the correct task trajectory. This methodological fusion of human oversight with autonomous decision-making aims to improve task success rates and operational efficiency.

Research Findings

In evaluations across five significant website domains, the CowPilot framework in human-agent collaborative mode demonstrated an impressive task success rate of 95%, with human intervention accounting for only 15.2% of the task steps. Notably, autonomous efforts by the LLM agent contributed to 84.8% of task actions, underscoring the effectiveness of guided agent actions even with minimal human input.

The ability of the LLM agent to independently reach successful task completion in up to 52% of the cases further emphasizes the potential of these collaborative frameworks. Compared to human-only settings, CowPilot increased the task success rate by 6%. This suggests a marked improvement in operational efficiency, providing that integrating intelligent agents in the collaborative mode can indeed surpass autonomous or human-only strategies.

Technical Details and Evaluation Metrics

CowPilot is based on a dual-agent model consisting of an LLM agent and a human agent, operating within a predefined action space. The sequential actions are assessed based on task accuracy and five key collaboration metrics: agent step count, human step count, total step count, human intervention occurrences, and agent-driven completion accuracy.

The approach employed to gather empirical evidence involved testing an easy-to-understand yet comprehensive scenario set across various categories like shopping and social interaction domains. This ensured that the evaluation was robust and applicable to real-world applications.

Practical and Theoretical Implications

From a practical perspective, CowPilot has vast implications for enhancing web automation tasks, especially in highly dynamic environments where human judgment is critical. CowPilot can serve as a useful tool not only in task execution but also as a data collection framework, allowing both task trajectory analysis and user feedback metrics to enrich future research.

Theoretically, CowPilot showcases an innovative intersection of human-computer interaction and LLM capabilities, paving the way for further investigations into task efficiency optimization. The framework lays a foundation for understanding adaptive user interface interactions influenced by contextual learning and agent suggestion alignment.

Future Developments and Considerations

Future research may explore increasingly sophisticated human-like decision-making paradigms, transitioning CowPilot from a human-assisted tool to a fully autonomous web navigation system. Moreover, the integration of stronger models with multiple LLMs and various user feedback mechanisms could provide more nuanced approaches toward agent reliability and task complexity management.

Careful attention to security implications is necessary, given the inherent privacy risks associated with tracking and learning from user actions. Ensuring robust safeguards and developing transparency protocols will be crucial to advancing the safe implementation of systems like CowPilot.

In conclusion, CowPilot illustrates the promising trajectory of enhancing LLM agents with strategic human intervention. The research effectively demonstrates improved task accuracy and efficiency, positioning CowPilot as a valuable tool and a benchmark for future developments in web-based AI systems.