Cocoa: Co-Planning and Co-Execution with AI Agents (2412.10999v3)

Abstract: Human collaboration benefits from continuous coordination -- planning, delegating tasks, sharing progress, and adjusting objectives -- to align on shared goals. However, agentic AI systems often limit users to previewing or reviewing an agent's plans for fully autonomous execution. While this may be useful for confirmation and correction, it does not support deeper collaboration between humans and AI agents. We present Cocoa, a system that introduces a novel design pattern -- interactive plans -- for collaborating with an AI agent on complex, multi-step tasks. Informed by a formative study ($n=9$), Cocoa builds on interaction designs from computational notebooks and document editors to support flexible delegation of agency through Co-planning and Co-execution, where users collaboratively compose and execute plans with an Agent. Using scientific research as a sample domain, our lab (n=16) and field deployment (n=7) studies found that Cocoa improved agent steerability without sacrificing ease-of-use compared to a strong chat baseline. Additionally, researchers valued Cocoa for real-world projects and saw the interleaving of co-planning and co-execution as an effective novel paradigm for human-AI collaboration.

• The paper introduces Cocoa, a system using interactive plans within a document interface to enable flexible human-AI co-planning and co-execution.
• User studies show Cocoa improves AI agent steerability and user control compared to chat-based interfaces without compromising usability.
• The structured plan environment in Cocoa enhances transparency of agent actions and holds potential for broader application beyond research contexts.

Co-Planning and Co-Execution with AI Agents: An Overview of Cocoa

The paper "Cocoa: Co-Planning and Co-Execution with AI Agents" introduces an innovative system designed to enhance human-AI collaboration through novel interaction patterns within a document editing environment. This system, Cocoa, proposes the use of "interactive plans" to facilitate the processes of co-planning and co-execution between human users and AI agents. The aim is to provide an interface where tasks, roles, and progress are shared directly within the document context, thus improving the synergy between the human and AI counterparts.

System Design and Methodology

Cocoa builds on prior research in AI and HCI by integrating AI agents into a document interface, which is a common medium for scientific researchers to organize and plan their work. The approach is informed by parallels with computational notebooks, which combine code, visual output, and commentary in a way that supports iterative and exploratory workflows. Similarly, Cocoa allows users to edit plans, assign tasks between themselves and agents, and iterate based on intermediate outputs. This design prioritizes both transparency and flexibility in the collaborative process.

To validate the system, the authors engaged 16 researchers in a user paper, using a chat-based approach as a baseline for comparison. Cocoa demonstrated improved agent steerability, suggesting that interactive plans can offer users more control over AI-driven processes without compromising usability.

Findings and Implications

The research highlights several key implications for both the design of AI systems and the broader field of human-AI collaboration:

• Improved Agency and Control: By allowing users to edit plan steps and assign tasks dynamically, Cocoa provides a more granular level of control over AI behavior compared to traditional chat interfaces. This segmentation of tasks can enhance performance, especially when human expertise is crucial for task completion.
• Interface and Usability: The results showed that while Cocoa did not compromise on ease of use, it substantially improved the ability of users to steer the AI agent effectively. This indicates that structured interactive elements can transform user interaction paradigms with AI systems.
• Transparency and Task Execution: The structured environment of Cocoa, with clearly defined plans and tasks, was noted to improve transparency. Users could follow agent actions more easily, which is critical in applications where AI reliability and accountability are paramount.
• Scalability and Generalizability: While the paper focused primarily on academic researchers, the authors suggest that the interaction design pattern implemented by Cocoa could be extended across different domains where users engage in complex planning within document-based environments.

Speculation on Future Developments

The results of this research point toward several potential directions for the future development of AI agents and interfaces. One significant area is the integration of interactive plans into more varied contexts outside academic research, such as enterprise resource planning or project management applications, where structured tasks are integral to operations. Additionally, combining the strengths of chat interfaces with interactive plans could lead to hybrid systems that offer unstructured exploration alongside structured planning when needed.

Further investigation could explore how such systems can be optimized for cost, efficiency, and safety, leveraging user input to handle tasks that are difficult for AI to manage autonomously. Moreover, the ability to handle multimodal inputs (e.g., text, images, code execution) remains an area for expansion, enhancing the utility of AI systems across diverse professional practices.

In summary, this paper extends our understanding of human-AI collaboration by introducing a promising system architecture for co-planning and co-execution, reshaping how users interact with AI agents within document environments.