WorkBench: a Benchmark Dataset for Agents in a Realistic Workplace Setting (2405.00823v2)

Abstract: We introduce WorkBench: a benchmark dataset for evaluating agents' ability to execute tasks in a workplace setting. WorkBench contains a sandbox environment with five databases, 26 tools, and 690 tasks. These tasks represent common business activities, such as sending emails and scheduling meetings. The tasks in WorkBench are challenging as they require planning, tool selection, and often multiple actions. If a task has been successfully executed, one (or more) of the database values may change. The correct outcome for each task is unique and unambiguous, which allows for robust, automated evaluation. We call this key contribution outcome-centric evaluation. We evaluate five existing ReAct agents on WorkBench, finding they successfully complete as few as 3% of tasks (Llama2-70B), and just 43% for the best-performing (GPT-4). We further find that agents' errors can result in the wrong action being taken, such as an email being sent to the wrong person. WorkBench reveals weaknesses in agents' ability to undertake common business activities, raising questions about their use in high-stakes workplace settings. WorkBench is publicly available as a free resource at https://github.com/olly-styles/WorkBench.

• The paper introduces a robust benchmark, WorkBench, that simulates realistic workplace tasks across five tool categories and 690 scenarios.
• The paper demonstrates that current agents, including GPT-4, correctly complete only 43% of tasks, highlighting significant performance limitations.
• The paper outlines future research directions focusing on improved multi-tool interactions, minimized side effects, and enhanced contextual understanding.

Exploring WorkBench: A Benchmark for Evaluating Agents in Realistic Workplace Settings

Introducing WorkBench

WorkBench is poised as a robust testing ground for evaluating the efficiency and reliability of autonomous agents in workplace environments. This benchmark encompasses a simulated sandbox environment featuring five databases, 26 toolkits, and 690 tasks that mirror real business functions such as email correspondence, scheduling, and data analysis. The uniqueness of WorkBench lies in its outcome-centric evaluation method, where agents are not just tested for their task understanding or completion but the actual results of their actions against predefined, ground-truth outcomes.

Challenges of Current Agents in WorkBench

The evaluations detailed in the paper shed light on the actual capabilities of present-day agents, revealing both promising potentials and significant shortcomings. For instance, the best-performing agent, built on the GPT-4 model, completed only 43% of the tasks correctly. This points to critical limitations in task handling, especially when dealing with complex scenarios requiring multiple tools and steps.

The Role of Tools and Domains

Tools are integral to the agents' operations within WorkBench, allowing interactions with the sandbox databases to perform tasks. The five categories include:

• Email: Managing and organizing emails.
• Calendar: Handling event scheduling and queries.
• Web Analytics: Analyzing visitor data on web platforms.
• CRM (Customer Relationship Management): Managing customer-related data.
• Project Management: Overseeing project-related tasks.

The effectiveness of these tools heavily influences the agent's performance, with additional challenges arising from the requirement to sometimes utilize multiple tools across different domains to complete a task.

Evaluating Agent Performances

One of the most notable findings from the agent evaluations is the high variability in performance across different tasks and agents:

• The best agent (GPT-4) scored only 43%, with significantly lower scores when all tools were accessible versus being provided only the necessary ones.
• Lesser-known or less sophisticated models like Llama2-70B struggled more substantially, often failing to exceed single-digit success percentages.

These findings underscore the complexity of comprehensive task coverage and the agents’ dependence on efficiently navigating and utilizing their toolkits.

Multi-step Problem Solving in WorkBench

Many tasks in WorkBench require not just a one-step operation but a sequence of actions that must be carried out in a specific order to achieve the correct result. The outcome-centric evaluation is particularly useful here, as it ensures that agents must not only perform actions but achieve the correct state change in the sandbox databases – a truer test of practical utility in real-world applications.

Implications and Future Directions

The results from WorkBench highlight several key areas for future research and improvement:

• Enhanced Tool Interaction: Better ways for models to interact with and between multiple toolkits could provide significant uplifts in task handling efficiency.
• Reduction of Side Effects: As agents interact with complex systems, minimizing unintended alterations (or side effects) becomes crucial, requiring more refined control mechanisms.
• Greater Context Understanding: Struggles with complex multi-step or multi-tool tasks suggest that deeper contextual understanding and memory could benefit future agents.

In tandem with these technical challenges, the theoretical understanding of autonomous agents' performance limitations also needs expansion, informed by ongoing empirical findings from benchmarks like WorkBench.

Conclusion

WorkBench serves as a critical mirror reflecting the current state of autonomous agent capabilities and their practical applications in business-like settings. While the results from various evaluations hint at exciting possibilities, they also soberly remind us of the long road ahead in AI research. As agents continue to evolve, so too will benchmarks like WorkBench, continually pushing the envelope on what is achievable in automated task execution.