$C^3$-Bench: The Things Real Disturbing LLM based Agent in Multi-Tasking (2505.18746v4)

Abstract: Agents based on LLMs leverage tools to modify environments, revolutionizing how AI interacts with the physical world. Unlike traditional NLP tasks that rely solely on historical dialogue for responses, these agents must consider more complex factors, such as inter-tool relationships, environmental feedback and previous decisions, when making choices. Current research typically evaluates agents via multi-turn dialogues. However, it overlooks the influence of these critical factors on agent behavior. To bridge this gap, we present an open-source and high-quality benchmark $C^3$-Bench. This benchmark integrates attack concepts and applies univariate analysis to pinpoint key elements affecting agent robustness. In concrete, we design three challenges: navigate complex tool relationships, handle critical hidden information and manage dynamic decision paths. Complementing these challenges, we introduce fine-grained metrics, innovative data collection algorithms and reproducible evaluation methods. Extensive experiments are conducted on 49 mainstream agents, encompassing general fast-thinking, slow-thinking and domain-specific models. We observe that agents have significant shortcomings in handling tool dependencies, long context information dependencies and frequent policy-type switching. In essence, $C^3$-Bench aims to expose model vulnerabilities through these challenges and drive research into the interpretability of agent performance. The benchmark is publicly available at https://github.com/TencentHunyuan/C3-Benchmark.

Analyzing Multi-Tasking Agent Capabilities with $C^3$-Bench

The paper introduces a benchmarking framework known as $C^3$-Bench, designed to rigorously evaluate the capabilities and robustness of LLM-based agents in multi-tasking scenarios. LLM-based agents increasingly interact with the real world by leveraging a variety of tools to complete complex tasks. Unlike static NLP tasks, these dynamic environments present additional challenges, including tool dependencies, environmental feedback, and decision path optimization.

Benchmark Structure and Challenges

$C^3$-Bench provides a structured, open-source methodology for assessing LLM-based agents. The benchmark focuses on three core challenges faced by these agents:

1. Complex Tool Relationships: Agents are tasked with navigating complex tool dependencies. The benchmark evaluates not just the correctness of tool usage but also the agent's ability to develop efficient tool invocation strategies, considering dependencies and connections between tools.
2. Critical Hidden Information: In multi-task environments, agents often rely on context or historical interactions that are not directly available. This challenge assesses an agent's ability to extract and apply relevant contextual information from previous exchanges, overcoming strategies like omitted information, referential references, and long-context incorporations.
3. Variable Decision Trajectories: Agents must demonstrate flexibility in adjusting decision-making processes based on changing task requirements and historical policies. This challenge examines how alterations in the decision trajectory affect an agent's performance, emphasizing policy-switching frequencies and the stability of decision processes.

Methodology and Data Collection

To generate accurate and high-quality data for testing, $C^3$-Bench employs an innovative multi-role data generation framework. It involves simulated roles such as a user, a planner, and a checker, among others, collaborating to create realistic scenarios that reflect diverse real-world applications. The generated data undergoes rigorous checks by human experts, ensuring that the test cases adequately cover the complexities of tool interactions, historical context dependencies, and varied decision-making paradigms.

Evaluation and Metrics

The benchmark utilizes novel evaluation methodologies, including depth-first search-based topological sorting for path correctness verification. It further introduces fine-grained metrics like the Optimal Path Rate (OP Rate), Accomplish Progress (AP), Volatility Factor (VF), and Debiased Descent Direction (DDD). These metrics are instrumental in assessing not only the success rate in task completion but also the efficiency and stability of an agent's decision-making process.

Experimental Insights

Comprehensive testing across 49 mainstream LLM-based agents reveals significant hurdles in tool relationship navigation and context utilization, with the highest performing model attaining an accuracy of 55.34%. The results highlight key factors affecting agent performance: tool dependencies, long-context information, task scaling, and frequent decision policy shifts. Interestingly, slow-thinking models generally outperform fast-thinking ones, indicating that thorough consideration of information can enhance task performance.

Conclusions and Future Directions

The implications of $C^3$-Bench are profound for the future design and training of LLM-based agents. It not only uncovers current limitations in multi-tasking scenarios but also provides a directional focus for improving agent interpretability and performance. Future developments could expand on the challenges and metrics introduced, fostering the creation of more robust, versatile agents that can seamlessly integrate into complex, real-world environments. As the field evolves, methodologies like those established by $C^3$-Bench will be central in advancing the capabilities and comprehensiveness of AI agents.