AgentBench: Benchmarking Autonomous LLM Agents

• AgentBench is a multi-dimensional benchmark suite designed to assess LLMs as autonomous agents in multi-turn, interactive environments.
• It incorporates eight diverse environments—spanning code-, game-, and web-grounded tasks—that challenge agents with planning, reasoning, and decision-making.
• Using metrics like success rate, F1 score, and reward functions, AgentBench provides fine-grained insights into LLM performance, revealing both strengths and limitations.

AgentBench is a multi-dimensional benchmark suite designed to quantitatively evaluate LLMs functioning as autonomous agents in interactive, real-world scenarios. Departing from traditional single-turn NLP assessments, AgentBench focuses on multi-turn, open-ended interactions, requiring sustained reasoning, planning, and decision-making. As LLMs increasingly pursue roles as decision-making agents across diverse settings, AgentBench provides a standardized framework for measuring and comparing their emergent pragmatic intelligence, highlighting both existing capabilities and critical limitations.

1. Benchmark Architecture and Task Environments

AgentBench comprises eight distinct environments, systematically grouped by the nature of the tasks and underlying modalities. These environments collectively simulate realistic challenges and demand that agents execute complex procedures over several interaction rounds. The environments are divided into three main categories:

• Code-Grounded Environments:
  • Operating System (OS): Agents translate natural language instructions into bash commands, performing shell operations such as file enumeration and manipulations.
  • Database (DB): Models generate SQL queries to interact with live MySQL multi-table databases, requiring multi-step query formulation.
  • Knowledge Graph (KG): Agents answer queries via basic knowledge graph tools in partially observable, large-scale KG environments.
• Game-Grounded Environments:
  • Digital Card Game (DCG): Based on the Aquawar framework, challenges include strategic card-based decision making in a turn-based setting.
  • Lateral Thinking Puzzles (LTP): Agents deduce concealed truths by iteratively posing binary questions, evaluating creativity and indirect reasoning.
  • House-holding (HH): Adapted from ALFWorld, this environment assesses task decomposition and sequential planning in simulated household settings.
• Web-Grounded Environments:
  • Web Shopping (WS): Agents interact with e-commerce sites, constrained by requirements like price or attribute matching, to locate and purchase products.
  • Web Browsing (WB): General web navigation tasks—clicking elements, parsing HTML, and executing sequences towards specified goals.

These environments have been engineered to be both challenging and representative of practical, high-value agent deployments, with tasks involving naturalistic instructions, large decision/action spaces, and multi-stage dependencies.

2. Evaluation Methodology and Metrics

The AgentBench protocol leverages a combination of task-specific and global metrics to comprehensively assess agentic competence:

• Success Rate: Proportion of tasks correctly completed, e.g., file manipulation accuracy in OS or correct product purchase in WS.
• F1 Score: Used for environments with information extraction or entity resolution requirements, e.g., KG question answering.
• Reward Function: Applied where reward or cumulative scoring is more informative, such as in the Web Shopping environment.
• Intermediate Failure Conditions: Explicit tracking of "Invalid Format," "Invalid Action," "Context Limit Exceeded," and "Task Limit Exceeded" enables fine-grained error analysis.

A robust decoupled evaluation toolkit based on a Server-Client architecture is provided. Task Servers and Agent Servers communicate via HTTP, and every agent/environment pair can be executed inside dedicated Docker containers for conflict isolation and reproducibility. Assignment of tasks is optimized via a max-flow algorithm (Edmonds–Karp), represented as a flow graph:

$$V = \{A_k \mid 1 \leq k \leq n\} \cup \{T_k \mid 1 \leq k \leq m\} \cup \{S, D\}$$

$$E = \{ (A_{x_k}, T_{y_k}, s_k) \mid 1 \leq k \leq l \} \cup \{ (S, A_k, w(A_k)) \mid 1 \leq k \leq n \} \cup \{ (T_k, D, w(T_k)) \mid 1 \leq k \leq m \}$$

Here, $A_k$ are agents, $T_k$ are tasks, $S$ and $D$ are source and sink nodes, and $w$ are weights/capacities.

3. Comparative Results and Failure Analysis

AgentBench has been applied to a spectrum of 27 LLMs—spanning commercial API-based systems (e.g., GPT-4, Claude-2, GPT-3.5-turbo) and widely-used open-source models (e.g., Llama2, Vicuna, CodeLlama). Testing reveals:

• Commercial models exhibit marked superiority in complex, interactive, multi-turn tasks. For example, GPT-4 achieves 78% success rate in the House-holding environment, demonstrating superior reasoning and low error ratios.
• Open-source models experience substantial performance gaps, particularly in nuanced tasks such as Knowledge Graph querying and the Digital Card Game, even when showing competitive results on traditional static NLP benchmarks.

Frequent failure types include:

• Context Limit Exceeded: Inadequate context windows influence multi-turn performance, notably in some commercial and most open-source models.
• Invalid Format and Invalid Action: Inability to adhere strictly to required output schemas or permitted action spaces.
• Task Limit Exceeded: Failure to complete tasks within allowed interaction rounds, often signaling deficits in long-term planning and memory.

Analysis traces these weaknesses largely to limitations in long-term reasoning, incomplete instruction following, and lack of robust recursive planning.

4. Performance Optimization and Methodological Advances

AgentBench-driven studies identify two primary levers for agentic improvement:

• Code-Oriented Training: Integrating code-based procedural data improves adherence to structured output requirements and sequential reasoning, particularly for tasks modeled after software operation (OS and WS). However, models heavily tuned on code can sometimes underperform in highly dynamic, strategic environments, suggesting a trade-off between rigidity and flexible reasoning.
• Multi-Turn Dialogue Alignment: Supervised fine-tuning with high-quality, multi-turn dialogue data—often generated from powerful models (e.g., GPT-4-originating interactions)—yields better instruction following, more consistent decision-making, and reduced error rates.

Recent research proposes data construction protocols utilizing simulated agent-environment dialogs, manual filtering for logical consistency, multi-path reasoning, and backtracking strategies. The following mixed loss function is deployed for balanced tuning:

$$\mathcal{L}(\theta) = \lambda \cdot \mathbb{E}_{(x, y) \in D_{agent}}[\log M_\theta(y|x)] + (1-\lambda) \cdot \mathbb{E}_{(x, y) \in D_{general}}[\log M_\theta(y|x)]$$

where $\lambda$ controls the ratio between agent-specific and general instruction tuning data. Low-Rank Adaptation (LoRA) techniques are further used to enable parameter-efficient supervised fine-tuning:

$$W' = W + \Delta W \quad \text{with} \quad \Delta W = A \times B$$

$A$ and $B$ are low-rank matrices, allowing for scalable, targeted adaptation.

5. Datasets, Tools, and Community Resources

Each environment in AgentBench is associated with a curated dataset reflecting genuine complexity and varying interaction depths:

• OS: ~1,200 well-documented tasks with accompanying scripts and checking utilities.
• DB: SQL datasets derived from WikiSQL and augmented for diversity.
• Other environments: Correspondingly complex interaction records for KG, DCG, LTP, HH, WS, and WB.

The toolkit and datasets are distributed via a public repository (https://github.com/THUDM/AgentBench), including Dockerized environments and documentation for ease of replication and extension. The server-client structure and containerization facilitate scalable, conflict-free benchmarking and agent evaluation.

6. Connections to the Broader LLM Agentic Research Landscape

AgentBench’s evaluation design and findings are directly relevant to the methodological recommendations from surveys and meta-analyses on LLM evaluation (Guo et al., 2023, Luo et al., 27 Mar 2025). By emphasizing reasoning, planning, tool use, alignment, and safety, it anchors itself within a suite of emerging agentic benchmarks (e.g., WebArena, ToolLLM, BattleAgentBench, SafeAgentBench) and supports the move toward holistic and multi-domain agent evaluation. AgentBench also serves as a proving ground for advances in cross-modal (code, language, vision) reasoning, memory optimization, and safety protocols.

7. Limitations and Prospects for Future Development

AgentBench highlights several open challenges and paths for future progress:

• Expanding diversity and realism of agent tasks and environments: The current scope, while multidimensional, is recognized as an evolving baseline.
• Refinement of evaluation metrics: There is active interest in reward functions that better capture intermediate reasoning, self-correction, and adaptation.
• Scaling effects: Early results suggest increasing model size alone does not ensure better agent performance—alignment and data diversity are equally necessary.
• Self-correcting and interactive feedback mechanisms: Automated error detection and recovery (e.g., on SQL or logic errors) are seen as crucial for robust agent deployments.
• Bridging open-source and commercial performance: Continuous improvements in training methodologies, especially in multi-turn interaction and code-based alignment, are required to close this gap.

AgentBench stands as a pivotal reference point for both benchmarking LLM agent capabilities and for structuring iterative improvement cycles within the rapidly expanding agentic AI landscape. By exposing both strengths and weaknesses of current models, it guides research toward more reliable, adaptive, and genuinely autonomous LLM-driven agents.