LiveAgentBench: Evaluating Autonomous Agents

Updated 5 July 2026

LiveAgentBench is a benchmark designed to evaluate AI agents on realistic, tool-dependent, multi-step tasks drawn from public online scenarios.
It employs the Social Perception-Driven Data Generation process to convert noisy social data into stable, verifiable tasks.
Empirical results show significant gaps between AI agents and human performance, underlining challenges in tool integration and environmental understanding.

LiveAgentBench is a benchmark for evaluating autonomous AI agents on realistic, tool-using, multi-step tasks drawn from actual user needs rather than synthetic or narrowly academic problem sets. It is presented as an open-source, dynamically updateable benchmark organized around 104 scenarios and 374 tasks, split into 125 validation tasks and 249 test tasks, and constructed from publicly sourced questions on social media and real-world products through the Social Perception-Driven Data Generation (SPDG) process (Li et al., 3 Mar 2026).

1. Definition and design objective

LiveAgentBench is designed to measure whether contemporary agents can handle the heterogeneous, tool-dependent, multimodal work that people actually request online. Its motivating claim is that many established evaluations remain too narrow, either because they target isolated capabilities such as math, code, or planning, or because they do not adequately represent the breadth of real human tasks. The benchmark therefore emphasizes three principles throughout construction: realistic relevance, challenge, and ease of validation (Li et al., 3 Mar 2026).

The benchmark’s critique of earlier evaluation practice has four parts. First, many benchmarks cover too narrow a capability range. Second, they provide insufficient coverage of real everyday environments, including settings such as phone use, video comprehension, and other heterogeneous scenarios. Third, they are difficult to maintain when websites and public resources change. Fourth, static benchmarks are vulnerable to training-data contamination. LiveAgentBench addresses these concerns by grounding tasks in public user cases, updating task collections over time, and rewriting tasks into stable forms with verifiable answers (Li et al., 3 Mar 2026).

A common misconception is that LiveAgentBench is “live” in the sense of unconstrained real-time interaction with an always-changing external world. The benchmark is better characterized as curated from dynamic real-world sources: its source material is live and public, but the released task instances are rewritten so that their answers remain stable and can be evaluated automatically (Li et al., 3 Mar 2026).

The benchmark’s construction pipeline is the Social Perception-Driven Data Generation (SPDG) method. SPDG has four phases: Data Collection, Data Screening, Task Construction, and Quality Control. The purpose of this pipeline is to convert noisy, open-ended social data into benchmark tasks that remain socially grounded, sufficiently difficult, tool-dependent, and easy to verify (Li et al., 3 Mar 2026).

Data collection begins from public online platforms, including Zhihu, Quora, Baidu Knows, Xiaohongshu, BiliBili, Douyin, Stack Overflow, CSDN, TikTok, and Kuaishou. The collection stage uses a combination of automatic and manual labeling, and LLM assistance is used to identify cases with attachments so that tasks involving images, audio, and video are not missed. The first filtering stage applies two explicit criteria: Non-retrievability and Tool dependency. After this screening, the benchmark retains 1112 user cases that satisfy those criteria (Li et al., 3 Mar 2026).

Task construction then extracts required capabilities and environments from each case. The paper describes a representative example in which the environment is a government website accessed through a browser, while the capability is reasoning over gathered information. Annotators with relevant expertise build benchmark tasks and labels from these extracted capability-environment pairs. To preserve complexity, annotators also label the correct execution steps. Because original user questions are often open-ended, annotators rewrite them so that the final answer is stable, concise, unambiguous, and easy to verify (Li et al., 3 Mar 2026).

Quality control imposes several explicit thresholds. If there is more than 50% mismatch between the original and rewritten task in environment and examined capabilities, the task is filtered and reconstructed. If planning steps are fewer than 2, or if no tools are required, the task is deleted from the dataset. Other annotators execute the labeled steps to verify solvability, and double-blind annotation is used to ensure result uniqueness; when two respondents disagree, a third adjudicates the case (Li et al., 3 Mar 2026).

3. Benchmark composition and task structure

The initial release contains 374 total tasks organized across 104 daily real-world scenarios. The paper groups these scenarios into five broad categories: Work and Study (WS), Daily Life (DL), Information Access and Processing (IA&P), Humanities and Social Science (H&SS), and Social Production (SP). It also reports capability-oriented slices for Text File, Image, Video, and Audio (Li et al., 3 Mar 2026).

LiveAgentBench is explicitly broader than a text-only browsing benchmark. The task environment may involve browser operation, file operation, Android/iOS system operation, audio and video comprehension, and multimodal processing. The benchmark description also refers to concrete environment types such as government websites, commercial websites, and music and video tools or platforms. At evaluation time, however, systems are prompted in zero-shot form and scored on the extracted final answer rather than on a released trajectory-level interaction protocol (Li et al., 3 Mar 2026).

This construction creates an important methodological compromise. The task origins and implied environments are dynamic and realistic, but the final benchmark instances are deliberately rewritten into closed-form tasks whose answers “do not change over time.” This makes LiveAgentBench more realistic than a purely synthetic static benchmark, while also making it less open-ended than a fully live interactive environment benchmark (Li et al., 3 Mar 2026).

4. Evaluation protocol and empirical results

The evaluation covers four system classes: LLMs, commercial agents, an open-source agent framework, and humans. The five LLMs are Qwen3-235B-A22B, Claude 3.5 Sonnet, GPT-4o, Gemini-2.5-pro, and DeepSeek-R1-671B. The commercial or productized agents are Gemini Deep Research, Manus, OpenAI Deep Research-4omini, Perplexity Research, and Coze Space. The framework baseline is AWorld, using Claude 3.5 Sonnet as planning and execution model. Evaluation uses the systems’ own capabilities and tools; if a system lacks a required capability such as attachment upload, the task is counted as a failure. The main metric is Pass@1, implemented by extracting the answer from the response and comparing it to ground truth with string matching, without an LLM judge (Li et al., 3 Mar 2026).

The headline finding is that current systems remain far from robust real-world agency. The best overall system is Manus at 35.29%, the best LLM is Gemini-2.5-pro at 16.85%, and the human reference is 69.25%, leaving a 33.96-point gap between humans and the strongest evaluated agent. The paper also states that agents with tools perform 56.51% better on average than standalone LLMs, with average LLM performance around 13.48% and average agent performance 23.85% (Li et al., 3 Mar 2026).

System class	Representative system	Overall score
LLM	Gemini-2.5-pro	16.85
Agent	Manus	35.29
Human	Human	69.25

Capability-level results show strong modality asymmetries. Several LLMs score 0 on audio and video, and some also score 0 on image. Gemini-2.5-pro is the strongest multimodal LLM among those tested, with 12.26 on Text File, 27.73 on Image, 16.0 on Video, and 0 on Audio. Manus is the broadest-performing agent, with 37.85 on Text File, 35.29 on Image, 16.0 on Video, and 33.33 on Audio (Li et al., 3 Mar 2026).

The benchmark also diagnoses why failures occur. Two primary causes are identified: tool instability and lack of environmental background knowledge. For AWorld, approximately 11.76% of task failures are attributed to instability, including inner-tool instability, and its success rate differs from other agents by around 8.34%, mainly due to stability. The second failure mode appears when an agent reaches the correct website or tool but cannot navigate the environment well enough to find the relevant sub-entry, especially on unfamiliar sites such as government websites (Li et al., 3 Mar 2026).

5. Position within the agent-benchmark landscape

LiveAgentBench is explicitly positioned against GAIA, AgentBench, and API-Bank. According to its comparison, it covers browser, text file, Android/iOS OS, audio, video, image, real-world scenarios, real-world cases, and regular dataset updates, whereas prior systems cover only subsets of those dimensions. The benchmark’s distinctive claim is therefore not merely broader modality support, but broader task origin and maintenance logic (Li et al., 3 Mar 2026).

In historical terms, AgentBench is a clear precursor. AgentBench introduced a multi-environment, multi-turn evaluation framework for LLMs as agents across 8 distinct environments, formalized interactive evaluation as a POMDP, and shifted attention from static QA toward tool use, decision-making, and environment feedback (Liu et al., 2023). LiveAgentBench extends that line of work toward publicly sourced real-world tasks, multimodal everyday scenarios, and periodic refresh through SPDG rather than a fixed suite of curated environments (Li et al., 3 Mar 2026).

Later work has highlighted that “live” can be interpreted more strongly than LiveAgentBench does. RealClawBench argues that realism is strongest when tasks originate in ongoing production traffic, with reconstructed environments and deterministic verifiers derived from real developer-agent sessions, rather than from public social-media questions rewritten into stable answerable tasks (Lv et al., 2 Jun 2026). This suggests that LiveAgentBench is live primarily through continuous refresh and socially grounded sourcing, not through ongoing production logs or real-time open-world execution.

Within the broader benchmark ecosystem, LiveAgentBench remains capability-centered. Other contemporaneous benchmarks extend evaluation along orthogonal axes: AgentLAB studies long-horizon attack robustness, AgentFairBench studies demographic disparity in agent actions, and AgentProcessBench studies step-level trajectory quality (Jiang et al., 18 Feb 2026, Morla et al., 15 Jun 2026, Fan et al., 15 Mar 2026). LiveAgentBench therefore occupies the role of a broad, socially sourced capability benchmark rather than a benchmark for fairness, security, or process supervision.

6. Limitations, interpretation, and later significance

The paper identifies three main limitations. First, the current benchmark focuses mainly on Chinese-language real-world tasks, so it lacks cultural diversity. Second, there is an irreducible tension between realism and evaluability: because real tasks are often open-ended and ambiguous, substantial disambiguation and rewriting were necessary, and this sometimes introduced unnatural details. Third, forcing tasks to have exactly one correct answer can make some prompts less natural than genuine human requests (Li et al., 3 Mar 2026).

These limitations are methodologically important because they clarify what LiveAgentBench does and does not measure. It measures performance on tasks that are socially grounded and multimodal, but also heavily normalized for stable automatic checking. A plausible implication is that high performance on LiveAgentBench would indicate strong competence on curated real-world-style tasks, not complete robustness in open, continuously changing environments.

The benchmark’s lasting significance lies in its attempt to connect agent evaluation to public demand rather than to isolated academic subskills. SPDG provides a repeatable human-LLM workflow for collecting real internet cases, filtering them for tool dependence and non-retrievability, rewriting them into closed tasks, and refreshing the dataset over time. The paper states that future work will collect more online corpora, continuously update the dataset, improve diversity, and reduce the unnaturalness introduced by disambiguation (Li et al., 3 Mar 2026). In that sense, LiveAgentBench established a benchmark family centered on socially sourced realism, multimodal task breadth, and continuous refresh, even while leaving open the question of how far benchmark liveness should extend beyond periodic reconstruction.