---
title: 'LiveAgentBench: Evaluating Autonomous Agents'
url: https://www.emergentmind.com/topics/liveagentbench
type: topic
---

# LiveAgentBench: Evaluating Autonomous Agents

LiveAgentBench is a benchmark for evaluating autonomous AI agents on realistic, tool-using, multi-step tasks drawn from actual user needs rather than synthetic or narrowly academic problem sets. It is presented as an open-source, dynamically updateable benchmark organized around **104 scenarios** and **374 tasks**, split into **125 validation tasks** and **249 test tasks**, and constructed from publicly sourced questions on social media and real-world products through the **Social Perception-Driven Data Generation (SPDG)** process [2603.02586].

## 1. Definition and design objective

LiveAgentBench is designed to measure whether contemporary agents can handle the heterogeneous, tool-dependent, multimodal work that people actually request online. Its motivating claim is that many established evaluations remain too narrow, either because they target isolated capabilities such as math, code, or planning, or because they do not adequately represent the breadth of real human tasks. The benchmark therefore emphasizes three principles throughout construction: **realistic relevance**, **challenge**, and **ease of validation** [2603.02586].

The benchmark’s critique of earlier evaluation practice has four parts. First, many benchmarks cover too narrow a capability range. Second, they provide insufficient coverage of real everyday environments, including settings such as **phone use**, **video comprehension**, and other heterogeneous scenarios. Third, they are difficult to maintain when websites and public resources change. Fourth, static benchmarks are vulnerable to training-data contamination. LiveAgentBench addresses these concerns by grounding tasks in public user cases, updating task collections over time, and rewriting tasks into stable forms with verifiable answers [2603.02586].

A common misconception is that LiveAgentBench is “live” in the sense of unconstrained real-time interaction with an always-changing external world. The benchmark is better characterized as **curated from dynamic real-world sources**: its source material is live and public, but the released task instances are rewritten so that their answers remain stable and can be evaluated automatically [2603.02586].

## 2. Social Perception-Driven Data Generation

The benchmark’s construction pipeline is the **Social Perception-Driven Data Generation (SPDG)** method. SPDG has four phases: **Data Collection**, **Data Screening**, **Task Construction**, and **Quality Control**. The purpose of this pipeline is to convert noisy, open-ended social data into benchmark tasks that remain socially grounded, sufficiently difficult, tool-dependent, and easy to verify [2603.02586].

Data collection begins from public online platforms, including **Zhihu, Quora, Baidu Knows, Xiaohongshu, BiliBili, Douyin, Stack Overflow, CSDN, TikTok, and Kuaishou**. The collection stage uses a combination of automatic and manual labeling, and LLM assistance is used to identify cases with attachments so that tasks involving **images, audio, and video** are not missed. The first filtering stage applies two explicit criteria: **Non-retrievability** and **Tool dependency**. After this screening, the benchmark retains **1112 user cases** that satisfy those criteria [2603.02586].

Task construction then extracts required capabilities and environments from each case. The paper describes a representative example in which the environment is a government website accessed through a browser, while the capability is reasoning over gathered information. Annotators with relevant expertise build benchmark tasks and labels from these extracted capability-environment pairs. To preserve complexity, annotators also label the correct execution steps. Because original user questions are often open-ended, annotators rewrite them so that the final answer is **stable, concise, unambiguous, and easy to verify** [2603.02586].

Quality control imposes several explicit thresholds. If there is **more than 50% mismatch** between the original and rewritten task in environment and examined capabilities, the task is filtered and reconstructed. If planning steps are **fewer than 2**, or if no tools are required, the task is deleted from the dataset. Other annotators execute the labeled steps to verify solvability, and **double-blind annotation** is used to ensure result uniqueness; when two respondents disagree, a third adjudicates the case [2603.02586].

## 3. Benchmark composition and task structure

The initial release contains **374 total tasks** organized across **104 daily real-world scenarios**. The paper groups these scenarios into five broad categories: **Work and Study (WS)**, **Daily Life (DL)**, **Information Access and Processing (IA&P)**, **Humanities and Social Science (H&SS)**, and **Social Production (SP)**. It also reports capability-oriented slices for **Text File**, **Image**, **Video**, and **Audio** [2603.02586].

LiveAgentBench is explicitly broader than a text-only browsing benchmark. The task environment may involve **browser operation**, **file operation**, **Android/iOS system operation**, **audio and video comprehension**, and multimodal processing. The benchmark description also refers to concrete environment types such as **government websites**, commercial websites, and music and video tools or platforms. At evaluation time, however, systems are prompted in **zero-shot** form and scored on the extracted final answer rather than on a released trajectory-level interaction protocol [2603.02586].

This construction creates an important methodological compromise. The task origins and implied environments are dynamic and realistic, but the final benchmark instances are deliberately rewritten into closed-form tasks whose answers “do not change over time.” This makes LiveAgentBench more realistic than a purely synthetic static benchmark, while also making it less open-ended than a fully live interactive environment benchmark [2603.02586].

## 4. Evaluation protocol and empirical results

The evaluation covers four system classes: **LLMs**, **commercial agents**, **an open-source agent framework**, and **humans**. The five LLMs are **Qwen3-235B-A22B**, **Claude 3.5 Sonnet**, **GPT-4o**, **Gemini-2.5-pro**, and **DeepSeek-R1-671B**. The commercial or productized agents are **Gemini Deep Research**, **Manus**, **OpenAI Deep Research-4omini**, **Perplexity Research**, and **Coze Space**. The framework baseline is **AWorld**, using **Claude 3.5 Sonnet** as planning and execution model. Evaluation uses the systems’ own capabilities and tools; if a system lacks a required capability such as attachment upload, the task is counted as a failure. The main metric is **Pass@1**, implemented by extracting the answer from the response and comparing it to ground truth with **string matching**, without an LLM judge [2603.02586].

The headline finding is that current systems remain far from robust real-world agency. The best overall system is **Manus** at **35.29%**, the best LLM is **Gemini-2.5-pro** at **16.85%**, and the human reference is **69.25%**, leaving a **33.96-point** gap between humans and the strongest evaluated agent. The paper also states that agents with tools perform **56.51% better** on average than standalone LLMs, with average LLM performance around **13.48%** and average agent performance **23.85%** [2603.02586].

| System class | Representative system | Overall score |
|---|---:|---:|
| LLM | Gemini-2.5-pro | 16.85 |
| Agent | Manus | 35.29 |
| Human | Human | 69.25 |

Capability-level results show strong modality asymmetries. Several LLMs score **0** on **audio** and **video**, and some also score **0** on **image**. **Gemini-2.5-pro** is the strongest multimodal LLM among those tested, with **12.26** on Text File, **27.73** on Image, **16.0** on Video, and **0** on Audio. **Manus** is the broadest-performing agent, with **37.85** on Text File, **35.29** on Image, **16.0** on Video, and **33.33** on Audio [2603.02586].

The benchmark also diagnoses why failures occur. Two primary causes are identified: **tool instability** and **lack of environmental background knowledge**. For **AWorld**, approximately **11.76%** of task failures are attributed to instability, including inner-tool instability, and its success rate differs from other agents by around **8.34%**, mainly due to stability. The second failure mode appears when an agent reaches the correct website or tool but cannot navigate the environment well enough to find the relevant sub-entry, especially on unfamiliar sites such as government websites [2603.02586].

## 5. Position within the agent-benchmark landscape

LiveAgentBench is explicitly positioned against **GAIA**, **AgentBench**, and **API-Bank**. According to its comparison, it covers **browser**, **text file**, **Android/iOS OS**, **audio**, **video**, **image**, **real-world scenarios**, **real-world cases**, and **regular dataset updates**, whereas prior systems cover only subsets of those dimensions. The benchmark’s distinctive claim is therefore not merely broader modality support, but broader **task origin** and **maintenance logic** [2603.02586].

In historical terms, **AgentBench** is a clear precursor. AgentBench introduced a multi-environment, multi-turn evaluation framework for LLMs as agents across **8 distinct environments**, formalized interactive evaluation as a POMDP, and shifted attention from static QA toward tool use, decision-making, and environment feedback [2308.03688]. LiveAgentBench extends that line of work toward publicly sourced real-world tasks, multimodal everyday scenarios, and periodic refresh through SPDG rather than a fixed suite of curated environments [2603.02586].

Later work has highlighted that “live” can be interpreted more strongly than LiveAgentBench does. **RealClawBench** argues that realism is strongest when tasks originate in ongoing production traffic, with reconstructed environments and deterministic verifiers derived from real developer-agent sessions, rather than from public social-media questions rewritten into stable answerable tasks [2606.03889]. This suggests that LiveAgentBench is live primarily through **continuous refresh and socially grounded sourcing**, not through ongoing production logs or real-time open-world execution.

Within the broader benchmark ecosystem, LiveAgentBench remains capability-centered. Other contemporaneous benchmarks extend evaluation along orthogonal axes: **AgentLAB** studies long-horizon attack robustness, **AgentFairBench** studies demographic disparity in agent actions, and **AgentProcessBench** studies step-level trajectory quality [2602.16901] [2606.16723] [2603.14465]. LiveAgentBench therefore occupies the role of a broad, socially sourced capability benchmark rather than a benchmark for fairness, security, or process supervision.

## 6. Limitations, interpretation, and later significance

The paper identifies three main limitations. First, the current benchmark focuses mainly on **Chinese-language real-world tasks**, so it lacks cultural diversity. Second, there is an irreducible tension between **realism and evaluability**: because real tasks are often open-ended and ambiguous, substantial disambiguation and rewriting were necessary, and this sometimes introduced **unnatural details**. Third, forcing tasks to have exactly one correct answer can make some prompts less natural than genuine human requests [2603.02586].

These limitations are methodologically important because they clarify what LiveAgentBench does and does not measure. It measures performance on tasks that are socially grounded and multimodal, but also heavily normalized for stable automatic checking. A plausible implication is that high performance on LiveAgentBench would indicate strong competence on curated real-world-style tasks, not complete robustness in open, continuously changing environments.

The benchmark’s lasting significance lies in its attempt to connect agent evaluation to public demand rather than to isolated academic subskills. SPDG provides a repeatable human-LLM workflow for collecting real internet cases, filtering them for tool dependence and non-retrievability, rewriting them into closed tasks, and refreshing the dataset over time. The paper states that future work will collect more online corpora, continuously update the dataset, improve diversity, and reduce the unnaturalness introduced by disambiguation [2603.02586]. In that sense, LiveAgentBench established a benchmark family centered on **socially sourced realism**, **multimodal task breadth**, and **continuous refresh**, even while leaving open the question of how far benchmark liveness should extend beyond periodic reconstruction.

Source: https://www.emergentmind.com/topics/liveagentbench