ClawBench: Real-World LLM Agent Benchmark

• ClawBench is a benchmark suite that evaluates large language model agents on live, dynamic web tasks.
• It utilizes a layered data-capture approach to intercept state-changing requests with 100% precision, ensuring safety and reproducibility.
• The benchmark reveals LLM agents' limitations in managing multi-step, stateful workflows under realistic web conditions.

ClawBench is a benchmark suite for evaluating LLM agents on realistic, state-changing everyday online tasks. Unlike prior agentic benchmarks that focus on read-only information retrieval or operate in static offline environments, ClawBench operates on production websites, preserving the full complexity and dynamism of the live web. This ecological validity is achieved by a layered data-capture and intervention architecture ensuring both safety and reproducibility. ClawBench is designed to measure the practical, general-purpose utility of LLM agents for automating the routine digital workflows commonly encountered by users across numerous online platforms (Zhang et al., 9 Apr 2026).

1. Motivation and Distinguishing Features

ClawBench’s primary motivation is to expose the gap between current LLM agent capabilities and the real demands of web-mediated task completion. Prior benchmarks such as WebArena, OSWorld, and Mind2Web either constrain agents to offline sandboxed HTML with static content or restrict evaluation to read-only tasks on the live web. These design choices omit the dynamic content, authentication protocols, client-side rendering, evolving UI layouts, pop-ups, and anti-bot defenses characteristic of production web services (Zhang et al., 9 Apr 2026). As a result, prior results may substantially overstate real-world agent readiness. ClawBench instead focuses on write-heavy, multi-step workflows that are central to everyday digital life:

• Booking flights, appointments, or rentals requiring cross-page context carryover and pop-up management.
• Placing orders and making purchases involving complex forms, payment logic, and dynamic price widgets.
• Submitting job or loan applications requiring accurate parsing of user-uploaded resumes and attached documents.

The benchmark ensures that interactions unfold on live websites, capturing all aspects of contemporary web application complexity and hazard.

2. Task Suite Construction

ClawBench comprises 153 unique tasks spanning 144 live platforms and 15 domain categories (grouped under Daily Life, Finance, Work, Development, Academic, Travel, Social, and Pets). Each task is modeled on actual recurring needs of end-users and involves altering platform state as would occur during genuine use.

Examples:

• Complete an e-commerce purchase (intercept the final buy request).
• Reserve medical or government appointments, including multi-modal calendar inputs.
• Submit job applications through dynamic multi-page forms with required uploads.
• Enroll in services that require multi-step verification, consent handling, and address validation.

Each task’s endpoint is precisely defined using ground-truth human annotation, specifying the exact HTTP request(s) that commit to irrevocable changes (e.g., POST /checkout, PUT /submit-form). All other traffic, such as auxiliary AJAX requests or image loads, is allowed to transit unmodified. This selective interception ensures that agents experience the full web environment without risk of causing actual user-account changes (Zhang et al., 9 Apr 2026).

Task Suite Table

Aspect	Value	Example Domains
Num. Tasks	153	e-commerce, travel, jobs
Platforms	144 unique live websites	finance, academic, social
Categories	15 (in 8 broad groups)	dev tools, pets, daily

3. Architecture and Data-Capture Methodology

A lightweight Chrome extension and server built atop the Chrome DevTools Protocol (CDP) logs every browser action (clicks, keystrokes, scroll events) and forwards them to an external server for orchestration and traceability. Human annotators supply an explicit schema for each task: endpoint URL, HTTP method, and canonical payload schema. At run time, all outbound browser requests are monitored, with exact-match interception of annotated submission endpoints. The intercepted final-state-changing requests are (i) fully logged (including headers, body, and query fields), (ii) blocked from transmission to prevent actual transactions, and (iii) stored for post-hoc evaluation.

Validation experiments demonstrate 100% precision and recall in submission interception, with zero false positives among navigation, preflight, and auxiliary traffic (Zhang et al., 9 Apr 2026).

Five-layer Recording Workflow

1. Session video (full, via Xvfb + FFmpeg).
2. Per-action screenshots after each agent event.
3. HTTP traffic log, with intercepted payloads.
4. Agent reasoning traces and tool invocations (JSON).
5. Low-level browser event log (coordinate + keycode).

This holistic capture enables rigorous evaluation, failure analysis, and reproducibility.

4. Evaluation Protocol, Scoring, and Baselines

Each task instance consists of a natural-language prompt $q^{(t)}$, an agent trajectory $\mathcal{T}_a^{(t)}$, and a reference human trajectory $\mathcal{T}_h^{(t)}$. The evaluation relies on an "Agentic Evaluator" (a Claude Code sub-agent) that aligns the agent’s actions to the human reference, verifies field-level form completion, and issues a binary success/failure verdict:

$$\text{Score}(t)\;=\;\mathcal{A}\bigl(q^{(t)},\,\mathcal{T}_a^{(t)},\,\mathcal{T}_h^{(t)}\bigr)\;\in\{0,1\}$$

$$\mathrm{SR}\;=\;\frac{1}{|T|}\sum_{t \in T}\text{Score}(t)$$

Partial credit is not permitted. Reproducibility is enforced using containerized browser environments, fixed startup flags, and precise human annotation of interception events.

Evaluated Models and Results:

Seven frontier LLM-based agents were tested. The highest aggregate success rate was 33.3% (Claude Sonnet 4.6). Performance varies considerably by task domain: for Sonnet 4.6, up to 50% on "Academic" tasks versus 11.1% on "Development." The majority of models, including GPT-5.4, failed to achieve non-trivial completion rates on most domains (Zhang et al., 9 Apr 2026).

Baseline Results Table

Model	Overall SR (%)	Notable Domain SR Max/Min
Claude Sonnet 4.6	33.3	50 / 11.1
GLM-5 (text-only)	24.2	-
Gemini 3 Flash	19.0	-
Claude Haiku 4.5	18.3	-
GPT-5.4	6.5	-
Gemini 3.1 Flash Lite	3.3	-
Kimi K2.5	0.7	-

5. Agent Capabilities and Failure Taxonomy

ClawBench tasks require robust document object model (DOM) understanding under rapidly changing layouts, precise form-filling and information extraction (e.g., from user-supplied PDFs), explicit management of multi-page workflow state, and resilience to unstructured interruptions such as cookie banners or pop-up prompts. Agents must route context across nonlinear flows (e.g., updating shipping details across reloads) and handle errors such as failed logins or backend validation (Zhang et al., 9 Apr 2026).

Empirical results indicate several characteristic failure modes:

• Mis-clicks on ambiguous or similarly labeled UI elements.
• Omission of required form fields, leading to failed submissions.
• Breakdown in multi-step sequence management (e.g., skipping confirmation screens).
• Incorrect parsing of user documents and misunderstanding of dynamic content or price breakdowns.
• Inability to overcome or bypass CAPTCHA, rate-limiting, or anti-bot challenges.

The full logging stack enables diagnosis at the action, page, and protocol levels for targeted agent improvement.

6. Comparative Context, Limitations, and Future Directions

ClawBench highlights a much lower ceiling for LLM agent capability on real-world, write-heavy web tasks compared to read-only or sandboxed tasks where top models achieve 65–75% (Zhang et al., 9 Apr 2026). The significant performance gap demonstrates that current systems lack robust generalization across varying real-world DOMs, error conditions, and interaction flows. Agents display brittle, domain-specific competence and do not yet approach the requirements for reliable automation of daily digital workflows.

The benchmark provides step-level traceability, facilitating both macro-level progress tracking and micro-level error analysis. Future recommended research directions include:

• Enhanced DOM and visual grounding to resist layout and structure changes.
• Advanced document parsing and in-context retrieval capabilities.
• Systematic error recovery routines and workflow reranking logic.
• Agent training explicitly targeting dynamic UI interaction and adversarial anti-bot environments.
• Expansion into additional international, subscription, and longitudinal user scenarios.

ClawBench establishes a rigorous, reproducible foundation for agentic evaluation at the intersection of natural language processing, web automation, and real-world usability. As agentic benchmarks evolve, ClawBench’s uncompromising ecological validity and layered observability will remain reference standards for community progress (Zhang et al., 9 Apr 2026).