---
title: 'ClawArena: Benchmark for Persistent AI Agents'
url: https://www.emergentmind.com/topics/clawarena
type: topic
---

# ClawArena: Benchmark for Persistent AI Agents

Searching arXiv for the primary ClawArena paper and closely related papers to ground the article.
ClawArena is a benchmark for evaluating AI agents as persistent assistants in evolving information environments, with the central objective of testing whether an agent can maintain correct beliefs over time when evidence is noisy, partial, heterogeneous, and sometimes contradictory [2604.04202]. It was introduced to address a gap in existing benchmarks, which largely assume static, single-authority settings and therefore do not evaluate whether an agent can track facts scattered across chats, files, logs, and updates; revise beliefs when later evidence invalidates earlier conclusions; and preserve user preferences that are learned implicitly through interaction history rather than supplied as explicit one-shot instructions [2604.04202].

## 1. Problem setting and motivating assumptions

ClawArena is built around a setting in which an agent interacts over multiple rounds with a changing information environment. The paper characterizes this environment through three coupled structural traits. First, it is **multi-source**: evidence is spread across heterogeneous sources that may disagree. Second, it is **dynamic**: later evidence can invalidate earlier conclusions. Third, it is **personalized**: user preferences emerge from interaction history, not from explicit one-shot instructions [2604.04202].

This framing shifts the task from simple retrieval to belief maintenance. The key difficulty is not only locating relevant evidence, but deciding what should still be believed as new evidence appears. A central design principle follows from this choice: answer correctness is judged against a complete hidden ground truth, not against any single observable source [2604.04202]. This makes ClawArena a benchmark of epistemic state tracking under uncertainty rather than a benchmark of source-following.

A further implication of the benchmark design is that the agent is evaluated under partial observability. The complete hidden ground truth exists, but the agent sees only a noisy, partial, and sometimes contradictory projection of it through session histories, workspace files, staged updates, and a personalization protocol [2604.04202]. This suggests that the benchmark is intended to stress belief revision and source arbitration rather than memorization of a fixed corpus.

## 2. Scenario construction and evidence environment

Each ClawArena scenario contains a **complete hidden ground truth** designated as **Layer 0**, which includes the objective timeline and contradiction map. The scenario is organized into six layers:

- **Layer 0**: hidden truth, objective timeline, contradiction map  
- **Layer 1**: workspace files  
- **Layer 2**: session histories  
- **Layer 3**: evaluation questions  
- **Layer 4**: staged updates  
- **Layer 5**: internal generation guide  

What the agent observes is a projection of Layer 0 through the visible artifacts and staged changes [2604.04202].

The evidence environment is deliberately dense and noisy. Scenarios typically include **5–7 channels** such as Slack, email, and WeChat, **200–400 messages**, and **4–8 workspace documents**, such as monitoring logs, sprint notes, and audit reports, together with staged updates that arrive later and can change the correct answer [2604.04202]. Some sources conflict, some are incomplete, and some are irrelevant but plausible. To keep the scenarios realistic, the benchmark uses real-world empirical distributions for message timing, contact frequency, and information overload [2604.04202].

The personalization component is not a mere annotation layer. ClawArena uses a **four-stage personalization protocol** that ends with silent-exam rounds, so user preferences must be inferred and retained from prior interaction rather than recovered from a contemporaneous instruction [2604.04202]. In operational terms, this means that the benchmark couples factual reasoning, temporal revision, and preference adherence within a single scenario rather than isolating them into separate tasks.

## 3. Challenge taxonomy and evaluation protocol

ClawArena evaluates three main dimensions:

- **Multi-source conflict reasoning (MS)**: the agent must decide which source claims are reliable when sources disagree.  
- **Dynamic belief revision (DU)**: the agent must update its answer when later evidence contradicts an earlier conclusion.  
- **Implicit personalization (P)**: the agent must infer and retain user preferences from earlier corrections and interaction patterns, then apply them later without reminders [2604.04202].

These dimensions generate **7 non-empty subsets**:

- MS  
- DU  
- P  
- MS × DU  
- MS × P  
- DU × P  
- MS × DU × P  

Each subset is split into a **recall variant** and a **reasoning variant**, yielding

$$
7 \times 2 = 14
$$

question categories in total [2604.04202]. The taxonomy is explicitly designed so that a system cannot succeed by handling only one aspect in isolation.

ClawArena uses two question formats. The first is **multi-choice / set-selection**, in which the agent selects the correct subset from **7–9 candidate statements**. In the cross-framework experiment, this format is scored by the per-option rule

$$
1 - (\mathrm{fp} + \mathrm{fn})/n
$$

whereas the cross-model experiment uses exact-match [2604.04202]. The second format is **shell-based executable checks**, in which the agent’s claim is tested by running sandboxed shell commands against the workspace files [2604.04202]. The intended separation is between reasoning about evidence and verifiable workspace grounding.

This two-format design matters because it prevents a plausible-text answer from being treated as equivalent to a grounded answer. The benchmark therefore does not collapse textual inference and file-level operational correctness into a single score.

## 4. Scale, released data, and principal empirical results

The released benchmark contains **64 scenarios** across **8 professional domains**, totaling **1,879 evaluation rounds** and **365 dynamic updates** [2604.04202]. Because each round involves a full multi-turn interaction, evaluation is expensive. For some experiments the paper therefore uses a **12-scenario subset** with **337 rounds**, covering all 8 domains and representing **17.9%** of the full benchmark. On GPT-5.2, this subset yields **58.9%** exact-match versus **55.4%** on the full set, a difference of only **3.5 points**, which the paper presents as evidence that the subset is representative [2604.04202].

The experimental study evaluates **five agent frameworks** and **five models**. The frameworks are **OpenClaw**, **Claude Code**, **NanoBot**, **PicoClaw**, and **MetaClaw**. The models are **Claude Opus 4.6**, **Claude Sonnet 4.6**, **Claude Haiku 4.5**, **GPT-5.2**, and **GPT-5.1** [2604.04202]. MetaClaw is described as a **self-evolving skill framework** that stores procedural skills distilled from past failures and injects them into prompts without changing model weights [2604.04202].

The paper’s headline quantitative result is that model capability has the larger effect on performance, but framework design remains substantial. It reports a **model-induced Overall range: 15.4%** and a **framework-induced Overall range: 9.2%** [2604.04202]. In the cross-model comparison under OpenClaw, the reported Overall scores are **0.735** for Opus 4.6, **0.708** for Sonnet 4.6, **0.614** for Haiku 4.5, and **0.581** for GPT-5.1, establishing the ordering **Opus > Sonnet > Haiku > GPT-5.1** [2604.04202].

Framework effects are nonetheless material. Under GPT-5.1, **MetaClaw** attains **0.603 Overall** versus **0.579 Overall** for **OpenClaw**, a **+4.1%** improvement over its executor, and it also achieves the best execution evaluation score, **0.511** versus **0.468** for OpenClaw [2604.04202]. The paper interprets this as evidence that skill injection is especially useful for workspace-grounded operations.

## 5. Diagnostics, failure modes, and interpretation

ClawArena’s diagnostic value lies in how it differentiates distinct failure sources that aggregate metrics can hide. The paper explicitly notes that aggregate scores can conceal qualitatively different failure modes and reports several such patterns [2604.04202].

One of the central findings is that belief-revision difficulty is determined by **update design strategy rather than the mere presence of updates**. In targeted, contradictory update settings, scores can drop by **0.28–0.36** after the first update, whereas in more distributed update settings the overall change can be much smaller; one comparison reports that Haiku changes only **+1.7%** overall [2604.04202]. The conclusion drawn in the paper is that concentrated, strategically placed contradictions are substantially harder than merely having many updates.

The two question formats expose another separation. Executable-check performance is often weak even when multi-choice performance is strong. The paper gives a concrete example in which Haiku scores **95.2%** on multi-choice but **0.0%** on executable checks in one hospital-administration scenario [2604.04202]. This demonstrates that reasoning quality and tool or workspace grounding are only partly coupled.

The reported diagnostics also identify model- and domain-specific effects. Domain variation is large, with performance varying by **over 60%** across domains for the same model, and GPT-5.2 performs better than Haiku on a Chinese-language enterprise scenario by **26.7%**, suggesting that language-specific training data matters [2604.04202]. Some models show narrative anchoring biases, and the paper notes that frameworks like Claude Code can reduce these biases by quoting source text before reasoning [2604.04202]. At the upper end of scenario difficulty, the hardest scenario defeats all models, with **below 30% exact-match** [2604.04202].

Taken together, these observations indicate that ClawArena is not primarily a benchmark of generic verbal competence. Its discriminative signal comes from the interaction among source conflict, temporal revision, and latent preference structure, plus the distinction between plausible reasoning and verifiable grounding.

## 6. Subsequent uses, conceptual boundaries, and distinct benchmarks

ClawArena has already been reused in later work, but the name also sits within a broader family of “claw” benchmarks whose scopes differ sharply. In **"Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents"**, ClawArena is one of the two main evaluation benchmarks and is used specifically for coding-agent tasks with an objective checker and an attached preference overlay [2606.13174]. That study uses **62 total scenario templates from ClawArena**, split into **32 scenarios from four families** for training and **30 scenarios from five entirely unseen families** for out-of-distribution evaluation, and reports that TRACE reduces preference violation from **100.0% to 37.6%** on in-distribution tasks and from **100.0% to 2.0%** on out-of-distribution tasks [2606.13174]. A plausible implication is that the ClawArena design is reusable as a substrate for studying preference compliance in multi-round coding workflows.

Several similarly named artifacts are distinct from ClawArena proper. **ClawArena-Team** is a separate benchmark for measuring whether a single LLM can act as a manager of subagents in multi-turn, multimodal, multi-directory scenarios; it contains **41 scenarios**, **258 evaluation rounds**, and **72 staged updates**, and its scoring is based on the **Subagent-Management Score (SMS)** rather than ClawArena’s question taxonomy [2606.31174]. **SafeClawArena** is likewise a separate benchmark, focused on security of persistent Claw-like agents; it contains **406 adversarial tasks** across four attack surfaces and evaluates containerized replicas of real agent platforms with automated taint tracking [2606.30755].

By contrast, some “claw” papers do **not** define or use ClawArena at all. **Claw-SWE-Bench** introduces a multilingual SWE-bench-style benchmark and adapter protocol for OpenClaw-style agent harnesses, and the named artifact in that paper is **Claw-SWE-Bench**, not ClawArena [2606.12344]. **SecureClaw** introduces a dual-boundary security architecture evaluated on **AgentDojo**, **AgentLeak**, and **ASB**, and the term ClawArena does not appear in that paper’s evaluated objects [2606.09549].

These distinctions matter because “ClawArena” can be misread as a generic label for any benchmark involving Claw-like agents. In the literature summarized here, however, ClawArena denotes a specific benchmark for maintaining correct beliefs in evolving information environments, whereas ClawArena-Team and SafeClawArena are separate benchmarks targeting managerial orchestration and system security, respectively.

## 7. Limitations and future directions

The main caveat stated in the original paper is that ClawArena remains a **constructed benchmark**, not a live environment [2604.04202]. Although it is designed to mimic realistic persistent-assistant settings, it uses **static files, staged updates, and predefined questions** rather than unconstrained interaction with live sources [2604.04202]. This limits ecological validity in exactly the way the benchmark seeks to expose: real deployed agents must often formulate their own queries, navigate open-ended source discovery, and react to information that is neither staged nor pre-indexed.

The paper therefore presents ClawArena as a step toward more realistic persistent-agent evaluation rather than a final statement of the problem. Its explicit future direction is to move beyond static files and staged updates toward **live, unconstrained environments** in which agents must **formulate their own queries** and interact with **real-time information sources** [2604.04202]. This suggests a research trajectory from hidden-ground-truth scenario design toward environments that preserve the same three coupled challenges—multi-source conflict reasoning, dynamic belief revision, and implicit personalization—while reducing the amount of benchmark-side scaffolding.

Within that trajectory, ClawArena’s lasting contribution is methodological. It defines persistent-assistant evaluation around hidden ground truth, multi-round evidence evolution, and coupled challenge dimensions, and it demonstrates empirically that both model capability and framework design affect whether an agent can sustain correct beliefs under such conditions [2604.04202].

Source: https://www.emergentmind.com/topics/clawarena