ClawArena: Benchmark for Persistent AI Agents

• ClawArena is a benchmark that tests AI agents’ ability to maintain accurate beliefs over time in dynamic, multi-source, and personalized environments.
• It assesses multi-source conflict reasoning, dynamic belief revision, and implicit personalization across realistic communication channels like chats, emails, and logs.
• Empirical results highlight that both model capability and framework design substantially influence performance, underscoring challenges in deploying persistent assistants.

Searching arXiv for the primary ClawArena paper and closely related papers to ground the article. ClawArena is a benchmark for evaluating AI agents as persistent assistants in evolving information environments, with the central objective of testing whether an agent can maintain correct beliefs over time when evidence is noisy, partial, heterogeneous, and sometimes contradictory (Ji et al., 5 Apr 2026). It was introduced to address a gap in existing benchmarks, which largely assume static, single-authority settings and therefore do not evaluate whether an agent can track facts scattered across chats, files, logs, and updates; revise beliefs when later evidence invalidates earlier conclusions; and preserve user preferences that are learned implicitly through interaction history rather than supplied as explicit one-shot instructions (Ji et al., 5 Apr 2026).

1. Problem setting and motivating assumptions

ClawArena is built around a setting in which an agent interacts over multiple rounds with a changing information environment. The paper characterizes this environment through three coupled structural traits. First, it is multi-source: evidence is spread across heterogeneous sources that may disagree. Second, it is dynamic: later evidence can invalidate earlier conclusions. Third, it is personalized: user preferences emerge from interaction history, not from explicit one-shot instructions (Ji et al., 5 Apr 2026).

This framing shifts the task from simple retrieval to belief maintenance. The key difficulty is not only locating relevant evidence, but deciding what should still be believed as new evidence appears. A central design principle follows from this choice: answer correctness is judged against a complete hidden ground truth, not against any single observable source (Ji et al., 5 Apr 2026). This makes ClawArena a benchmark of epistemic state tracking under uncertainty rather than a benchmark of source-following.

A further implication of the benchmark design is that the agent is evaluated under partial observability. The complete hidden ground truth exists, but the agent sees only a noisy, partial, and sometimes contradictory projection of it through session histories, workspace files, staged updates, and a personalization protocol (Ji et al., 5 Apr 2026). This suggests that the benchmark is intended to stress belief revision and source arbitration rather than memorization of a fixed corpus.

2. Scenario construction and evidence environment

Each ClawArena scenario contains a complete hidden ground truth designated as Layer 0, which includes the objective timeline and contradiction map. The scenario is organized into six layers:

• Layer 0: hidden truth, objective timeline, contradiction map
• Layer 1: workspace files
• Layer 2: session histories
• Layer 3: evaluation questions
• Layer 4: staged updates
• Layer 5: internal generation guide

What the agent observes is a projection of Layer 0 through the visible artifacts and staged changes (Ji et al., 5 Apr 2026).

The evidence environment is deliberately dense and noisy. Scenarios typically include 5–7 channels such as Slack, email, and WeChat, 200–400 messages, and 4–8 workspace documents, such as monitoring logs, sprint notes, and audit reports, together with staged updates that arrive later and can change the correct answer (Ji et al., 5 Apr 2026). Some sources conflict, some are incomplete, and some are irrelevant but plausible. To keep the scenarios realistic, the benchmark uses real-world empirical distributions for message timing, contact frequency, and information overload (Ji et al., 5 Apr 2026).

The personalization component is not a mere annotation layer. ClawArena uses a four-stage personalization protocol that ends with silent-exam rounds, so user preferences must be inferred and retained from prior interaction rather than recovered from a contemporaneous instruction (Ji et al., 5 Apr 2026). In operational terms, this means that the benchmark couples factual reasoning, temporal revision, and preference adherence within a single scenario rather than isolating them into separate tasks.

3. Challenge taxonomy and evaluation protocol

ClawArena evaluates three main dimensions:

• Multi-source conflict reasoning (MS): the agent must decide which source claims are reliable when sources disagree.
• Dynamic belief revision (DU): the agent must update its answer when later evidence contradicts an earlier conclusion.
• Implicit personalization (P): the agent must infer and retain user preferences from earlier corrections and interaction patterns, then apply them later without reminders (Ji et al., 5 Apr 2026).

These dimensions generate 7 non-empty subsets:

• MS
• DU
• P
• MS × DU
• MS × P
• DU × P
• MS × DU × P

Each subset is split into a recall variant and a reasoning variant, yielding

$7 \times 2 = 14$

question categories in total (Ji et al., 5 Apr 2026). The taxonomy is explicitly designed so that a system cannot succeed by handling only one aspect in isolation.

ClawArena uses two question formats. The first is multi-choice / set-selection, in which the agent selects the correct subset from 7–9 candidate statements. In the cross-framework experiment, this format is scored by the per-option rule

$1 - (\mathrm{fp} + \mathrm{fn})/n$

whereas the cross-model experiment uses exact-match (Ji et al., 5 Apr 2026). The second format is shell-based executable checks, in which the agent’s claim is tested by running sandboxed shell commands against the workspace files (Ji et al., 5 Apr 2026). The intended separation is between reasoning about evidence and verifiable workspace grounding.

This two-format design matters because it prevents a plausible-text answer from being treated as equivalent to a grounded answer. The benchmark therefore does not collapse textual inference and file-level operational correctness into a single score.

4. Scale, released data, and principal empirical results

The released benchmark contains 64 scenarios across 8 professional domains, totaling 1,879 evaluation rounds and 365 dynamic updates (Ji et al., 5 Apr 2026). Because each round involves a full multi-turn interaction, evaluation is expensive. For some experiments the paper therefore uses a 12-scenario subset with 337 rounds, covering all 8 domains and representing 17.9% of the full benchmark. On GPT-5.2, this subset yields 58.9% exact-match versus 55.4% on the full set, a difference of only 3.5 points, which the paper presents as evidence that the subset is representative (Ji et al., 5 Apr 2026).

The experimental study evaluates five agent frameworks and five models. The frameworks are OpenClaw, Claude Code, NanoBot, PicoClaw, and MetaClaw. The models are Claude Opus 4.6, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.2, and GPT-5.1 (Ji et al., 5 Apr 2026). MetaClaw is described as a self-evolving skill framework that stores procedural skills distilled from past failures and injects them into prompts without changing model weights (Ji et al., 5 Apr 2026).

The paper’s headline quantitative result is that model capability has the larger effect on performance, but framework design remains substantial. It reports a model-induced Overall range: 15.4% and a framework-induced Overall range: 9.2% (Ji et al., 5 Apr 2026). In the cross-model comparison under OpenClaw, the reported Overall scores are 0.735 for Opus 4.6, 0.708 for Sonnet 4.6, 0.614 for Haiku 4.5, and 0.581 for GPT-5.1, establishing the ordering Opus > Sonnet > Haiku > GPT-5.1 (Ji et al., 5 Apr 2026).

Framework effects are nonetheless material. Under GPT-5.1, MetaClaw attains 0.603 Overall versus 0.579 Overall for OpenClaw, a +4.1% improvement over its executor, and it also achieves the best execution evaluation score, 0.511 versus 0.468 for OpenClaw (Ji et al., 5 Apr 2026). The paper interprets this as evidence that skill injection is especially useful for workspace-grounded operations.

5. Diagnostics, failure modes, and interpretation

ClawArena’s diagnostic value lies in how it differentiates distinct failure sources that aggregate metrics can hide. The paper explicitly notes that aggregate scores can conceal qualitatively different failure modes and reports several such patterns (Ji et al., 5 Apr 2026).

One of the central findings is that belief-revision difficulty is determined by update design strategy rather than the mere presence of updates. In targeted, contradictory update settings, scores can drop by 0.28–0.36 after the first update, whereas in more distributed update settings the overall change can be much smaller; one comparison reports that Haiku changes only +1.7% overall (Ji et al., 5 Apr 2026). The conclusion drawn in the paper is that concentrated, strategically placed contradictions are substantially harder than merely having many updates.

The two question formats expose another separation. Executable-check performance is often weak even when multi-choice performance is strong. The paper gives a concrete example in which Haiku scores 95.2% on multi-choice but 0.0% on executable checks in one hospital-administration scenario (Ji et al., 5 Apr 2026). This demonstrates that reasoning quality and tool or workspace grounding are only partly coupled.

The reported diagnostics also identify model- and domain-specific effects. Domain variation is large, with performance varying by over 60% across domains for the same model, and GPT-5.2 performs better than Haiku on a Chinese-language enterprise scenario by 26.7%, suggesting that language-specific training data matters (Ji et al., 5 Apr 2026). Some models show narrative anchoring biases, and the paper notes that frameworks like Claude Code can reduce these biases by quoting source text before reasoning (Ji et al., 5 Apr 2026). At the upper end of scenario difficulty, the hardest scenario defeats all models, with below 30% exact-match (Ji et al., 5 Apr 2026).

Taken together, these observations indicate that ClawArena is not primarily a benchmark of generic verbal competence. Its discriminative signal comes from the interaction among source conflict, temporal revision, and latent preference structure, plus the distinction between plausible reasoning and verifiable grounding.

6. Subsequent uses, conceptual boundaries, and distinct benchmarks

ClawArena has already been reused in later work, but the name also sits within a broader family of “claw” benchmarks whose scopes differ sharply. In "Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents", ClawArena is one of the two main evaluation benchmarks and is used specifically for coding-agent tasks with an objective checker and an attached preference overlay (Zhou et al., 11 Jun 2026). That study uses 62 total scenario templates from ClawArena, split into 32 scenarios from four families for training and 30 scenarios from five entirely unseen families for out-of-distribution evaluation, and reports that TRACE reduces preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks (Zhou et al., 11 Jun 2026). A plausible implication is that the ClawArena design is reusable as a substrate for studying preference compliance in multi-round coding workflows.

Several similarly named artifacts are distinct from ClawArena proper. ClawArena-Team is a separate benchmark for measuring whether a single LLM can act as a manager of subagents in multi-turn, multimodal, multi-directory scenarios; it contains 41 scenarios, 258 evaluation rounds, and 72 staged updates, and its scoring is based on the Subagent-Management Score (SMS) rather than ClawArena’s question taxonomy (Xiong et al., 30 Jun 2026). SafeClawArena is likewise a separate benchmark, focused on security of persistent Claw-like agents; it contains 406 adversarial tasks across four attack surfaces and evaluates containerized replicas of real agent platforms with automated taint tracking (Niu et al., 29 Jun 2026).

By contrast, some “claw” papers do not define or use ClawArena at all. Claw-SWE-Bench introduces a multilingual SWE-bench-style benchmark and adapter protocol for OpenClaw-style agent harnesses, and the named artifact in that paper is Claw-SWE-Bench, not ClawArena (Zheng et al., 10 Jun 2026). SecureClaw introduces a dual-boundary security architecture evaluated on AgentDojo, AgentLeak, and ASB, and the term ClawArena does not appear in that paper’s evaluated objects (Ma et al., 8 Jun 2026).

These distinctions matter because “ClawArena” can be misread as a generic label for any benchmark involving Claw-like agents. In the literature summarized here, however, ClawArena denotes a specific benchmark for maintaining correct beliefs in evolving information environments, whereas ClawArena-Team and SafeClawArena are separate benchmarks targeting managerial orchestration and system security, respectively.

7. Limitations and future directions

The main caveat stated in the original paper is that ClawArena remains a constructed benchmark, not a live environment (Ji et al., 5 Apr 2026). Although it is designed to mimic realistic persistent-assistant settings, it uses static files, staged updates, and predefined questions rather than unconstrained interaction with live sources (Ji et al., 5 Apr 2026). This limits ecological validity in exactly the way the benchmark seeks to expose: real deployed agents must often formulate their own queries, navigate open-ended source discovery, and react to information that is neither staged nor pre-indexed.

The paper therefore presents ClawArena as a step toward more realistic persistent-agent evaluation rather than a final statement of the problem. Its explicit future direction is to move beyond static files and staged updates toward live, unconstrained environments in which agents must formulate their own queries and interact with real-time information sources (Ji et al., 5 Apr 2026). This suggests a research trajectory from hidden-ground-truth scenario design toward environments that preserve the same three coupled challenges—multi-source conflict reasoning, dynamic belief revision, and implicit personalization—while reducing the amount of benchmark-side scaffolding.

Within that trajectory, ClawArena’s lasting contribution is methodological. It defines persistent-assistant evaluation around hidden ground truth, multi-round evidence evolution, and coupled challenge dimensions, and it demonstrates empirically that both model capability and framework design affect whether an agent can sustain correct beliefs under such conditions (Ji et al., 5 Apr 2026).