UniClawBench: Capability-Driven Benchmark

• UniClawBench is a benchmark for proactive agents in dynamic real-world settings with 400 bilingual tasks.
• It organizes tasks around five core capabilities to pinpoint performance bottlenecks in skill usage, exploration, and reasoning.
• Evaluation employs live Docker execution with fine-grained checkpoint scoring and a closed-loop feedback system.

UniClawBench is a benchmark for proactive agents operating in dynamic real-world environments. It is introduced as the first capability-driven benchmark for this setting and contains 400 bilingual tasks organized around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Unlike benchmarks that depend on static, pre-recorded answers or single-turn evaluation, UniClawBench runs agents in live Docker containers, scores them with fine-grained completion checkpoints, and evaluates them through a closed-loop interaction among an executor agent, a hidden supervisor agent, and a user simulator agent (Chen et al., 9 Jul 2026).

1. Definition and benchmark position

UniClawBench targets a class of systems described as proactive agents: agents that autonomously interleave reasoning, tool calls, and action, manage skills, and maintain persistent state across sessions. The benchmark is motivated by three limitations attributed to prior agent evaluations: reliance on sandboxed environments, single-turn evaluation paradigms, and scenario-based taxonomies that mix multiple capabilities inside the same task category, thereby obscuring the source of failure (Chen et al., 9 Jul 2026).

Its design occupies a distinctive position within the broader Claw-style benchmark landscape. WildClawBench emphasizes native-runtime, long-horizon evaluation in real CLI harnesses, with 60 bilingual multimodal tasks and explicit harness sensitivity, including shifts of up to 18 points for a single model under different harnesses (Ding et al., 11 May 2026). ClawMark emphasizes multi-turn, multi-day coworker workflows in stateful sandboxed services, with 100 tasks and 1537 deterministic Python checkers over post-execution state (Meng et al., 26 Apr 2026). UniClawBench differs from both by making the capability taxonomy itself the primary organizing principle, so that failure can be attributed to a specific bottleneck rather than to a broad scenario label (Chen et al., 9 Jul 2026).

This suggests a diagnostic orientation rather than a purely leaderboard-oriented one. A task is assigned to the capability whose absence would make success impossible, which makes capability-specific performance a first-class object of analysis rather than a secondary breakdown (Chen et al., 9 Jul 2026).

2. Capability-driven taxonomy

The benchmark is partitioned into five foundational capabilities, each represented by 80 tasks. The taxonomy is intended to separate bottlenecks that scenario-driven benchmarks often conflate (Chen et al., 9 Jul 2026).

Capability	Evaluated ability	Illustrative tasks
Skill Usage	Discover, inspect, understand, and correctly invoke declared tools or APIs	"task_05_org_chart_mermaid", "task_19_github_activity_digest"
Exploration	Investigate uncertain, noisy, or underspecified environments	"task_01_zshrc_prompt_refactor", "task_18_license_provenance_audit"
Long-Context Reasoning	Integrate heterogeneous evidence across long contexts and trajectories	"task_03_slack_openclaw_assistant_setup", "task_07_santa_monica_trip_budget"
Multimodal Understanding	Extract and ground information from images, video, and audio	"task_01_scaling_laws_figure5_aspect_ratio", "task_22_met_art_room_match"
Cross-Platform Coordination	Coordinate state across multiple platforms, apps, and interfaces	"task_01_rag_survey_gui_zotero_obsidian", "task_03_rust_tokyo_meetup_gui_calendar"

Skill Usage tasks evaluate whether an agent can discover and understand declared tools, skills, or APIs, invoke them correctly on provided data, and produce verifiable artifacts such as JSON, CSV, Markdown reports, logs, or diagrams. The key constraint is that the bottleneck is not source discovery but correct tool application. The cited examples include an OCR-based boarding-pass parser, a Mermaid org-chart task from a proxy statement, and a live GitHub activity digest that must authenticate with $GITHUB_TOKEN and produce both digest.md and activity_index.csv (Chen et al., 9 Jul 2026).

Exploration tasks shift the burden from execution to investigation. They require agents to search uncertain or noisy environments, identify relevant evidence, reject distractors, and document negative findings. The .zshrc audit task requires validation across eight scenarios and real shell behavior, while the license-provenance task requires detection of traps such as dual licenses, vendored code, generated headers, and registry-versus-LICENSE mismatches, with outputs including license_audit_summary.json, license_file_matrix.json, license_rejection_traps.json, a methodology file, and an audit script with run log (Chen et al., 9 Jul 2026).

Long-Context Reasoning tasks require evidence integration over extended trajectories and heterogeneous sources. The Slack-plus-OpenClaw setup task combines a YouTube instructional video, real Slack, Outlook webmail, CAPTCHA handling, and bot setup. The Santa Monica travel-planning task requires coherent itinerary and budget construction across multiple airline, hotel, rental, and reservation sources, with daily planning and budget trade-offs preserved consistently (Chen et al., 9 Jul 2026).

Multimodal Understanding tasks are classified as such only when correctness depends on non-text modalities rather than on textual surrogates. Examples include recreating the “Aspect Ratio” line chart from Figure 1 of a paper found on arXiv, with Python and matplotlib output plus revision notes, and selecting a painting for room style matching using screenshots and metadata from three Metropolitan Museum pages (Chen et al., 9 Jul 2026).

Cross-Platform Coordination tasks require synchronized use of multiple platforms and apps, with side effects verified in the applications themselves. The Zotero-plus-Obsidian task requires a real Zotero desktop entry, a BibTeX export keyed as gao2024retrieval, and an Obsidian literature note linked to that citation key. The Rust Global Tokyo task requires event verification across several web pages, creation of a real event in GNOME Calendar or an equivalent GUI calendar, and a screenshot of the event in the application view (Chen et al., 9 Jul 2026).

3. Dataset structure, task packaging, and curation

UniClawBench contains 400 tasks, distributed as 5 capabilities times 80 tasks each, with 40 English and 40 Chinese tasks per capability. The corpus is manually curated from realistic day-to-day scenarios spanning research, office work, shopping, travel, DevOps, multimedia, and civic settings (Chen et al., 9 Jul 2026).

The benchmark is bilingual in a strict operational sense. Prompts and evaluation rubrics are written in the respective language, and the user simulator is instructed to respond in the same language as the original public task. This means bilingualism is embedded not only in task text but also in the closed-loop evaluation process (Chen et al., 9 Jul 2026).

Heatmaps described for the benchmark indicate coverage across application domains such as media and entertainment, AI/ML and research, travel and events, business and finance, productivity and knowledge management, consumer and shopping, games and puzzles, data and analytics, design and creative work, and public or civic contexts. Inputs span tables and JSON, CSV and spreadsheets, logs and email, PDFs and documents, videos, screenshots and images, while outputs include Markdown reports, CSV files, JSON audits, diagrams, calendar files, BibTeX exports, and screenshots (Chen et al., 9 Jul 2026). This suggests that the task distribution is intended to force interaction with heterogeneous representational formats rather than only with text-centric corpora.

Each task is a package rather than a prompt-answer pair. The public contract is a YAML file containing fields such as task_id, category, task, task_snapshot, sources, skills, services, pre_exec, timeouts, success thresholds, and CodeX-related configuration. The hidden evaluation side is an eval_rule.md file organized into nine sections: grading philosophy; task contract; source-selection and target-resolution rules; ground-truth snapshot; checkpoint rubric; scoring policy and caps; continue-versus-fail guidance; hidden reference assets inventory; and dynamic-content notes (Chen et al., 9 Jul 2026).

The scoring structure implied by the hidden rubric is checkpoint-based:

$S_{\text{task}}(a) = \sum_{i=1}^{K} w_i\, c_i(a),$

where $a$ is an agent attempt, $c_i(a) \in [0,1]$ is checkpoint completion, and $\sum_i w_i = 1.0$. Section-level caps then bound the maximum attainable score under specified conditions, such as restricted search breadth (Chen et al., 9 Jul 2026).

Evaluation reliability is supported by a 50-trajectory study with 3 human experts. Automatic pass/fail agrees with human majority at 92%, and checkpoint-based Average Score correlates with human scores at Pearson $r = 0.71$ and Spearman $\rho = 0.68$ (Chen et al., 9 Jul 2026). This does not eliminate evaluator bias, but it indicates that the hidden-supervisor scoring procedure tracks human judgments reasonably closely.

4. Live Docker execution and closed-loop supervision

The benchmark executes agents in fresh live Docker containers rather than in static replay environments. The execution environment includes a real browser with Agent-Browser, a terminal and bash environment, GUI applications such as Zotero, Obsidian, GNOME Calendar, and PDF viewers, a local filesystem rooted at /tmp_workspace, and task-specific skills and services such as advanced packaging tools, web search, and desktop control (Chen et al., 9 Jul 2026).

This live-execution choice places UniClawBench alongside native-runtime evaluation efforts rather than sandbox-only benchmarks. WildClawBench similarly argues for evaluation in reproducible Docker containers hosting real agent harnesses and real tools rather than mock services (Ding et al., 11 May 2026). UniClawBench extends that orientation by coupling live execution with capability-driven task assignment and multi-turn feedback (Chen et al., 9 Jul 2026).

Its evaluation loop uses three roles. The executor agent is the evaluated system and has access only to the public task contract, visible files, tools, and the results directory. The hidden supervisor agent operates in a separate workspace containing public task material, visible evidence, hidden references, and eval_rule.md; it evaluates the attempt and emits structured decisions such as verdict, attempt_state, recoverable, score, and confidence. The user simulator agent sees only the public task, visible executor evidence, and a minimal handoff from the supervisor—specifically coarse fields such as verdict, attempt_state, recoverable, and score—and then generates natural-language follow-up feedback (Chen et al., 9 Jul 2026).

A strict information firewall separates the three roles. Hidden rubrics, rationale, missing artifacts, guidance tags, and reference assets never appear in the user simulator workspace, and supervisor rationale is never passed to the executor. The user simulator’s feedback is further sanitized by a deterministic feedback rewriter before being shown to the executor (Chen et al., 9 Jul 2026). This architecture is meant to preserve the realism of user feedback while avoiding label leakage.

The interaction protocol is iterative. The executor receives the initial task, the supervisor evaluates the outcome, and the loop ends on pass, irrecoverable fail, infra_error, timeout, or follow-up-budget exhaustion. If the supervisor returns continue and the attempt remains recoverable, the user simulator produces a follow-up instruction and the executor proceeds to another turn (Chen et al., 9 Jul 2026). A plausible implication is that UniClawBench treats recovery under feedback as part of agent competence rather than as an extraneous convenience.

Timeouts are capability-sensitive. Standard tasks use a 30-minute global timeout with 20 minutes per turn; long-context tasks use a 45-minute global timeout with 30 minutes per turn. If timeout occurs, the highest score among completed turns is retained (Chen et al., 9 Jul 2026).

5. Scoring methodology and empirical results

UniClawBench reports two primary aggregate metrics. Pass Rate is the fraction of tasks ending with verdict = pass:

$$\text{PR} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\text{task}_i \text{ passed}].$$

Average Score is the mean checkpoint-based task score:

$$\text{AS} = \frac{1}{N}\sum_{i=1}^{N} S_{\text{task}_i}.$$

Capability-specific variants $\text{PR}_c$ and $\text{AS}_c$ are defined by restricting the average to tasks whose category equals capability $a$0 (Chen et al., 9 Jul 2026).

Under OpenClaw, the strongest overall Pass Rate reported is for Claude Opus-4.8 at $a$1, with $a$2. GPT-5.4 records $a$3 and $a$4, while Claude Sonnet-4.6 records $a$5 and $a$6. Open-source or regionally hosted models narrow the gap more than a simple pass/fail summary would suggest: Kimi-2.6 reaches $a$7, $a$8, and Qwen-3.5-Plus reaches $a$9, $c_i(a) \in [0,1]$0 (Chen et al., 9 Jul 2026).

The benchmark’s diagnostic structure reveals uneven capability profiles. Models are relatively strong on Skill Usage and Exploration, with Exploration Pass Rates above 0.75 for several systems, including Gemini-3.0-Flash at 0.800 and Claude Sonnet-4.6 at 0.812. Performance is substantially weaker on Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. For example, GPT-5.4 records Long-Context $c_i(a) \in [0,1]$1 and Multimodal $c_i(a) \in [0,1]$2, while Claude Opus-4.8 records Long-Context $c_i(a) \in [0,1]$3 and Cross-Platform $c_i(a) \in [0,1]$4 (Chen et al., 9 Jul 2026).

A central empirical pattern is the “halfway failure” phenomenon: Average Scores are materially higher than Pass Rates. This means agents often complete many checkpoints and make substantial partial progress but still fail to satisfy the full task contract because of late-stage inconsistency or an irrecoverable error (Chen et al., 9 Jul 2026). This pattern parallels findings from ClawMark, where weighted score was much higher than strict Task Success and complete end-to-end workflow completion remained rare (Meng et al., 26 Apr 2026).

Cross-framework evaluation shows that framework design materially shapes outcomes. For GPT-5.4, OpenClaw yields $c_i(a) \in [0,1]$5, $c_i(a) \in [0,1]$6; EDICT yields $c_i(a) \in [0,1]$7, $c_i(a) \in [0,1]$8; Nanobot yields $c_i(a) \in [0,1]$9, $\sum_i w_i = 1.0$0. For Claude Opus-4.8, OpenClaw yields $\sum_i w_i = 1.0$1, EDICT 0.415, Nanobot 0.385. For Kimi-2.6, OpenClaw yields $\sum_i w_i = 1.0$2, EDICT 0.278, and Nanobot 0.290 (Chen et al., 9 Jul 2026). The authors attribute OpenClaw’s advantage primarily to unified trajectory and context management, EDICT’s pattern to multi-agent coordination friction and context loss, and Nanobot’s profile to token efficiency paired with reduced evidence coverage (Chen et al., 9 Jul 2026).

Token consumption rises especially for Long-Context and Multimodal tasks, and performance improves over additional feedback cycles. The reported progression indicates that Pass Rate begins around 23.8% after cycle 0 and then increases across later cycles (Chen et al., 9 Jul 2026). This supports the benchmark’s claim that multi-turn recovery is a meaningful capability axis that static first-answer evaluation suppresses.

6. Broader significance, limitations, and future directions

UniClawBench’s main contribution is methodological: it combines capability-driven taxonomy, live Docker execution, checkpoint-based scoring, and closed-loop user feedback within a single framework. This combination is unusual within the broader literature. RealClawBench anchors evaluation to real OpenClaw developer sessions and reconstructed execution environments (Lv et al., 2 Jun 2026); Claw-SWE-Bench makes harnesses comparable on coding tasks through a fixed adapter protocol and shows that harness choice can shift Pass@1 by 27.4 percentage points under fixed models (Zheng et al., 10 Jun 2026); WildClawBench demonstrates that native-runtime agent evaluation is highly sensitive to harness design (Ding et al., 11 May 2026). UniClawBench inherits the same concern with disentangling model capability from framework behavior, but operationalizes it through a capability-first task taxonomy and a structured feedback loop (Chen et al., 9 Jul 2026).

This design has practical implications. Researchers can use capability-level Pass Rates and Average Scores to target specific weaknesses such as long-horizon memory, multimodal grounding, or cross-platform state synchronization. Framework designers can use the same tasks to study context management, orchestration overhead, and evidence preservation. The benchmark’s separation of executor, supervisor, and user simulator also provides a template for future multi-turn agent evaluation in which recovery matters but label leakage is tightly controlled (Chen et al., 9 Jul 2026).

The authors note several limitations. The task set, while large, still covers only 400 tasks; live environments introduce instability because websites and services may change; and both supervisor and user simulator rely on GPT-5.4 Codex agents, which may inject systematic evaluator bias (Chen et al., 9 Jul 2026). Future work is described in terms of scaling to more domains and languages, adding capability dimensions such as safety or adversarial robustness, integrating other evaluation paradigms, and refining framework-level analysis (Chen et al., 9 Jul 2026).

Taken together, UniClawBench can be understood as a universal benchmark not because it covers every possible agent setting, but because it proposes a general evaluation logic for proactive agents: identify the primary capability bottleneck, run the agent in a live environment, score the trajectory through fine-grained checkpoints, and allow realistic but sanitized feedback to test recovery as well as first-pass execution (Chen et al., 9 Jul 2026).