ClawMark Benchmark Overview

• ClawMark benchmark is a comprehensive testbed that evaluates digital coworker agents in multi-turn, multi-day tasks within evolving office environments.
• It employs deterministic, rule-based verification across sandboxed services to simulate real-world professional scenarios with multimodal inputs.
• The benchmark reveals challenges in state tracking, environmental adaptation, and compliance, offering actionable insights and detailed failure analysis.

ClawMark is a benchmark for evaluating language-model coworkers operating in persistent, dynamic, and multimodal office workflows. It is designed to measure agent proficiency across multi-turn, multi-day tasks in environments subject to exogenous state changes, requiring robust adaptation and integration of diverse information modalities. ClawMark encompasses a comprehensive set of professional scenarios interfaced via stateful sandboxed services, and employs fully deterministic, rule-based evaluation, removing reliance on LLM-as-judge paradigms. The benchmark supports rigorous, reproducible testing and detailed failure analysis for agent systems operating as long-lived digital coworkers (Meng et al., 26 Apr 2026).

1. Motivation, Scope, and Distinguishing Features

ClawMark addresses several unmet demands in coworker-agent evaluation:

• Persistence and Multi-Day Operation: Realistic deployments require agents to operate over sequences of days, tracking evolving contexts rather than processing atomic, one-shot prompts.
• Exogenous Environmental Change: The office environment mutates independently: new emails, modified calendar events, updated knowledge bases, and newly surfaced evidence in images, PDFs, audio, or video.
• Untranscribed Multimodal Inputs: Professional workflows necessitate grounding in raw, cross-modal artifacts (e.g., scanned forms, spreadsheet data, videos), not solely structured or pre-transcribed text.
• Rigorous Rule-Based Verification: Prior benchmarks commonly depend on static, text-centric episodes and/or LLMs for judgment. ClawMark replaces these with executable, deterministic Python-based checkers.
• Benchmark Gap: No existing benchmark simultaneously tests state-tracking, environmental adaptation, and robust multimodal reasoning over multi-day workflows.

The overarching goal is to establish a testbed that reveals the strengths and limits of LLMs and agentic systems functioning as real-world digital coworkers in evolving professional contexts (Meng et al., 26 Apr 2026).

2. Corpus Structure, Task Format, and Multimodal Assets

Corpus Characteristics:

• Scale: 100 tasks spanning 13 professional scenarios (e.g., operations, insurance, human resources), distributed across 87 distinct in-task roles.
• Turn Structure: Tasks unfold over 2–6 turns (mean 3.6), each representing a working day.
• Sandboxed Services: Five stateful backend services:
  • Filesystem (Docker-mounted)
  • Email (GreenMail SMTP/IMAP)
  • Calendar (Radicale CalDAV)
  • Knowledge Base (Notion-compatible API)
  • Spreadsheet (Google Sheets–compatible API)
• Multimodal Artifacts: 1,072 assets spanning scanned PDFs, images, audio, video, and spreadsheet formats, made available solely in raw, untranslated form.

Between-Turn Dynamics:

• Loud Events: Explicitly announced changes included in the turn’s wake-up prompt.
• Silent Mutations: Unannounced exogenous state changes (e.g., file system edits, new calendar entries, appended emails) requiring the agent to refresh and rediscover contextual changes.

Verification Rubric:

• Each task is accompanied by 6–29 weighted rule-based Python checkers (mean 15.4 per task, 1,537 total), among which 55 are “red-line” constraints that must never be violated (e.g., premature decisions, unauthorized data exfiltration).

3. Task Execution Model and Evaluation Harness

The ClawMark execution model emulates in-universe working days:

• Turn Loop:

1. The system delivers a wake-up prompt, outlining that day’s objectives and listing any loud events.
2. Before the agent’s first tool call, the framework applies all silent mutations to the environment.
3. The agent interacts synchronously with sandboxed services (via tool calls) until signaling completion.
4. Upon agent completion, all checker scripts are executed sequentially against the post-turn service state.

Agents must infer and adapt to environmental drift, as cached prior observations do not reflect between-turn changes. Each task is isolated within a docker-compose-managed stack, ensuring repeatability and deterministic state.

Orchestration Details:

• Components include a loader for task parsing and asset injection, an executor that stages and runs environment and agent loops, a checker runner for automated deterministic evaluation, and an aggregator for structured per-turn/task output (Meng et al., 26 Apr 2026).

4. Scoring Methodologies and Metrics

ClawMark employs two principal metrics:

• Weighted Score: Measures partial progress. For model $m$ on task $\tau$,

$$\text{score}(m, \tau) = \frac{\sum_{c \in C(\tau)} w_c \cdot 1[\text{pass}_c(m, \tau)]}{\sum_{c \in C(\tau)} w_c}$$

Scaled to [0, 1], typically reported as a percentage.

• Strict Task Success: All-or-nothing measurement. For the corpus $T$,

$$\text{Succ}(m) = \left( \frac{100}{|T|} \right) \sum_{\tau\in T} 1[\forall c \in C(\tau): \text{pass}_c(m, \tau)]$$

• LLM-Free Verification: All verdicts reflect pure programmatic state checks, excluding any LLM-as-judge components. All red-line checkers carry high fixed weights under weighted scoring, but count equally for strict task success.
• Checker Taxonomy: There are four principal checker types: filesystem/artifact inspection, backend state queries, email state queries, and numeric/semantic equivalence. All checker verdicts must be identical across multiple runs.

5. Empirical Results and Agent Performance Analysis

Evaluation Suite:

• Frontier Models Benchmarked: Claude Sonnet 4.6, Claude Opus 4.6, GPT-5.4 (high), Gemini 3.1 Pro Preview, Qwen 3.6 Plus, Kimi K2.6 (open-source), Kimi K2.5 (public endpoint).
• Uniform Setup: Single-sweep (no per-model prompt tuning), identical tool interfaces (OpenClaw scaffold), and consistent, isolated stack execution.

Leaderboard (Weighted Score / Task Success):

Model	Score (%)	Task Success (%)
Claude Sonnet 4.6	75.8	14.0
Claude Opus 4.6	74.6	20.0
GPT-5.4 (high)	72.0	9.0
Kimi K2.6	68.4	7.0
Gemini 3.1 Pro	68.2	8.0
Qwen 3.6 Plus	57.2	5.0
Kimi K2.5	56.0	0.0

• Key Trends: Highest weighted score observed (75.8) is offset by a rare strict end-to-end success (max 20.0%), indicating agents reliably deliver partial progress, but end-to-end multi-day completion remains elusive.
• Scenario Difficulty: Project management tasks consistently pose the greatest challenge (<44.0 mean score).
• Efficiency: No monotonic correlation between resource usage (tokens/tool calls) and aggregate score.
• Turn-Level Trajectory: In 73 tasks with three turns: after exogenous environmental changes (Day 2), all models except Qwen 3.6 experience a marked performance drop (6–11 pp on average), with only partial recovery on Day 3.

Failure Taxonomy: (Evaluated across 10,759 checker calls; overall failure rate 31.6%)

Failure Mode	Fail Rate (%)
Silent-change detection	56.5
Backend writeback	53.6
Cross-source consistency	34.0
Deliverable correctness	31.4
Evidence extraction	23.6
Compliance guardrails	21.5
Red-line violations	7.1

Notable Case Studies:

• Positive Multimodal Chain: E.g., task “content_operation_task7” involving Whisper (audio), video-frame vision, PDF parsing, and spreadsheet cross-analysis.
• Red-Line Breach: E.g., insurance_task1, where an agent approves a claim before necessary evidence (technical report) is received, violating compliance rules.

6. Construction Pipeline and Implementation Details

Service Architecture:

• Each task is encapsulated within a per-task docker-compose deployment, instantiating the five service backends.
• The orchestrator pipeline consists of loading, execution, deterministic checking, and aggregation components, guaranteeing reproducibility.

Corpus Construction:

• Phase 1: Task Authoring — task.py with explicit turn structure, inject hooks, and specified checker rubrics (including mandatory silent mutation tests and multi-modal contradictions).
• Phase 2: Evidence Sourcing — raw data collected from publicly available documents, original recordings, and targeted AI-generated assets.
• Phase 3: Review Loop — 3–5 audit cycles mixing human and AI evaluation, checker hacking, task-checker correspondence validation, and runtime reviewer agent checks for ambiguous or fragile tasks.
• Phase 4: Release Gate — task only published if all artifacts are human-audited, no audit or runtime flaws remain, and checker verdicts are bit-identical across at least two reference-run trajectories.

7. Insights, Open Challenges, and Future Directions

Key Findings and Open Problems:

• Adaptation Deficit: The two largest failure modes (silent-change detection, backend writeback) indicate present agents are ill-equipped to refresh or synchronize internal state after exogenous updates.
• Partial Completion vs. End-to-End Success: High weighted scores obscure the underlying prevalence of unfinished subtasks; strict task success remains infrequent, necessitating both granular and holistic metrics.
• Multimodal Integration: While some models can exploit cross-modality (e.g., audio→vision→text→tabular), consistency is restricted to leading systems.
• Red-Line Enforcement: Even highest-scoring agents sporadically breach critical constraints, highlighting the need for stringent compliance mechanisms beyond raw task specification.

Future Benchmark Trajectories:

• Incorporation of advanced state-management heuristics or explicit “refresh” protocols.
• Expansion of service emulation (web APIs, database, interactive chat) to better mirror professional systems.
• Exploration of hybrid verification (combining rule-based and learned approach) to accommodate open-ended or fuzzy outputs.
• Development of multi-agent and human-in-the-loop benchmark tracks.

The ClawMark corpus, agent harness, and full construction toolkit are publicly released, providing an extensible foundation for measuring and advancing long-lived, adaptable, and multimodal agentic systems in evolving digital work environments (Meng et al., 26 Apr 2026).