Papers
Topics
Authors
Recent
Search
2000 character limit reached

ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents

Published 26 Apr 2026 in cs.CV and cs.SE | (2604.23781v1)

Abstract: Language-model agents are increasingly used as persistent coworkers that assist users across multiple working days. During such workflows, the surrounding environment may change independently of the agent: new emails arrive, calendar entries shift, knowledge-base records are updated, and evidence appears across images, scanned PDFs, audio, video, and spreadsheets. Existing benchmarks do not adequately evaluate this setting because they typically run within a single static episode and remain largely text-centric. We introduce \bench{}, a benchmark for coworker agents built around multi-turn multi-day tasks, a stateful sandboxed service environment whose state evolves between turns, and rule-based verification. The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet) and scored by 1537 deterministic Python checkers over post-execution service state; no LLM-as-judge is invoked during scoring. We benchmark seven frontier agent systems. The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0\%, indicating that partial progress is common while complete end-to-end workflow completion remains rare. Turn-level analysis shows that performance drops after the first exogenous environment update, highlighting adaptation to changing state as a key open challenge. We release the benchmark, evaluation harness, and construction pipeline to support reproducible coworker-agent evaluation.

Summary

  • The paper introduces a benchmark—ClawMark—that evaluates LLM coworker agents on persistent, multi-day, multimodal workflows using rule-based Python checkers.
  • It employs deterministic evaluations over dynamically evolving environments with images, audio, PDFs, and spreadsheets to measure state refresh and tool integration.
  • Empirical results reveal agents often make partial progress yet face significant challenges in achieving full workflow completion due to state refresh and action writeback issues.

ClawMark: A Benchmark for Evaluating Persistent, Multimodal Coworker Agents

Motivation and Context

The ClawMark benchmark addresses critical gaps in the current evaluation of language-model-based agents, particularly in settings where agents act as long-term, stateful coworkers collaborating with humans across multi-turn, multi-day professional workflows. Existing benchmarks predominantly assess static, single-episode performances, lack exogenous state evolution between agent actions, and are primarily text-centered, thereby failing to measure an agent's ability to adapt to dynamic environments and process multimodal information essential for real-world office tasks. ClawMark is designed to fill this void by providing a systematic, rule-based, reproducible framework for evaluating LLM agents on persistent, evolving, multimodal tasks under realistic conditions (2604.23781).

Benchmark Design and Novel Aspects

ClawMark introduces a rigorous evaluation environment constructed around three principal axes absent in prior work:

  • Multi-Turn, Multi-Day Timelines: Each task simulates a real-world workflow spanning two to six in-universe working days, with every turn representing one working day. This setup requires sustained state tracking and cumulative workflow progression, going beyond one-shot or single-episode paradigms.
  • Dynamic, Exogenously-Mutating Environments: The benchmark features a sandbox with five stateful services (filesystem, email, calendar, knowledge base, spreadsheet), where the environment evolves independently between agent actions through both explicit (loud) and implicit (silent) updates. Agents are thus challenged to correctly refresh context and avoid stale cache assumptions.
  • Full Multimodal Evidence: ClawMark incorporates 1,072 raw artifacts encompassing images, audio, scanned PDFs, video, and spreadsheets. Evidence is presented in native modalities, not via pre-transcriptions, compelling agents to autonomously utilize appropriate toolchains (e.g., Whisper, ffmpeg, PyMuPDF) for extraction and reasoning.

Each task includes deterministic, rule-based Python checkers (1,537 total, including red-line compliance constraints) for evaluation, eschewing any language-model-provided judgments. This enables deterministic, reproducible, and interpretable assessment over post-execution state, a significant methodological advance over LLM-as-judge paradigms common in the literature.

Evaluation Metrics and Protocol

ClawMark introduces dual evaluation metrics:

  1. Weighted Score: A real-valued, normalized measure reflecting partial credit over all rubric items, suitable for high-resolution leaderboard distinction across tasks with varying checker complexity.
  2. Strict Task Success: A binary all-or-nothing metric denoting full end-to-end workflow completion; all rubric constraints, including red-lines, must be satisfied. This metric reflects deployment-ready reliability and is sensitive to long-tail or red-line failures.

Tasks and checkers are engineered via a pipeline ensuring every silent change is checker-linked, cross-modal contradictions are intentional, and red-lines map to deterministic state rather than text. Each task is admitted only after consensus is achieved on checker runs, ensuring robustness and release-gate invariance.

Experimental Results

Seven state-of-the-art agent models (five proprietary, two open-source) are benchmarked under a uniform agent framework (OpenClaw) with no per-model prompt engineering, supporting a controlled comparison. Key findings include:

  • Performance Distribution: The top weighted score is 75.8 (Claude Sonnet 4.6), but the strict Task Success peaks at just 20.0 (Claude Opus 4.6). This demonstrates that while agents frequently make partial progress, full, robust end-to-end completion remains rare even among frontier models.
  • Scenario-specific Differentiation: No single model dominates; best scores per scenario are distributed among Claude models, GPT-5.4, and Gemini 3.1 Pro Preview. Difficult scenarios (e.g., Project Management, Insurance, Clinical Assistant) reveal the nontriviality of sustained, safe adaptation in dynamic, multimodal environments.
  • Adaptation Challenge: Turn-level trajectory analysis shows that performance commonly drops after the first exogenous update (Day 2), with only partial recovery in subsequent turns. This underscores the open research challenge of dynamic state refreshing and adaptation in persistent multi-turn settings.
  • Failure Modes: The dominant sources of error are silent-change detection (56.5% fail rate) and backend writeback (53.6%), as opposed to compliance guardrails (21.5%) or basic extraction. Red-line violations are infrequent but non-negligible, concentrated in a subset of high-stakes tasks, indicating potential compliance and safety exposures even for models with high aggregate scores.

Implications and Future Directions

ClawMark's deployment marks a transition from episodic, static, text-centric benchmarks to a regime supporting realistic, persistent coworker agents. Its rigorous, reproducible, and multimodal evaluation protocol exposes fundamental limitations in current LLM agents—especially their inability to reliably maintain and update state across exogenously-mutating timelines and their frequent failures to commit critical actions back to host services.

These findings suggest several future research avenues:

  • Advanced State Management: Methodologies for robustly detecting and adapting to external state mutations, possibly integrating persistent memory architectures or state contrastive learning.
  • Safe and Interpretable Action Planning: Mechanisms for strictly enforcing compliance constraints, reducing surface-complete but safety-violating behaviors undetectable through output text alone.
  • Benchmarks for Specialized Domains: Extension of the benchmark corpus into other high-compliance or safety-sensitive domains (e.g., medical, legal, financial), leveraging ClawMark's modular, scenario-first pipeline.
  • Agent Toolchain Generalization: Further research on universal, sandbox-compatible tool schemas to maximize agent framework applicability and minimize framework-induced variance in evaluation.

ClawMark's approach—especially its deterministic, post-execution state evaluation—establishes a new standard for reproducibility, transparency, and reliability in agent benchmarking.

Conclusion

ClawMark provides a definitive framework for assessing next-generation LLM agents in persistent, dynamic, multimodal coworker scenarios. Empirical results across multiple agent architectures expose a significant performance gap for end-to-end workflow completion, primarily due to failures in state adaptation and service action integration. The benchmark’s methodology and findings signal essential directions for advancing robust, trustworthy AI coworkers and for evaluating their fitness for complex, real-world deployments (2604.23781).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What this paper is about

This paper introduces ClawMark, a big “test world” for checking how well AI assistants (called coworker agents) work alongside people over several days. Unlike many tests that only use text and run once, ClawMark looks like a small, fake office with real moving parts: emails come in, calendars change, files get updated, and important clues can be in images, PDFs, audio, video, and spreadsheets. The goal is to see if AI coworkers can keep up with a changing workplace, not just answer a single question.

The main questions the paper asks

  • Can AI coworkers handle tasks that last several days, not just one session?
  • Can they notice and adapt when the world changes on its own (like a new email arriving) between workdays?
  • Can they use information that isn’t just text—such as images, audio, video, and scans?
  • Can we grade their work in a clear, fair, and repeatable way without asking another AI to “judge” them?

How ClawMark tests AI coworkers

Think of ClawMark like a practice office where everything actually runs but is safely contained.

Here’s what it includes and how it works:

  • Multi-day tasks with turns: Each task is a small project that lasts 2–6 “in-universe” workdays. Each day is one turn. The agent wakes up, sees what’s new, and continues the job.
  • A living office environment: Five working “apps” are provided inside a sandbox:
    • Filesystem (folders and files)
    • Email (sending and receiving)
    • Calendar (events that can move)
    • Knowledge base (like a team wiki)
    • Spreadsheet (like Google Sheets)
    • These services keep their history and can change over time.
  • Changes you’re told about and changes you aren’t:
    • Loud events: Announced changes the agent is told about in the morning.
    • Silent mutations: Unannounced changes the agent must discover on its own (for example, a new file appears or a spreadsheet row changes overnight).
  • Multimodal evidence: Important details live in photos, scanned PDFs, audio, video, and spreadsheets—just like a real office. The agent must open and figure these out, not rely on pre-typed transcripts.
  • Clear, rule-based grading: Instead of asking another AI if the agent did well, ClawMark uses 1,537 small, deterministic “checkers” (Python scripts) that inspect the actual state of the office apps after the agent finishes. If the correct email was sent, or the right cells were updated, the checker passes. This makes grading consistent and fair every time.
  • Two score types:
    • Weighted score: Partial credit is given for completing parts of the task well.
    • Task Success: All-or-nothing—every checker must pass for full success.
  • Red lines: Certain “never do this” rules (like approving a claim before a required report arrives). Breaking a red line counts heavily against you.

Analogy: Imagine a school project that lasts a week. Each day, teachers might change the instructions or add materials. You must notice those changes, use resources like videos and documents, and turn in updated work. The grading is done by a detailed checklist that looks at the final files, not by someone’s opinion.

What they built and tested

  • 100 tasks across 13 job areas (like clinical assistant, legal assistant, insurance, investment analysis, journalism, HR, and more).
  • Each task runs inside the five real, stateful services listed above.
  • They tested 7 advanced AI agent systems using the same setup.
  • Everything (the benchmark, scoring tools, and how tasks are made) is released for others to use and reproduce results.

Main findings and why they matter

  • Good, but not great, scores overall:
    • The top weighted score was about 75.8 out of 100.
    • But strict Task Success (finishing everything perfectly) was much lower: the best model fully completed only 20% of tasks.
    • This means partial progress is common, but end-to-end “done and correct” is still rare.
  • Adapting to change is hard:
    • When the environment first changed between days (like new info arriving), most models’ performance dropped on Day 2.
    • Some recovered a bit by Day 3, but usually not back to Day 1 levels.
  • The two biggest failure types:
    • Missing silent changes: Agents often failed to notice unannounced updates.
    • Not saving work back correctly: Agents reasoned correctly but forgot to commit changes to the right app (like not actually updating the spreadsheet).
  • Safety and compliance:
    • “Red-line” violations (things you must never do) were uncommon overall, but some models broke them more than others.
    • The hardest red lines involved bypassing compliance rules and mishandling confidential data—areas that matter in real jobs.
  • No single model wins everywhere:
    • Different models did best in different job areas. This suggests strengths vary by scenario.

Why this research is important

  • Realistic evaluation: ClawMark pushes beyond one-off, text-only tests. It checks whether AI assistants can be dependable coworkers over time, with changing conditions and many kinds of information.
  • Clear, repeatable grading: Because scoring is done by deterministic rules, results are reproducible and fair.
  • Guidance for building better agents: The biggest problems—missing silent updates and not writing back changes—show what developers need to fix first if they want reliable office helpers.
  • Safer deployments: Red-line checks help catch dangerous behaviors early, which is crucial if these tools are used in real workplaces.

In short, ClawMark gives the AI community a tough, realistic playground to train and measure AI coworkers. It shows that today’s best systems can make progress but still struggle with staying up-to-date and finishing tasks perfectly. The tools and tasks are public, so others can use them to build more reliable, adaptable, and safe AI assistants.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper leaves several areas unexplored; the most salient are:

  • External validity of sandboxed services: Results are obtained on five simulated backends (filesystem, SMTP/IMAP, Notion-like KB, Sheets-like spreadsheet, CalDAV) with deterministic behavior; missing evaluation against live, brittle APIs (auth flows, permission scopes, latency, flaky I/O, rate limits, partial failures, schema drift, and concurrent human edits).
  • Browser and web-workflow coverage: No web/browser tasks, form-filling, authentication handshakes, or dynamic site interactions are evaluated, limiting conclusions about agents that operate across SaaS tools, CRMs, ticketing systems, and intranet portals.
  • Time-horizon realism: Tasks span 2–6 “days”; the benchmark does not test retention and recovery over weeks/months, multi-week projects, or long gaps between turns where memory decay and context migration matter.
  • Single-agent, no human-in-the-loop: The benchmark omits human-agent interaction (clarifications, preference updates, critiques), preference learning, and multi-agent collaboration—key dynamics in coworker settings.
  • Limited concurrency and asynchrony: Environment changes occur only between turns; no within-day asynchronous notifications (e.g., mid-shift emails), simultaneous conflicting updates, or race conditions are simulated.
  • Modality toolchain dependence: Agents must handle raw audio/video/PDFs, but the paper does not standardize or ablate ASR/OCR/vision tool choices; performance may hinge on the specific tool stack bundled with the harness (e.g., Whisper, PyMuPDF), not the model’s reasoning.
  • Natural-language output quality: Rule-based checks target state changes and coarse deliverable correctness; nuanced qualities (tone, clarity, professionalism, legal style, email etiquette) are not measured without LLM-as-judge or human review.
  • Checker weight subjectivity and Goodhart risk: Per-checker weights are author-set; the paper does not justify weight calibration or audit whether agents can “game” rubrics by satisfying minimal state conditions while producing subpar deliverables.
  • Red-line coverage and validation: Only 55 red-line checks across 8 scenarios; selection criteria and weightings are not externally validated (e.g., by compliance officers), and coverage of subtle privacy/ethics issues (e.g., data minimization, insider sharing) is limited.
  • Robustness to adversarial or corrupted inputs: No evaluation of prompt injection via documents/emails, poisoned PDFs, misleading spreadsheets, or malicious attachments; security posture of tool invocation and sandbox isolation is untested.
  • Determinism vs real-world stochasticity: All mutations and checks are deterministic; agents are not evaluated under stochastic or adversarial environment changes, intermittent tool failures, or non-deterministic backend responses.
  • Statistical reliability of results: Each model is run once; run-to-run variance, statistical significance, and sensitivity to API nondeterminism are unreported, limiting confidence in small leaderboard gaps.
  • Confounds in protocol settings: Provider-default parameters differ (e.g., “extended thinking” availability, prompt caching); no controlled ablation ensures apples-to-apples comparison across models.
  • Cost-efficiency comparability: The paper reports tokens and tool calls but not dollar costs or energy/carbon; standardized cost-normalized performance and time-to-completion analyses are missing.
  • Adaptation analysis scope: Turn-level analysis focuses on 3-turn tasks only; the effects of change magnitude/type, density of silent vs loud events, and position of updates within a workflow are not systematically ablated.
  • Failure cause diagnosis for writeback: Backend writeback is the largest failure bucket, but the paper does not instrument root causes (planning vs permission errors vs API misuse vs validation failures) or propose diagnostic benchmarks.
  • Language and locale diversity: Tasks appear predominantly English; no evaluation on multilingual artifacts, mixed scripts, locale-specific formats (dates, currencies), or right-to-left layouts.
  • Scenario balance and coverage: Some domains are thin (e.g., EDA has one task); scenario difficulty and red-line density vary widely, complicating cross-scenario comparability and generalization claims.
  • Memory and state-management strategies: The study does not compare memory architectures (ephemeral vs persistent notes, embeddings, episodic memory) or measure how internal memory use affects cross-day performance.
  • Framework dependency: Although the tool schema is said to be harness-agnostic, all reported results use a single framework (OpenClaw); sensitivity to other scaffolds, tool availability, and orchestration policies is unknown.
  • Hidden-test and overfitting risk: Public release of tasks and deterministic checkers invites benchmark-specific tuning; no hidden test split or rolling-update protocol is described to mitigate overfitting.
  • Cross-service breadth: Only five backends are covered; missing common coworker surfaces (issue trackers, chat platforms like Slack/Teams, code repos, cloud storage, CRM) that require richer orchestration and permission handling.
  • Intra-day scheduling and load: No evaluation of task prioritization under deadlines, interruptions, or multi-task juggling; the benchmark assesses single-chain workflows per day rather than realistic task queues.
  • Deliverable schema brittleness: The extent to which rubric checks rely on strict string/path schemas is unclear; fragile matching could penalize legitimate variations, while permissive checks could miss important errors.
  • Data and artifact realism: Some artifacts are AI-synthesized; the paper does not quantify how synthetic vs real evidence affects difficulty nor validate that synthesized artifacts avoid exploitable regularities.
  • Ethical and societal impacts: Beyond selected red-lines, the benchmark does not probe bias, fairness, or harmful content risks in agent outputs (e.g., discriminatory recommendations, manipulative emails).
  • Reproducibility under provider evolution: API updates (tokenization, reasoning modes, caching) can change behavior; the study lacks version pinning strategies and longitudinal robustness checks.
  • Metric interpretability: Weighted score aggregates tasks with 6–29 checks; despite rationale, comparability across tasks remains imperfect; alternative normalizations or per-capability sub-scores are not explored.
  • Silent/loud annotation consistency: The paper notes borderline cases and author-supplied labels; a formal taxonomy and automated annotation protocol for mutation types is not provided.
  • Generalization to unseen domains: No explicit out-of-distribution tests (new scenarios, tools, or artifact types) to assess how agents extend beyond the 13 curated scenarios.
  • Release transparency: While 700 execution traces are mentioned, it is unclear whether all traces/artifacts are released with sufficient metadata for independent error analysis and replication.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are concrete, deployable applications that leverage ClawMark’s dynamic, multimodal, rule-based evaluation and its findings (notably the high fail rates on silent-change detection and backend writeback, and the value of red-line, no-LLM-as-judge scoring).

  • Agent vendor benchmarking and procurement scoring
    • Sectors: cross-industry; IT, software, BPO; enterprises evaluating “AI coworker” tools
    • Tools/products/workflows: RFP-style bake-offs using ClawMark’s harness; per-scenario scorecards; “Agent Fitness Report” summarizing weighted score, Task Success, red-line incidents, and failure taxonomy
    • Assumptions/dependencies: Access to candidate agents via API; mapping the five tool schemas to each vendor’s tool interface; reproducible single-sweep runs
  • Pre-deployment Agent CI/CD gate
    • Sectors: software/product orgs, internal IT; any team shipping agent features to end users
    • Tools/products/workflows: Convert ClawMark checkers into “tests” in CI; block release on red-line failures; regressions tracked by scenario; nightly sweeps in a docker-compose sandbox
    • Assumptions/dependencies: Stable staging environment (filesystem/email/calendar/KB/spreadsheet) mirrored from production schemas; allowance for benchmark wall time in CI; secrets handling
  • Compliance guardrails and red-team exercises
    • Sectors: healthcare, insurance, finance, legal, HR
    • Tools/products/workflows: Map domain policies into red-line checkers (premature-decision, compliance-bypass, data-exfiltration, irreversible-write); run periodic red-team drills; attach “policy-as-code” guardrails to deployment pipelines
    • Assumptions/dependencies: Legal/policy teams codify norms as deterministic checks; representative domain artifacts available; oversight for false positives/negatives
  • Targeted agent retraining and prompt/process fixes for top failures
    • Sectors: cross-industry; any multimodal workflow
    • Tools/products/workflows: Curriculum derived from failure taxonomy (silent-change refresh routines; explicit commit-and-verify steps for backend writeback); prompt patterns that force “state refresh” at each turn; finetuning data built from failed turns
    • Assumptions/dependencies: Ability to modify agent scaffolding/prompts or finetune models; telemetry on tool calls and writebacks; access to ASR/OCR/video parsers for multimodal
  • Tool-adapter QA for enterprise systems
    • Sectors: IT/integration, SaaS vendors; Google Workspace/Microsoft 365 admins
    • Tools/products/workflows: Run adapter-level tests for IMAP/SMTP, CalDAV/Graph, Notion/Confluence, Sheets/Excel; idempotency and reconciliation checks; numeric tolerance and semantic equivalence checks from ClawMark
    • Assumptions/dependencies: Mappings from sandbox services to production APIs; test accounts and mock data; change-management policy for connectors
  • Multimodal document operations benchmarking
    • Sectors: journalism, claims processing, legal discovery, research operations
    • Tools/products/workflows: Evaluate OCR/ASR/video transcription and cross-modal reasoning on raw PDFs, scans, audio, video; swap in alternate pipelines (Whisper, PyMuPDF, ffmpeg, vision models) and compare deliverable correctness
    • Assumptions/dependencies: Rights-cleared artifacts or synthetic equivalents; consistent decoding and pre-processing; GPU capacity for multimodal inference
  • Agent cost/efficiency tuning with tool-call and token telemetry
    • Sectors: platform engineering, FinOps
    • Tools/products/workflows: Track score per 1k tool calls/tokens; cap/tool budgeting; identify wasteful browsing or redundant state reads; prompt-caching policies
    • Assumptions/dependencies: Provider token accounting; stable inference settings; comparable tool schemas across models
  • Domain onboarding playbooks built from scenario rubrics
    • Sectors: insurance adjudication, clinical assistants, investment research, real estate, HR, project management
    • Tools/products/workflows: Turn ClawMark rubrics into role playbooks and SOPs; design deliverable templates (e.g., claim adjudication checklist) tied to deterministic checks; embed silent-change polling steps
    • Assumptions/dependencies: Tailoring to local policy and data schemas; stakeholder review; localization for jurisdictional rules
  • Benchmark-as-a-Service for internal stakeholders
    • Sectors: enterprise centers of excellence, model governance
    • Tools/products/workflows: Self-serve portal to submit agents/builds to ClawMark; versioned results, trajectory diffs, and red-line incident reports; model selection dashboard
    • Assumptions/dependencies: DevOps ownership of the harness; quota on model API usage; privacy review if using internal artifacts
  • Academic reproducibility and coursework labs
    • Sectors: academia and training programs
    • Tools/products/workflows: Course labs on non-stationary environments; assignments on designing deterministic checkers and multi-turn tasks; replication studies with turned-off LLM-as-judge
    • Assumptions/dependencies: Teaching infrastructure with containers; institutional GPU/CPU quotas; open-source models with tool use
  • Personal/daily-life assistant pre-check
    • Sectors: consumer; prosumers; SMBs
    • Tools/products/workflows: Run a personal assistant in the sandbox to vet calendar/email behaviors and writeback safety before granting live access; “dry-run” of recurring tasks over simulated days
    • Assumptions/dependencies: Basic technical proficiency to run the harness; mapping from consumer services to sandbox (e.g., export/import calendar)
  • Policy pilots for public-sector procurement
    • Sectors: government, regulated industries’ oversight bodies
    • Tools/products/workflows: Use rule-based, deterministic scoring to set minimal acceptance thresholds; require no-LLM-as-judge evaluation artifacts in submissions
    • Assumptions/dependencies: Clear authority to set standards; publicly reviewable tasks and artifacts; avoiding sensitive data in evaluations

Long-Term Applications

The following applications build on ClawMark’s design principles (multi-day timelines, dynamic environments, full multimodal evidence, deterministic checks) but require further research, scaling, integration, or standardization.

  • Certification standard for coworker agents
    • Sectors: cross-industry; certifying bodies; insurers of AI systems
    • Tools/products/workflows: Third-party certification based on scenario coverage, red-line incident rate, turn-level recovery after exogenous updates; periodic re-certification
    • Assumptions/dependencies: Industry consensus on scenarios/thresholds; auditability of runs; evolving corpus maintenance
  • Live “shadow mode” with safety-gated writebacks
    • Sectors: enterprise IT, compliance-heavy verticals
    • Tools/products/workflows: Read-only live connections + sandboxed mirrors; escalate to human review or two-phase commit for writebacks; automatic rollback if a checker fails post-commit
    • Assumptions/dependencies: Robust data mirroring; transaction logs and reconciliation; low-latency checkers; organizational tolerance for staged deployment
  • Agent architectures for non-stationary environments
    • Sectors: agent R&D, platform teams
    • Tools/products/workflows: Event subscription APIs, periodic state-diff policies, and “refresh mental model” routines; memory that invalidates stale beliefs after exogenous changes
    • Assumptions/dependencies: New benchmarks for continual adaptation; tool vendors exposing change streams; model support for structured state refresh
  • Transaction-safe backend writeback frameworks
    • Sectors: software, finance ops, HRIS, CRM
    • Tools/products/workflows: Agent-side transaction abstraction (pre-commit validation, post-commit verification), idempotent retries, and reconciliation checkers; “Writeback Verifier” library
    • Assumptions/dependencies: Service-side consistency guarantees; fine-grained permissions; audit logging
  • Policy-as-code DSL for red-lines and compliance
    • Sectors: legal, healthcare, insurance, public sector
    • Tools/products/workflows: A domain-specific language to express red-lines and convert them into deterministic checkers; compliance profiles per jurisdiction/role
    • Assumptions/dependencies: Formalization of policy rules; governance over updates; legal validation
  • RL and self-play training from deterministic checkers
    • Sectors: ML research labs, product teams
    • Tools/products/workflows: Optimize agents with RL using checker rewards; train curriculum on tasks that introduce exogenous changes; trajectory-level credit assignment
    • Assumptions/dependencies: Stable, fast simulation; sample efficiency; safe exploration in multimodal tasks
  • Interoperable tool schemas and agent interface standards
    • Sectors: SaaS ecosystem, open standards bodies
    • Tools/products/workflows: Standardized schemas for email/calendar/KB/spreadsheets/tools; “Agent Tool Adapter” conformance tests built from ClawMark patterns
    • Assumptions/dependencies: Vendor buy-in; reference adapters; versioning and deprecation policy
  • Domain-grade extensions and certification (e.g., EHR-grade clinical assistants)
    • Sectors: healthcare, insurance, legal, investment analysis
    • Tools/products/workflows: Link to EHR/claims/edgar-like corpora; HIPAA/GDPR privacy audits; scenario-specific red-lines (e.g., no coding/billing without documentation)
    • Assumptions/dependencies: Data-sharing agreements; de-identification; regulator engagement
  • Multi-agent coworker orchestration across roles
    • Sectors: project management, media production, research labs
    • Tools/products/workflows: Role-specialized agents with handoffs; checker suites for cross-role consistency and deliverable completeness; day-by-day milestone tracking
    • Assumptions/dependencies: Coordination protocols; conflict resolution; provenance tracking
  • Autonomous “silent-change sentinels”
    • Sectors: operations, SRE, RevOps
    • Tools/products/workflows: Background agents that monitor mail/calendar/KB/spreadsheets for exogenous changes and alert primary agents; diff-based triggers for replanning
    • Assumptions/dependencies: Eventing or polling access; cost controls; noise filtering
  • Education and workforce upskilling on agentic work
    • Sectors: universities, corporate L&D
    • Tools/products/workflows: Capstones on designing deterministic rubrics, safe writebacks, and multimodal evidence handling; simulated internships based on multi-day tasks
    • Assumptions/dependencies: Partnerships with industry; sustainable compute budgets; curated case libraries
  • Regulatory frameworks for dynamic-environment reliability
    • Sectors: policy, standards, consumer protection
    • Tools/products/workflows: Guidance that mandates evaluation under exogenous updates and multi-day timelines; incident reporting keyed to red-line categories; minimum bar for Task Success in safety-critical settings
    • Assumptions/dependencies: Stakeholder consensus; public test suites; governance of benchmark updates
  • Privacy-preserving, on-device coworker agents validated offline
    • Sectors: consumer devices, edge computing, defense
    • Tools/products/workflows: Run agents locally against synthetic-but-realistic tasks to verify behaviors before limited live access; constrained multimodal pipelines
    • Assumptions/dependencies: Sufficient on-device compute; local toolchain for OCR/ASR; secure artifact handling
  • Benchmark-driven procurement marketplaces
    • Sectors: enterprise software marketplaces, MSPs
    • Tools/products/workflows: Listings that include ClawMark scenario scores, red-line incident rates, and efficiency metrics; filters by domain readiness (e.g., “Insurance 90+”)
    • Assumptions/dependencies: Trust in benchmark governance; continuous retesting; vendor consent to disclose

Notes on cross-cutting feasibility

  • Data and privacy: Replace real data with synthetic/curated artifacts during evaluation; ensure de-identification and secure sandboxing.
  • Integration: Mapping the benchmark’s five services to live enterprise systems (e.g., Exchange/Graph API, Google Workspace, Confluence/Notion, Sheets/Excel) requires adapter engineering.
  • Compute/cost: Multi-day, multimodal runs are compute- and token-intensive; budget for provider variability and caching.
  • Governance: Deterministic, rule-based scoring simplifies audits but requires continuous maintenance of rubrics, artifacts, and checker code to track evolving policies and tools.
View Paper Prompt  View All Prompts 

Glossary

  • backend writeback: Committing changes to external backends (e.g., email, calendar, KB, spreadsheet) so that the system state reflects the agent’s intended actions. "Backend writeback is also the single largest absolute failure bucket, contributing 567 failures, or 16.7\% of all failures."
  • bit-identical: Exactly identical at the bit level across runs, ensuring determinism and reproducibility. "two independent re-runs produce bit-identical checker verdicts and diagnostic messages"
  • CalDAV: An Internet standard (Calendaring Extensions to WebDAV) for accessing calendar data on a server. "a Radicale CalDAV server"
  • compliance-bypass: A class of prohibited behavior where compliance rules are intentionally circumvented. "compliance-bypass, data-exfiltration, and irreversible-write"
  • compliance guardrail: A rubric or checker designed to prevent should-not-do actions in sensitive workflows. "Compliance guardrail (should-not-do)"
  • coworker agents: Language-model agents that persist across days, assisting users across evolving tasks and environments. "ClawMark, a benchmark for coworker agents built around multi-turn multi-day tasks"
  • cross-modal contradiction: A designed discrepancy where evidence in different modalities conflicts, used to test reasoning robustness. "every cross-modal contradiction spans at least two modalities"
  • cross-source consistency: Ensuring information agrees across multiple data sources (e.g., email vs. spreadsheet). "Cross-source consistency"
  • data-exfiltration: Unauthorized or inappropriate transfer of data outside approved boundaries. "compliance-bypass, data-exfiltration, and irreversible-write"
  • deterministic Python checkers: Fixed, rule-based test functions that examine system state without randomness to score tasks. "scored by 1{,}537 deterministic Python checkers"
  • docker-compose: A tool to define and run multi-container Docker applications in isolation. "inside an isolated docker-compose group"
  • Docker-mounted filesystem: A filesystem volume mounted into a Docker container to provide persistent, inspectable state. "Docker-mounted filesystem, GreenMail for SMTP/IMAP, a Notion-compatible knowledge base, a Google-Sheets-compatible spreadsheet, and a Radicale CalDAV server"
  • evaluation harness: The software scaffold that runs tasks and scoring procedures in a standardized, reproducible way. "We release the benchmark, evaluation harness, and construction pipeline"
  • exogenous between-turn environment changes: State mutations that occur independently of the agent’s actions between turns/days. "exogenous between-turn environment changes"
  • Google-Sheets-compatible spreadsheet: A spreadsheet service that emulates Google Sheets behavior for tool-based interactions. "a Google-Sheets-compatible spreadsheet"
  • GreenMail: A test-friendly email server implementing SMTP/IMAP protocols for reproducible email interactions. "GreenMail for SMTP/IMAP"
  • harness-agnostic: Designed to work across different agent frameworks without changing the tool interface. "the tool schema for our five services is harness-agnostic by design"
  • inject/stage{N} directory: A per-turn injection folder used to add files/events to the workspace as the environment evolves. "An inject/stage{N\/} directory (legacy field name; one entry per turn) drops new files into the workspace"
  • loud events: Announced, turn-entry environment changes the agent is explicitly told about. "announced events (which we call loud events)"
  • LLM-as-judge: An evaluation approach where a LLM subjectively scores agent outputs, as opposed to rule-based checks. "no LLM-as-judge is invoked during scoring"
  • Notion-compatible knowledge base: A knowledge-base service with Notion-like interfaces used for structured data storage and queries. "a Notion-compatible knowledge base"
  • numeric-tolerance checks: Scoring checks that allow small numeric deviations within predefined bounds. "numeric-tolerance or semantic-equivalence checks."
  • post-turn service state: The full state of all services after a turn completes, used for deterministic scoring. "inspects post-turn service state"
  • prompt caching: Provider-side caching of prompts/results to reduce latency or cost on repeated calls. "prompt caching enabled where providers offer it."
  • Radicale CalDAV server: An open-source server implementing the CalDAV standard for calendar data. "a Radicale CalDAV server"
  • red-line constraints: High-weight, must-not-violate checks capturing compliance-sensitive prohibitions. "A subset of 55 checkers (3.6\%) are designated red-line constraints"
  • release gate: A final acceptance step that a task must pass (determinism, audits, reviews) before inclusion in the corpus. "Phase 4: Release gate."
  • semantic-equivalence checks: Scoring that accepts outputs with the same meaning even if text differs. "numeric-tolerance or semantic-equivalence checks."
  • silent mutations: Unannounced environment changes that require the agent to refresh state without prompts. "as silent mutations that appear in the services without notification"
  • stateful sandboxed services: Isolated, persistent services (email, calendar, KB, etc.) whose mutable state agents must manage. "executed against five stateful sandboxed services (filesystem, email, calendar, knowledge base, spreadsheet)"
  • Task Success: A strict, all-or-nothing metric that requires every checker for a task to pass. "the best strict Task Success is only 20.0\%"
  • tool schema: The formal interface specification for tools that agents can call to interact with services. "the tool schema for our five services is harness-agnostic by design"
  • trajectory-aware scoring: Evaluation that accounts for the full sequence of agent actions rather than just final outputs. "Claw-Eval adds trajectory-aware scoring"
  • trajectory review: A review process that examines complete end-to-end runs to detect design flaws or brittleness. "Every task alternates between task review and trajectory review."
  • wake-up message: A per-turn prompt that signals the start of a new in-universe working day. "delivered as a wake-up message"
  • weight-normalised pass rate: The weighted fraction of rubric checks passed, used as the primary continuous score. "the task score is their weight-normalised pass rate on [0,1][0, 1]."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 91 likes about this paper.

Sign Up for Free