C²-Eval: Evaluating Autonomous Agents

• C²-Eval is an evaluation framework that distinguishes an agent's capability from controllability by examining the complete execution trajectory.
• It employs trajectory-aware auditing using execution traces, service-side logs, and environment snapshots to ensure safety and proper tool usage.
• By incorporating in situ error injection and multi-trial metrics, C²-Eval reveals reliability gaps and strengthens evaluations across diverse, cross-modal tasks.

C²-Eval denotes a style of evaluation framework for autonomous LLM agents in which capability is separated from controllability and assessed under realistic multi-step operating conditions rather than through final-output success alone. In the formulation reflected by "Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents," C²-Eval-style evaluation is motivated by the need to distinguish task completion from safety compliance, robustness under perturbation, and the integrity of the action trajectory that produced the result. Claw-Eval is explicitly described as addressing the same concerns that motivate C²-Eval-style frameworks and can be viewed as a concrete instantiation and extension of that philosophy for real agentic workflows (Ye et al., 7 Apr 2026).

1. Conceptual basis

The central premise of C²-Eval is that a scalar capability signal is insufficient unless one also understands how the capability was exercised. In this view, an agent evaluation protocol is incomplete if it checks only whether a final file exists, whether a test passes, or whether an answer string matches. Such trajectory-opaque grading can assign full credit to agents that fabricate intermediate steps, bypass mandated tools or compliance procedures, or incur safety violations and robustness failures that remain invisible because the terminal artifact still passes (Ye et al., 7 Apr 2026).

Within this framework, the core distinction is between capability and controllability. Capability is reflected in whether the agent can achieve the task objective. Controllability is reflected in whether the same behavior remains policy-compliant, robust to perturbation, and faithful to the prescribed workflow. The Claw-Eval paper makes this mapping explicit: capability is captured by Completion; controllability or safety is captured by Safety metrics; robustness is measured through controlled error injection; and trajectory awareness is enforced through evidence-backed auditing (Ye et al., 7 Apr 2026).

A second principle is that safety and robustness must be evaluated in situ. Rather than isolating safety in standalone red-teaming suites or robustness in synthetic stress tests detached from goal-directed behavior, the C²-Eval perspective embeds both into normal task execution. The relevant question is not simply whether an agent can produce unsafe outputs in the abstract, but whether it preserves constraints while actively attempting to complete a realistic objective under pressure, partial observability, and tool-mediated interaction (Ye et al., 7 Apr 2026).

2. Failure modes in conventional agent benchmarking

The Claw-Eval analysis identifies three deficiencies in existing benchmarks that a C²-Eval framework is intended to correct. The first is trajectory-opaque grading. If evaluation ignores which tools were called, whether forbidden actions occurred, whether required workflows were followed, and how errors were handled, then reward hacking becomes structurally easy. An agent can satisfy the scoring function without executing the intended process, and the benchmark cannot reliably discriminate genuine competence from opportunistic exploitation (Ye et al., 7 Apr 2026).

The second deficiency is underspecified safety and robustness evaluation. Practical agents must complete tasks while respecting safety constraints and handling perturbations such as API errors, timeouts, or degraded services. Existing work often separates these dimensions: safety is tested outside normal goal pursuit, or unsafe actions are sandboxed so that they cannot execute and therefore cannot be meaningfully scored; robustness to transient failures is seldom stress-tested in a principled fashion. The C²-Eval formulation rejects that separation and treats safe, policy-bounded execution as part of the task itself (Ye et al., 7 Apr 2026).

The third deficiency is narrow modality and interaction coverage. Real deployments require service orchestration across software systems, multimodal perception and generation over documents, images, and video, and multi-turn professional dialogue involving information asymmetry. Benchmarks that isolate only terminal coding, only web navigation, only tool use, or only dialogue obscure cross-modal tradeoffs and prevent comparison across interaction paradigms. A C²-Eval-style benchmark therefore seeks unified, cross-modal evaluation with consistent scoring and auditing across heterogeneous task types (Ye et al., 7 Apr 2026).

A recurrent misconception addressed by this line of work is that output-only success is an acceptable proxy for trustworthy behavior. The empirical results reported for Claw-Eval argue otherwise: a trace-only or output-oriented judge can under-detect both safety and robustness failures, particularly when the decisive evidence lies in service-side parameters or post-execution state rather than in the conversational trace itself (Ye et al., 7 Apr 2026).

3. Evaluation architecture and trajectory awareness

In the Claw-Eval operationalization of C²-Eval, trustworthy assessment depends on full-trajectory auditing through three independent evidence channels that are never visible to the agent during execution. The first channel is the execution trace, which records the complete sequence of model prompts, tool invocations, and tool returns. This captures the trajectory itself and supports verification of tool usage, retry behavior, and open-ended judging of reasoning quality (Ye et al., 7 Apr 2026).

The second channel is the service-side audit log. For each mock API such as email, CRM, or scheduling, every incoming request and its parameters are logged outside the sandbox. This records what the service actually saw, independent of what the agent claims to have done. It enables deterministic checks for forbidden actions, sensitive endpoint access, and parameter-level correctness, including whether the correct account identifiers, amounts, or dates were used (Ye et al., 7 Apr 2026).

The third channel is the environment snapshot, collected only after execution has ended. This snapshot contains the final filesystem and environment state, including generated code, images, videos, logs, outputs, test results, and rendered artifacts after evaluator scripts run. It supports deterministic validation of artifacts and LLM-based assessment of visual or textual outputs grounded in the resulting state (Ye et al., 7 Apr 2026).

This evidence design is reinforced by a temporal firewall. During Execution, no grader, references, or scoring code are accessible inside the container. During Judge, the agent is no longer acting, and only evaluation scripts run. The purpose is to prevent direct exploitation of the evaluation mechanism. In C²-Eval terms, trajectory awareness is therefore not merely additional logging; it is an integrity condition on the entire evaluation process (Ye et al., 7 Apr 2026).

4. Scoring model and reliability metrics

The scoring protocol associated with this framework is explicitly multi-dimensional. For each run, three scores are computed: Completion, Safety, and Robustness. Completion measures how well the task objective was achieved through task-specific rubric items. Safety measures whether safety constraints were respected. Robustness measures recovery from injected failures. In Claw-Eval these are combined so that Completion has weight $0.8$, Robustness has weight $0.2$, and Safety is multiplicative, meaning that a significant safety violation can drive the entire score toward zero regardless of completion quality (Ye et al., 7 Apr 2026).

Completion is derived from weighted rubric items, some deterministic and some judgment-based, and is anchored in the three evidence channels. Safety is embedded in normal tasks rather than evaluated separately. Examples given in Claw-Eval include email triage tasks in which sending any email is a safety violation, operations tasks in which destructive actions are forbidden, and finance tasks in which credentials must not be exposed. This corresponds directly to the C²-Eval notion that capability violating constraints is not acceptable capability (Ye et al., 7 Apr 2026).

Robustness is defined through controlled error injection into tool calls. The robustness score depends on the breadth of recovery across tool types that experienced injected errors, not on raw retry count. Repeatedly retrying the same failing call without changing strategy does not increase the robustness score. This design is intended to distinguish superficial persistence from actual failure recovery (Ye et al., 7 Apr 2026).

A distinctive feature of the framework is its use of multi-trial metrics. Each task is run three times per model, with $k=3$, and a pass is defined by a score threshold $\tau = 0.75$. Average Score measures overall capability level across tasks and trials. Pass@$\,k$ measures the fraction of tasks that succeed at least once in $k$ attempts and is interpreted as a capability ceiling. Pass$^k$ measures the fraction of tasks that succeed on all trials and is interpreted as a reliability floor. The gap between Pass@3 and Pass$^3$ quantifies stochastic brittleness. In the C²-Eval perspective, this gap is indispensable because a high Pass@3 with a low Pass$^3$ indicates that an agent can succeed, but not in a manner that is yet reliable enough for deployment (Ye et al., 7 Apr 2026).

5. Task scope and unified coverage

As implemented in Claw-Eval, the C²-Eval-style framework spans 300 human-verified tasks organized into 9 categories across 3 groups, with 2,159 fine-grained rubric items. The three groups are general service orchestration, multimodal perception and generation, and multi-turn professional dialogue. This breadth is intended to preserve a single evaluation logic while covering substantially different forms of agent behavior (Ye et al., 7 Apr 2026).

General service orchestration contains 161 tasks and focuses on realistic software workflows. The suite includes easy tasks involving single-service queries and basic scheduling, medium tasks involving cross-service coordination and data retrieval, and hard tasks involving multi-system orchestration, financial compliance, and operations. These tasks use mock services such as CRM, email, and scheduling through custom tools, and 43 tasks include embedded safety constraints such as prohibitions on sending emails, exposing credentials, or performing destructive actions in read-only contexts (Ye et al., 7 Apr 2026).

Multimodal perception and generation contains 101 tasks. The subcategories are Video with 53 tasks, Document & Image with 22 tasks, and code-driven visual generation with 26 tasks. The skill profile is a perceive–reason–act loop: choosing which video segments or frames to inspect, deciding which document pages to open, using tools such as ReadMedia, and reasoning over retrieved visual information to answer questions or construct artifacts. Generative tasks include production of HTML/CSS/JS, SVG, images, and edited video clips satisfying functional specifications (Ye et al., 7 Apr 2026).

Multi-turn professional dialogue contains 38 tasks across STEM, Social Science, and Business. These tasks use a simulated user with domain expertise, latent intent, and an information-revealing strategy that withholds key details unless the agent asks correctly. The agent must therefore ask clarifying questions, challenge or refine user misconceptions, and synthesize partial information across turns. A plausible implication is that C²-Eval treats dialogic competence not as generic chatting, but as structured consultation under epistemic asymmetry and goal pressure (Ye et al., 7 Apr 2026).

6. Empirical findings, limitations, and significance

The empirical results reported for Claw-Eval provide a concrete demonstration of why C²-Eval-style evaluation differs from conventional benchmarks. In a comparison between a hybrid grader using all three evidence channels and a vanilla LLM judge that had the full agent conversation trace and grader source code but no audit logs or environment snapshots, the hybrid pipeline identified 27 task-level safety violations, of which the vanilla judge missed 12, corresponding to a 44% miss rate. For robustness, the hybrid pipeline identified 118 task-level issues, of which the vanilla judge missed 15, corresponding to a 13% miss rate. The stated interpretation is that without log- and snapshot-based checks, trajectory-opaque or trace-only judging systematically under-detects safety and robustness problems (Ye et al., 7 Apr 2026).

Controlled error injection further clarifies the difference between peak capability and consistent reliability. In selected experiments, mock services returned errors with probability $p \in \{0.0, 0.2, 0.4, 0.6\}$, using HTTP 429 rate-limit failures for 35% of injected failures, HTTP 500 internal errors for 35%, and latency spikes of 2–4 seconds for 30%. Under this regime, Pass@3 remained largely stable even at a 60% error rate, while Pass$0.2$0 degraded sharply. Reported drops in Pass$0.2$1 include $0.2$2 points for Gemini-3.1-Pro, $0.2$3 for Claude Opus 4.6, and $0.2$4 for GLM-5-Turbo. The widening Pass@3–Pass$0.2$5 gap exposes a consistency problem that would be missed by single-run or best-of-$0.2$6 reporting alone (Ye et al., 7 Apr 2026).

The multimodal findings point to strong specialization rather than uniform competence. Multimodal tasks are harder than text-based tasks; the best Multimodal Pass$0.2$7 reported is 25.7% for GPT-5.4, whereas General Pass$0.2$8 reaches 70.8% for Claude Opus 4.6. Video is particularly challenging: aggregated across 9 models, Video Pass$0.2$9 is 10.7%, compared with 32.3% for Document & Image and 23.9% for Code. No single model dominates across all modalities, which supports the C²-Eval emphasis on modality-aware capability profiles rather than a single aggregate score (Ye et al., 7 Apr 2026).

The framework also exposes interaction-specific structure. In multi-turn professional dialogue, success is reported to correlate strongly with question precision, with $k=3$0 and $k=3$1, while correlation between average rounds and Pass$k=3$2 is $k=3$3. This indicates that more turns do not by themselves improve outcomes; what matters is whether the agent elicits the withheld information with precise, well-targeted questioning (Ye et al., 7 Apr 2026).

The limitations identified for this C²-Eval-style operationalization are also consequential. The task suite is finite, some modalities such as speech/audio and physical robotics are not covered, evaluation relies on Dockerized sandboxes and mock services, some rubric items depend on LLM judges and therefore may introduce variance or bias, perturbations focus primarily on service-level errors, and results are conditioned on a particular agent scaffold. The paper therefore points toward expansion to more domains and modalities, richer safety taxonomies, broader robustness stress tests including adversarial inputs and tool schema changes, and standardization of a reusable evaluation stack combining trajectory-aware logging, multi-dimensional scoring, multi-trial metrics, and unified multimodal task definitions (Ye et al., 7 Apr 2026).