Harness-Bench: Diagnostic Agent Benchmark

• The paper introduces Harness-Bench as a diagnostic benchmark that isolates execution-layer variation in agent workflows by evaluating model–harness configurations.
• It operationalizes the agent as a combination of a model backend and an execution harness, using 106 sandboxed offline tasks to ensure reproducible evaluation.
• Empirical findings over 5,194 trajectories reveal substantial differences in task completion, process quality, and efficiency, underscoring harness dependence on overall performance.

Harness-Bench is a diagnostic benchmark and evaluation protocol for measuring configuration-level harness effects in realistic agent workflows. It treats an agent as the combination of a model backend and a harness, where the harness is the execution layer that manages context, tools, state, constraints, permissions, tracing, and recovery. Introduced as a suite of 106 sandboxed offline tasks evaluated across multiple harnesses and model backends, it is designed to isolate execution-layer variation under shared task environments, budgets, and evaluators while preserving each harness’s native execution behavior. Across 5,194 execution trajectories, it reports substantial variation in completion, process quality, efficiency, and failure behavior, and argues that agent capability should be attributed to the model–harness configuration rather than to the base model alone (Yao et al., 27 May 2026).

1. Conceptual framing

Harness-Bench starts from the decomposition

$\text{Agent} = \text{Model} + \text{Harness}.$

In this formulation, the model is the LLM backend, while the harness is the system layer that constructs prompts and context windows, exposes and calls tools, manages workspace and state, enforces constraints and permissions, controls budgets and timeouts, handles errors and retries, and manages tracing, logging, and evaluation hooks (Yao et al., 27 May 2026). The benchmark therefore treats execution as a first-class object of study rather than as incidental scaffolding around inference.

This framing is consistent with broader harness-centric work. “From Model Scaling to System Scaling: Scaling the Harness in Agentic AI” defines the harness as “the structured system layer surrounding a foundation model: the tool interface, control loop, context constructor, memory store, skill-routing mechanism, and verification-and-governance layer,” and represents long-horizon performance conceptually as

$$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$

with the harness corresponding to the non-model components (Gu, 25 May 2026). “PTCG-Bench” makes the same point more explicitly by parameterizing the policy as

$a_t \sim \pi_{\theta,\eta}(\cdot \mid o_t, h_t),$

where $\theta$ denotes backbone LLM parameters and $\eta$ denotes harness design; the benchmark objective is written as $J(\theta,\eta)$, making harness choice a formal variable rather than an implementation detail (Hua et al., 28 May 2026).

Harness-Bench operationalizes this perspective for realistic workflows. A run is conceptualized as

$$R = \mathrm{Run}(M, H, E, T), \qquad \mathrm{TaskScore} = \mathrm{Eval}(R; J),$$

where $M$ is the model backend, $H$ the harness configuration, $E$ the sandboxed environment, $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$0 the task, and $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$1 the evaluator (Yao et al., 27 May 2026). The scientific target is therefore not only end-task success, but the measurable contribution of the execution stack to that success.

2. Task suite and benchmark construction

The benchmark contains 106 sandboxed offline tasks “constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity” (Yao et al., 27 May 2026). Tasks are end-to-end workflows rather than isolated tool calls, and they require agents to produce final artifacts in specified locations within controlled local sandboxes.

The task categories are as follows.

Category	Tasks
Software Engineering & Codebase Maintenance	22
Data, BI & Finance Analytics	14
Workspace & Tool Use / Multimodal Operations	15
Knowledge, Evidence & Retrieval	13
Office & Business Communication	12
Vertical Professional Workflows	12
Long-running Autonomy & State Adaptation	11
SRE, DevOps & Release Ops	7

The benchmark’s “sandboxed offline” design means that each task runs in its own isolated file-system sandbox with fixed input fixtures and local tools such as Python and SQLite, without external network calls or drifting live services (Yao et al., 27 May 2026). This choice is central to reproducibility: every run begins from the same initial state, and evaluators can remain stable over time.

Task design emphasizes artifact-producing workflows. The benchmark description includes a multi-layer iterative bug-fixing task that requires repairing a Python program layer by layer using tests and minimal edits; a SQLite migration safety task that requires repairing a migration, ensuring idempotency and data preservation, and generating preflight, rollback, and postcheck scripts; a research-claim evidence audit that requires classifying claims as supported, contradicted, overstated, unsupported, or not reproducible while citing precise locations in offline sources; a document-and-spreadsheet workflow that requires consistent JSON and DOCX outputs; and a customer-support routing task that requires policy-cited routing, priorities, reply templates, and escalation notes (Yao et al., 27 May 2026). These examples illustrate the benchmark’s emphasis on verifiable workspace state rather than answer-only evaluation.

The task inclusion criteria are unusually explicit. “Realism” requires plausibility as a real workflow; “solvability” requires that the task be completable using only the provided sandbox resources; “oracle-checkability” requires deterministic or rubric-backed evaluation of the final workspace state; and “integrity” requires protection against reading hidden answers, modifying protected fixtures, or bypassing evaluation logic (Yao et al., 27 May 2026). This makes Harness-Bench not merely a task collection, but a controlled experimental environment for execution-layer analysis.

3. Harness configurations, model backends, and execution protocol

A harness configuration in Harness-Bench is a black-box runtime around a model backend, comprising prompting and action formats, context-window management, tool interfaces, state tracking, error handling and recovery, budget enforcement, and tracing (Yao et al., 27 May 2026). The main factorial evaluation uses six configurable harnesses—OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, and NanoBot—plus Codex as a model-bound coding-agent reference rather than part of the main configurable matrix (Yao et al., 27 May 2026).

The benchmark pairs those harnesses with eight API model backends: anthropic/claude-opus-4.6, anthropic/claude-sonnet-4.6, google/gemini-3.1-pro-preview, qwen/qwen3.6-plus, z-ai/glm-5.1, moonshot/kimi-k2.5, openai/gpt-5.4, and deepseek/deepseek-v4-flash (Yao et al., 27 May 2026). The resulting design is a full tasks × harnesses × models factorial matrix over the 106 tasks, plus the Codex reference runs.

Fairness is achieved by fixing external conditions while preserving internal harness behavior. For each task, Harness-Bench fixes the task prompt and fixtures, the initial sandbox state, the budget and timeout, and the evaluator. It varies the model backend and the harness configuration. At the same time, each harness retains its native prompting and action format, tool interface and state policy, and retry and recovery behavior; only a minimal required tool set is enabled for each harness (Yao et al., 27 May 2026). This design rejects a common normalization strategy in agent benchmarking—forcing all systems through a single runtime—and instead compares systems under a shared external contract.

Each run records four primary artifacts: the final workspace state, the execution trace, usage statistics, and validator outputs (Yao et al., 27 May 2026). The final workspace state captures the concrete artifacts that determine completion. The execution trace includes model requests and responses, tool calls and outputs, workspace changes, and higher-level session structure. Usage statistics include token counts, turns, and time usage. Validator outputs include deterministic oracle checks and process-level rubric scores. This logging design enables post hoc analysis of completion, efficiency, and process quality from the same trajectory data.

4. Scoring, process quality, and execution-alignment failures

Harness-Bench separates outcome quality from execution quality. For task $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$2, the overall score is

$$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$3

with

$$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$4

Here, $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$5 is a gate that drops to zero if an explicit security or permission violation occurs; $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$6 is task-specific correctness over final outputs; and the three process subscores are normalized to $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$7 (Yao et al., 27 May 2026).

The process dimensions are explicitly defined. Robustness measures how well the agent handles tool or environment failures, including recovery and replanning after errors. ToolUse measures whether tools are chosen and used appropriately, neither ignored when necessary nor misused when irrelevant. Consistency measures alignment between actions and observations, intermediate states and final outputs, and workspace state and user constraints (Yao et al., 27 May 2026). Completion is often determined by deterministic scripts—tests passing, database invariants holding, output schemas matching—but process scoring also uses a fixed LLM judge for rubric-based assessment.

This scoring design is tied to the benchmark’s central notion of execution alignment: whether an agent’s reasoning remains coupled to tool feedback, workspace state, evidence, and verifiable output contracts (Yao et al., 27 May 2026). Harness-Bench identifies recurring execution-alignment failures among failed runs. The reported failure symptoms are contract or format violations (36.4%), tool or recovery failures (24.6%), evidence or grounding issues (14.6%), artifact commitment failures (11.1%), and state or continuation failures (9.3%) (Yao et al., 27 May 2026).

These categories distinguish semantically plausible reasoning from mechanically successful execution. Contract or format violations include malformed JSON, incomplete or misnamed output files, and outputs that cannot be consumed by the oracle. Tool or recovery failures arise when tool errors are ignored or retries are ineffective. Evidence or grounding issues occur when claims are not supported by the sources actually consulted. Artifact commitment failures capture the familiar pattern in which an agent appears to have solved the task but fails to write or preserve the required file. State or continuation failures appear in longer workflows when durable progress is not preserved across rounds (Yao et al., 27 May 2026). The benchmark’s diagnostic power lies precisely in making these mid-trajectory failures measurable.

5. Empirical findings

Across 5,194 execution trajectories, Harness-Bench reports substantial performance differences across harnesses under the same benchmark conditions (Yao et al., 27 May 2026). Aggregated over the eight model backends, the configurable harnesses achieve the following overall scores and completion rates.

Harness	Score	Completion
OpenClaw	52.4%	60.0%
ZeroClaw	61.4%	69.9%
NullClaw	64.4%	75.9%
Moltis	68.8%	78.4%
Hermes	71.2%	80.4%
NanoBot	76.2%	81.6%

All configurable harnesses report 100% security under the benchmark’s explicit violation criteria, so the observed differences are driven by completion and process quality rather than by security-gate failures (Yao et al., 27 May 2026). As a reference point, Codex achieves 80.4% score and 86.5% completion, with an average of 86.1K tokens and 5 turns per run (Yao et al., 27 May 2026).

The configurable harnesses also differ substantially in efficiency. Average token usage ranges from 68.7K for NanoBot to 175.1K for NullClaw, while mean turn counts range from about 5 for OpenClaw to 22.6 for Hermes (Yao et al., 27 May 2026). Harness-Bench therefore does not portray harness effects as purely accuracy-related; they are also differences in computational cost, interaction length, and process structure.

The paper further defines harness dependence as the extent to which a model’s average score varies across harnesses. Stronger models tend to achieve higher mean scores and exhibit lower variance across harnesses, whereas weaker models show larger harness-dependent variation (Yao et al., 27 May 2026). This is one of the benchmark’s central findings: harness effects are not constant across model backends, and a single model can occupy materially different positions in the ranking depending on the runtime around it. That result underwrites the benchmark’s core claim that performance should be reported at the model–harness configuration level.

6. Position in the harness-centric literature and limitations

Harness-Bench belongs to a broader shift from model-centric evaluation to harness-aware evaluation. Domain-specific benchmarks have independently arrived at similar conclusions. In repository-level coding, “Claw-SWE-Bench” fixes task set, prompt, runtime budget, workspace contract, patch extraction, and evaluator, then varies harnesses and models; with the same GLM 5.1 backbone, an OpenClaw bare adapter scores 19.1% Pass@1 while the full adapter reaches 73.4%, and harness choice under a fixed Qwen 3.6-flash model changes Pass@1 by 27.4 percentage points (Zheng et al., 10 Jun 2026). In strategic gameplay, “PTCG-Bench” formalizes harness design as $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$8 in $$\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),$$9 and shows that removing legal-action masking or history drops Glicko-2 rating by 115–118 points, about twice the median adjacent LLM backbone gap of 55 (Hua et al., 28 May 2026). In multi-user governance, “Harness-MU” evaluates the same base models under different infrastructures on Muses-Bench-style scenarios and reports perfect privacy, 1.000, across all access-control attacks together with utility gains of about 0.28–0.39 over a standard baseline (Fan et al., 20 Jun 2026).

Relative to these systems, Harness-Bench is more explicitly benchmark-centric than domain-centric. It is not a coding-only, game-only, or governance-only evaluation, but a general diagnostic benchmark for realistic workflows. At the same time, it inherits several limitations. The benchmark focuses on offline sandboxed workflows rather than live services or evolving websites; its 106 tasks cover many patterns but do not exhaust agent-use domains; it evaluates complete harness configurations rather than isolating every internal mechanism; and part of its process scoring depends on a fixed LLM judge rather than purely deterministic scripts (Yao et al., 27 May 2026). A plausible implication, consistent with the broader system-scaling agenda, is that future harness-aware benchmarks will extend this design toward longitudinal evaluation, memory hygiene, context efficiency, verification cost, and safe evolution over time (Gu, 25 May 2026).

Harness-Bench’s lasting significance lies in its change of unit of analysis. Instead of asking whether a model is strong in the abstract, it asks how a concrete execution stack behaves under controlled workloads. That shift—already visible in adjacent work on coding adapters, game harness ablations, and multi-principal governance—makes the harness an experimentally legible object and establishes configuration-level evaluation as a necessary condition for rigorous agent benchmarking (Yao et al., 27 May 2026).