---
title: 'Harness-Bench: Diagnostic Agent Benchmark'
url: https://www.emergentmind.com/topics/harness-bench
type: topic
---

# Harness-Bench: Diagnostic Agent Benchmark

Harness-Bench is a diagnostic benchmark and evaluation protocol for measuring configuration-level harness effects in realistic agent workflows. It treats an agent as the combination of a model backend and a harness, where the harness is the execution layer that manages context, tools, state, constraints, permissions, tracing, and recovery. Introduced as a suite of 106 sandboxed offline tasks evaluated across multiple harnesses and model backends, it is designed to isolate execution-layer variation under shared task environments, budgets, and evaluators while preserving each harness’s native execution behavior. Across 5,194 execution trajectories, it reports substantial variation in completion, process quality, efficiency, and failure behavior, and argues that agent capability should be attributed to the model–harness configuration rather than to the base model alone [2605.27922].

## 1. Conceptual framing

Harness-Bench starts from the decomposition
$$
\text{Agent} = \text{Model} + \text{Harness}.
$$
In this formulation, the model is the LLM backend, while the harness is the system layer that constructs prompts and context windows, exposes and calls tools, manages workspace and state, enforces constraints and permissions, controls budgets and timeouts, handles errors and retries, and manages tracing, logging, and evaluation hooks [2605.27922]. The benchmark therefore treats execution as a first-class object of study rather than as incidental scaffolding around inference.

This framing is consistent with broader harness-centric work. “From Model Scaling to System Scaling: Scaling the Harness in Agentic AI” defines the harness as “the structured system layer surrounding a foundation model: the tool interface, control loop, context constructor, memory store, skill-routing mechanism, and verification-and-governance layer,” and represents long-horizon performance conceptually as
$$
\mathcal{P}_H = \Phi(\mathcal{R}, \mathcal{M}, \mathcal{C}, \mathcal{S}, \mathcal{O}, \mathcal{G}),
$$
with the harness corresponding to the non-model components [2605.26112]. “PTCG-Bench” makes the same point more explicitly by parameterizing the policy as
$$
a_t \sim \pi_{\theta,\eta}(\cdot \mid o_t, h_t),
$$
where $\theta$ denotes backbone LLM parameters and $\eta$ denotes harness design; the benchmark objective is written as $J(\theta,\eta)$, making harness choice a formal variable rather than an implementation detail [2605.29653].

Harness-Bench operationalizes this perspective for realistic workflows. A run is conceptualized as
$$
R = \mathrm{Run}(M, H, E, T), \qquad \mathrm{TaskScore} = \mathrm{Eval}(R; J),
$$
where $M$ is the model backend, $H$ the harness configuration, $E$ the sandboxed environment, $T$ the task, and $J$ the evaluator [2605.27922]. The scientific target is therefore not only end-task success, but the measurable contribution of the execution stack to that success.

## 2. Task suite and benchmark construction

The benchmark contains 106 sandboxed offline tasks “constructed from practical agent-use patterns and manually reviewed for realism, solvability, oracle-checkability, and integrity” [2605.27922]. Tasks are end-to-end workflows rather than isolated tool calls, and they require agents to produce final artifacts in specified locations within controlled local sandboxes.

The task categories are as follows.

| Category | Tasks |
|---|---:|
| Software Engineering & Codebase Maintenance | 22 |
| Data, BI & Finance Analytics | 14 |
| Workspace & Tool Use / Multimodal Operations | 15 |
| Knowledge, Evidence & Retrieval | 13 |
| Office & Business Communication | 12 |
| Vertical Professional Workflows | 12 |
| Long-running Autonomy & State Adaptation | 11 |
| SRE, DevOps & Release Ops | 7 |

The benchmark’s “sandboxed offline” design means that each task runs in its own isolated file-system sandbox with fixed input fixtures and local tools such as Python and SQLite, without external network calls or drifting live services [2605.27922]. This choice is central to reproducibility: every run begins from the same initial state, and evaluators can remain stable over time.

Task design emphasizes artifact-producing workflows. The benchmark description includes a multi-layer iterative bug-fixing task that requires repairing a Python program layer by layer using tests and minimal edits; a SQLite migration safety task that requires repairing a migration, ensuring idempotency and data preservation, and generating preflight, rollback, and postcheck scripts; a research-claim evidence audit that requires classifying claims as supported, contradicted, overstated, unsupported, or not reproducible while citing precise locations in offline sources; a document-and-spreadsheet workflow that requires consistent JSON and DOCX outputs; and a customer-support routing task that requires policy-cited routing, priorities, reply templates, and escalation notes [2605.27922]. These examples illustrate the benchmark’s emphasis on verifiable workspace state rather than answer-only evaluation.

The task inclusion criteria are unusually explicit. “Realism” requires plausibility as a real workflow; “solvability” requires that the task be completable using only the provided sandbox resources; “oracle-checkability” requires deterministic or rubric-backed evaluation of the final workspace state; and “integrity” requires protection against reading hidden answers, modifying protected fixtures, or bypassing evaluation logic [2605.27922]. This makes Harness-Bench not merely a task collection, but a controlled experimental environment for execution-layer analysis.

## 3. Harness configurations, model backends, and execution protocol

A harness configuration in Harness-Bench is a black-box runtime around a model backend, comprising prompting and action formats, context-window management, tool interfaces, state tracking, error handling and recovery, budget enforcement, and tracing [2605.27922]. The main factorial evaluation uses six configurable harnesses—OpenClaw, ZeroClaw, Hermes, Moltis, NullClaw, and NanoBot—plus Codex as a model-bound coding-agent reference rather than part of the main configurable matrix [2605.27922].

The benchmark pairs those harnesses with eight API model backends: anthropic/claude-opus-4.6, anthropic/claude-sonnet-4.6, google/gemini-3.1-pro-preview, qwen/qwen3.6-plus, z-ai/glm-5.1, moonshot/kimi-k2.5, openai/gpt-5.4, and deepseek/deepseek-v4-flash [2605.27922]. The resulting design is a full tasks × harnesses × models factorial matrix over the 106 tasks, plus the Codex reference runs.

Fairness is achieved by fixing external conditions while preserving internal harness behavior. For each task, Harness-Bench fixes the task prompt and fixtures, the initial sandbox state, the budget and timeout, and the evaluator. It varies the model backend and the harness configuration. At the same time, each harness retains its native prompting and action format, tool interface and state policy, and retry and recovery behavior; only a minimal required tool set is enabled for each harness [2605.27922]. This design rejects a common normalization strategy in agent benchmarking—forcing all systems through a single runtime—and instead compares systems under a shared external contract.

Each run records four primary artifacts: the final workspace state, the execution trace, usage statistics, and validator outputs [2605.27922]. The final workspace state captures the concrete artifacts that determine completion. The execution trace includes model requests and responses, tool calls and outputs, workspace changes, and higher-level session structure. Usage statistics include token counts, turns, and time usage. Validator outputs include deterministic oracle checks and process-level rubric scores. This logging design enables post hoc analysis of completion, efficiency, and process quality from the same trajectory data.

## 4. Scoring, process quality, and execution-alignment failures

Harness-Bench separates outcome quality from execution quality. For task $i$, the overall score is
$$
\mathrm{TaskScore}_i = \mathrm{Security}_i \cdot \mathrm{Completion}_i \cdot \mathrm{Process}_i,
$$
with
$$
\mathrm{Process}_i = \frac{\mathrm{Robustness}_i + \mathrm{ToolUse}_i + \mathrm{Consistency}_i}{3}.
$$
Here, $\mathrm{Security}_i \in \{0,1\}$ is a gate that drops to zero if an explicit security or permission violation occurs; $\mathrm{Completion}_i \in [0,1]$ is task-specific correctness over final outputs; and the three process subscores are normalized to $[0,1]$ [2605.27922].

The process dimensions are explicitly defined. Robustness measures how well the agent handles tool or environment failures, including recovery and replanning after errors. ToolUse measures whether tools are chosen and used appropriately, neither ignored when necessary nor misused when irrelevant. Consistency measures alignment between actions and observations, intermediate states and final outputs, and workspace state and user constraints [2605.27922]. Completion is often determined by deterministic scripts—tests passing, database invariants holding, output schemas matching—but process scoring also uses a fixed LLM judge for rubric-based assessment.

This scoring design is tied to the benchmark’s central notion of **execution alignment**: whether an agent’s reasoning remains coupled to tool feedback, workspace state, evidence, and verifiable output contracts [2605.27922]. Harness-Bench identifies recurring execution-alignment failures among failed runs. The reported failure symptoms are contract or format violations (36.4%), tool or recovery failures (24.6%), evidence or grounding issues (14.6%), artifact commitment failures (11.1%), and state or continuation failures (9.3%) [2605.27922].

These categories distinguish semantically plausible reasoning from mechanically successful execution. Contract or format violations include malformed JSON, incomplete or misnamed output files, and outputs that cannot be consumed by the oracle. Tool or recovery failures arise when tool errors are ignored or retries are ineffective. Evidence or grounding issues occur when claims are not supported by the sources actually consulted. Artifact commitment failures capture the familiar pattern in which an agent appears to have solved the task but fails to write or preserve the required file. State or continuation failures appear in longer workflows when durable progress is not preserved across rounds [2605.27922]. The benchmark’s diagnostic power lies precisely in making these mid-trajectory failures measurable.

## 5. Empirical findings

Across 5,194 execution trajectories, Harness-Bench reports substantial performance differences across harnesses under the same benchmark conditions [2605.27922]. Aggregated over the eight model backends, the configurable harnesses achieve the following overall scores and completion rates.

| Harness | Score | Completion |
|---|---:|---:|
| OpenClaw | 52.4% | 60.0% |
| ZeroClaw | 61.4% | 69.9% |
| NullClaw | 64.4% | 75.9% |
| Moltis | 68.8% | 78.4% |
| Hermes | 71.2% | 80.4% |
| NanoBot | 76.2% | 81.6% |

All configurable harnesses report 100% security under the benchmark’s explicit violation criteria, so the observed differences are driven by completion and process quality rather than by security-gate failures [2605.27922]. As a reference point, Codex achieves 80.4% score and 86.5% completion, with an average of 86.1K tokens and 5 turns per run [2605.27922].

The configurable harnesses also differ substantially in efficiency. Average token usage ranges from 68.7K for NanoBot to 175.1K for NullClaw, while mean turn counts range from about 5 for OpenClaw to 22.6 for Hermes [2605.27922]. Harness-Bench therefore does not portray harness effects as purely accuracy-related; they are also differences in computational cost, interaction length, and process structure.

The paper further defines **harness dependence** as the extent to which a model’s average score varies across harnesses. Stronger models tend to achieve higher mean scores and exhibit lower variance across harnesses, whereas weaker models show larger harness-dependent variation [2605.27922]. This is one of the benchmark’s central findings: harness effects are not constant across model backends, and a single model can occupy materially different positions in the ranking depending on the runtime around it. That result underwrites the benchmark’s core claim that performance should be reported at the model–harness configuration level.

## 6. Position in the harness-centric literature and limitations

Harness-Bench belongs to a broader shift from model-centric evaluation to harness-aware evaluation. Domain-specific benchmarks have independently arrived at similar conclusions. In repository-level coding, “Claw-SWE-Bench” fixes task set, prompt, runtime budget, workspace contract, patch extraction, and evaluator, then varies harnesses and models; with the same GLM 5.1 backbone, an OpenClaw bare adapter scores 19.1% Pass@1 while the full adapter reaches 73.4%, and harness choice under a fixed Qwen 3.6-flash model changes Pass@1 by 27.4 percentage points [2606.12344]. In strategic gameplay, “PTCG-Bench” formalizes harness design as $\eta$ in $\pi_{\theta,\eta}$ and shows that removing legal-action masking or history drops Glicko-2 rating by 115–118 points, about twice the median adjacent LLM backbone gap of 55 [2605.29653]. In multi-user governance, “Harness-MU” evaluates the same base models under different infrastructures on Muses-Bench-style scenarios and reports perfect privacy, 1.000, across all access-control attacks together with utility gains of about 0.28–0.39 over a standard baseline [2606.21856].

Relative to these systems, Harness-Bench is more explicitly benchmark-centric than domain-centric. It is not a coding-only, game-only, or governance-only evaluation, but a general diagnostic benchmark for realistic workflows. At the same time, it inherits several limitations. The benchmark focuses on offline sandboxed workflows rather than live services or evolving websites; its 106 tasks cover many patterns but do not exhaust agent-use domains; it evaluates complete harness configurations rather than isolating every internal mechanism; and part of its process scoring depends on a fixed LLM judge rather than purely deterministic scripts [2605.27922]. A plausible implication, consistent with the broader system-scaling agenda, is that future harness-aware benchmarks will extend this design toward longitudinal evaluation, memory hygiene, context efficiency, verification cost, and safe evolution over time [2605.26112].

Harness-Bench’s lasting significance lies in its change of unit of analysis. Instead of asking whether a model is strong in the abstract, it asks how a concrete execution stack behaves under controlled workloads. That shift—already visible in adjacent work on coding adapters, game harness ablations, and multi-principal governance—makes the harness an experimentally legible object and establishes configuration-level evaluation as a necessary condition for rigorous agent benchmarking [2605.27922].

Source: https://www.emergentmind.com/topics/harness-bench