Harness-Aware Evaluation Framework

• Harness-Aware Evaluation Framework is an approach that evaluates agent performance as a combined property of both the model and its execution harness.
• It benchmarks systems by varying components like context handling, tool routing, verification, and state management to isolate the influence of the harness.
• Empirical studies reveal non-monotonic, model-specific performance differences, paving the way for harness optimization and self-improvement.

A harness-aware evaluation framework is an evaluation methodology that treats agent performance as a property of a model–harness configuration rather than of a base model alone. In this literature, the harness is the system layer that conditions model calls and turns model outputs into actions in an external workspace, while also managing context, tools, state, constraints, permissions, tracing, and recovery (Yao et al., 27 May 2026). Closely related work makes the same point in two compact formalisms: $\text{Agent} = \text{Model} + \text{Harness}$ and $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$, both of which recast evaluation away from model-only capability and toward composed system behavior (Yao et al., 27 May 2026, Zhong et al., 13 May 2026).

1. Conceptual scope and object of evaluation

A harness-aware framework begins by redefining what is being measured. Rather than scoring a base model in isolation, it evaluates the runtime substrate that mediates observation, action, memory, verification, and control. Recent work uses “harness” broadly. In realistic agent workflows it includes prompt templates, action formats, context construction, tool routing, workspace access, state handling, permissions, constraints, budget control, tracing, and recovery (Yao et al., 27 May 2026). In algorithm discovery it additionally includes prompt construction, data and memory design, evaluation pipelines, parent selection logic, execution management, parallelization, safety controls, and persistence of search history (Ishibashi et al., 13 May 2026). In autonomous software engineering it is decomposed into task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording (Zhong et al., 13 May 2026).

This reframing is motivated by a recurring empirical and methodological claim: observed agent capability is not stably attributable to the model when the execution layer changes. The most explicit statement is the Binding Constraint Thesis, which holds that for long-horizon tasks evaluated across models with comparable frontier capability, performance variance is often governed more by harness configuration than by model choice (Zhang et al., 7 May 2026). A similar point is made in model-specific form by studies showing that the same base model can behave very differently under different harnesses, and that different model families benefit from different harness edits rather than from a universal scaffold (Zhang et al., 8 Jun 2026).

A further conceptual distinction appears in several frameworks: the harness is not the same thing as the task instruction. In HEAT-24, for example, the raw task instruction always contains the core required action, and file-modification tasks always include the write-marker requirement; harness complexity is the additional planning, scoping, verification, and reporting structure layered around that base instruction (Cho, 26 May 2026). This distinction matters because it prevents task formulation and harness design from being conflated.

2. Benchmark designs and controlled protocols

Harness-aware evaluation has been operationalized through several distinct benchmark styles. One style fixes external task conditions while varying the harness. Harness-Bench follows this design on 106 sandboxed offline tasks across eight workflow categories, using 6 configurable harnesses, 8 API model backends, and 5,194 execution trajectories. All compared systems share the same task prompt, initial sandbox state, budget, timeout, and evaluator, while each harness preserves its native prompting, tool interface, state policy, and recovery behavior (Yao et al., 27 May 2026). This is a configuration-level diagnostic design rather than an attempt to force all systems into a common internal runtime.

A second style varies harness strictness under deterministic workspace verification. HEAT-24 is a synthetic 24-task suite with a fixed 12-file prompt-injected workspace, git reinitialization before each run, deterministic workspace-level verification, and three harness conditions—light, balanced, and strict—that differ only in surrounding process and verification scaffolding. The full-factorial design is 6 models × 3 harnesses × 24 tasks = 432 runs, with $n=24$ per model–harness cell and $k=1$ repeat (Cho, 26 May 2026). This setup is explicitly designed to isolate harness effects from tool noise, because there is no browser, shell, or flaky API layer.

A third style treats the harness as the execution infrastructure of an optimization loop. In Vesper, used for algorithm discovery on Circle Packing, the search object is a repository rather than a single function string. Evolution proceeds over Git branches; each search cycle selects a parent, creates a Git worktree, launches a coding agent, evaluates the candidate, detects hacks, and updates a program database. Comparisons against OpenEvolve are normalized by a fixed 40M-token budget, because agentic mutation and stateless mutation consume very different tokens per candidate (Ishibashi et al., 13 May 2026). This makes token allocation itself a harness variable.

A fourth style studies harnesses jointly with training-time adaptation. The ALFWorld extension in “The Interplay of Harness Design and Post-Training in LLM Agents” treats harness informativeness, tool schema, and task type as separate axes. It defines three harnesses—h-low, h-mid, and h-high—and supports both task shift and tool-environment shift through schema versions v1.0, v1.1, and v2.0 (Kim et al., 24 Jun 2026). This turns harness design into part of the train/eval interface rather than a purely inference-time wrapper.

Taken together, these designs suggest a common protocol template: fix tasks, environments, budgets, and evaluators where possible; vary harnesses explicitly; preserve native harness behavior when configuration-level comparison is the goal; and log enough execution evidence to distinguish outcome changes caused by the harness from those caused by the model.

3. Metrics, traces, and diagnostic structures

A central feature of harness-aware evaluation is that it scores more than final task completion. Harness-Bench makes this explicit with

$$\mathrm{TaskScore}_i = \mathrm{Security}_i \cdot \mathrm{Completion}_i \cdot \mathrm{Process}_i, \qquad \mathrm{Process}_i = \frac{\mathrm{Robustness}_i + \mathrm{ToolUse}_i + \mathrm{Consistency}_i}{3},$$

with $\mathrm{Security}_i \in \{0,1\}$ and all non-binary scores normalized to $[0,1]$ (Yao et al., 27 May 2026). The paper also defines execution alignment as “the degree to which a harness preserves correspondence among the agent's reasoning, the observed workspace state, the actions taken through tools, and the conditions checked by the evaluator,” and reports recurring failure symptoms among failed trajectories such as Contract / format — 36.4%, Tool / recovery — 24.6%, Evidence / grounding — 14.6%, Artifact commitment — 11.1%, and State / continuation — 9.3% (Yao et al., 27 May 2026).

HEAT-24 uses simpler outcome metrics—TSR and VTSR—but supplements them with an automated six-label failure taxonomy: format_violation, wrong_answer, wrong_file, missing_change, unrelated_change, and tests_still_fail (Cho, 26 May 2026). Because verification is deterministic and binary, aggregate pass rates can be linked directly to mechanism-level failure labels rather than to manual adjudication.

Other frameworks enlarge the trace model itself. “AI Harness Engineering” proposes a trace-based protocol that packages each episode into auditable artifacts: action trace, tool trace, context trace, verification trace, failure-attribution log, intervention log, entropy audit, and outcome record, all stored as JSONL traces plus patch and verification report (Zhong et al., 13 May 2026). HarnessFix goes further by compiling heterogeneous traces and harness code into a Harness-aware Trace Intermediate Representation (HTIR). HTIR normalizes TraceStep objects, adds role and execution-status annotations, and links steps through temporal edges, input-provenance links, and control-flow links so that failures can be attributed to responsible steps and then mapped to harness layers (Chen et al., 4 Jun 2026).

Release-oriented work adds a separate aggregation layer. The LLM Readiness Harness computes scenario-weighted readiness using

$R = \sum_i w_i \cdot m_i$

and, when some metrics are missing,

$$R = \frac{\sum_{i \in \mathcal{P}} w_i m_i}{\sum_{i \in \mathcal{P}} w_i},$$

where $\mathcal{P}$ is the set of present metrics (Maiorano, 28 Mar 2026). This makes readiness an operational score rather than a pure benchmark score.

For harnesses that are themselves executable artifacts, correctness can also be evaluated directly before downstream performance is trusted. QuartetFuzz formalizes harness correctness through P1 Logic Correctness, P2 API Protocol Compliance, P3 Security Boundary Respect, and P4 Entry Point Adequacy, with pass/fail acceptance only when all four hold (Sheng et al., 20 May 2026). This source-level notion of correctness is distinct from coverage or crash yield.

4. Empirical interaction between models and harnesses

The strongest empirical finding across the literature is that harness sensitivity is often non-monotone and strongly model-specific. HEAT-24 directly rejects the assumption that weaker models benefit most from strict harnesses while stronger models need less structure. The clearest summary is the per-model VTSR table: Gemini 2.5 Flash scores 95.8% / 58.3% / 66.7% under light / balanced / strict; Qwen3.5-122B-A10B scores 87.5% / 75.0% / 91.7%; GPT-OSS-120B scores 95.8% / 95.8% / 87.5%; Qwen3.5:2B scores 0.0% / 58.3% / 4.2%; LLaMA 3.2 scores 16.7% / 4.2% / 20.8%; and Gemma4:e2B remains at 91.7% across all three harnesses (Cho, 26 May 2026). The paper therefore identifies non-monotonicity as the strongest defensible claim and shows that failure mechanism also shifts with capability: format_violation dominates harness-induced failures in capable models, while wrong_file dominates lower-capability failures.

Harness-Bench reaches a similar conclusion at larger scale. Across the same task suite and model pool, configurable harnesses differ by 23.8 points in aggregate score: NanoBot 76.2 versus OpenClaw 52.4, with substantial differences in tokens and turns as well. The paper also finds that stronger backends tend to have higher mean scores and lower cross-harness variance, whereas weaker or less robust backends show larger variance across harnesses (Yao et al., 27 May 2026). This frames harness dependence as an interaction effect rather than as an additive nuisance term.

The most explicit variance-decomposition result appears in the “Binding Constraint Thesis” paper. In a controlled $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$0 model-by-harness experiment on a 100-task subset of SWE-bench Verified, the average harness variance is reported as 18.48 pp$$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$1, the average model variance as 2.37 pp$$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$2, the ratio as 7.80×, and the number of ranking-reversal pairs as 6 out of 9 (Zhang et al., 7 May 2026). The same paper formalizes performance as $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$3 and defines

$$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$4

This is the clearest argument that model rankings on long-horizon tasks can be unstable under harness substitution.

Training-time interaction produces another version of the same phenomenon. In the ALFWorld extension, more informative harnesses generally improve both zero-shot and post-trained performance, but simply swapping in a better harness after training recovers much less benefit than training under that harness. For Qwen2.5-7B + GRPO, the paper reports a -20.7 gap between training-time h-mid and post-hoc h-mid, and -22.5 between training-time h-high and post-hoc h-high. Under strong tool-schema shift $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$5, Qwen2.5-7B + GRPO trained under h-low collapses to 2.7 overall success, while the corresponding zero-shot base under $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$6-low is 13.5 (Kim et al., 24 Jun 2026). This suggests that the harness is part of the effective training environment, not merely an inference-time decorator.

A common misconception addressed across these results is that “more structure is better.” The HEAT-24 results explicitly reject that folk rule, while the variance-decomposition and post-training studies show that harness value depends on model family, task type, and whether the harness is present during adaptation (Cho, 26 May 2026, Zhang et al., 7 May 2026, Kim et al., 24 Jun 2026).

5. Harness optimization and self-improvement

Once harnesses are treated as first-class evaluation objects, several papers move from measuring them to optimizing them. Self-Harness defines an iterative loop over a fixed model and evolving harness $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$7 with three stages: Weakness Mining, Harness Proposal, and Proposal Validation. Candidate edits are accepted only if they are non-regressive on both held-in and held-out splits and improve at least one of them. Across three base models on Terminal-Bench-2.0, held-out pass rates rise from 40.5% to 61.9% for MiniMax M2.5, 23.8% to 38.1% for Qwen3.5-35B-A3B, and 42.9% to 57.1% for GLM-5 (Zhang et al., 8 Jun 2026). The paper emphasizes that the accepted changes are model-specific rather than generic instruction accretions.

HarnessFix approaches the same problem from trace-guided diagnosis. It compiles raw traces and harness code into HTIR, attributes failures to responsible steps and ETCLOVG layers, consolidates recurring diagnoses into flaw records, maps those flaws to scoped repair operators, and accepts patches only if they reduce target flaws on validation tasks without unacceptable regressions. On held-out tests it improves over the initial harness by 26.7% on SWE-Bench Verified, 50.0% on Terminal-Bench 2.0 Verified, 42.3% on GAIA, and 15.2% on AppWorld (Chen et al., 4 Jun 2026).

A separate line of work asks how to evaluate the optimizer rather than only the optimized agent. “Towards Direct Evaluation of Harness Optimizers via Priority Ranking” introduces priority ranking, a step-level task in which the optimizer ranks the four harness components—prompt, tool, memory, workflow—by expected impact of modifying them next. The benchmark Shor contains 182 human-verified optimization scenarios, and ranking quality correlates with actual multi-step optimization ability with Pearson $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$8, $$C_{\text{system}} = F(C_{\text{model}}, C_{\text{harness}}, C_{\text{environment}}, T)$$9. The paper also reports that priority ranking is 8× cheaper and 17× faster than end-improvement observation (Ong et al., 21 May 2026). This changes the evaluation object from final gain to intermediate decision quality.

At the most ambitious level, “The Last Harness You’ll Ever Build” proposes a two-level automation stack: an inner Harness Evolution Loop that iteratively improves a worker harness $n=24$0, and an outer Meta-Evolution Loop that optimizes the evolution protocol $n=24$1 across tasks (Seong et al., 22 Apr 2026). The paper formalizes this as a meta-learning analogue and proposes convergence speed, final performance, and robustness on held-out tasks as the relevant evaluation quantities. The proposal is algorithmic rather than empirically validated, but it extends the same harness-aware logic upward: not only the worker harness, but also the protocol that edits the harness, becomes an evaluable object.

6. Operational deployment, determinism, and limitations

In deployment-oriented settings, harness-aware evaluation becomes a release-control mechanism rather than a descriptive benchmark. The LLM Readiness Harness turns evaluation into a deployment-decision workflow by combining automated benchmarks, OpenTelemetry observability, scenario-weighted readiness scores, Pareto frontiers, and promptfoo-based CI quality gates. It reports 162/162 valid cells across datasets, scenarios, retrieval depths, seeds, and models in its Azure matrix, and shows that ticket-routing regression gates consistently reject unsafe prompt variants instead of merely reporting offline scores (Maiorano, 28 Mar 2026). Here, policy compliance is a hard gate, not just another dimension in a weighted score.

Safety-critical work pushes the same idea toward deterministic enforcement. CAAF replaces open-loop agent behavior with Fail-Safe Determinism through Recursive Atomic Decomposition, Harness as an Asset implemented via machine-readable registries and a deterministic Unified Assertion Interface (UAI), and Structured Semantic Gradients with State Locking. On the SAE Level 3 autonomous driving paradox benchmark, CAAF-all-GPT-4o-mini achieves 100% paradox detection, while monolithic GPT-4o achieves 0%, even at temperature $n=24$2. In pharmaceutical continuous flow reactor design, the Mono+UAI ablation reaches 95%, while debate and sequential-checking architectures remain at 0% across 80 trials, which the paper interprets as evidence that reliability derives primarily from the deterministic assertion harness rather than from multi-agent orchestration alone (Zhang, 18 Apr 2026).

QuartetFuzz applies harness-aware evaluation to fuzz harness generation. It treats fuzz harnesses as artifacts that must satisfy P1–P4 before downstream fuzzing is trusted, and it validates this stance empirically through both generation and audit. In deployment across 23 open-source projects, QuartetFuzz submits 42 bug reports, of which 29 are fixed or confirmed upstream, including 3 CVEs, with only 2 rejected reports for a 4.8% FP rate. During generation, P1/P2 checks automatically intercept 58 harness-induced crashes that would otherwise have been false positives, and an audit of 586 existing production harnesses identifies 53 violations (Sheng et al., 20 May 2026). This shows that harness-aware evaluation can be operationally useful even when the “agent” is itself a code-generation system.

The literature also converges on substantial limitations. HEAT-24 emphasizes that each capability tier is represented by a single model, so apparent tier effects are provisional and model-specific (Cho, 26 May 2026). Harness-Bench is restricted to controlled, sandboxed offline workflows and reports configuration-level diagnostics rather than mechanism-level causal effects (Yao et al., 27 May 2026). Vesper is centered on one benchmark, Circle Packing $n=24$3, and its hack detection remains comparatively simple (Ishibashi et al., 13 May 2026). The ALFWorld study varies only a narrow harness family and remains within one environment (Kim et al., 24 Jun 2026). “The Last Harness You’ll Ever Build” provides the formal loops but no empirical validation (Seong et al., 22 Apr 2026). Taken together, these limitations suggest that the field has established the necessity of harness-aware evaluation more convincingly than it has established any single universal protocol.

Across these works, the durable synthesis is consistent. Harness-aware evaluation treats the harness as an experimental factor, a diagnostic object, and, increasingly, a deployable asset. It replaces model-only benchmarking with configuration-level measurement; supplements pass/fail with traces, failure taxonomies, and operational telemetry; and, in stronger forms, uses the resulting evidence to block releases, detect unsatisfiable specifications, or automatically repair the harness itself. The underlying methodological change is straightforward but consequential: in executable AI systems, the harness is not background infrastructure. It is part of what is being evaluated.