---
title: 'HarnessFix: Trace-Guided Agent Repair'
url: https://www.emergentmind.com/topics/harnessfix
type: topic
---

# HarnessFix: Trace-Guided Agent Repair

HarnessFix is a trace-guided framework for diagnosing agent failures and repairing agent harnesses. It is grounded in the view that an executable agent is not reducible to a base model alone, but is instead a model–harness system in which the harness is “the system layer that conditions model calls and turns model outputs into actions in an external workspace” [2606.06324]. In the surrounding 2026 literature, this view aligns with the decomposition
$$
\text{Agent} = \text{Model} + \text{Harness},
$$
and with the argument that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone [2605.27922].

## 1. Definition and conceptual scope

HarnessFix, in the narrow sense, denotes the framework introduced in “From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws” [2606.06324]. Its purpose is to take failed LLM-agent runs, identify where the responsible evidence lies in the trajectories and which harness layer causes the unreliable behavior, and then generate bounded, regression-aware harness patches. In a broader design reading, the term also denotes a methodology for treating the harness as a first-class object of measurement, diagnosis, and controlled improvement across executable agent systems [2605.27922].

Within this literature, the harness includes prompt templates and action formats; context construction and conversation/session management; tool invocation and tool interface policy; workspace access; permissions and security boundaries; budget control; tracing and logging; recovery and retry behavior; and state management and continuation logic [2605.27922]. This places HarnessFix downstream of prompt engineering and upstream of model retraining: it operates on the execution layer that mediates how a fixed model observes, acts, verifies, and recovers.

A central implication is that many observed failures are harness failures rather than pure reasoning failures. This suggests that reliable improvement often requires modifying the runtime substrate—tools, validators, state handling, orchestration, and observability—rather than only modifying prompts or model parameters [2606.06324].

## 2. Harness as a layered runtime substrate

HarnessFix assumes that the harness is layered. The seven-layer ETCLOVG taxonomy used by the framework comprises Execution Environment & Sandbox, Tool Interface, Context & Memory, Lifecycle & Orchestration, Observability, Verification & Evaluation, and Governance & Security [2606.06324]. Reliability failures are mapped into these layers rather than treated as undifferentiated “agent mistakes.”

A related formalization in “AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents” identifies eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording [2605.13357]. This broader runtime-substrate view is compatible with HarnessFix: both approaches treat the harness as the engineering locus where latent model capability is converted into auditable behavior.

The layered perspective matters because repair operators are layer-specific. A missing required API argument is a Tool Interface defect; swallowed execution errors are an Observability defect; accepting completion without evidence of state change is a Lifecycle and Verification defect. HarnessFix therefore aims at localized interventions rather than broad, indirect edits [2606.06324].

## 3. HTIR, trace abstraction, and failure attribution

The core technical artifact in HarnessFix is the Harness-aware Trace Intermediate Representation, or HTIR. HTIR compiles raw execution traces and harness code into a unified representation that normalizes fragmented trajectory evidence and captures step-level provenance and control-flow relations [2606.06324].

Execution is represented as a sequence of TraceSteps. Each TraceStep contains a request message and a response message, plus three derived attributes: Role, Execution Status, and Artifact/state effect. HTIR also adds three classes of links. Temporal links preserve execution order. Input provenance links explain how later requests reuse earlier content and harness logic. Control flow links explain why the harness transitioned from one step to another. For each step, HTIR constructs node-local evidence bundles combining provenance, control-flow evidence, and artifact/state effect evidence [2606.06324].

This structure enables the diagnosis pipeline. The diagnosis agent performs symptom localization using the external evaluation result and the last TraceStep’s evidence; evidence backtracking over temporal, provenance, control-flow, and artifact/state links; candidate adjudication over responsible steps; and layer mapping through ETCLOVG responsibility facets [2606.06324]. The output is a diagnosis record containing a failure signal, an attribution result, and a harness diagnosis.

Single diagnoses are then consolidated into flaw records. A flaw record groups repeated diagnosis records by harness layer and common root cause, and becomes the unit that the repair agent acts on. The framework thereby moves from one-off failure inspection to recurring harness-flaw patterns [2606.06324].

## 4. Scoped repair operators and regression-aware patching

HarnessFix does not permit unconstrained rewriting of the harness. It introduces scoped repair operators grouped by layer. Examples include sandbox-boundary tightening and state-diff logging for Execution; tool-schema narrowing, argument validation, tool-menu ranking/retrieval, and transactional tool wrapping for Tool Interface; failure-tail evidence preservation and task-constraint refresh for Context & Memory; loop guarding, retry/timeout bounding, workflow checkpointing, and verification-gated finalization for Lifecycle; structured trace export and API-error & state-delta logging for Observability; effect-evidence completion guarding and stronger finalization checks for Verification; and least-privilege credentialing and out-of-scope action blocking for Governance [2606.06324].

For each flaw record, selected operators are instantiated into a repair specification. The specification contains four field groups: target and scope; edit constraints; required behavior; and validation and risk [2606.06324]. Its validation conditions are expressed as
$$
\Delta D_{\text{target}} \ge \delta_{\min}
$$
for target-flaw reduction and
$$
R_{\text{new}} \le r_{\max}
$$
for regression control, where \(R_{\text{new}}\) is the count of validation tasks that the original harness solved but the patched harness fails [2606.06324].

The AppWorld example illustrates the method. A run called `POST /venmo/payment_requests` without the required `receiver user_email`; API errors were caught silently; and `apis.supervisor.complete_task()` was accepted despite zero new `venmo.PaymentRequest` or `venmo.Notification` records [2606.06324]. HarnessFix attributes these to Tool Interface, Observability, and Lifecycle/Verification. The resulting repair combines argument validation, API-error and state-delta logging, and effect-evidence completion guarding. The patch is accepted only if it reduces the diagnosed flaw on a validation split without introducing unacceptable regressions [2606.06324].

## 5. Empirical performance and recurring failure patterns

On held-out test splits, HarnessFix reports 57 (57.0%) on SWE-Bench Verified, 9 (26.5%) on Terminal-Bench 2.0 Verified, 37 (61.7%) on GAIA, and 38 (42.2%) on AppWorld [2606.06324]. Relative improvements over the initial harnesses are 26.7% on SWE-Bench, 50.0% on Terminal-Bench, 42.3% on GAIA, and 15.2% on AppWorld [2606.06324]. In the reported ablations, prompt-only repair, removal of trace-grounded diagnosis, removal of scoped repair operators, and removal of regression-aware acceptance all underperform the full system [2606.06324].

The broader harness-measurement literature reinforces the same diagnosis. Harness-Bench evaluates 106 sandboxed offline tasks and, across 5,194 execution trajectories, reports substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings [2605.27922]. Its analysis introduces “execution alignment” as “the degree to which a harness preserves correspondence among the agent’s reasoning, the observed workspace state, the actions taken through tools, and the conditions checked by the evaluator” [2605.27922].

Among failed runs in Harness-Bench, the reported failure modes are contract / format failures (36.4%), tool / recovery failures (24.6%), evidence / grounding failures (14.6%), artifact commitment failures (11.1%), and state / continuation failures (9.3%) [2605.27922]. This suggests that HarnessFix’s trace-guided, layer-specific repair logic is addressing a broad empirical class of execution drift rather than a benchmark-specific anomaly.

## 6. Evolutionary, self-supervised, and test-time variants

HarnessFix belongs to a wider family of systems that modify harnesses from execution feedback. “Adaptive Auto-Harness” decomposes the gap to an oracle harness into evolution loss and adaptation loss, and addresses them with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal [2606.01770]. The design target is sustained construction under open-ended task streams rather than one fixed monolithic harness.

“Retrospective Harness Optimization” optimizes the harness using only past trajectories. It selects a diverse coreset of challenging tasks from past trajectories, re-solves them in parallel, diagnoses them with self-validation and self-consistency, generates candidate harness updates, and selects the most effective one by pairwise self-preference [2606.05922]. In one optimization round it improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading [2606.05922].

“Test-Time Harness Evolution” shifts harness optimization into evaluation itself. It maintains a population of candidate harnesses, refines them through agentic proposers that reason over execution traces, and commits one harness for subsequent inputs through a judge that uses execution-derived proxy signals [2607.08124]. Reported gains include 12.0% to 50.0% on BIRD, 30.0% to 38.3% on LiveCodeBench, 20.0% to 35.0% on SWE-bench Verified, 38.0% to 44.0% on DS-1000, and 48.9% to 69.8% mean score on claw-eval [2607.08124].

“Harness-Aware Self-Evolving” unifies solution generation and harness editing inside a single agentic RL process. A single Qwen3-8B model can generate task solutions or edit selected harness components in a multi-turn action space; it matches the text-classification performance of a GPT-OSS-120B model that uses Claude Code as the harness proposer, repairs imperfect evaluation components, and converges to state-of-the-art performance in circle-packing algorithm discovery [2607.03935]. “AutoHarness” sits at the code-synthesis end of the same spectrum: Gemini-2.5-Flash automatically synthesizes a code harness through iterative refinement from environment feedback, preventing all illegal moves in 145 different TextArena games [2603.03329].

## 7. Formalizations, domain transfer, and limitations

Several 2026 works push HarnessFix toward a more general theory. “Harness Engineering as Categorical Architecture” identifies the harness with an Architecture triple
$$
A = (G,\mathrm{Know},\Phi),
$$
where \(G\) is syntactic wiring, \(\mathrm{Know}\) is a knowledge structure of structural properties and certificates, and \(\Phi\) is a deployment map from abstract capability slots to concrete models and tools [2605.12239]. In that formulation, compiler functors preserve certificate types by identity or replay, and LangGraph preserves the same certificates through its shared per-stage execution path [2605.12239]. “Harnesses for Inference-Time Alignment over Execution Trajectories” instead models a harness as
$$
h = (\kappa,\lambda,\psi),
$$
separating task decomposition from guided execution and analyzing over-decomposition, over-pruning, hallucinated execution, and partial harnesses through inference-time trajectory alignment [2605.21516].

The concept is also domain-general. “ReFlect” externalizes error detection and recovery as a deterministic wrapper around the model, is model-agnostic and training-free, and reports task success rates ranging from 41% on gpt-4o-mini to 56% on Claude Sonnet 4.5 across six models, with SWE-bench patch-structural quality rising from 0% to between 82% and 87% [2605.05737]. In fuzzing, “Quality-Assured Fuzz Harness Generation via the Four Principles Framework” defines Logic Correctness, API Protocol Compliance, Security Boundary Respect, and Entry Point Adequacy as source-level harness correctness conditions, and reports that built-in P1/P2 checks intercepted 58 harness-induced crashes that would otherwise have been false positives [2605.21824]. In robotics, “Harness VLA” exposes a frozen VLA as a retryable contact-rich primitive and reports improvements of 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, reaching 58.4% on RoboTwin C2R [2607.08448].

Reported limitations remain substantial. HarnessFix itself is computationally heavy, depends on trace quality, and can mis-attribute intertwined failures [2606.06324]. Harness-Bench is restricted to offline sandboxed workflows and partly relies on LLM-based process assessment [2605.27922]. TTHE identifies proxy reliability as a central challenge for robust unsupervised agent improvement [2607.08124]. Adaptive Auto-Harness leaves branch lifecycle management and router quality as open problems [2606.01770]. Taken together, these limitations suggest that HarnessFix is best understood not as a finished solution, but as a rapidly consolidating research program: treat the harness as a first-class, executable, inspectable, and formally analyzable object, and improve it with the same rigor traditionally reserved for models and tasks.

Source: https://www.emergentmind.com/topics/harnessfix