HarnessFix: Trace-Guided Agent Repair

• HarnessFix is a trace-guided framework that treats the agent harness as a layered runtime substrate for diagnosing and repairing failures.
• It utilizes the Harness-aware Trace Intermediate Representation (HTIR) to compile execution traces and map specific failures to distinct harness layers.
• Empirical results demonstrate significant performance improvements across benchmarks by applying localized, regression-aware repair operators.

HarnessFix is a trace-guided framework for diagnosing agent failures and repairing agent harnesses. It is grounded in the view that an executable agent is not reducible to a base model alone, but is instead a model–harness system in which the harness is “the system layer that conditions model calls and turns model outputs into actions in an external workspace” (Chen et al., 4 Jun 2026). In the surrounding 2026 literature, this view aligns with the decomposition

$\text{Agent} = \text{Model} + \text{Harness},$

and with the argument that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone (Yao et al., 27 May 2026).

1. Definition and conceptual scope

HarnessFix, in the narrow sense, denotes the framework introduced in “From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws” (Chen et al., 4 Jun 2026). Its purpose is to take failed LLM-agent runs, identify where the responsible evidence lies in the trajectories and which harness layer causes the unreliable behavior, and then generate bounded, regression-aware harness patches. In a broader design reading, the term also denotes a methodology for treating the harness as a first-class object of measurement, diagnosis, and controlled improvement across executable agent systems (Yao et al., 27 May 2026).

Within this literature, the harness includes prompt templates and action formats; context construction and conversation/session management; tool invocation and tool interface policy; workspace access; permissions and security boundaries; budget control; tracing and logging; recovery and retry behavior; and state management and continuation logic (Yao et al., 27 May 2026). This places HarnessFix downstream of prompt engineering and upstream of model retraining: it operates on the execution layer that mediates how a fixed model observes, acts, verifies, and recovers.

A central implication is that many observed failures are harness failures rather than pure reasoning failures. This suggests that reliable improvement often requires modifying the runtime substrate—tools, validators, state handling, orchestration, and observability—rather than only modifying prompts or model parameters (Chen et al., 4 Jun 2026).

2. Harness as a layered runtime substrate

HarnessFix assumes that the harness is layered. The seven-layer ETCLOVG taxonomy used by the framework comprises Execution Environment & Sandbox, Tool Interface, Context & Memory, Lifecycle & Orchestration, Observability, Verification & Evaluation, and Governance & Security (Chen et al., 4 Jun 2026). Reliability failures are mapped into these layers rather than treated as undifferentiated “agent mistakes.”

A related formalization in “AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents” identifies eleven component responsibilities: task specification, context selection, tool access, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording (Zhong et al., 13 May 2026). This broader runtime-substrate view is compatible with HarnessFix: both approaches treat the harness as the engineering locus where latent model capability is converted into auditable behavior.

The layered perspective matters because repair operators are layer-specific. A missing required API argument is a Tool Interface defect; swallowed execution errors are an Observability defect; accepting completion without evidence of state change is a Lifecycle and Verification defect. HarnessFix therefore aims at localized interventions rather than broad, indirect edits (Chen et al., 4 Jun 2026).

3. HTIR, trace abstraction, and failure attribution

The core technical artifact in HarnessFix is the Harness-aware Trace Intermediate Representation, or HTIR. HTIR compiles raw execution traces and harness code into a unified representation that normalizes fragmented trajectory evidence and captures step-level provenance and control-flow relations (Chen et al., 4 Jun 2026).

Execution is represented as a sequence of TraceSteps. Each TraceStep contains a request message and a response message, plus three derived attributes: Role, Execution Status, and Artifact/state effect. HTIR also adds three classes of links. Temporal links preserve execution order. Input provenance links explain how later requests reuse earlier content and harness logic. Control flow links explain why the harness transitioned from one step to another. For each step, HTIR constructs node-local evidence bundles combining provenance, control-flow evidence, and artifact/state effect evidence (Chen et al., 4 Jun 2026).

This structure enables the diagnosis pipeline. The diagnosis agent performs symptom localization using the external evaluation result and the last TraceStep’s evidence; evidence backtracking over temporal, provenance, control-flow, and artifact/state links; candidate adjudication over responsible steps; and layer mapping through ETCLOVG responsibility facets (Chen et al., 4 Jun 2026). The output is a diagnosis record containing a failure signal, an attribution result, and a harness diagnosis.

Single diagnoses are then consolidated into flaw records. A flaw record groups repeated diagnosis records by harness layer and common root cause, and becomes the unit that the repair agent acts on. The framework thereby moves from one-off failure inspection to recurring harness-flaw patterns (Chen et al., 4 Jun 2026).

4. Scoped repair operators and regression-aware patching

HarnessFix does not permit unconstrained rewriting of the harness. It introduces scoped repair operators grouped by layer. Examples include sandbox-boundary tightening and state-diff logging for Execution; tool-schema narrowing, argument validation, tool-menu ranking/retrieval, and transactional tool wrapping for Tool Interface; failure-tail evidence preservation and task-constraint refresh for Context & Memory; loop guarding, retry/timeout bounding, workflow checkpointing, and verification-gated finalization for Lifecycle; structured trace export and API-error & state-delta logging for Observability; effect-evidence completion guarding and stronger finalization checks for Verification; and least-privilege credentialing and out-of-scope action blocking for Governance (Chen et al., 4 Jun 2026).

For each flaw record, selected operators are instantiated into a repair specification. The specification contains four field groups: target and scope; edit constraints; required behavior; and validation and risk (Chen et al., 4 Jun 2026). Its validation conditions are expressed as

$\Delta D_{\text{target}} \ge \delta_{\min}$

for target-flaw reduction and

$R_{\text{new}} \le r_{\max}$

for regression control, where $R_{\text{new}}$ is the count of validation tasks that the original harness solved but the patched harness fails (Chen et al., 4 Jun 2026).

The AppWorld example illustrates the method. A run called POST /venmo/payment_requests without the required receiver user_email; API errors were caught silently; and apis.supervisor.complete_task() was accepted despite zero new venmo.PaymentRequest or venmo.Notification records (Chen et al., 4 Jun 2026). HarnessFix attributes these to Tool Interface, Observability, and Lifecycle/Verification. The resulting repair combines argument validation, API-error and state-delta logging, and effect-evidence completion guarding. The patch is accepted only if it reduces the diagnosed flaw on a validation split without introducing unacceptable regressions (Chen et al., 4 Jun 2026).

5. Empirical performance and recurring failure patterns

On held-out test splits, HarnessFix reports 57 (57.0%) on SWE-Bench Verified, 9 (26.5%) on Terminal-Bench 2.0 Verified, 37 (61.7%) on GAIA, and 38 (42.2%) on AppWorld (Chen et al., 4 Jun 2026). Relative improvements over the initial harnesses are 26.7% on SWE-Bench, 50.0% on Terminal-Bench, 42.3% on GAIA, and 15.2% on AppWorld (Chen et al., 4 Jun 2026). In the reported ablations, prompt-only repair, removal of trace-grounded diagnosis, removal of scoped repair operators, and removal of regression-aware acceptance all underperform the full system (Chen et al., 4 Jun 2026).

The broader harness-measurement literature reinforces the same diagnosis. Harness-Bench evaluates 106 sandboxed offline tasks and, across 5,194 execution trajectories, reports substantial variation in completion, process quality, efficiency, and failure behavior across model-harness pairings (Yao et al., 27 May 2026). Its analysis introduces “execution alignment” as “the degree to which a harness preserves correspondence among the agent’s reasoning, the observed workspace state, the actions taken through tools, and the conditions checked by the evaluator” (Yao et al., 27 May 2026).

Among failed runs in Harness-Bench, the reported failure modes are contract / format failures (36.4%), tool / recovery failures (24.6%), evidence / grounding failures (14.6%), artifact commitment failures (11.1%), and state / continuation failures (9.3%) (Yao et al., 27 May 2026). This suggests that HarnessFix’s trace-guided, layer-specific repair logic is addressing a broad empirical class of execution drift rather than a benchmark-specific anomaly.

6. Evolutionary, self-supervised, and test-time variants

HarnessFix belongs to a wider family of systems that modify harnesses from execution feedback. “Adaptive Auto-Harness” decomposes the gap to an oracle harness into evolution loss and adaptation loss, and addresses them with a stateful multi-agent evolver, a harness tree with solve-time routing, and human-steering hooks for cases where history lacks the needed signal (Liu et al., 1 Jun 2026). The design target is sustained construction under open-ended task streams rather than one fixed monolithic harness.

“Retrospective Harness Optimization” optimizes the harness using only past trajectories. It selects a diverse coreset of challenging tasks from past trajectories, re-solves them in parallel, diagnoses them with self-validation and self-consistency, generates candidate harness updates, and selects the most effective one by pairwise self-preference (Pan et al., 4 Jun 2026). In one optimization round it improves the pass rate on SWE-Bench Pro from 59% to 78% without any external grading (Pan et al., 4 Jun 2026).

“Test-Time Harness Evolution” shifts harness optimization into evaluation itself. It maintains a population of candidate harnesses, refines them through agentic proposers that reason over execution traces, and commits one harness for subsequent inputs through a judge that uses execution-derived proxy signals (Nie et al., 9 Jul 2026). Reported gains include 12.0% to 50.0% on BIRD, 30.0% to 38.3% on LiveCodeBench, 20.0% to 35.0% on SWE-bench Verified, 38.0% to 44.0% on DS-1000, and 48.9% to 69.8% mean score on claw-eval (Nie et al., 9 Jul 2026).

“Harness-Aware Self-Evolving” unifies solution generation and harness editing inside a single agentic RL process. A single Qwen3-8B model can generate task solutions or edit selected harness components in a multi-turn action space; it matches the text-classification performance of a GPT-OSS-120B model that uses Claude Code as the harness proposer, repairs imperfect evaluation components, and converges to state-of-the-art performance in circle-packing algorithm discovery (Luo et al., 4 Jul 2026). “AutoHarness” sits at the code-synthesis end of the same spectrum: Gemini-2.5-Flash automatically synthesizes a code harness through iterative refinement from environment feedback, preventing all illegal moves in 145 different TextArena games (Lou et al., 10 Feb 2026).

7. Formalizations, domain transfer, and limitations

Several 2026 works push HarnessFix toward a more general theory. “Harness Engineering as Categorical Architecture” identifies the harness with an Architecture triple

$A = (G,\mathrm{Know},\Phi),$

where $G$ is syntactic wiring, $\mathrm{Know}$ is a knowledge structure of structural properties and certificates, and $\Phi$ is a deployment map from abstract capability slots to concrete models and tools (Banu, 12 May 2026). In that formulation, compiler functors preserve certificate types by identity or replay, and LangGraph preserves the same certificates through its shared per-stage execution path (Banu, 12 May 2026). “Harnesses for Inference-Time Alignment over Execution Trajectories” instead models a harness as

$h = (\kappa,\lambda,\psi),$

separating task decomposition from guided execution and analyzing over-decomposition, over-pruning, hallucinated execution, and partial harnesses through inference-time trajectory alignment (Wang et al., 15 May 2026).

The concept is also domain-general. “ReFlect” externalizes error detection and recovery as a deterministic wrapper around the model, is model-agnostic and training-free, and reports task success rates ranging from 41% on gpt-4o-mini to 56% on Claude Sonnet 4.5 across six models, with SWE-bench patch-structural quality rising from 0% to between 82% and 87% (Huang, 7 May 2026). In fuzzing, “Quality-Assured Fuzz Harness Generation via the Four Principles Framework” defines Logic Correctness, API Protocol Compliance, Security Boundary Respect, and Entry Point Adequacy as source-level harness correctness conditions, and reports that built-in P1/P2 checks intercepted 58 harness-induced crashes that would otherwise have been false positives (Sheng et al., 20 May 2026). In robotics, “Harness VLA” exposes a frozen VLA as a retryable contact-rich primitive and reports improvements of 38.6 and 25.4 percentage points on LIBERO-Pro and RoboCasa365, respectively, reaching 58.4% on RoboTwin C2R (Zhang et al., 9 Jul 2026).

Reported limitations remain substantial. HarnessFix itself is computationally heavy, depends on trace quality, and can mis-attribute intertwined failures (Chen et al., 4 Jun 2026). Harness-Bench is restricted to offline sandboxed workflows and partly relies on LLM-based process assessment (Yao et al., 27 May 2026). TTHE identifies proxy reliability as a central challenge for robust unsupervised agent improvement (Nie et al., 9 Jul 2026). Adaptive Auto-Harness leaves branch lifecycle management and router quality as open problems (Liu et al., 1 Jun 2026). Taken together, these limitations suggest that HarnessFix is best understood not as a finished solution, but as a rapidly consolidating research program: treat the harness as a first-class, executable, inspectable, and formally analyzable object, and improve it with the same rigor traditionally reserved for models and tasks.