Automatic Harness Evolution

• Automatic Harness Evolution is a framework of iterative, evidence-driven processes that refine harness components such as prompts, code, and configurations in diverse systems.
• It employs structured methods including Bayesian optimization, staged decomposition, and RL-guided search to boost branch coverage, bug detection, and overall system performance.
• Empirical results demonstrate significant gains, with up to 4.26× higher branch coverage and improved accuracy in agent performance, validating its cross-domain applicability.

Searching arXiv for the cited topic and closely related papers to ground the article. Automatic Harness Evolution denotes a family of automated processes that modify, synthesize, or optimize a “harness” around a target system. In current arXiv usage, a harness may be a fuzzing driver that translates byte-level inputs into semantically valid program states (Zhang et al., 8 Jul 2026), the structured execution layer around a base model, including prompts, tools, memory, middleware, runtime state, and the model-tool interaction loop (Zheng et al., 16 Jun 2026), the executable code that wraps an LLM and determines what to store, retrieve, and present to the model (Lee et al., 30 Mar 2026), or a cable-harness layout in a customizable 3D environment (Karlsson et al., 2023). Across these settings, the shared idea is to replace manual, one-shot, or static construction with iterative, evidence-driven procedures that generate, validate, diagnose, and revise harness artifacts.

1. Conceptual scope and mutable surfaces

For agent systems, the formal reduction Agent = Model + Harness makes the optimization target explicit (Seong et al., 22 Apr 2026). In this literature, a harness is not a single prompt but a composite runtime substrate. Representative definitions include prompting, tool/action schemas, tool implementations, runtime control, error handling and orchestration, logging and tracing, scoring and acceptance logic, caching and memory, environment setup, and skill libraries (Lin et al., 28 Apr 2026). HarnessX expresses the same idea as a typed runtime interface comprising prompts, tools, memory, and control flow, organized as typed processors bound to hook points (Chen et al., 12 Jun 2026).

This scope makes Automatic Harness Evolution a broader category than prompt optimization. Some systems search over file-scoped components and middleware edits (Lin et al., 28 Apr 2026), some over mixed Boolean, categorical, and continuous flag spaces (Sengupta et al., 22 Apr 2026), some over executable harness code (Lee et al., 30 Mar 2026), and some over self-authored patches proposed by the same model whose behavior is being scaffolded (Zhang et al., 8 Jun 2026). A plausible implication is that “harness evolution” is best understood as a problem family indexed by the mutable object: code, configuration, typed processors, runtime policies, or task-specific branches.

The literature also separates harness evolution from model-weight tuning. HASE explicitly makes selected harness components part of the editable environment and action space, while Continual Harness evolves prompt, sub-agents, reusable skills, and memory online without resets (Luo et al., 4 Jul 2026, Karten et al., 11 May 2026). EvoTrainer extends the concept to the training side, treating diagnostics, reward shaping, validators, intervention libraries, backtesting, and skill memory as an adaptive training harness that co-evolves with the policy (Chen et al., 2 Jun 2026).

2. Function-oriented fuzzing harnesses

In fuzzing, Automatic Harness Evolution addresses the problem that one-turn LLM harness generation suffers from hallucinations, coarse-grained targeting, and misaligned workflows (Zhang et al., 8 Jul 2026). SynapseFlow replaces monolithic generation with a structured process grounded in a Structural Flow Graph, $G = (V, E)$, and coherent Function Triplets, $FT = (I, P, H)$. The SFG abstracts structure-centric dataflow rather than call edges, and the FT constrains one harness to one Input Stream Function with Process Functions and Helper Functions in the dataflow closure of the input structure. Synthesis is then decomposed into four stages—API documentation, structure snippet stitching, rough code assembly, and fuzz-target transformation—governed by a finite-state machine with staged rollback, $M = (S, \Sigma, \delta, s_0, F)$ (Zhang et al., 8 Jul 2026).

The measured outcome is branch coverage and bug detection rate. With llvm-cov, coverage is defined as $C = |CoveredBranches| / |TotalBranches|$, and relative improvement over a baseline $B$ is $R_C = C_{SynapseFlow} / C_{Baseline}$. On 25 real-world open-source software projects, SynapseFlow achieved $3.07\times$, $1.71\times$, and $4.26\times$ higher branch coverage than OSS-Fuzz-Gen, CKGFuzzer, and PromeFuzz, and $1.77\times$, $FT = (I, P, H)$0, and $FT = (I, P, H)$1 higher bug detection rates; it discovered 7 previously unreported bugs, 5 assigned CVEs (Zhang et al., 8 Jul 2026). The paper’s ablation attributes these gains to both SFG-backed coherent triplet selection and staged decomposition with rollback, with coverage dropping by 29.6% when replacing the SFG with heuristic grouping and by 46.7% when disabling staged decomposition and rollback (Zhang et al., 8 Jul 2026).

HarnessAgent addresses a related scaling problem for arbitrary OSS-Fuzz targets through a tool-augmented, iterative LLM pipeline rather than state-machine-guided decomposition (Yang et al., 3 Dec 2025). Its core elements are a rule-based strategy to identify and minimize compilation errors, a hybrid tool pool for robust symbol retrieval, and an enhanced validation pipeline that detects fake definitions. On 243 target functions from OSS-Fuzz projects, it improves the three-shot success rate by approximately 20% compared to state-of-the-art techniques, reaching 87% for C and 81% for C++; more than 75% of the generated harnesses increase the target function coverage, and the hybrid tool-pool system achieves a response rate of over 90% for source code retrieval, outperforming Fuzz Introspector by more than 30% (Yang et al., 3 Dec 2025). Here, “evolution” is iterative repair under compile/link, runtime, and authenticity constraints.

3. Agent and model harness optimization

A major line of work treats the harness around an LLM agent as the primary optimization object. HARBOR formalizes automated harness optimization as constrained noisy Bayesian optimization over a mixed-variable, cost-heterogeneous configuration space with cold-start-corrected rewards and a posterior chance-constrained safety check (Sengupta et al., 22 Apr 2026). The objective is

$FT = (I, P, H)$2

with a block-additive SAAS surrogate, multi-fidelity cost-aware acquisition, and TuRBO trust regions. In the codex-py case study on Terminal-Bench 2, the baseline scored 15/89, a manual five-flag configuration reached 17/89, and HARBOR returned a two-flag configuration scoring 17/89 in 122 min at concurrency 4, matching the manual peak with fewer flags (Sengupta et al., 22 Apr 2026).

Meta-Harness instead searches over executable harness code through an outer-loop coding-agent proposer that has filesystem access to the source code, scores, and execution traces of all prior candidates (Lee et al., 30 Mar 2026). This preserves far more diagnostic information than score-only or summary-only optimizers. On online text classification, the best discovered harness reaches 48.6% test accuracy, surpassing ACE by 7.7 points while using only 11.4K additional context tokens versus 50.8K for ACE; on retrieval-augmented math reasoning, a single discovered harness improves accuracy on 200 IMO-level problems by 4.7 points on average across five held-out models; on TerminalBench-2, discovered harnesses surpass the best hand-engineered baselines (Lee et al., 30 Mar 2026).

Self-Harness internalizes this search loop: the same base model, under its current harness, performs Weakness Mining, Harness Proposal, and Proposal Validation (Zhang et al., 8 Jun 2026). The harness remains non-parametric, and the evaluator remains fixed. Across three base models on Terminal-Bench-2.0, held-out pass rates increase from 40.5% to 61.9% for MiniMax M2.5, from 23.8% to 38.1% for Qwen3.5-35B-A3B, and from 42.9% to 57.1% for GLM-5 (Zhang et al., 8 Jun 2026). HarnessX generalizes the same ambition into typed harness primitives, a substitution algebra, and AEGIS, a four-role trace-driven evolution engine with deterministic gating; across five benchmarks it yields an average gain of +14.5%, with gains up to +44.0% (Chen et al., 12 Jun 2026).

These systems differ primarily in what they regard as a safe and meaningful search space. HARBOR searches flags, Meta-Harness searches code, Self-Harness searches small auditable self-patches, and HarnessX searches typed builders and processor substitutions. This suggests that the central design choice in Automatic Harness Evolution is not whether to search, but where to place structure: in configuration manifolds, code repositories, typed composition, or self-editing control loops.

4. Observability, attribution, and evaluation protocols

Because harness edits can affect many tasks simultaneously, recent work emphasizes observability and attribution. Agentic Harness Engineering instruments three matched observability pillars—component observability, experience observability, and decision observability—and ties every edit to a falsifiable prediction in a change_manifest.json (Lin et al., 28 Apr 2026). Over ten iterations, this lifted pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, exceeding Codex-CLI at 71.9% and the self-evolving baselines ACE at 68.9% and TF-GRPO at 72.3%; the frozen evolved harness then transferred to SWE-bench-verified with 12% fewer tokens than the seed (Lin et al., 28 Apr 2026). SEAGym makes related concerns explicit at the evaluation level by separating update-validation, held-out ID transfer, OOD transfer, replay diagnostics, and cost records; under its shared protocol, AHE is the only baseline improving validation, ID, and OOD together in the reported main comparison (Zheng et al., 16 Jun 2026).

HarnessFix sharpens attribution further by compiling raw traces and harness code into a Harness-aware Trace Intermediate Representation, $FT = (I, P, H)$3, with temporal, provenance, and control-flow edges, and then mapping responsible steps to ETCLOVG layers: Execution environment and sandbox, Tool interface, Context and memory, Lifecycle and orchestration, Observability, Verification and evaluation, and Governance and security (Chen et al., 4 Jun 2026). Its regression-aware repair loop improves held-out test performance over the initial harnesses by 45 → 57 resolved instances on SWE-Bench Verified, 6 → 9 passed tasks on Terminal-Bench 2.0 Verified, 26 → 37 exact matches on GAIA, and 33 → 38 successful tasks on AppWorld (Chen et al., 4 Jun 2026). Harness Handbook addresses a different but related bottleneck—behavior localization—by synthesizing a behavior-centric document tree and register view, then using Behavior-Guided Progressive Disclosure to map requests to verified code sites. On 30 requests per harness, Handbook-assisted planning improves win rates from 28.3% to 38.3% on Codex and from 26.7% to 45.6% on Terminus-2 while reducing planner tokens by 12.7% and 8.6% (Wang et al., 14 Jul 2026).

A contrasting line of work argues that evaluation protocols themselves have overstated the benefits of harness evolution. “Rethinking the Evaluation of Harness Evolution for Agents” compares automatic harness evolution with parallel sampling and sequential refinement under matched feedback and inference budgets on Terminal-Bench 2.1 and reports that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization (Wang et al., 14 Jul 2026). In the held-out split, the initial harness averages 67.7 pass@1 across Claude Opus 4.6 and GPT-5.4, while the evolved harness reaches 68.3, a marginal +0.6 (Wang et al., 14 Jul 2026). One response is to harden credit assignment: “Self-Evolving Agent Harnesses via Gated Semantic Quality-Diversity” uses validity, activation, and statistical-significance gates and reports sealed-test gains of +9 to +15.5 pp with retention of 86–147% of the training gain (Luo et al., 15 Jul 2026). The disagreement is methodological rather than terminological: both lines treat overfitting, noisy self-feedback, and false credit as central hazards.

5. Online adaptation, co-evolution, and meta-evolution

Several systems push Automatic Harness Evolution beyond offline harness search into online adaptation or coupled policy evolution. AutoHarness synthesizes legality-enforcing code harnesses for TextArena games through iterative code refinement, with Thompson sampling over candidate programs and a legal-action success-rate heuristic (Lou et al., 10 Feb 2026). The resulting harness prevents all illegal moves in 145 different TextArena games, achieving 100% legal action success rate; in the harness-as-policy variant on 16 one-player games, the synthesized code policy attains an average reward of 0.870, compared with 0.844 for GPT-5.2-High and 0.707 for Gemini-2.5-Pro (Lou et al., 10 Feb 2026). HASE goes further by allowing a single Qwen3-8B policy to edit selected guidance and evaluation components, co-evolving its own weights, object-level solutions, and parts of the harness in one RL-driven action space (Luo et al., 4 Jul 2026). It reaches 86.98% ± 0.38 accuracy on the Symptom2Disease split, achieves AER 14.35% and IR 1.70 in alpha factor mining, and repairs imperfect evaluators while converging to a circle-packing score of 2.635983 (Luo et al., 4 Jul 2026).

Continual Harness targets reset-free embodied deployment rather than episodic benchmarks (Karten et al., 11 May 2026). Starting from only a minimal environment interface, it alternates between acting and refining prompt, sub-agents, skills, and memory in place. On Emerald with Gemini 3 Pro, from-scratch Continual Harness reaches 100% of milestones at a $FT = (I, P, H)$4215; on lower-capability models, however, every Continual Harness variant on Flash-Lite falls to 3–13% versus a baseline of 20%, indicating a capability floor (Karten et al., 11 May 2026). Adaptive Auto-Harness applies similar ideas to open-ended task streams and decomposes expected regret into an evolution loss and an adaptation loss,

$FT = (I, P, H)$5

then addresses them with a stateful multi-agent evolver, a harness tree, solve-time routing, and targeted human steering (Liu et al., 1 Jun 2026). On PolyBench it reports 97.9% Coverage, 80.9% Accuracy, and +330% Return; on CTF-Dojo, 50.2% Pass; on FutureX, 49.5% Pass for the multi-agent configuration (Liu et al., 1 Jun 2026).

A separate extension places harness evolution itself inside a higher-level optimization loop. “The Last Harness You’ll Ever Build” defines an inner Harness Evolution Loop over the worker harness $FT = (I, P, H)$6 and an outer Meta-Evolution Loop over the evolution protocol $FT = (I, P, H)$7, explicitly aligning the framework with meta-learning (Seong et al., 22 Apr 2026). EvoTrainer applies the co-evolution idea on the training side rather than the deployment side, revising diagnostics, reward shaping, validators, intervention libraries, and backtests as policy training proceeds; on SWE-9B, EvoTrainer reaches 38.16 BC%, exceeding the human-engineered RL reference at 33.77 under the same data, codebase, and evaluation protocol (Chen et al., 2 Jun 2026). The common pattern is a shift from static harnesses to harnesses that learn from trajectories, and from offline selection to continual or nested adaptation.

6. Physical cable-harness routing and broader outlook

In mechanical and electrical engineering, “Automatic Harness Evolution” denotes something different: the automated exploration and improvement of cable-harness layouts in a 3D routing environment (Karlsson et al., 2023). The cable harness routing problem is formulated as a multi-objective optimization over a discretized graph $FT = (I, P, H)$8, with objectives for length minimization, bundling preference, and spatial preferences or penalties. The paper’s weighted scalarization is

$FT = (I, P, H)$9

subject to flow constraints and edge-activation constraints (Karlsson et al., 2023). The deterministic HRH, SHRH, and $M = (S, \Sigma, \delta, s_0, F)$0-SPHRH procedures generate many topology candidates and approximate the Pareto front, which the paper explicitly interprets as an evolutionary exploration of layouts.

This physical-routing line is semantically distinct from software or agent harness evolution, but it shares two abstractions: explicit objective trade-offs and iterative candidate generation under structural constraints. On three industrial-sized 3D cases, SHRH finds near-optimal solutions in 19 s, 222 s, and 115 min, with duality gaps of 0%–0.36%, 0.01%–3.97%, and 0.01%–11.90%; $M = (S, \Sigma, \delta, s_0, F)$1-SPHRH is faster but yields fewer candidates and slightly worse best objective values (Karlsson et al., 2023). A plausible implication is that the phrase “Automatic Harness Evolution” has already become cross-domain, naming not a single algorithmic school but a recurring engineering move: expose the harness as an explicit optimization object, couple it to measurements that matter, and evolve it under verifiable constraints.

Across the software-facing literature, future directions are correspondingly convergent. SynapseFlow proposes dynamic feedback, cross-project transfer, richer analyses, and broader targets such as C++ (Zhang et al., 8 Jul 2026). AutoHarness highlights formal verification integration and hybrid symbolic-ML validators (Lou et al., 10 Feb 2026). AHE and HarnessX emphasize interaction-aware evolution, multi-objective optimization, and transfer across models and benchmarks (Lin et al., 28 Apr 2026, Chen et al., 12 Jun 2026). The field’s outstanding questions concern attribution, generalization, and governance: which harness changes genuinely transfer, which only absorb search budget or task-specific quirks, and how to keep an evolving harness auditable as it accumulates prompts, tools, memory, runtime policies, and verification logic.