HarnessForge: Evolving Agent Systems

• HarnessForge is a meta-adaptive framework that defines agent systems as coupled harness–policy pairs, enabling joint evolution of structure and reasoning.
• It decomposes agent systems into planning, action, and memory modules, allowing fault-guided harness tailoring and harness-conditioned policy alignment.
• Empirical results demonstrate up to 12% performance gains over baseline methods across benchmarks like ToolHop, SearchQA, and API-Bank.

Searching arXiv for the specified paper and closely related harness literature to ground the article. HarnessForge is a meta-adaptive framework for evolving LLM agent systems by treating the system as a coupled harness–policy pair rather than as a fixed model wrapped by a static runtime. In this formulation, the harness specifies execution structure—planning, action, and memory—while the policy specifies the reasoning behavior executed within that structure. HarnessForge makes the adaptation space between these two layers explicit, then performs harness–policy co-evolution through fault-guided harness tailoring and harness-conditioned policy alignment. Across five benchmarks spanning tool use, retrieval-heavy multi-hop question answering, REST-style environments, and tool-augmented APIs, it improves both Qwen3-4B and Qwen3-8B backbones, outperforming harness-only and policy-only baselines and showing that executable compatibility between the external harness and internal reasoner is a central determinant of agent-system performance (Chen et al., 1 Jun 2026).

1. Conceptual definition and adaptation space

HarnessForge defines an agent system as

$$\mathcal{G} = (\mathcal{H}, \mathcal{R}_\delta), \quad \mathcal{H} = (\mathcal{P}, \mathcal{A}, \mathcal{M}), \quad \mathcal{R}_\delta = \mathcal{R}_{\theta_0 + \delta}.$$

Here, $\mathcal{H}$ is the external execution harness, factored into planning $\mathcal{P}$, action $\mathcal{A}$, and memory $\mathcal{M}$, while $\mathcal{R}_\delta$ is the LLM-based reasoning policy implemented as a LoRA adapter $\delta$ on top of a frozen base model $\mathcal{R}_{\theta_0}$ (Chen et al., 1 Jun 2026).

This decomposition is motivated by heterogeneous task regimes that demand distinct execution paradigms. The framework is explicitly positioned against approaches that adapt only the external harness or only the internal policy. Harness-only methods search over workflows, tool routing, and memory organization while leaving the executor fixed; policy-only methods train the reasoner under a fixed interface. HarnessForge instead isolates a stable adaptation space between structure and execution behavior, then optimizes both sides jointly (Chen et al., 1 Jun 2026).

The central notion is executable compatibility. A harness may expose planning stages, memory slots, or strict tool schemas, but these structural affordances are only useful if the policy can reliably operate under them. Conversely, a stronger policy can remain bottlenecked by a poor interface. This suggests that the effective unit of adaptation is not the model or harness independently, but the pair $(\mathcal{H}, \mathcal{R}_\delta)$. A plausible implication is that benchmark gains attributed to “agent architecture” or “post-training” can be mischaracterized when harness–policy compatibility is not controlled explicitly.

The broader harness literature reinforces this framing from different angles. “AI Harness Engineering” formalizes the harness as the runtime substrate governing context, tools, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording (Zhong et al., 13 May 2026). “Harness-Bench” likewise argues that agent capability should be reported at the model–harness configuration level rather than attributed to the base model alone (Yao et al., 27 May 2026). HarnessForge is more specific in making the compatibility between harness and policy itself the object of optimization.

2. Harness representation: planning, action, and memory

HarnessForge’s harness representation is deliberately minimal and structured. The harness is partitioned into three modules.

Planning $\mathcal{P}$ governs task decomposition, multi-step planning, replanning behavior, and termination conditions. In domains such as ToolHop and retrieval-heavy multi-hop question answering, this includes explicit subgoal decomposition and conditions for revising plans.

Action $\mathcal{H}$0 governs tool interfaces, argument schemas, role assignment, orchestration rules, and external control logic. In REST-style environments and API-Bank-style tasks, this layer constrains how endpoints are chosen, how arguments are normalized, and how multi-step API routes are sequenced.

Memory $\mathcal{H}$1 governs what is written into memory, how information is retrieved, summarized, and exposed back to the reasoner, and how stale or noisy information is filtered (Chen et al., 1 Jun 2026).

The representation is not a generic software architecture taxonomy; it is a search space over executable agent-system components. Harness-level adaptation is restricted to these modules and associated wiring. The paper states that only execution-layer components are edited, including:

• planning_module/provider.py
• action_module/provider.py
• memory_module/provider.py
• builder.py
• prompts and constants (Chen et al., 1 Jun 2026)

This restriction defines a tractable and auditable edit surface. It also distinguishes HarnessForge from more general harness foundries. For example, “HarnessX” models harnesses as typed processor pipelines attached to lifecycle hooks and organized across nine behavioral dimensions, including context assembly, tool ecosystem, execution environment, evaluation, observability, and training bridge (Chen et al., 12 Jun 2026). HarnessForge is narrower: its search space is centered on planning, action, and memory as the minimal execution interface most responsible for harness–policy compatibility.

This suggests a useful interpretive shorthand: HarnessForge’s harness is an “execution interface” rather than a full runtime substrate. That interpretation is consistent with its benchmark suite, which emphasizes long-horizon tool use, retrieval, and control-flow-sensitive API tasks rather than repository-scale verification or deployment automation.

3. Co-evolution algorithm

HarnessForge maintains a population over rounds: $\mathcal{H}$2 Round $\mathcal{H}$3 begins from a single manually designed base harness and a frozen base reasoner. Each subsequent round alternates between harness evolution and policy evolution (Chen et al., 1 Jun 2026).

The harness-evolution operator is denoted

$\mathcal{H}$4

where:

• $\mathcal{H}$5 performs fault attribution
• $\mathcal{H}$6 performs archive-guided improvement planning
• $\mathcal{H}$7 performs candidate harness generation (Chen et al., 1 Jun 2026)

For each parent system and batch $\mathcal{H}$8, the system first collects trajectories $\mathcal{H}$9 and computes batch fitness

$\mathcal{P}$0

where $\mathcal{P}$1 aggregates final response quality, negative token cost, and negative latency (Chen et al., 1 Jun 2026).

Fault attribution is then defined as

$\mathcal{P}$2

The resulting fault report localizes failures to planning, action, or memory. Archive-guided planning retrieves relevant historical harnesses from $\mathcal{P}$3 and produces an improvement brief

$\mathcal{P}$4

This brief specifies which failures to prioritize, which module owns each fix, and which historical patterns to reuse or avoid (Chen et al., 1 Jun 2026).

Candidate harnesses are generated as

$\mathcal{P}$5

The full candidate pool is subjected to progressive half-selection using stagewise Pareto filtering over subsets $\mathcal{P}$6, optimizing jointly for task performance, token cost, and latency (Chen et al., 1 Jun 2026).

This co-evolution loop differs from neighboring auto-harness systems in emphasis. “Adaptive Auto-Harness” organizes sustained harness construction around a multi-agent evolver, a harness tree, and solve-time routing to address open-ended task streams and non-stationarity (Liu et al., 1 Jun 2026). HarnessForge instead assumes a round-based co-evolution regime on benchmark batches and focuses on pairwise harness–policy compatibility rather than branch routing.

4. Harness-conditioned policy alignment

Once survivor harnesses $\mathcal{P}$7 are selected, HarnessForge adapts the policy separately for each survivor harness: $\mathcal{P}$8 This alignment is lineage-specific: each surviving harness carries forward its own policy branch (Chen et al., 1 Jun 2026).

Initialization follows a parent-child adapter pattern. A new adapter is created by materializing the parent adapter and adding a new harness-specific LoRA component: $\mathcal{P}$9 Only the new adapter $\mathcal{A}$0 is trained; the base model remains frozen (Chen et al., 1 Jun 2026).

Training data are drawn from successful trajectories only: $\mathcal{A}$1 Each successful trajectory is decomposed into step-level tuples $\mathcal{A}$2, where $\mathcal{A}$3 contains the task, active harness specification, observation history, current memory state, and available actions, and $\mathcal{A}$4 is the next reasoning step, tool call, memory operation, or final answer. This produces a harness-conditioned dataset

$\mathcal{A}$5

In the main instantiation, policy alignment uses supervised fine-tuning: $\mathcal{A}$6 with

$\mathcal{A}$7

The same framework can also instantiate GRPO or RLOO in place of SFT (Chen et al., 1 Jun 2026).

The distinctive feature is not simply that the model is fine-tuned. It is that the training examples are specific to a harness-defined execution interface. The aligned policy learns the exact API formats, memory conventions, planning discipline, and control-flow expectations induced by its paired harness. This is closely related in spirit to “HarnessBridge,” which inserts a learnable bidirectional controller between agent and environment and learns both observation projection and action projection as harness behaviors (Wang et al., 11 Jun 2026). The difference is that HarnessBridge learns the harness itself as a small model, whereas HarnessForge keeps the harness as explicit code/configuration and adapts the reasoner to that evolving interface.

5. Empirical behavior and benchmark performance

HarnessForge is evaluated on five datasets grouped into four benchmark families:

• ToolHop
• SearchQA, comprising HotpotQA and 2WikiMultiHopQA
• RestBench–TMDB
• API-Bank (Chen et al., 1 Jun 2026)

The benchmark metrics are task-specific. ToolHop uses answer accuracy and subgoal path coverage. SearchQA uses token-level F1 on HotpotQA and 2WikiMultiHopQA and averages them. TMDB uses binary success and path match under ordered subsequence criteria. API-Bank uses success, path F1, and API call accuracy (Chen et al., 1 Jun 2026).

Two backbones are used: Qwen3-4B and Qwen3-8B. Policy adaptation uses LoRA on attention and MLP layers with rank 8, $\mathcal{A}$8, dropout 0.05, and learning rate $\mathcal{A}$9 (Chen et al., 1 Jun 2026).

Across all benchmarks and both backbones, HarnessForge outperforms harness-only and policy-only baselines. The paper reports:

• average +3.56% over the strongest baseline per metric
• gains of up to 12.0% absolute on TMDB Success with Qwen3-4B (Chen et al., 1 Jun 2026)

Selected reported results are summarized below.

Setting	Strongest baseline	HarnessForge
Qwen3-4B, ToolHop Answer	49.74	52.82
Qwen3-4B, SearchQA Overall	39.67	42.00
Qwen3-4B, TMDB Success	64.00	76.00
Qwen3-4B, API-Bank APIAcc	76.60	82.27
Qwen3-8B, ToolHop Answer	51.28	54.87
Qwen3-8B, SearchQA Overall	41.67	42.83
Qwen3-8B, TMDB Success	74.00	80.00
Qwen3-8B, API-Bank APIAcc	73.76	78.01

Rollout-efficiency analyses place HarnessForge on or near the Pareto frontier for all benchmarks, with policy-only RL baselines requiring substantially more rollouts while matching or underperforming the co-evolution approach (Chen et al., 1 Jun 2026).

Ablations attribute most of the gain to harness evolution, but policy alignment remains necessary. For example, removing harness evolution causes a larger performance drop than removing policy evolution, and the gap widens over rounds (Chen et al., 1 Jun 2026). This supports the claim that harness tailoring is the principal driver, while policy alignment converts structural improvements into executable behavior.

6. Compatibility, lineage, and broader significance

HarnessForge’s most distinctive empirical finding is that matched harness–policy pairs outperform mismatched combinations. Compatibility matrices on API-Bank and ToolHop show that the diagonal, corresponding to co-evolved pairs from the same lineage, systematically outperforms off-diagonal cross-pairings. On API-Bank, the base pair starts at 69.30% success, the final co-evolved pair reaches 77.19%, while final-harness/early-policy and final-policy/early-harness mismatches remain notably lower (Chen et al., 1 Jun 2026).

This result is central because it rules out two simpler interpretations. The gains do not arise solely from discovering a universally better harness, nor solely from obtaining a universally better policy. They arise from the pairwise fit between execution structure and reasoning behavior.

Qualitative case studies illustrate how compatibility emerges. On ToolHop, later rounds add finer-grained subgoals, more consistent memory exposure, improved replan triggers, tool-chain preservation, and more selective memory retrieval. The paired policy then learns to fill those subgoal slots, use the memory interface correctly, and respect the stricter action formats (Chen et al., 1 Jun 2026). On API-Bank, harness changes harden authentication and deletion routes while the aligned policy learns the revised route discipline. On HotpotQA and 2Wiki, harness changes emphasize evidence-grounded answering, and the aligned policy learns to populate those structures (Chen et al., 1 Jun 2026).

This pattern situates HarnessForge within a broader research trend that treats the harness as a first-class optimization object. “HARBOR” formalizes automated harness optimization as constrained noisy Bayesian optimization over a mixed-variable configuration space with multi-fidelity cost-aware acquisition and chance-constrained safety (Sengupta et al., 22 Apr 2026). “HarnessX” treats harnesses as typed, composable processor pipelines and aligns harness evolution with an RL-style operational mirror (Chen et al., 12 Jun 2026). HarnessForge differs in focusing specifically on joint harness and policy evolution for medium-scale agent systems and in demonstrating that compatibility between the two layers is not incidental but operationally decisive.

The principal limitation stated in the paper is scope: evaluations are on Qwen3-4B and Qwen3-8B, with repeated rollout costs and a restricted harness edit space centered on planning, action, and memory modules (Chen et al., 1 Jun 2026). This suggests that future extensions could broaden the editable runtime surface or combine co-evolution with more explicit deployment-layer harnesses. A plausible implication is that systems such as DMAS-Forge, which separate logical agent graphs from distributed deployment choices, and HarnessForge, which separates execution harness from policy behavior, could be viewed as complementary strata in a multi-layer agent-systems stack (Cornacchia et al., 13 Oct 2025).