---
title: 'HarnessForge: Evolving Agent Systems'
url: https://www.emergentmind.com/topics/harnessforge
type: topic
---

# HarnessForge: Evolving Agent Systems

Searching arXiv for the specified paper and closely related harness literature to ground the article.
HarnessForge is a meta-adaptive framework for evolving large language model agent systems by treating the system as a coupled **harness–policy pair** rather than as a fixed model wrapped by a static runtime. In this formulation, the **harness** specifies execution structure—planning, action, and memory—while the **policy** specifies the reasoning behavior executed within that structure. HarnessForge makes the adaptation space between these two layers explicit, then performs **harness–policy co-evolution** through **fault-guided harness tailoring** and **harness-conditioned policy alignment**. Across five benchmarks spanning tool use, retrieval-heavy multi-hop question answering, REST-style environments, and tool-augmented APIs, it improves both Qwen3-4B and Qwen3-8B backbones, outperforming harness-only and policy-only baselines and showing that executable compatibility between the external harness and internal reasoner is a central determinant of agent-system performance [2606.01779].

## 1. Conceptual definition and adaptation space

HarnessForge defines an agent system as
\[
\mathcal{G} = (\mathcal{H}, \mathcal{R}_\delta), \quad \mathcal{H} = (\mathcal{P}, \mathcal{A}, \mathcal{M}), \quad \mathcal{R}_\delta = \mathcal{R}_{\theta_0 + \delta}.
\]
Here, \(\mathcal{H}\) is the external execution harness, factored into **planning** \(\mathcal{P}\), **action** \(\mathcal{A}\), and **memory** \(\mathcal{M}\), while \(\mathcal{R}_\delta\) is the LLM-based reasoning policy implemented as a LoRA adapter \(\delta\) on top of a frozen base model \(\mathcal{R}_{\theta_0}\) [2606.01779].

This decomposition is motivated by heterogeneous task regimes that demand distinct execution paradigms. The framework is explicitly positioned against approaches that adapt only the external harness or only the internal policy. Harness-only methods search over workflows, tool routing, and memory organization while leaving the executor fixed; policy-only methods train the reasoner under a fixed interface. HarnessForge instead isolates a stable adaptation space between **structure** and **execution behavior**, then optimizes both sides jointly [2606.01779].

The central notion is **executable compatibility**. A harness may expose planning stages, memory slots, or strict tool schemas, but these structural affordances are only useful if the policy can reliably operate under them. Conversely, a stronger policy can remain bottlenecked by a poor interface. This suggests that the effective unit of adaptation is not the model or harness independently, but the pair \((\mathcal{H}, \mathcal{R}_\delta)\). A plausible implication is that benchmark gains attributed to “agent architecture” or “post-training” can be mischaracterized when harness–policy compatibility is not controlled explicitly.

The broader harness literature reinforces this framing from different angles. “AI Harness Engineering” formalizes the harness as the runtime substrate governing context, tools, project memory, task state, observability, failure attribution, verification, permissions, entropy auditing, and intervention recording [2605.13357]. “Harness-Bench” likewise argues that agent capability should be reported at the **model–harness configuration level** rather than attributed to the base model alone [2605.27922]. HarnessForge is more specific in making the compatibility between harness and policy itself the object of optimization.

## 2. Harness representation: planning, action, and memory

HarnessForge’s harness representation is deliberately minimal and structured. The harness is partitioned into three modules.

**Planning \(\mathcal{P}\)** governs task decomposition, multi-step planning, replanning behavior, and termination conditions. In domains such as ToolHop and retrieval-heavy multi-hop question answering, this includes explicit subgoal decomposition and conditions for revising plans.

**Action \(\mathcal{A}\)** governs tool interfaces, argument schemas, role assignment, orchestration rules, and external control logic. In REST-style environments and API-Bank-style tasks, this layer constrains how endpoints are chosen, how arguments are normalized, and how multi-step API routes are sequenced.

**Memory \(\mathcal{M}\)** governs what is written into memory, how information is retrieved, summarized, and exposed back to the reasoner, and how stale or noisy information is filtered [2606.01779].

The representation is not a generic software architecture taxonomy; it is a search space over executable agent-system components. Harness-level adaptation is restricted to these modules and associated wiring. The paper states that only **execution-layer components** are edited, including:
- `planning_module/provider.py`
- `action_module/provider.py`
- `memory_module/provider.py`
- `builder.py`
- prompts and constants [2606.01779]

This restriction defines a tractable and auditable edit surface. It also distinguishes HarnessForge from more general harness foundries. For example, “HarnessX” models harnesses as typed processor pipelines attached to lifecycle hooks and organized across nine behavioral dimensions, including context assembly, tool ecosystem, execution environment, evaluation, observability, and training bridge [2606.14249]. HarnessForge is narrower: its search space is centered on planning, action, and memory as the minimal execution interface most responsible for harness–policy compatibility.

This suggests a useful interpretive shorthand: HarnessForge’s harness is an “execution interface” rather than a full runtime substrate. That interpretation is consistent with its benchmark suite, which emphasizes long-horizon tool use, retrieval, and control-flow-sensitive API tasks rather than repository-scale verification or deployment automation.

## 3. Co-evolution algorithm

HarnessForge maintains a population over rounds:
\[
\mathbb{G}^{(r)} = \left\{
\mathcal{G}_i^{(r)} =
\left(\mathcal{H}_i^{(r)}, \mathcal{R}_{\delta_i}^{(r)}\right)
\right\}_{i \in I^{(r)}}.
\]
Round \(r=0\) begins from a single manually designed base harness and a frozen base reasoner. Each subsequent round alternates between **harness evolution** and **policy evolution** [2606.01779].

The harness-evolution operator is denoted
\[
T_\psi = (\mathbb{L}_\omega, \mathbb{R}_\omega, \Gamma_\omega),
\]
where:
- \(\mathbb{L}_\omega\) performs **fault attribution**
- \(\mathbb{R}_\omega\) performs **archive-guided improvement planning**
- \(\Gamma_\omega\) performs **candidate harness generation** [2606.01779]

For each parent system and batch \(B_r\), the system first collects trajectories \(\mathcal{T}_i^{(r)}\) and computes batch fitness
\[
\mathbf{J}(\mathcal{G}; B) = \frac{1}{|B|} \sum_{x \in B} \boldsymbol{\phi}(\tau_x(\mathcal{G}), x),
\]
where \(\boldsymbol{\phi}\) aggregates final response quality, negative token cost, and negative latency [2606.01779].

Fault attribution is then defined as
\[
\mathbf{F}_{\mathcal{H}_i}^{(r)} = \mathbb{L}_\omega\left(
\mathcal{H}_i^{(r)}, \mathcal{T}_i^{(r)}, \mathbf{J}(\mathcal{G}_i^{(r)}; B_r)
\right).
\]
The resulting fault report localizes failures to planning, action, or memory. Archive-guided planning retrieves relevant historical harnesses from \(\mathcal{Z}^{(r)}\) and produces an improvement brief
\[
\mathbf{I}_{\mathcal{H}_i}^{(r)} =
\mathbb{R}_\omega\left(
\mathcal{H}_i^{(r)}, \mathbf{F}_{\mathcal{H}_i}^{(r)}, \mathcal{S}_{\mathcal{H}_i}^{(r)}
\right).
\]
This brief specifies which failures to prioritize, which module owns each fix, and which historical patterns to reuse or avoid [2606.01779].

Candidate harnesses are generated as
\[
\mathcal{C}_{\mathcal{H}_i}^{(r)} = \Gamma_\omega\left(
\mathcal{H}_i^{(r)}, \mathbf{I}_{\mathcal{H}_i}^{(r)}
\right), \quad |\mathcal{C}_{\mathcal{H}_i}^{(r)}| = K_{\text{gen}}.
\]
The full candidate pool is subjected to **progressive half-selection** using stagewise Pareto filtering over subsets \(B_{r,t}\), optimizing jointly for task performance, token cost, and latency [2606.01779].

This co-evolution loop differs from neighboring auto-harness systems in emphasis. “Adaptive Auto-Harness” organizes sustained harness construction around a multi-agent evolver, a harness tree, and solve-time routing to address open-ended task streams and non-stationarity [2606.01770]. HarnessForge instead assumes a round-based co-evolution regime on benchmark batches and focuses on pairwise harness–policy compatibility rather than branch routing.

## 4. Harness-conditioned policy alignment

Once survivor harnesses \(\{\mathcal{H}_k^{(r+1)}\}\) are selected, HarnessForge adapts the policy separately for each survivor harness:
\[
\mathcal{R}^{(r+1)}_{\delta_k} = E_\eta\left(
(\mathcal{H}_k^{(r+1)}, \mathcal{R}^{(r)}_{\delta_k}), B_r
\right).
\]
This alignment is **lineage-specific**: each surviving harness carries forward its own policy branch [2606.01779].

Initialization follows a parent-child adapter pattern. A new adapter is created by materializing the parent adapter and adding a new harness-specific LoRA component:
\[
\delta_k^{(r+1)} = \text{Merge}(\delta_k^{(r)}) \oplus \Delta\delta_k^{(r+1)}.
\]
Only the new adapter \(\Delta\delta_k^{(r+1)}\) is trained; the base model remains frozen [2606.01779].

Training data are drawn from successful trajectories only:
\[
\mathcal{T}_k^{+} = \{ \tau_x \in \mathcal{T}_k^{(r+1)} \mid S(\tau_x) = 1 \}.
\]
Each successful trajectory is decomposed into step-level tuples \((z_t, y_t)\), where \(z_t\) contains the task, active harness specification, observation history, current memory state, and available actions, and \(y_t\) is the next reasoning step, tool call, memory operation, or final answer. This produces a harness-conditioned dataset
\[
\mathcal{D}_{\mathcal{H}_k} = \{(z_t, y_t)\}.
\]
In the main instantiation, policy alignment uses supervised fine-tuning:
\[
\Delta\delta_k^{(r+1)} =
\arg\min_{\Delta\delta}
\mathcal{L}\big(
\mathcal{R}^{(r)}_{\delta_k} \oplus \Delta\delta; \mathcal{D}_{\mathcal{H}_k}
\big),
\]
with
\[
\mathcal{L}_{\text{sft}} = - \sum_{(z_t, y_t) \in \mathcal{D}_{\mathcal{H}_k}}
\log p_{\mathcal{R}^{(r)}_{\delta_k} \oplus \Delta\delta}(y_t \mid z_t).
\]
The same framework can also instantiate GRPO or RLOO in place of SFT [2606.01779].

The distinctive feature is not simply that the model is fine-tuned. It is that the training examples are specific to a harness-defined execution interface. The aligned policy learns the exact API formats, memory conventions, planning discipline, and control-flow expectations induced by its paired harness. This is closely related in spirit to “HarnessBridge,” which inserts a learnable bidirectional controller between agent and environment and learns both observation projection and action projection as harness behaviors [2606.12882]. The difference is that HarnessBridge learns the harness itself as a small model, whereas HarnessForge keeps the harness as explicit code/configuration and adapts the reasoner to that evolving interface.

## 5. Empirical behavior and benchmark performance

HarnessForge is evaluated on five datasets grouped into four benchmark families:
- **ToolHop**
- **SearchQA**, comprising **HotpotQA** and **2WikiMultiHopQA**
- **RestBench–TMDB**
- **API-Bank** [2606.01779]

The benchmark metrics are task-specific. ToolHop uses answer accuracy and subgoal path coverage. SearchQA uses token-level F1 on HotpotQA and 2WikiMultiHopQA and averages them. TMDB uses binary success and path match under ordered subsequence criteria. API-Bank uses success, path F1, and API call accuracy [2606.01779].

Two backbones are used: **Qwen3-4B** and **Qwen3-8B**. Policy adaptation uses LoRA on attention and MLP layers with rank 8, \(\alpha=16\), dropout 0.05, and learning rate \(2\times10^{-6}\) [2606.01779].

Across all benchmarks and both backbones, HarnessForge outperforms harness-only and policy-only baselines. The paper reports:
- average **+3.56%** over the strongest baseline per metric
- gains of up to **12.0%** absolute on TMDB Success with Qwen3-4B [2606.01779]

Selected reported results are summarized below.

| Setting | Strongest baseline | HarnessForge |
|---|---:|---:|
| Qwen3-4B, ToolHop Answer | 49.74 | 52.82 |
| Qwen3-4B, SearchQA Overall | 39.67 | 42.00 |
| Qwen3-4B, TMDB Success | 64.00 | 76.00 |
| Qwen3-4B, API-Bank APIAcc | 76.60 | 82.27 |
| Qwen3-8B, ToolHop Answer | 51.28 | 54.87 |
| Qwen3-8B, SearchQA Overall | 41.67 | 42.83 |
| Qwen3-8B, TMDB Success | 74.00 | 80.00 |
| Qwen3-8B, API-Bank APIAcc | 73.76 | 78.01 |

Rollout-efficiency analyses place HarnessForge on or near the Pareto frontier for all benchmarks, with policy-only RL baselines requiring substantially more rollouts while matching or underperforming the co-evolution approach [2606.01779].

Ablations attribute most of the gain to harness evolution, but policy alignment remains necessary. For example, removing harness evolution causes a larger performance drop than removing policy evolution, and the gap widens over rounds [2606.01779]. This supports the claim that harness tailoring is the principal driver, while policy alignment converts structural improvements into executable behavior.

## 6. Compatibility, lineage, and broader significance

HarnessForge’s most distinctive empirical finding is that matched harness–policy pairs outperform mismatched combinations. Compatibility matrices on API-Bank and ToolHop show that the diagonal, corresponding to co-evolved pairs from the same lineage, systematically outperforms off-diagonal cross-pairings. On API-Bank, the base pair starts at **69.30%** success, the final co-evolved pair reaches **77.19%**, while final-harness/early-policy and final-policy/early-harness mismatches remain notably lower [2606.01779].

This result is central because it rules out two simpler interpretations. The gains do not arise solely from discovering a universally better harness, nor solely from obtaining a universally better policy. They arise from the pairwise fit between execution structure and reasoning behavior.

Qualitative case studies illustrate how compatibility emerges. On ToolHop, later rounds add finer-grained subgoals, more consistent memory exposure, improved replan triggers, tool-chain preservation, and more selective memory retrieval. The paired policy then learns to fill those subgoal slots, use the memory interface correctly, and respect the stricter action formats [2606.01779]. On API-Bank, harness changes harden authentication and deletion routes while the aligned policy learns the revised route discipline. On HotpotQA and 2Wiki, harness changes emphasize evidence-grounded answering, and the aligned policy learns to populate those structures [2606.01779].

This pattern situates HarnessForge within a broader research trend that treats the harness as a first-class optimization object. “HARBOR” formalizes automated harness optimization as constrained noisy Bayesian optimization over a mixed-variable configuration space with multi-fidelity cost-aware acquisition and chance-constrained safety [2604.20938]. “HarnessX” treats harnesses as typed, composable processor pipelines and aligns harness evolution with an RL-style operational mirror [2606.14249]. HarnessForge differs in focusing specifically on **joint harness and policy evolution** for medium-scale agent systems and in demonstrating that compatibility between the two layers is not incidental but operationally decisive.

The principal limitation stated in the paper is scope: evaluations are on Qwen3-4B and Qwen3-8B, with repeated rollout costs and a restricted harness edit space centered on planning, action, and memory modules [2606.01779]. This suggests that future extensions could broaden the editable runtime surface or combine co-evolution with more explicit deployment-layer harnesses. A plausible implication is that systems such as DMAS-Forge, which separate logical agent graphs from distributed deployment choices, and HarnessForge, which separates execution harness from policy behavior, could be viewed as complementary strata in a multi-layer agent-systems stack [2510.11872].

Source: https://www.emergentmind.com/topics/harnessforge