---
title: Trial-to-Harness Conversion
url: https://www.emergentmind.com/topics/trial-to-harness-behavior-conversion
type: topic
---

# Trial-to-Harness Conversion

Searching arXiv for the cited harness-related papers to ground the article in current literature.
Trial-to-harness-behavior conversion denotes a family of procedures in which trial artifacts—such as input–output tests, execution traces, failed trajectories, simulator rollouts, or adversarial multi-turn transcripts—are converted into persistent harness-mediated behavior. In this literature, the harness is the executable layer around a model or policy that governs prompting, tool use, state updates, validation, recovery, tracing, permissions, and control flow. The conversion therefore does not merely detect failure; it rewrites or instantiates the interface that shapes subsequent behavior, whether as Python test code, runtime validators, file-level harness edits, executable gates, or learnable controllers [2511.01104][2605.27922][2606.12882].

## 1. Conceptual scope and core definitions

Across the literature, the harness is treated as a first-class object rather than an informal wrapper. Harness-Bench defines a harness \(H\) as the system layer that transforms raw model outputs into concrete actions on an external workspace and maintains the execution protocol, including prompting, tool calls, state updates, permissions, tracing, and recovery. It writes end-to-end execution as \(R=\mathrm{Run}(M,H,T)\), contrasting this with a raw trial \(O_{\mathrm{raw}}=M(T)\), which yields text but no managed workflow [2605.27922].

Several papers formalize trial-to-harness conversion as a mapping from execution evidence to interface logic. In HarnessLLM, the classical trial \(T=(x,\hat y)\) is replaced by a generated Python harness \(H\) that synthesizes inputs and validates outputs, formally \(H:\{q,f\}\mapsto(\{x_i\},v_H)\), where \(v_H(x,y)\) raises an assertion if \(y\) is deemed incorrect [2511.01104]. Life-Harness defines a trajectory-to-intervention transform \(H:\{\tau\}\rightarrow\mathcal{I}\), mapping failed deterministic-environment trajectories into interventions at the Contract, Skill, ActionRealization, or TrajectoryRegulation layers [2605.22166]. Sibyl-AutoResearch introduces an explicitly harness-side conversion function,
\[
\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},
\]
where a recurring process failure class \(f\) observed at iteration \(t\) produces an update action \(u\) at iteration \(t+k\) [2605.22343].

The concept extends beyond language-agent interfaces. HARBOR frames robot RL automation as a harness-engineering problem and packages the workflow as
\[
H_{RL}=(H_A,C,M,G,K),
\]
where isolated agents, standardized commands, mutable artifacts, executable gates, and reusable knowledge collectively turn long-horizon automation into verifiable micro-projects [2606.08610]. HarnessX, by contrast, treats the runtime harness as a typed object \(\mathcal{H}\) subject to symbolic edits \(e:\mathcal{H}\mapsto\mathcal{H}\), and then places those edits inside an MDP whose states are harness configurations and trace stores [2606.14249].

## 2. Forms of the converted harness

One major form is the **code harness**. HarnessLLM asks an LLM to emit Python code containing up to five `generate_input_i() -> List[str]` functions and a `check_output(input, output)` function. This converts a single trial into a small executable program that can synthesize many \(x_i\), use reference implementations, and assert invariants such as palindrome structure [2511.01104]. AutoHarness similarly synthesizes Python wrappers around an LLM agent, including action-verifier, action-filter, and harness-as-policy styles. Its formal target is a code harness
\[
H:(\mathcal{S}\to\mathcal{A})\rightarrow(\mathcal{S}\to\mathcal{A}_{\rm safe}),
\]
so that the wrapped policy never outputs an illegal action [2603.03329].

A second form is the **runtime intervention harness**. Life-Harness keeps model weights frozen and evolves the runtime layer from training trajectories. The editable intervention space includes environment contracts, procedural skills, action realization, and trajectory regulation. The resulting harness remains fixed during held-out evaluation, which the paper presents as evidence that the learned structure is environment-side rather than model-specific [2605.22166]. Self-Harness likewise holds the base model fixed and defines the harness \(h\) as system and user instructions, declared tools, and runtime policies; only \(h\) changes across iterations, while the language model \(M\) and evaluator \(\mathcal{E}\) are held constant [2606.09498].

A third form is the **file-backed evolvable harness**. Agentic Harness Engineering represents a harness as an explicit set of files \(H=\{f_1,\dots,f_n\}\), covering seven editable component types such as system prompt, tool description files, tool implementations, middleware modules, skills, sub-agent configs, and long-term memory. This makes the action space “git edit this file” or “git add this file,” and every single logical change becomes a revertible file-level diff [2604.25850]. “The Last Harness You’ll Ever Build” uses a broader decomposition \(\mathcal{H}=(P,T,O,S)\), consisting of prompts, tools, orchestration logic, and evaluation criteria, and embeds that harness in an outer evolution loop and a meta-evolution loop [2604.21003].

A fourth form is the **learnable harness controller**. HarnessBridge parameterizes the interface between a fixed generator \(\pi_g\) and the environment \(\mathcal{E}\) as a learnable harness policy \(\pi_h\) factored into observation and action projections. Observation projection \(\phi_{\mathrm{obs}}\) assigns each history unit to `Pass`, `Compress`, or `Drop`, while action projection \(\phi_{\mathrm{act}}\) maps a proposed action to `Pass` or `Reject` with trajectory-grounded feedback \((\mathrm{concern},\mathrm{evidence},\mathrm{suggestion})\) [2606.12882].

## 3. Conversion mechanisms from trials to harness behavior

A recurring pattern is **collection, normalization, and diagnosis of trial evidence**. HarnessFix compiles raw execution logs and harness code into a Harness-aware Trace Intermediate Representation (HTIR), consisting of `TraceStep` records plus provenance and control-flow links. It then assigns each failed trace a responsibility score based on provenance paths, control-flow paths, and status-effect mismatches, identifies the responsible step, maps implicated ETCLOVG layers, and consolidates recurring diagnoses into `FlawRecord`s [2606.06324]. AHE performs a related compression step through experience observability: raw trajectories \(T_t\) are cleaned into \(\hat T_t\), distilled into `overview.md`, per-task detail files, and cleaned JSON traces, and then consumed by the Evolve Agent instead of multi-million-token logs [2604.25850].

A second pattern is **recurrence-based abstraction**. Life-Harness groups failures by signature \(\sigma_f(\tau)\) and only evolves an intervention when the recurrence threshold \(|\{\tau:\sigma_f(\tau)=s\}|\ge N_{\min}\) is met [2605.22166]. Self-Harness computes failure signatures \(\phi(r_i)=(c_i,q_i,m_i)\), clusters failures by exact signature match, and sorts the resulting structured patterns by cluster size and actionability to form an evidence bundle \(B_t\) [2606.09498]. Sibyl routes reflection outputs into evolution memory and triggers harness self-evolution when a normalized failure class recurs above a threshold or at strategic project-phase gates [2605.22343].

A third pattern is **proposal generation over a constrained edit space**. HarnessLLM uses a two-stage training pipeline: supervised fine-tuning on approximately \(15\) K verified harnesses over approximately \(4.4\) K unique problems, followed by RLVR with reward \(r(H;f,g)\in\{1,0.1,0\}\) and optimization with GRPO [2511.01104]. AutoHarness uses an LLM-driven code-refinement loop organized as a tree search with Thompson sampling over candidate harness programs \(c\in\mathcal{C}\), optimizing legal-move success rate [2603.03329]. RHO selects a diverse coreset with a DPP kernel, re-solves the tasks in parallel, generates candidate harnesses by editing the harness directory, and chooses the best by pairwise self-preference rather than external labels [2606.05922]. HarnessX uses AEGIS, whose Digester, Planner, Evolver, and Critic operate over typed harness primitives and ship only one safe edit per round after a deterministic gate [2606.14249].

A fourth pattern is **validation under anti-regression constraints**. Self-Harness accepts a candidate edit \(\Delta_j\) iff \(\Delta_{\mathrm{in}}^{(j)}\ge0\), \(\Delta_{\mathrm{ho}}^{(j)}\ge0\), and \(\max(\Delta_{\mathrm{in}}^{(j)},\Delta_{\mathrm{ho}}^{(j)})>0\), thereby requiring non-regression on both held-in and held-out splits [2606.09498]. HarnessFix validates patches using both target improvement \(\Delta D\ge\delta_{\min}\) and a regression bound \(R_{\mathrm{new}}\le r_{\max}\) [2606.06324]. AHE compares predicted fixes and risk tasks with actual fixes and regressions, issuing a per-edit verdict of `KEEP` or `ROLLBACK`; this makes every edit a falsifiable contract [2604.25850]. HARBOR enforces executable gates ranging from dependency checks to rollout and logging conditions before progression to later stages [2606.08610].

## 4. Representative instantiations

| System | Trial substrate | Resulting harness behavior |
|---|---|---|
| HarnessLLM | Input–output trials and program executions | Python test harness with input generators and `check_output` [2511.01104] |
| AutoHarness | Illegal-move failure traces in TextArena | Code harness preventing illegal actions or full code policy [2603.03329] |
| Life-Harness | Deterministic-environment trajectories | Reusable runtime interventions across four layers [2605.22166] |
| Self-Harness | Execution traces on Terminal-Bench-2.0 | Minimal prompt, tool, and policy edits accepted by regression testing [2606.09498] |
| HarnessFix | Raw logs plus harness code | Scoped patches generated from HTIR-based flaw attribution [2606.06324] |
| HARBOR | Simulator trials and short RL runs | Bounded-stage RL workflow with commands, artifacts, gates, and reusable knowledge [2606.08610] |
| HarnessBridge | Raw long-horizon histories and proposed actions | Learnable observation compression and action rejection controller [2606.12882] |

These systems differ in their trial objects, but the conversion logic is structurally similar. A raw episode is not treated as a terminal score; it is compiled into an executable scaffold that changes future interaction. This suggests that “behavior” in this literature is often mediated behavior: not only what a model predicts, but how a harness permits, blocks, reshapes, or validates those predictions.

## 5. Evaluation criteria and empirical findings

In software testing, HarnessLLM evaluates generated harnesses with Good Input rate, Invalid Test Rate, and True Bug Rate, averaged over eight sampled harnesses per \((q,f)\). The paper reports that RL-trained HarnessLLM with Qwen3-4B achieves TBR \(74.4\%\) on Codeforces versus \(72.2\%\) for an input–output RL baseline; on LiveCodeBench Unseen, TBR is \(37.2\%\) versus \(33.3\%\); unique-input ratio is \(77.1\%\) versus \(48.6\%\); input length range log-span is \(8.90\) versus \(1.00\); and test-time scaling raises the original pass@1 of Qwen3-32B on LiveCodeBench from \(63.5\%\) to \(69.5\%\) [2511.01104].

In deterministic agent settings, Life-Harness reports improvements in \(116\) out of \(126\) model–environment settings across \(18\) model backbones, with an average relative improvement of \(88.5\%\). Example benchmark results include AgentBench/ALFWorld Pass@1 from \(41.1\%\) to \(75.7\%\), \(τ\)-bench/Retail Pass@1 from \(56.2\%\) to \(61.8\%\), and \(τ^2\)-bench/Telecom Pass@1 from \(55.3\%\) to \(69.0\%\) [2605.22166]. Self-Harness reports held-out pass-rate improvements from \(40.5\%\) to \(61.9\%\) for MiniMax M2.5, from \(23.8\%\) to \(38.1\%\) for Qwen3.5-35B-A3B, and from \(42.9\%\) to \(57.1\%\) for GLM-5 [2606.09498].

In agent safety and execution compliance, AutoHarness reports \(100\%\) illegal-move success rate across \(145\) TextArena environments, with an average of \(14.5\) LLM-calls per game and a worst case of \(64\) calls for Chess. On \(16\) sampled two-player games, Gemini-2.5-Flash+Harness won \(56.3\%\) versus Gemini-2.5-Pro’s \(38.2\%\), and the code-only policy reached \(0.870\) average reward on \(16\) one-player games [2603.03329].

In harness evolution and diagnosis, AHE raises pass@1 on Terminal-Bench 2 from \(69.7\%\) to \(77.0\%\), surpassing Codex-CLI at \(71.9\%\), and reports frozen-harness cross-family gains of \(+5.1\) to \(+10.1\) percentage points [2604.25850]. HarnessFix improves held-out performance from \(45.0\%\) to \(57.0\%\) on SWE-Bench Verified, from \(17.6\%\) to \(26.5\%\) on Terminal-Bench 2.0 Verified, from \(43.3\%\) to \(61.7\%\) on GAIA, and from \(36.7\%\) to \(42.2\%\) on AppWorld [2606.06324]. RHO reports that one optimization round improves the pass rate on SWE-Bench Pro from \(59\%\) to \(78\%\) without external grading [2606.05922].

In runtime and systems benchmarks, Harness-Bench evaluates \(106\) sandboxed offline tasks and \(5{,}194\) execution trajectories, reporting a \(23.8\) percentage-point gap between the best- and worst-performing harnesses and arguing that capability should be reported at the model × harness configuration level [2605.27922]. HarnessBridge reports \(33.7\%\) success on Terminal-Bench 2.0 at \(1.23\) M tokens versus Terminus 2 at \(30.3\%\) and \(2.31\) M, and \(60.2\%\) on SWE-bench Verified at \(1.13\) M tokens [2606.12882]. HARBOR reports \(88\)–\(100\%\) success in simulation and \(60\)–\(90\%\) on the real robot across \(16\) simulator-task settings and \(6\) backends [2606.08610]. HarnessX reports an average gain of \(+14.5\%\), up to \(+44.0\%\), across ALFWorld, GAIA, WebShop, \(τ^3\)-Bench, and SWE-bench Verified [2606.14249].

## 6. Misconceptions, limitations, and open directions

A common misconception is that a harness is merely a prompt. The surveyed papers consistently use a broader definition. Harnesses include tools, schemas, runtime policies, mutable artifacts, middleware, memory, validators, permissions, observability hooks, and governance layers [2605.27922][2604.25850][2606.06324]. Another misconception is that behavior change must come from updating model weights. Life-Harness, Self-Harness, AutoHarness, AHE, HarnessFix, RHO, and Sibyl all present harness-side improvement as a complementary or standalone alternative to model-centric training [2605.22166][2606.09498][2605.22343].

The literature also imposes visible limits. AutoHarness synthesizes a separate harness per environment, and cross-game generalization remains future work [2603.03329]. HarnessBridge evaluates only coding domains, reports one deterministic run per task, and uses static SFT rather than reinforcement learning [2606.12882]. Life-Harness is evaluated on deterministic environments [2605.22166]. HARBOR targets sim-to-real robot RL automation rather than general agent execution [2606.08610]. ProofAgent Harness is primarily an adversarial evaluation infrastructure, converting a `TrialScenario` into an evidence-linked report through multi-juror scoring, rather than directly repairing the tested agent [2605.24134].

Several papers emphasize that final outcomes alone are insufficient. HarnessFix argues that broad or poorly scoped changes arise when one cannot diagnose where the responsible evidence lies in failed trajectories [2606.06324]. AHE’s experience observability and decision observability are designed precisely to avoid collapsing into trial-and-error [2604.25850]. Sibyl’s retrospective audit found eight high-confidence conversion events, with a median latency of one iteration and a maximum latency of three iterations, but explicitly states that these traces do not establish a comparative performance claim [2605.22343]. This suggests that the field treats auditability, scoped repair, and regression control as constitutive parts of conversion rather than ancillary implementation detail.

Taken together, the literature describes trial-to-harness-behavior conversion as a shift from ephemeral trial evidence to executable, persistent interface structure. The converted artifact may be a verifier, a validator, a gate, a workflow, a controller, a file-level diff, or a model-conditioned set of runtime policies. What unifies these cases is the claim that reliable behavior emerges not only from better models, but from better mechanisms for turning trials into harnesses that subsequently govern action.

Source: https://www.emergentmind.com/topics/trial-to-harness-behavior-conversion