Trial-to-Harness Conversion

• Trial-to-harness-behavior conversion is a process that transforms trial outputs into executable harnesses, enforcing controlled execution through code, interventions, or learnable controllers.
• It integrates components like prompt governance, tool validation, state updates, and runtime policies to convert raw model outputs into actionable, verified behavior.
• Empirical findings demonstrate improved pass rates, reduced illegal actions, and increased system reliability across diverse agent and testing environments.

Searching arXiv for the cited harness-related papers to ground the article in current literature. Trial-to-harness-behavior conversion denotes a family of procedures in which trial artifacts—such as input–output tests, execution traces, failed trajectories, simulator rollouts, or adversarial multi-turn transcripts—are converted into persistent harness-mediated behavior. In this literature, the harness is the executable layer around a model or policy that governs prompting, tool use, state updates, validation, recovery, tracing, permissions, and control flow. The conversion therefore does not merely detect failure; it rewrites or instantiates the interface that shapes subsequent behavior, whether as Python test code, runtime validators, file-level harness edits, executable gates, or learnable controllers (Liu et al., 2 Nov 2025, Yao et al., 27 May 2026, Wang et al., 11 Jun 2026).

1. Conceptual scope and core definitions

Across the literature, the harness is treated as a first-class object rather than an informal wrapper. Harness-Bench defines a harness $H$ as the system layer that transforms raw model outputs into concrete actions on an external workspace and maintains the execution protocol, including prompting, tool calls, state updates, permissions, tracing, and recovery. It writes end-to-end execution as $R=\mathrm{Run}(M,H,T)$, contrasting this with a raw trial $O_{\mathrm{raw}}=M(T)$, which yields text but no managed workflow (Yao et al., 27 May 2026).

Several papers formalize trial-to-harness conversion as a mapping from execution evidence to interface logic. In HarnessLLM, the classical trial $T=(x,\hat y)$ is replaced by a generated Python harness $H$ that synthesizes inputs and validates outputs, formally $H:\{q,f\}\mapsto(\{x_i\},v_H)$, where $v_H(x,y)$ raises an assertion if $y$ is deemed incorrect (Liu et al., 2 Nov 2025). Life-Harness defines a trajectory-to-intervention transform $H:\{\tau\}\rightarrow\mathcal{I}$, mapping failed deterministic-environment trajectories into interventions at the Contract, Skill, ActionRealization, or TrajectoryRegulation layers (Xu et al., 21 May 2026). Sibyl-AutoResearch introduces an explicitly harness-side conversion function,

$$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$

where a recurring process failure class $R=\mathrm{Run}(M,H,T)$0 observed at iteration $R=\mathrm{Run}(M,H,T)$1 produces an update action $R=\mathrm{Run}(M,H,T)$2 at iteration $R=\mathrm{Run}(M,H,T)$3 (Wang et al., 21 May 2026).

The concept extends beyond language-agent interfaces. HARBOR frames robot RL automation as a harness-engineering problem and packages the workflow as

$R=\mathrm{Run}(M,H,T)$4

where isolated agents, standardized commands, mutable artifacts, executable gates, and reusable knowledge collectively turn long-horizon automation into verifiable micro-projects (Li et al., 7 Jun 2026). HarnessX, by contrast, treats the runtime harness as a typed object $R=\mathrm{Run}(M,H,T)$5 subject to symbolic edits $R=\mathrm{Run}(M,H,T)$6, and then places those edits inside an MDP whose states are harness configurations and trace stores (Chen et al., 12 Jun 2026).

2. Forms of the converted harness

One major form is the code harness. HarnessLLM asks an LLM to emit Python code containing up to five generate_input_i() -> List[str] functions and a check_output(input, output) function. This converts a single trial into a small executable program that can synthesize many $R=\mathrm{Run}(M,H,T)$7, use reference implementations, and assert invariants such as palindrome structure (Liu et al., 2 Nov 2025). AutoHarness similarly synthesizes Python wrappers around an LLM agent, including action-verifier, action-filter, and harness-as-policy styles. Its formal target is a code harness

$R=\mathrm{Run}(M,H,T)$8

so that the wrapped policy never outputs an illegal action (Lou et al., 10 Feb 2026).

A second form is the runtime intervention harness. Life-Harness keeps model weights frozen and evolves the runtime layer from training trajectories. The editable intervention space includes environment contracts, procedural skills, action realization, and trajectory regulation. The resulting harness remains fixed during held-out evaluation, which the paper presents as evidence that the learned structure is environment-side rather than model-specific (Xu et al., 21 May 2026). Self-Harness likewise holds the base model fixed and defines the harness $R=\mathrm{Run}(M,H,T)$9 as system and user instructions, declared tools, and runtime policies; only $O_{\mathrm{raw}}=M(T)$0 changes across iterations, while the LLM $O_{\mathrm{raw}}=M(T)$1 and evaluator $O_{\mathrm{raw}}=M(T)$2 are held constant (Zhang et al., 8 Jun 2026).

A third form is the file-backed evolvable harness. Agentic Harness Engineering represents a harness as an explicit set of files $O_{\mathrm{raw}}=M(T)$3, covering seven editable component types such as system prompt, tool description files, tool implementations, middleware modules, skills, sub-agent configs, and long-term memory. This makes the action space “git edit this file” or “git add this file,” and every single logical change becomes a revertible file-level diff (Lin et al., 28 Apr 2026). “The Last Harness You’ll Ever Build” uses a broader decomposition $O_{\mathrm{raw}}=M(T)$4, consisting of prompts, tools, orchestration logic, and evaluation criteria, and embeds that harness in an outer evolution loop and a meta-evolution loop (Seong et al., 22 Apr 2026).

A fourth form is the learnable harness controller. HarnessBridge parameterizes the interface between a fixed generator $O_{\mathrm{raw}}=M(T)$5 and the environment $O_{\mathrm{raw}}=M(T)$6 as a learnable harness policy $O_{\mathrm{raw}}=M(T)$7 factored into observation and action projections. Observation projection $O_{\mathrm{raw}}=M(T)$8 assigns each history unit to Pass, Compress, or Drop, while action projection $O_{\mathrm{raw}}=M(T)$9 maps a proposed action to Pass or Reject with trajectory-grounded feedback $T=(x,\hat y)$0 (Wang et al., 11 Jun 2026).

3. Conversion mechanisms from trials to harness behavior

A recurring pattern is collection, normalization, and diagnosis of trial evidence. HarnessFix compiles raw execution logs and harness code into a Harness-aware Trace Intermediate Representation (HTIR), consisting of TraceStep records plus provenance and control-flow links. It then assigns each failed trace a responsibility score based on provenance paths, control-flow paths, and status-effect mismatches, identifies the responsible step, maps implicated ETCLOVG layers, and consolidates recurring diagnoses into FlawRecords (Chen et al., 4 Jun 2026). AHE performs a related compression step through experience observability: raw trajectories $T=(x,\hat y)$1 are cleaned into $T=(x,\hat y)$2, distilled into overview.md, per-task detail files, and cleaned JSON traces, and then consumed by the Evolve Agent instead of multi-million-token logs (Lin et al., 28 Apr 2026).

A second pattern is recurrence-based abstraction. Life-Harness groups failures by signature $T=(x,\hat y)$3 and only evolves an intervention when the recurrence threshold $T=(x,\hat y)$4 is met (Xu et al., 21 May 2026). Self-Harness computes failure signatures $T=(x,\hat y)$5, clusters failures by exact signature match, and sorts the resulting structured patterns by cluster size and actionability to form an evidence bundle $T=(x,\hat y)$6 (Zhang et al., 8 Jun 2026). Sibyl routes reflection outputs into evolution memory and triggers harness self-evolution when a normalized failure class recurs above a threshold or at strategic project-phase gates (Wang et al., 21 May 2026).

A third pattern is proposal generation over a constrained edit space. HarnessLLM uses a two-stage training pipeline: supervised fine-tuning on approximately $T=(x,\hat y)$7 K verified harnesses over approximately $T=(x,\hat y)$8 K unique problems, followed by RLVR with reward $T=(x,\hat y)$9 and optimization with GRPO (Liu et al., 2 Nov 2025). AutoHarness uses an LLM-driven code-refinement loop organized as a tree search with Thompson sampling over candidate harness programs $H$0, optimizing legal-move success rate (Lou et al., 10 Feb 2026). RHO selects a diverse coreset with a DPP kernel, re-solves the tasks in parallel, generates candidate harnesses by editing the harness directory, and chooses the best by pairwise self-preference rather than external labels (Pan et al., 4 Jun 2026). HarnessX uses AEGIS, whose Digester, Planner, Evolver, and Critic operate over typed harness primitives and ship only one safe edit per round after a deterministic gate (Chen et al., 12 Jun 2026).

A fourth pattern is validation under anti-regression constraints. Self-Harness accepts a candidate edit $H$1 iff $H$2, $H$3, and $H$4, thereby requiring non-regression on both held-in and held-out splits (Zhang et al., 8 Jun 2026). HarnessFix validates patches using both target improvement $H$5 and a regression bound $H$6 (Chen et al., 4 Jun 2026). AHE compares predicted fixes and risk tasks with actual fixes and regressions, issuing a per-edit verdict of KEEP or ROLLBACK; this makes every edit a falsifiable contract (Lin et al., 28 Apr 2026). HARBOR enforces executable gates ranging from dependency checks to rollout and logging conditions before progression to later stages (Li et al., 7 Jun 2026).

4. Representative instantiations

System	Trial substrate	Resulting harness behavior
HarnessLLM	Input–output trials and program executions	Python test harness with input generators and check_output (Liu et al., 2 Nov 2025)
AutoHarness	Illegal-move failure traces in TextArena	Code harness preventing illegal actions or full code policy (Lou et al., 10 Feb 2026)
Life-Harness	Deterministic-environment trajectories	Reusable runtime interventions across four layers (Xu et al., 21 May 2026)
Self-Harness	Execution traces on Terminal-Bench-2.0	Minimal prompt, tool, and policy edits accepted by regression testing (Zhang et al., 8 Jun 2026)
HarnessFix	Raw logs plus harness code	Scoped patches generated from HTIR-based flaw attribution (Chen et al., 4 Jun 2026)
HARBOR	Simulator trials and short RL runs	Bounded-stage RL workflow with commands, artifacts, gates, and reusable knowledge (Li et al., 7 Jun 2026)
HarnessBridge	Raw long-horizon histories and proposed actions	Learnable observation compression and action rejection controller (Wang et al., 11 Jun 2026)

These systems differ in their trial objects, but the conversion logic is structurally similar. A raw episode is not treated as a terminal score; it is compiled into an executable scaffold that changes future interaction. This suggests that “behavior” in this literature is often mediated behavior: not only what a model predicts, but how a harness permits, blocks, reshapes, or validates those predictions.

5. Evaluation criteria and empirical findings

In software testing, HarnessLLM evaluates generated harnesses with Good Input rate, Invalid Test Rate, and True Bug Rate, averaged over eight sampled harnesses per $H$7. The paper reports that RL-trained HarnessLLM with Qwen3-4B achieves TBR $H$8 on Codeforces versus $H$9 for an input–output RL baseline; on LiveCodeBench Unseen, TBR is $H:\{q,f\}\mapsto(\{x_i\},v_H)$0 versus $H:\{q,f\}\mapsto(\{x_i\},v_H)$1; unique-input ratio is $H:\{q,f\}\mapsto(\{x_i\},v_H)$2 versus $H:\{q,f\}\mapsto(\{x_i\},v_H)$3; input length range log-span is $H:\{q,f\}\mapsto(\{x_i\},v_H)$4 versus $H:\{q,f\}\mapsto(\{x_i\},v_H)$5; and test-time scaling raises the original pass@1 of Qwen3-32B on LiveCodeBench from $H:\{q,f\}\mapsto(\{x_i\},v_H)$6 to $H:\{q,f\}\mapsto(\{x_i\},v_H)$7 (Liu et al., 2 Nov 2025).

In deterministic agent settings, Life-Harness reports improvements in $H:\{q,f\}\mapsto(\{x_i\},v_H)$8 out of $H:\{q,f\}\mapsto(\{x_i\},v_H)$9 model–environment settings across $v_H(x,y)$0 model backbones, with an average relative improvement of $v_H(x,y)$1. Example benchmark results include AgentBench/ALFWorld Pass@1 from $v_H(x,y)$2 to $v_H(x,y)$3, $v_H(x,y)$4-bench/Retail Pass@1 from $v_H(x,y)$5 to $v_H(x,y)$6, and $v_H(x,y)$7-bench/Telecom Pass@1 from $v_H(x,y)$8 to $v_H(x,y)$9 (Xu et al., 21 May 2026). Self-Harness reports held-out pass-rate improvements from $y$0 to $y$1 for MiniMax M2.5, from $y$2 to $y$3 for Qwen3.5-35B-A3B, and from $y$4 to $y$5 for GLM-5 (Zhang et al., 8 Jun 2026).

In agent safety and execution compliance, AutoHarness reports $y$6 illegal-move success rate across $y$7 TextArena environments, with an average of $y$8 LLM-calls per game and a worst case of $y$9 calls for Chess. On $H:\{\tau\}\rightarrow\mathcal{I}$0 sampled two-player games, Gemini-2.5-Flash+Harness won $H:\{\tau\}\rightarrow\mathcal{I}$1 versus Gemini-2.5-Pro’s $H:\{\tau\}\rightarrow\mathcal{I}$2, and the code-only policy reached $H:\{\tau\}\rightarrow\mathcal{I}$3 average reward on $H:\{\tau\}\rightarrow\mathcal{I}$4 one-player games (Lou et al., 10 Feb 2026).

In harness evolution and diagnosis, AHE raises pass@1 on Terminal-Bench 2 from $H:\{\tau\}\rightarrow\mathcal{I}$5 to $H:\{\tau\}\rightarrow\mathcal{I}$6, surpassing Codex-CLI at $H:\{\tau\}\rightarrow\mathcal{I}$7, and reports frozen-harness cross-family gains of $H:\{\tau\}\rightarrow\mathcal{I}$8 to $H:\{\tau\}\rightarrow\mathcal{I}$9 percentage points (Lin et al., 28 Apr 2026). HarnessFix improves held-out performance from $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$0 to $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$1 on SWE-Bench Verified, from $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$2 to $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$3 on Terminal-Bench 2.0 Verified, from $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$4 to $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$5 on GAIA, and from $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$6 to $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$7 on AppWorld (Chen et al., 4 Jun 2026). RHO reports that one optimization round improves the pass rate on SWE-Bench Pro from $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$8 to $$\mathrm{Conv}_h:\mathcal{F}\times\mathbb{N}\rightarrow\mathcal{U}\times\mathbb{N},$$9 without external grading (Pan et al., 4 Jun 2026).

In runtime and systems benchmarks, Harness-Bench evaluates $R=\mathrm{Run}(M,H,T)$00 sandboxed offline tasks and $R=\mathrm{Run}(M,H,T)$01 execution trajectories, reporting a $R=\mathrm{Run}(M,H,T)$02 percentage-point gap between the best- and worst-performing harnesses and arguing that capability should be reported at the model × harness configuration level (Yao et al., 27 May 2026). HarnessBridge reports $R=\mathrm{Run}(M,H,T)$03 success on Terminal-Bench 2.0 at $R=\mathrm{Run}(M,H,T)$04 M tokens versus Terminus 2 at $R=\mathrm{Run}(M,H,T)$05 and $R=\mathrm{Run}(M,H,T)$06 M, and $R=\mathrm{Run}(M,H,T)$07 on SWE-bench Verified at $R=\mathrm{Run}(M,H,T)$08 M tokens (Wang et al., 11 Jun 2026). HARBOR reports $R=\mathrm{Run}(M,H,T)$09–$R=\mathrm{Run}(M,H,T)$10 success in simulation and $R=\mathrm{Run}(M,H,T)$11–$R=\mathrm{Run}(M,H,T)$12 on the real robot across $R=\mathrm{Run}(M,H,T)$13 simulator-task settings and $R=\mathrm{Run}(M,H,T)$14 backends (Li et al., 7 Jun 2026). HarnessX reports an average gain of $R=\mathrm{Run}(M,H,T)$15, up to $R=\mathrm{Run}(M,H,T)$16, across ALFWorld, GAIA, WebShop, $R=\mathrm{Run}(M,H,T)$17-Bench, and SWE-bench Verified (Chen et al., 12 Jun 2026).

6. Misconceptions, limitations, and open directions

A common misconception is that a harness is merely a prompt. The surveyed papers consistently use a broader definition. Harnesses include tools, schemas, runtime policies, mutable artifacts, middleware, memory, validators, permissions, observability hooks, and governance layers (Yao et al., 27 May 2026, Lin et al., 28 Apr 2026, Chen et al., 4 Jun 2026). Another misconception is that behavior change must come from updating model weights. Life-Harness, Self-Harness, AutoHarness, AHE, HarnessFix, RHO, and Sibyl all present harness-side improvement as a complementary or standalone alternative to model-centric training (Xu et al., 21 May 2026, Zhang et al., 8 Jun 2026, Wang et al., 21 May 2026).

The literature also imposes visible limits. AutoHarness synthesizes a separate harness per environment, and cross-game generalization remains future work (Lou et al., 10 Feb 2026). HarnessBridge evaluates only coding domains, reports one deterministic run per task, and uses static SFT rather than reinforcement learning (Wang et al., 11 Jun 2026). Life-Harness is evaluated on deterministic environments (Xu et al., 21 May 2026). HARBOR targets sim-to-real robot RL automation rather than general agent execution (Li et al., 7 Jun 2026). ProofAgent Harness is primarily an adversarial evaluation infrastructure, converting a TrialScenario into an evidence-linked report through multi-juror scoring, rather than directly repairing the tested agent (Bousetouane, 22 May 2026).

Several papers emphasize that final outcomes alone are insufficient. HarnessFix argues that broad or poorly scoped changes arise when one cannot diagnose where the responsible evidence lies in failed trajectories (Chen et al., 4 Jun 2026). AHE’s experience observability and decision observability are designed precisely to avoid collapsing into trial-and-error (Lin et al., 28 Apr 2026). Sibyl’s retrospective audit found eight high-confidence conversion events, with a median latency of one iteration and a maximum latency of three iterations, but explicitly states that these traces do not establish a comparative performance claim (Wang et al., 21 May 2026). This suggests that the field treats auditability, scoped repair, and regression control as constitutive parts of conversion rather than ancillary implementation detail.

Taken together, the literature describes trial-to-harness-behavior conversion as a shift from ephemeral trial evidence to executable, persistent interface structure. The converted artifact may be a verifier, a validator, a gate, a workflow, a controller, a file-level diff, or a model-conditioned set of runtime policies. What unifies these cases is the claim that reliable behavior emerges not only from better models, but from better mechanisms for turning trials into harnesses that subsequently govern action.