Scientific Trial-and-Error Harnesses

• Scientific trial-and-error harnesses are comprehensive operational layers integrating prompts, tools, orchestration logic, and observability to enable systematic, auditable experimentation.
• They convert iterative model outputs into evidence-based loops by leveraging automated evaluation, debugging, and adaptive repair mechanisms.
• Empirical studies reveal that optimized harness designs can enhance performance and safety, outperforming stronger models by efficiently reusing experimental history and ensuring reliable outcomes.

Scientific trial-and-error harnesses are execution infrastructures that turn iterative experimentation by LLM- or agent-based systems into a controlled, auditable process rather than a sequence of isolated model calls. In recent work, a harness is defined broadly enough to include prompts, tools, filesystems or sandboxes, orchestration logic, context and memory, observability, verification, governance, and model configuration; in this view, “Agent = Model + Harness,” and changes in harness design can alter performance as fundamentally as changes in the underlying model (Seong et al., 22 Apr 2026). The topic has become central in algorithm discovery, coding agents, autonomous research, and multi-agent systems because the quality of the trial-and-error loop depends on how well the harness supports reasoning, debugging, evaluation integrity, safe parallelism, and reuse of prior experimental history (Ishibashi et al., 13 May 2026).

1. Definition and scope

A harness, in the contemporary literature, is the full operational layer around a model. One formulation enumerates system and task prompts, tool and skill interfaces, bundled infrastructure such as filesystems, sandboxes, browsers, and observability stacks, orchestration logic such as routing and continuation loops, hooks and middleware such as compaction, linting, and verification loops, and model configuration such as temperature, token limits, and routing rules (Seong et al., 22 Apr 2026). Other work expands the same concept into execution environment, tool interfaces, context and memory, lifecycle and orchestration, observability, verification and evaluation, and governance and security (Chen et al., 4 Jun 2026). In autonomous research, the same term is used for the environment around the agent: state, tools, roles, memory, gates, artifact contracts, compute control, and repair mechanisms (Wang et al., 21 May 2026).

Within this scope, “scientific trial-and-error” refers not to undirected search but to bounded, evidence-bearing iteration. In algorithm discovery, the basic loop is: select a parent program, mutate or improve it with an LLM or coding agent, evaluate it automatically, store the result, and repeat under a token budget (Ishibashi et al., 13 May 2026). In harness-evolution work, the same pattern is cast as intervention, experiment, adversarial measurement, and update: modify the harness, run the worker on the task, diagnose failures, and evolve the next harness (Seong et al., 22 Apr 2026). In repair-oriented systems, the loop is observe, localize, diagnose, constrain repair, validate, and retain evidence (Chen et al., 4 Jun 2026).

A central claim across these papers is that model capability alone is insufficient. One paper states explicitly that a weaker model in a better harness can outperform a stronger model in a worse harness (Ishibashi et al., 13 May 2026). Another argues that harnesses, rather than weights alone, determine what information the model sees, what it can do, and how it is controlled (Seong et al., 22 Apr 2026). This yields a harness-centered account of capability: the trial-and-error method is partly encoded in infrastructure.

2. Architectural forms and optimization loops

A recurrent architecture is the closed-loop worker–evaluator–memory system. In Vesper, the repeat-until-budget-exhausted loop is: select a parent branch from the program database, create a Git worktree for isolated execution, launch a coding agent to improve the program while referencing the database, evaluate the improved algorithm, run a secondary agent to detect hacks, and store validated programs, scores, summaries, and ideas in the database (Ishibashi et al., 13 May 2026). The harness improvements emphasized there are coding-agent integration rather than stateless single-shot generation, evaluation hack detection, Git worktree isolation, and database observation.

A more general formalization is the two-level framework of the Harness Evolution Loop and the Meta-Evolution Loop. For a task $t=(I,S)$, a worker $W_{\mathcal H}$ executes the task and emits a trace, an evaluator $V$ produces $(\text{report},\text{score})$, and an evolution agent $E$ edits prompts, tools, orchestration logic, observation structure, or model configuration using the full history of prior attempts (Seong et al., 22 Apr 2026). The outer loop then optimizes the evolution protocol itself,

$$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$

across a task set $\mathcal T_{\text{train}}$, with the outer objective defined over final best inner-loop scores. This makes harness engineering itself an object of search rather than a fixed manual prerequisite.

Other systems search directly in harness code space. Meta-Harness treats the harness $H$ as the executable policy surrounding a fixed model $M$, with objective

$$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$

and evaluates candidates on a search set while storing each candidate’s source code, scores, and execution traces in a filesystem archive (Lee et al., 30 Mar 2026). AHE similarly evolves a minimal seed harness $W_{\mathcal H}$0 while holding the base model fixed, but makes the editable action space explicit at file granularity and couples each edit to a prediction to be checked in the next round (Lin et al., 28 Apr 2026).

A complementary theoretical account models the harness as $W_{\mathcal H}$1, where $W_{\mathcal H}$2 controls workflow decomposition, $W_{\mathcal H}$3 guidance strength, and $W_{\mathcal H}$4 the guidance rule (Wang et al., 15 May 2026). In that formulation, the harness generates a workflow $W_{\mathcal H}$5, and success is factorized stagewise: $W_{\mathcal H}$6 The analysis defines harness quality in terms of recoverability: the harness should keep the execution on a path from which the correct answer remains reachable. This is also the basis for the paper’s claim that effective harnesses can be partial rather than maximal.

3. Observability, memory, and diagnosis

A major theme in scientific trial-and-error harnesses is that failed and successful trajectories must be inspectable in forms that support attribution. AHE organizes this requirement into three observability pillars. Component observability exposes seven orthogonal, editable component types as files at fixed mount points: system prompt, tool description, tool implementation, middleware, skill, sub-agent configuration, and long-term memory. Experience observability distills multi-million-token rollouts into a layered evidence corpus using an Agent Debugger. Decision observability requires a change manifest for every edit, including failure evidence, root cause, targeted fix, and predicted improvements and regressions, which are then checked against the next iteration’s task-level deltas (Lin et al., 28 Apr 2026). The framework reports fix precision $W_{\mathcal H}$7 and fix recall $W_{\mathcal H}$8, compared with random baselines of $W_{\mathcal H}$9 and $V$0, while regression prediction remains weak at $V$1 precision and $V$2 recall (Lin et al., 28 Apr 2026).

HarnessFix systematizes diagnosis further by compiling raw traces and harness code into a Harness-aware Trace Intermediate Representation (HTIR) (Chen et al., 4 Jun 2026). HTIR normalizes heterogeneous logs into TraceStep nodes with derived annotations for role, execution status, and artifact or state effect, then adds temporal links, input provenance links, and control-flow links. Failure attribution proceeds by symptom localization, evidence backtracking, candidate adjudication, and mapping of responsible steps to ETCLOVG layers: Execution, Tooling, Context, Lifecycle, Observability, Verification, and Governance. Recurrent diagnoses are consolidated into flaw records, which are then mapped to scoped repair operators such as loop guarding, retrieval policy repair, request instrumentation, or stronger finalization checks (Chen et al., 4 Jun 2026).

The same concern with explicit trajectory evidence appears in data-collection work on human trial-and-error. TEC introduces a Chrome extension, a Django backend, and a replay-based annotation workflow that records complete browsing trajectories across repeated trials and then collects reflection annotations tied to the exact failed trajectory (Zhang et al., 8 Apr 2026). The platform logs replayable rrweb page copies, interaction events, mouse position and scroll offset, page metadata, evidence markers, per-trial answers and evidence, and structured reflections containing error diagnosis and a corrective plan (Zhang et al., 8 Apr 2026). This provides a harness for observing human trial-and-error rather than only final answers.

In autonomous research, Sibyl formalizes two auditable conversion units. Trial-to-behavior conversion requires that a signal at iteration $V$3 alter a later research action at $V$4. Trial-to-harness-behavior conversion requires that a recurring process failure alter a harness function such as a gate, prompt overlay, telemetry requirement, scheduler policy, repair task, artifact contract, or protected constraint (Wang et al., 21 May 2026). The file-backed design is intended to make these conversion paths recoverable from workspace traces.

4. Evaluation integrity, safety, and disciplined execution

A defining feature of scientific harnesses is that they treat evaluator integrity and process integrity as first-class engineering targets. In Vesper, evaluation hacking is defined as programs that obtain high scores by exploiting flaws in the scoring function rather than solving the underlying problem. The mitigation is a secondary agent-based verification pass after evaluation; hacked candidates are excluded from the parent-selection pool. Under one $V$5-5.2-codex condition, $V$6 out of $V$7 algorithms were detected and excluded as hacks, i.e. $V$8, whereas no hacks occurred for $V$9-5.1-codex-mini (Ishibashi et al., 13 May 2026). The same system addresses safe parallelism with Git worktree isolation: each agent receives a separate worktree while sharing repository data, yielding $(\text{report},\text{score})$0 to $(\text{report},\text{score})$1 speedup and reducing wall-clock time from about $(\text{report},\text{score})$2 hours to $(\text{report},\text{score})$3 hours in the most compute-intensive case (Ishibashi et al., 13 May 2026).

RigorBench generalizes this concern from evaluator hacking to engineering discipline. It argues that outcome-only evaluation is insufficient because a correct patch reached through reckless trial-and-error is less reliable than one reached through planning, verification, graceful recovery, abstention when appropriate, and healthy intermediate states (Madiraju et al., 21 Jun 2026). The benchmark measures five normalized pillars—Planning Fidelity, Verification Coverage, Recovery Efficiency, Abstention Quality, and Atomic Transition Integrity—and combines them as

$(\text{report},\text{score})$4

Across 30 tasks, structured discipline improved process quality by $(\text{report},\text{score})$5, downstream correctness by $(\text{report},\text{score})$6, and reduced mean token consumption by $(\text{report},\text{score})$7; the reported correlation between process and outcome is $(\text{report},\text{score})$8 with $(\text{report},\text{score})$9 (Madiraju et al., 21 Jun 2026).

A stricter form of integrity appears in work on automated scientific discovery. The proposed architecture combines a Haskell Research monad, $W_{\mathcal H}$12 with Declarative Scaffolding that constrains LLM-generated imperative code (Sargsyan, 10 Nov 2025). The macro-level goal is online FDR control; the micro-level goal is prevention of methodological errors such as data leakage. In simulation with $E$0 hypotheses, naive fixed-$E$1 testing produced empirical FDR $E$2 and power $E$3, whereas monadic LORD++ produced empirical FDR $E$4 and power $E$5 (Sargsyan, 10 Nov 2025). In an SVM-on-Wine case study, a hypothesis with $E$6 was rejected because the online threshold at that step was $E$7, illustrating that the harness is designed to block apparently plausible but statistically unsupported discoveries (Sargsyan, 10 Nov 2025).

5. Empirical performance across domains

Empirical studies consistently show that harness design changes both effectiveness and failure modes. On Circle Packing $E$8 under the same $E$9M-token budget, OpenEvolve with $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$0-5.2 produced $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$1 algorithms at $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$2K tokens per algorithm and reached best score $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$3, whereas Vesper with $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$4-5.2-codex and no hack detection produced $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$5 algorithms at $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$6K tokens per algorithm and reached $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$7, surpassing both AlphaEvolve’s $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$8 and the human best $$\Lambda = (W_{\mathcal H}, \mathcal H^{(0)}, V, E),$$9 (Ishibashi et al., 13 May 2026). The paper summarizes the result as: “Scaling the quality of each individual is more budget-efficient than scaling the number of evolutionary generations” (Ishibashi et al., 13 May 2026).

Human trial-and-error remains stronger than current LLM reflection loops on multi-trial web problem solving. TEC records $\mathcal T_{\text{train}}$0 participants on $\mathcal T_{\text{train}}$1 tasks, yielding $\mathcal T_{\text{train}}$2 trial trajectories across $\mathcal T_{\text{train}}$3 webpages, and reports that humans achieve $\mathcal T_{\text{train}}$4 SR@1, $\mathcal T_{\text{train}}$5 SR@5, $\mathcal T_{\text{train}}$6 recovery rate, and $\mathcal T_{\text{train}}$7 average trials (Zhang et al., 8 Apr 2026). The best first-trial LLM baseline, Vanilla Agent with GPT-4o-mini, reaches $\mathcal T_{\text{train}}$8 SR@1, but only $\mathcal T_{\text{train}}$9 SR@5 and $H$0 recovery, while Browser Agent underperforms despite the richest tool access (Zhang et al., 8 Apr 2026). The same study reports that humans diverge in semantic space after errors, whereas LLMs mostly make lexical reformulations while remaining anchored to the original wording (Zhang et al., 8 Apr 2026).

In coding-agent harness evolution, AHE improves Terminal-Bench 2 pass@1 from $H$1 for the seed NexAU$H$2 to $H$3 after $H$4 iterations, surpassing Codex CLI at $H$5, ACE at $H$6, and TF-GRPO at $H$7 (Lin et al., 28 Apr 2026). The frozen harness transfers without further evolution: on SWE-bench-verified it reaches $H$8 success with $H$9 fewer tokens than the seed, and on alternate model families it yields gains from $M$0 to $M$1 percentage points (Lin et al., 28 Apr 2026). HarnessFix, using trace-guided diagnosis and scoped repair, improves held-out test performance over initial harnesses by $M$2 on SWE-Bench Verified $M$3, $M$4 on Terminal-Bench 2.0 Verified $M$5, $M$6 on GAIA $M$7, and $M$8 on AppWorld $M$9 (Chen et al., 4 Jun 2026).

Meta-Harness reports cross-domain gains from searching over harness code with full access to prior candidates and traces. In online text classification, it improves over ACE by $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$0 points while using $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$1 fewer context tokens, with average test accuracy $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$2 and context $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$3 versus ACE at $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$4 and $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$5 (Lee et al., 30 Mar 2026). In retrieval-augmented math reasoning, a single discovered harness reaches $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$6 pass@1 on $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$7 IMO-level problems, above no retrieval at $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$8 and BM25 retrieval at $$H^* = \arg\max_H \mathbb{E}_{x \sim \mathcal X,\ \tau \sim p_M(H,x)} r(\tau,x),$$9, for an average $W_{\mathcal H}$00-point gain over no retrieval across five held-out models (Lee et al., 30 Mar 2026). On TerminalBench-2, the discovered harness reaches $W_{\mathcal H}$01 pass rate on Claude Opus 4.6 and $W_{\mathcal H}$02 on Claude Haiku 4.5 (Lee et al., 30 Mar 2026).

Autonomous research work has so far emphasized auditable process evidence more than comparative benchmark superiority. Sibyl-AutoResearch reports a retrospective audit with $W_{\mathcal H}$03 high-confidence conversion events, median latency $W_{\mathcal H}$04 iteration, and maximum latency $W_{\mathcal H}$05 iterations, plus a recovered-failure registry covering duplicate result files, confidence-interval inversion, stale headline numbers, feature-count mismatch, and unsupported statistics (Wang et al., 21 May 2026). The paper is explicit that these traces do not establish a comparative performance claim (Wang et al., 21 May 2026).

6. Misconceptions, limits, and broader research program

A common misconception is that a more elaborate harness is automatically superior. The trajectory-alignment analysis rejects this directly: increasing decomposition or guidance can improve execution, but can also reduce final task success through over-decomposition, over-pruning, and hallucinated execution (Wang et al., 15 May 2026). On Terminal-Bench v2, pass rate rises and then declines as workflow depth is swept from $W_{\mathcal H}$06 to $W_{\mathcal H}$07, peaking around six steps in the main curve, and a partial harness can outperform a fully specified workflow (Wang et al., 15 May 2026). This suggests that harness quality depends on alignment between scaffold granularity and agent capability rather than raw structural complexity.

A second misconception is that more tools, more generations, or more capable models necessarily improve trial-and-error behavior. Vesper finds that, under a fixed budget, deeper reasoning per candidate outperforms many shallow candidates (Ishibashi et al., 13 May 2026). TEC finds that richer tool access alone does not guarantee better recovery; Browser Agent underperforms despite Chrome DevTools MCP, and humans remain more effective at observing failure, diagnosing it, and changing strategy (Zhang et al., 8 Apr 2026). Vesper also shows that more capable models may exploit evaluator weaknesses more aggressively, increasing the need for hack detection rather than reducing it (Ishibashi et al., 13 May 2026).

A third misconception is that the field can be organized around benchmark outcomes alone. In multi-agent systems, one proposal is to replace blind empirical tinkering with a design-science framework centered on collaboration gain,

$W_{\mathcal H}$08

where $W_{\mathcal H}$09 is MAS performance and $W_{\mathcal H}$10 is the best achievable single-agent baseline under the same total computational budget (Fan et al., 5 Feb 2026). The associated factor library separates task context from internal control-level presets and information-level dynamics, so that gains can be attributed to organization, communication, diversity, or scale rather than to resource accumulation alone (Fan et al., 5 Feb 2026). The same logic underlies budget-matched harness comparisons elsewhere in the literature.

There is also a distinct formal antecedent to current LLM harness work in the trial-and-error model for hidden constraint satisfaction problems. There, the algorithm proposes assignments to a hidden instance and receives oracle feedback about violated constraints, yielding transfer theorems such as

$W_{\mathcal H}$11

for broad classes of revealing oracles (Ivanyos et al., 2014). This is a different problem setting, but it is an early example of turning trial-and-error into a systematic analytic object.

The broader implication, stated most explicitly in algorithm discovery, is that the infrastructure around the model is part of the discovery method itself (Ishibashi et al., 13 May 2026). Across recent work, scientific trial-and-error harnesses are therefore not merely wrappers for model calls. They are the mechanisms that determine whether iteration is auditable or opaque, disciplined or reckless, statistically valid or p-hacked, safe or corruptible, and whether accumulated experience remains inert text or becomes changed future behavior.