HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?

Abstract: As agents move from research prototypes to deployed tools, their capability increasingly depends on model-external execution infrastructure, commonly termed the agent harness. Changing this harness while holding model weights fixed can substantially alter task performance. Current agent evaluations typically report downstream performance under a chosen harness, leaving a model's ability to develop the harness itself comparatively underexplored. We introduce HarnessDev, a benchmark that shifts the unit of evaluation from task outputs to runnable infrastructure. HarnessDev covers two stages. In Creation, the agent starts from a minimal seed and a small number of cases, then builds a complete execution system. In Evolution, it starts from its own created harness and iteratively revises it using downstream execution feedback, with the goal of improving benchmark performance. We then evaluate each constructed harness on capability (task success on held-out benchmarks) and efficiency (execution-token cost). The reported Creation results cover six creator LLMs, four domains, and five downstream benchmarks totaling 2,207 unique downstream instances, with hidden evaluation tasks withheld from development. We find that generated harnesses remain substantially behind mature human-engineered references on code and on search and research, while matching or exceeding the selected references on writing and machine-learning experimentation, with large variation in execution cost. Evolution produces some performance gains, but they are unstable and transfer only partially to held-out tasks. Experiments with a fixed runtime model further show that the gains depend strongly on the model executing the harness, indicating limited transfer across models.

• The paper evaluates LLMs' ability to create and improve execution harnesses from a weak seed, showing variable performance across domains and models with minimum Creation avg@3 scores of 22.8 and maximum 67.8.
• Harnesses created by LLMs often fail due to incomplete control mechanisms and validation, with only 32.7% of failures attributable to insufficient domain reasoning.
• Feedback-driven evolution can improve harnesses, but the impact is often limited and unstable, and deployment costs can vary dramatically by executor-token consumption

The paper reframes agent evaluation around a component that conventional benchmarks usually hold fixed: the agent harness, namely the model-external software responsible for execution control, tool invocation, context construction, state, recovery, verification, and artifact production. Rather than evaluating only whether an LLM solves downstream tasks inside a predefined scaffold, HarnessDev evaluates whether an LLM can construct and subsequently improve the scaffold itself (2609.01437). This formulation is closely related to work on automated agent design and executable harness optimization, including “Automated Design of Agentic Systems” (Hu et al., 2024), “HarnessOpt-Bench” (Ursekar et al., 6 Aug 2026), and “Harness Updating Is Not Harness Benefit” (Lin et al., 28 May 2026), but differs in connecting from-scratch creation with feedback-driven evolution, separating creator and executor models, and measuring both capability and execution-token cost.

Evaluation object and benchmark design

HarnessDev defines a generated artifact as a frozen, runnable harness $H$ produced by a creator model in a development environment. After development, a separate executor model runs the harness on downstream tasks, and an evaluator scores the resulting task artifact. This separation is central: the creator is responsible for engineering the execution substrate, whereas the executor is responsible for using that substrate after it has been frozen. The benchmark therefore distinguishes model capability, harness quality, creator–harness co-adaptation, and executor compatibility.

The benchmark has two stages:

• Creation (RQ1): the creator starts from a deliberately weak but runnable seed and builds a complete harness using a task-family specification and one to three development cases.
• Evolution (RQ2): the creator starts from its own Creation harness and iteratively modifies it using downstream execution feedback.

The seed provides interface compatibility, passive tools, configuration parsing, logging, and artifact-writing support, but no agent loop, task decomposition, context management, persistent state, verifier, recovery policy, or stopping rule. Its unmodified score is zero on every downstream benchmark. This establishes a useful causal boundary: nonzero Creation performance must result from execution logic added by the creator rather than from reproducing a mature scaffold.

The benchmark evaluates four Creation domains across five downstream suites: SWE-bench Pro and Terminal-Bench 2.1 for code, MLE-bench for machine-learning experimentation, EQ-Bench3 for writing, and BrowseComp for research and retrieval. The Creation corpus contains 2,207 unique downstream instances. Six creator LLMs are evaluated: Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro. Each creator independently constructs three harnesses per creator–benchmark pair, and results are reported as avg@3 to reduce the influence of individual stochastic constructions.

The two executor regimes answer different questions. In the self-runtime condition, the creator also executes its harness, measuring model–harness co-design. In the fixed-runtime condition, every generated harness is executed by Gemini 3.1 Pro, isolating portability and reducing executor variation. Human-engineered systems are used as external references, but the paper correctly emphasizes that these references are not paired controls under a common executor. Consequently, exceeding a reference score does not establish superiority over human engineering or over the corresponding model in a controlled comparison.

The benchmark’s harness interface spans six functional responsibilities: execution, tools, context, state, lifecycle, and verification. The artifact must also emit auditable trajectories, result metadata, logs, and domain-specific final outputs. Constraint compliance is enforced through the scoring path rather than through self-reported harness status: SWE-Pro credit comes from the repository diff, Terminal-Bench credit from final environment state, and the recorded source and trajectories are audited for prohibited behavior.

Creation performance across domains

Creation demonstrates that current LLMs can construct nontrivial runnable harnesses, but performance is highly domain-dependent. Under self-runtime evaluation, Opus 4.8 obtains the highest unweighted average score, 67.8, compared with an external human-engineered reference of 86.2. The gap is not uniform: generated harnesses approach the selected writing reference and exceed the selected MLE-bench reference, while code and especially research/search remain substantially weaker.

The strongest reported self-runtime scores are:

Creator	SWE-Pro	Terminal-Bench	MLE-bench medal rate	EQ-Bench3	BrowseComp	Average
Opus 4.8	69.3	64.8	32.9	84.6	52.4	67.8
GPT-5.5	32.8	52.1	19.1	83.0	52.6	55.1
Gemini 3.1 Pro	43.6	68.8	32.4	74.8	35.2	55.6
DeepSeek V4 Pro	28.9	35.6	19.6	75.4	40.9	45.2
Qwen 3.7 Max	33.5	41.3	3.1	68.7	32.3	44.0
Seed 2.0 Pro	10.8	6.0	5.3	71.1	3.2	22.8
Human-engineered reference	80.0	88.8	24.0	83.7	92.2	86.2

The paper reports that 77.8% of failed MLE-bench tasks are attributable to harness defects. This is an important result because it rejects a purely executor-centric interpretation of failure: in this setting, many failures arise from orchestration, artifact handling, validation, or lifecycle control rather than from insufficient domain reasoning by the runtime model. At the same time, the comparison with human references must be interpreted cautiously because the reference systems use different harness–executor combinations.

Figure 1: Normalized distance between generated harnesses and selected human-engineered reference systems across downstream benchmarks.

The most substantial deficit occurs in BrowseComp, where long-horizon information seeking, query management, evidence synthesis, and termination control are required. Code also remains difficult because the harness must coordinate repository inspection, editing, test execution, patch generation, and final-state verification over many turns. By contrast, writing tasks impose less demanding execution-state requirements, and MLE-bench performance benefits from harnesses that can sustain long-running tool interactions and produce scorer-readable artifacts.

The implementation analysis reinforces that code volume is not a proxy for harness quality. The 18 Code harnesses add 17,111 net lines in aggregate, but Gemini’s harness adds only 1,006 lines while achieving the highest Terminal-Bench score, 68.8. The implication is that targeted control-flow changes and reliable verification matter more than architectural size. Similarly, the paper finds no meaningful relationship between self-test count and downstream score; the reported Spearman correlation is only 0.13–0.26 and is not significant. Revision calls, by contrast, reach a correlation of 0.57 with downstream score at $p \le 0.0005$, suggesting that testing becomes useful only when it is connected to diagnosis, targeted modification, and re-verification.

Architectural coverage and failure modes

The generated harnesses generally implement an explicit execution loop, but important control mechanisms are either incomplete or unreachable. Among 18 Code artifacts, tools, lifecycle control, and verification are complete in 13, 13, and 15 artifacts, respectively. State and memory are the clearest weakness: 11 artifacts define a state class, but only one exposes a state-saving interface and only one implements periodic checkpointing. No checkpoint event appears in 26,679 recorded task trajectories.

This distinction between declared mechanisms and active mechanisms is essential. Of 108 Code component instances, only 72 are observed on real execution paths; 18 have partial evidence and 18 are never observed. All unobserved instances concern state and memory. The same issue appears in other domains: 124 of 587 Writing features are classified as dead code, while 36 Data mechanisms lie on dead paths. The results indicate that architectural presence is not equivalent to behavioral participation.

Figure 2: Evidence density for execution, tools, context, state, lifecycle, and verification mechanisms across Creation and Evolution harnesses.

Validation is also weak. Of 2,325 executed Data tasks, 441 produce degenerate submissions that no harness detects. This finding directly links architectural incompleteness to downstream errors: a harness may call the executor repeatedly and produce syntactically valid output while failing to establish whether the final artifact is semantically usable.

Executor-specific assumptions produce another major failure mode. Hard-coded step limits, output budgets, message formats, and stopping rules can make a harness effective with its creator model but brittle under another executor. The most pronounced example is an Opus Code harness whose SWE-Pro score falls from 69.3 under Opus to 33.0 under Gemini. In the Opus Search harness, the duplicate-query rate increases from 10.1% to 88.2% after changing the executor. The harness is runnable under the new model, but its control policy is no longer compatible with the model’s behavior.

Figure 3: Harness portability under a fixed Gemini executor compared with self-runtime execution.

The fixed-executor results also show that portability is not uniformly negative. Qwen gains 17.6 points on BrowseComp and 12.9 points on MLE-bench under Gemini, indicating that its original executor can be a bottleneck. Thus, a common executor does not merely rank harnesses by intrinsic quality; it reveals interaction effects between harness design, prompt protocol, tool semantics, context policy, and runtime-model behavior. The paper’s stronger claim is that a runnable harness is not necessarily a transferable software asset.

Efficiency further complicates the capability comparison. MLE-bench token use varies by approximately nineteen-fold among creators. GPT-5.5 obtains a medal rate of 19.1 using 29.3 million executor tokens, whereas DeepSeek obtains 19.6 using 208.4 million tokens. Similar downstream quality can therefore entail radically different deployment costs.

Figure 4: Downstream performance as a function of executor-token consumption under self and fixed-executor evaluation.

Evolution under downstream feedback

Evolution begins from each creator’s frozen Code harness and exposes a 100-task SWE-Pro feedback set together with all 89 Terminal-Bench tasks. Creators may submit up to ten post-baseline evaluation pairs, with each official candidate required to complete both benchmark legs. They may also use at most two five-task probes between charged pairs. Every official version is subsequently evaluated on a disjoint 630-task SWE-Pro held-out split whose scores are never revealed during development.

This protocol separates online adaptation from generalization. All five self-runtime creators improve on the visible feedback pair, but improvements are smaller on held-out tasks. Opus obtains the largest self-runtime held-out improvement, +4.44 points. Under fixed Gemini, only Opus improves on held-out tasks; Qwen, DeepSeek, and GPT-5.5 regress.

Setting	Creator	Feedback pair gain	Held-out-630 gain
Self	Gemini 3.1 Pro	+8.8	+2.70
Self	Opus 4.8	+3.0	+4.44
Self	Qwen 3.7 Max	+13.9	+1.43
Self	DeepSeek V4 Pro	+13.4	+3.17
Self	GPT-5.5	+5.9	+3.81
Fixed Gemini	Opus 4.8	+9.7	+2.70
Fixed Gemini	Qwen 3.7 Max	+1.1	-1.11
Fixed Gemini	DeepSeek V4 Pro	+6.5	-2.38
Fixed Gemini	GPT-5.5	+2.4	-10.32

The result is not simply that evolution fails. Rather, it produces useful local changes but lacks reliable selection and transfer. Across 64 adjacent official-version switches, eight regress on both benchmarks, 16 regress on one benchmark, three create cross-benchmark trade-offs, seven show no measurable change, 27 produce gains within the repeated-run noise band, two show clear positive evidence beyond the noise band, and one contains no executable code change. A single commit can vary by approximately $\pm 4.75$ pair-score points, making small observed changes difficult to attribute causally.

Figure 5: Feedback-set trajectories across successive frozen harness versions for SWE-Pro and Terminal-Bench.

The declared final version is frequently not the held-out optimum. Feedback and held-out scores move in the same direction in only 34 of 64 comparable switches, or 53.1%, and only two of nine declared versions are optimal on the held-out set. All five self-runtime declarations improve over their respective baselines on held-out tasks, with gains from +1.43 to +4.44 points and a mean gain of +3.11. This advantage disappears under fixed Gemini for three of four creators. The immediate implication is that feedback optimization is useful for local search but unreliable for model-independent version selection.

Figure 6: Visible SWE-Pro-100 feedback trajectories overlaid with post-freeze SWE-Pro-630 held-out performance.

The edit statistics explain why evolution is unstable. Across official switches, 58 modify execution or control flow, 37 modify tools, 17 modify lifecycle recovery, 16 modify context, and only four modify state. No switch modifies a standalone verifier. Of 169 new functions or classes, 113 are reachable from the entry point, 31 are reachable only through dead code, and 25 have no caller. Evolution therefore often adds nominal functionality without establishing that it participates in the task path.

The paper gives a representative successful diagnosis from Opus: 99 of 100 runs reported success, but only 48 passed the actual verifier. Opus traced this discrepancy to premature completion and added a completion check. This is qualitatively different from indiscriminate code expansion because the change connects an observed failure pattern to an execution-path intervention and is verified end to end. Conversely, Qwen’s message sanitizer breaks valid Gemini tool-result sequences, demonstrating that a locally plausible reliability mechanism can damage executor compatibility.

Figure 7: Executor-token consumption across frozen Evolution versions under self and fixed-Gemini runtimes.

Limitations and open questions

The benchmark’s conclusions are constrained by several design limitations. Human references are heterogeneous system-level results rather than paired controls, so the reference gaps cannot be interpreted as clean causal estimates of human versus model-built harness quality. Creation uses six frontier models and four domains, but Evolution focuses on Code and provides only one trajectory per creator–runtime cell, preventing uncertainty estimates or population-level conclusions.

The held-out Evolution evaluation covers only SWE-Pro, while Terminal-Bench remains part of the visible feedback signal. Consequently, the benchmark does not establish whether Evolution generalizes across genuinely different task families. The 630-task held-out split is substantial, but it is drawn from the same public SWE-Pro distribution as the 100-task feedback set, so the result is a within-benchmark generalization test rather than a cross-domain transfer test.

The fixed development environment is another important assumption. The creator always operates through the prescribed external workbench; the evolved harness is not itself used as the development environment for later evolution. Whether recursive harness development can remain stable when the evolving system must also provide its own engineering interface is therefore unresolved. Finally, HarnessDev measures model-external adaptation and explicitly does not claim that heuristic harness evolution replaces parameter training.

Conclusion

HarnessDev establishes runnable agent infrastructure as an explicit research object. Its Creation experiments show that LLMs can construct functional harnesses from a weak seed, but that performance depends strongly on domain, execution cost, architectural reachability, and creator–executor compatibility. Its Evolution experiments show that downstream feedback can produce real local improvements, while also exposing instability, noisy version selection, weak held-out transfer, and substantial dependence on the runtime model.

The central empirical conclusion is that harness development is not reducible to code generation or task solving. Effective development requires diagnosing execution traces, identifying structural failure modes, modifying the control substrate, and verifying that the modification remains active and compatible with the executor. HarnessDev provides a concrete protocol for measuring those capabilities and shows that current models possess them only inconsistently (2609.01437).