Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement

Abstract: This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/

• The paper introduces Harness-of-Harness (HoH), a framework that enhances autonomous software development through continual, evidence-integrated improvement over multiple iterations, by repeatedly inviting planning, development, and QA stages.
• The HoH architecture outperforms Vanilla implementations on GameCraft-Bench, FrontierSWE, and ProgramBench with absolute mean score gain to 72% on GameCraft-Bench.
• The study highlights the importance of integrating planning, development, and QA roles and traces through benchmarks over three coding passes, revealing continuous improvements on crucial areas and project scopes.

The paper presents Harness-of-Harness (HoH) as an orchestration layer for autonomous software development with LLM-based coding agents. Its central premise is that long-horizon development is not adequately addressed by extending a single coding-agent episode. As the project evolves, the system must preserve validated behavior, retain unresolved defects, revise priorities in response to new evidence, and distinguish capability expansion from regression repair. HoH addresses these requirements by repeatedly invoking a fixed harness–model configuration in three roles: Project Planner, Developer, and QA Tester (2609.01481).

Problem setting and research objective

The paper distinguishes conventional human-in-the-loop coding from autonomous software development. In the former, an agent generates or modifies code while humans continuously supply task decomposition, design guidance, review, and intervention. In the latter, the system receives only high-level requirements and must independently produce a complete, functional, and deployable software artifact.

Figure 1: Human-in-the-loop development compared with autonomous software development.

This distinction changes the technical problem. Repository-level issue resolution can often be framed as a bounded repair task, whereas greenfield development requires sustained translation between requirements, architecture, implementation, execution behavior, and quality assessment. A long trajectory creates several failure modes: requirements may be forgotten, locally valid changes may violate global constraints, previously repaired behavior may regress, and an agent may repeatedly inspect or repair the same subsystem without increasing overall completeness. The paper therefore treats autonomous development as a state-maintenance and evidence-integration problem rather than simply a problem of allocating more inference time.

HoH operates on existing coding-agent harnesses rather than replacing or retraining them. The model, underlying harness, role contracts, and runtime policy remain fixed during a run. What changes across iterations are the artifact, the development document, and the accumulated execution evidence. The claimed contribution is consequently architectural: HoH supplies persistent cross-iteration coordination and independent verification around a standard coding agent.

HoH architecture

A HoH iteration consists of planning, implementation, and quality assurance. The Project Planner receives the original specification, the previous evidence state, and read-only access to the current artifact. It selects a bounded but locally complete objective, identifies functionality that must be preserved, and specifies observable acceptance conditions. The Developer then modifies the existing artifact under a single-writer boundary. Finally, the QA Tester receives a frozen, read-only copy of the resulting artifact and independently evaluates it against the specification and the current development document.

Figure 2: HoH repeatedly invokes planning, development, and independent QA around an evolving artifact under deterministic runtime control.

The role decomposition is important because the three stages make different claims. Planning determines what should change next; development determines how to realize the change; QA determines whether the resulting behavior is actually present. HoH prevents these decisions from collapsing into a single self-reporting invocation. In particular, the Developer’s local tests and completion claims are not treated as acceptance evidence.

The runtime constrains inputs, permissions, write access, and output schemas while leaving the agent free to choose its internal reasoning process, tools, and implementation strategy. This design attempts to constrain verifiable outcomes without prescribing a rigid workflow. Structured outputs that violate the role schema trigger retries. The Developer performs shift-left testing through baseline–change–retest cycles, while QA combines black-box execution with white-box inspection of source, configuration, runtime state, logs, resources, and rendered output.

HoH maintains two distinct cross-iteration state channels:

• Artifact state: source code, configuration, assets, metadata, and the current executable project.
• Evidence state: verified behaviors, unresolved gaps, regressions, insufficiently supported claims, and execution records.

This distinction is more than an implementation detail. The artifact records what the project currently is, but not why a feature was prioritized or which behaviors have already been verified. Evidence records project knowledge, but cannot substitute for the implementation itself. The next planner uses both channels to choose work that addresses outstanding defects while preserving validated behavior.

The framework also uses progressive disclosure rather than a dedicated memory module. Development documents, QA reports, histories, and evidence packets are persisted to the filesystem and exposed initially through concise indexes. Detailed content is retrieved only when relevant. In the open-ended case study, the system additionally uses GitHub commits, issue histories, test records, and domain-specific skills for Godot development, asset handling, UI design, and interactive testing.

Benchmark evaluation

The benchmark study evaluates HoH on GameCraft-Bench, FrontierSWE, and ProgramBench using three harness–model configurations: Codex with GPT-5.5 at high reasoning effort, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3. Vanilla denotes one ordinary development pass, whereas HoH@1–3 denotes one to three complete planning–coding–testing iterations. The main comparison uses 45 stratified GameCraft-Bench tasks, 15 FrontierSWE tasks, and the reported ProgramBench evaluation protocol.

The aggregate results show consistent improvements across all configurations and benchmarks.

Configuration	GameCraft-Bench Vanilla → HoH@3	FrontierSWE Vanilla → HoH@3	ProgramBench Vanilla → HoH@3
Codex + GPT-5.5	49.58 → 71.52	0.31 → 0.54	60.41 → 66.50
OpenCode + DeepSeek-V4-Pro	26.90 → 48.98	0.23 → 0.31	45.27 → 57.56
Pi + MiniMax-M3	42.16 → 58.78	0.26 → 0.55	35.83 → 52.68

On GameCraft-Bench, the absolute gains after three iterations are 21.93 points for Codex, 22.08 for OpenCode, and 16.62 for Pi. On FrontierSWE, the reported gains are 0.23, 0.08, and 0.29, respectively; Pi obtains the largest improvement. On ProgramBench, HoH raises average hidden-test pass rates by 6.09, 12.29, and 16.85 percentage points. The improvement is therefore not restricted to the strongest baseline system. HoH also substantially improves the weakest starting configuration, although the final ranking remains configuration-dependent.

The GameCraft-Bench result is especially informative because the overall score incorporates Core Mechanics, Content Depth, Functional Visuals, and Art and Presentation, with non-runnable artifacts receiving zero. HoH@3 improves all four dimensions for all three harness–model pairs. For Codex, Functional Visuals increases from 48.67 to 74.23 and Art and Presentation from 45.28 to 65.28. This indicates that the protocol does not merely increase compile or smoke-test success; its gains extend to content completeness, player-facing feedback, and presentation quality.

Figure 3: HoH@3 improves mechanics, content, functional visuals, and presentation across all three harness–model configurations.

The task-level results show that improvement is broad but not universal at the individual-task level. For example, OpenCode’s HoH@3 score decreases on Autobattler relative to Vanilla, and Pi’s score decreases on several tasks including Spire Descent, Pipe Crisis, Space Colony, and Border Check. Thus, the claim supported by the experiments is an improvement in unweighted aggregate performance, not monotonic improvement on every task or every iteration.

The qualitative comparisons illustrate the nature of these gains. In Momentum Lab, HoH produces clearer terrain and visual cues for the intended wall-jump route. In Kitchen Rush, the resulting project exposes a complete workflow involving pickup, preparation, plating, and disposal. In Ant Empire, the project includes specialist caste counts, seasonal state, and outcome state that make longer-term progression legible. The corresponding Codex scores rise from 34.05 to 70.61, 42.62 to 73.38, and 65.52 to 87.88.

Figure 4: HoH produces more complete gameplay loops and more legible player-facing state than the Vanilla baseline.

The FrontierSWE extended-run experiment examines whether the effect persists beyond three loops. With Codex and GPT-5.5, HoH continues to improve over ten iterations. The paper reports a dominance score of 72.67% at HoH@10, compared with 27.33% for Vanilla, while the best checkpoint reaches 76.00% at HoH@9. The comparison pool includes Vanilla and HoH@1–10, so this metric is not equivalent to an absolute task-success rate; it measures average pairwise performance against the evaluated checkpoints. Nevertheless, the result supports the narrower claim that additional evidence-guided loops can continue improving performance on these tasks rather than causing immediate degradation.

Controlling for additional development passes

A central methodological question is whether HoH’s gains arise merely from running the coding agent more than once. The paper addresses this with Vanilla Continuation, which performs two or three sequential coding passes using the same harness and model but without separate planning and independent QA.

On GameCraft-Bench with Codex and GPT-5.5, the results are:

Method	Development passes	Mean score	Mean tokens per task
Vanilla	1	49.58	2.59M
Vanilla Continuation	2	54.99	4.56M
Vanilla Continuation	3	58.24	6.33M
HoH	1	59.71	2.88M
HoH	2	64.84	5.67M
HoH	3	71.52	8.41M

HoH outperforms Vanilla Continuation at every matched pass count. After three passes, the advantage is 13.28 points. More strongly, HoH@2 reaches 64.84 with 5.67 million tokens, exceeding the 58.24 obtained by three-pass Vanilla Continuation with 6.33 million tokens. The reported score gained per additional million tokens is 3.77 for HoH versus 2.32 for three-pass Vanilla Continuation.

These results support the paper’s claim that the planning–coding–testing structure contributes beyond simply increasing inference volume. They do not, however, isolate every possible source of the advantage. HoH invokes three role-specific calls per iteration and supplies additional structured context, so the comparison establishes the benefit of the full protocol rather than attributing the gain to one particular component in isolation.

Ablation analysis

The ablations remove one cross-iteration mechanism at a time while preserving the three-iteration budget. The variants are:

• Without plan update: reuses the initial development document in later iterations.
• Without evidence feedback: replans without the preceding QA evidence.
• Without warm-start: reconstructs the artifact from the initial workspace at every iteration.

Using Codex with GPT-5.5 on GameCraft-Bench, full HoH achieves 71.52. Removing plan updates reduces the score to 63.39, removing evidence feedback reduces it to 65.23, and removing warm-start reduces it to 63.67. The warm-start ablation also increases token use from 8.41M to 11.12M per task because the system repeatedly reconstructs project functionality.

Figure 5: Removing plan revision, evidence feedback, or artifact continuity reduces final quality; removing warm-start also increases token use.

The ablations support three distinct mechanisms. Plan revision allows the development objective to change as the artifact and failure profile evolve. Evidence feedback allows QA observations to influence subsequent prioritization. Warm-start allows validated implementation to accumulate rather than being recreated. The fact that all three variants underperform full HoH indicates that the observed gain is not attributable solely to persistent code or solely to iterative prompting.

The paper also reports token distributions for individual harness invocations. These distributions should be compared only within each harness–model configuration because providers differ in cache accounting and token-reporting conventions.

Figure 6: Per-invocation token distributions across GameCraft-Bench tasks, reported using provider-specific accounting.

Multi-day autonomous development case study

The open-ended case study evaluates whether HoH can sustain coherent project evolution over a much longer trajectory than the controlled benchmark runs. Starting from an empty workspace and a product requirements document, the system develops Fusepoint, a single-player narrative first-person shooter in Godot. The specification requires a five-minute bomb-defusal mission, sequential capture of two control points, a three-stage final defusal, 18 enemies distributed across three encounter regions, and distinct success and detonation branches.

The case uses Codex CLI with GPT-5.6-Sol at high reasoning effort, although the benchmark experiments use GPT-5.5. Human intervention is limited to restoring network or API availability; planning, coding, debugging, testing, and acceptance are performed by HoH. This difference in model version and the addition of domain-specific tools mean that the case study should not be interpreted as a direct replication of the benchmark condition.

Figure 7: Across more than 70 iterations, HoH develops a playable FPS with integrated mechanics, narrative progression, interface systems, visual assets, animation, and audio.

The system equips the roles with Godot MCP tools, asset retrieval and generation skills, UI/UX guidance, testing procedures, and project-state management. The project is versioned after each agent stage, and QA records are stored as issue tables and evidence packets. The development trajectory exhibits three phases: initial construction, capability expansion, and stabilization. Early feature additions increase the issue backlog because newly executable behavior exposes defects and missing requirements. Later iterations shift toward repair and regression management.

After 70 loops, 65 of 81 recorded issues have been closed, 16 remain unresolved, and 17 issues have been reopened after earlier closure. The non-monotonic issue history is significant: it demonstrates that the system does not simply accumulate features while assuming prior functionality remains valid. Reopened issues preserve the link between a regression and its earlier verification history, allowing subsequent planning to treat regression repair as explicit work.

The resulting game is described as human-playable and includes a coherent storyline, combat, weapon and enemy interactions, player guidance, HUD and menus, cinematic animation, visual presentation, and integrated audio. This is a strong systems-level demonstration of sustained artifact evolution. It is not, by itself, a controlled measurement of product quality against a human-developed baseline, and the case does not establish that every requirement was satisfied: 16 issues remained unresolved at the reported cutoff.

Limitations and open questions

The experimental evidence has several limitations. Each task–condition pair is represented by one valid run, and the selected clients do not provide a common reproducible generation seed. Consequently, the reported means do not quantify variance across independent generations of the same task. The main benchmark evaluation also uses relatively small subsets: 45 of 140 GameCraft-Bench tasks and 15 of 17 FrontierSWE tasks. Two FrontierSWE tasks are excluded for execution-environment reasons, and the FrontierSWE sample contains only two Research tasks, making category-level conclusions sensitive to individual outcomes.

The evaluation protocol also couples HoH to the particular role prompts, runtime contracts, evidence schemas, and harness–model configurations implemented by the authors. The ablations establish that the complete protocol is useful, but they do not determine whether planner quality, QA reliability, context organization, role separation, or the structured artifact schema is the dominant causal factor. In addition, QA is performed by another invocation of the same underlying model–harness configuration. This provides procedural independence and read-only isolation, but not independence in the statistical or model-family sense. A systematic evaluator bias or shared blind spot could therefore persist across development and assessment.

The open-ended case study provides traceability and a substantial 70-loop trajectory, but it is a single project in one engine and one application domain. Its result is also not directly comparable with the benchmark results because it uses GPT-5.6-Sol, additional tools, specialized skills, external assets, and a different evaluation procedure. The case leaves open whether the same state-management protocol can maintain correctness in non-game software with stronger semantic, security, compatibility, and performance requirements. It also leaves open how iteration budgets should be selected, when continued loops begin to cause regressions, and how evidence quality should be calibrated when requirements are ambiguous or difficult to observe.

Conclusion

HoH frames autonomous software development as continual, evidence-grounded artifact evolution. Its principal mechanism is a persistent loop that separates objective selection, implementation, and acceptance while carrying both the evolving artifact and structured execution evidence across iterations. Across three benchmarks and three harness–model configurations, HoH@3 consistently improves aggregate quality over Vanilla, with GameCraft-Bench gains of 16.62–22.08 points, FrontierSWE gains of 0.08–0.29, and ProgramBench gains of 6.09–16.85 percentage points. Matched-pass comparisons and ablations indicate that these gains cannot be explained solely by repeated coding or additional token expenditure. The multi-day Fusepoint deployment further demonstrates that the framework can preserve, revise, and repair a complex interactive project over more than 70 autonomous loops, while also making clear that substantial unresolved issues remain at the reported endpoint (2609.01481).