Harness Scaling in Agent Systems

• Harness scaling is the treatment and optimization of the structured execution layer—including prompts, memory, and skill routing—as a central design element, not just an afterthought.
• It emphasizes module-specific scaling where components like memory, context, and orchestration are individually tuned using composable, typed architectures to achieve end-to-end performance.
• Optimization strategies such as Bayesian tuning, iterative self-improvement, and reinforcement learning are applied to enhance empirical gains across diverse applications from agent benchmarks to fuzzing harnesses.

Searching arXiv for the provided harness-scaling papers and closely related recent work to ground the article. Harness scaling is the treatment of the harness—the structured execution layer around a foundation model—as a first-class object of design, evaluation, and optimization rather than a secondary implementation detail. In the recent agentic-AI literature, the harness includes prompts, tools, memory, control flow, orchestration, verification, and governance; in adjacent settings it also denotes task-specific execution infrastructure such as algorithm-discovery harnesses and fuzzing harnesses. The central claim across this literature is that end-to-end capability depends not only on model scale, but on how the harness exposes context, routes skills, constrains actions, and converts execution traces into further improvement (Gu, 25 May 2026, Chen et al., 12 Jun 2026, Ishibashi et al., 13 May 2026).

1. Conceptual foundations

The clearest formalization of harness scaling appears in work that contrasts model scaling with system scaling. There, the “agent harness” is the structured execution layer surrounding a foundation model, comprising the memory substrate $\mathcal{M}$, context constructor $\mathcal{C}$, skill-routing layer $\mathcal{S}$, orchestration loop $\mathcal{O}$, verification and governance layer $\mathcal{G}$, and foundation-model interface or reasoning substrate $\mathcal{R}$. End-to-end long-horizon performance is written as $$P_H = \Phi(\mathcal{R},\mathcal{M},\mathcal{C},\mathcal{S},\mathcal{O},\mathcal{G})$$, emphasizing that capability emerges from interaction among these modules rather than from $\mathcal{R}$ alone (Gu, 25 May 2026).

This formulation makes “scaling” a module-specific notion. Memory scaling is described in terms of precision, durability, retrievability, and verifiability. Context scaling is described through relevance, compactness, traceability, and refresh policy. Skill routing is scaled through specificity, selectivity, composability, and verifiability. Orchestration scaling concerns asynchronous workflows, parallel subagent scheduling, and retry policies, while governance scaling concerns permission models, rule engines, and human-in-the-loop checkpoints. The principal bottlenecks identified in this framework are context governance, trustworthy memory, and dynamic skill routing, together with the orchestration and governance mechanisms that coordinate them (Gu, 25 May 2026).

The same literature argues that model-centric evaluation is inadequate when long-horizon behavior depends on memory hygiene, context efficiency, post-condition checks, and auditability. This is reflected in the CheetahClaws reference harness, an open-source, Python-native implementation with structured memory entries carrying explicit confidence and recency fields, context scopes at the user/project/session level, plugin-based skill routing with post-condition schemas, and audit logging of every tool invocation and memory write. The comparison with Claude Code and OpenClaw is used to show that empirical differences can arise from harness choices rather than from underlying model APIs (Gu, 25 May 2026).

2. Harness as a composable architecture

A second line of work treats the harness itself as a typed, composable software object. HarnessX defines harness primitives as lightly typed “processors” spanning prompts, tools, memory modules, and control-flow processors, each attached to one of eight well-typed hook events. A harness configuration is expressed as $\mathcal{C}=(\mathbf{P},\mathbf{S})$, where $\mathbf{P}$ maps hooks to processor lists and $\mathcal{C}$0 stores shared singleton slots such as the tool registry or workspace. Because processors at a hook consume and yield the same event type, insertion, removal, or replacement preserves end-to-end type-correctness, and typed edits can modify one or more dimensions subject to hook and singleton-group constraints (Chen et al., 12 Jun 2026).

HarnessX’s AEGIS engine then casts harness evolution as an MDP over symbolic artifacts. The state is the current harness configuration and trace history; the actions are code-level harness edits; the reward is verifier score; and transition is a deterministic acceptance gate that either commits or rejects the proposed harness. Its roll-out, digester, planner, evolver, and critic-plus-gate structure is explicitly designed to defend against reward hacking, catastrophic forgetting, and under-exploration. This framing places harness scaling close to program transformation and constrained policy search rather than simple prompt tuning (Chen et al., 12 Jun 2026).

The code-centric survey literature extends the same idea from a single agent to a multi-agent ecosystem. There, a scaled harness $\mathcal{C}$1 must provide a shared substrate and coordinated control: agents read and write a common program-centric workspace, while the harness orchestrates planning, action, and verification across agent boundaries. The survey emphasizes sparse topologies—chain, star, tree, or adaptive graphs—rather than naïve fully connected communication, and describes file-only, repository-based, execution-based, and blackboard/shared-state representations as alternative substrates for scaled harnesses. It further highlights transactional concerns such as locking, atomic commits, semantic merges, and regression tests for harness mutations (Ning et al., 18 May 2026).

A plausible implication is that harness scaling becomes easier to automate when the harness is represented in typed, editable, and inspectable units. In this view, observability and edit locality are not ancillary engineering conveniences; they are prerequisites for reliable composition and evolution.

3. Optimization and self-improvement mechanisms

One approach to harness scaling is direct search over a bounded configuration space. HARBOR formalizes automated harness optimization as constrained, noisy Bayesian optimization over a mixed-variable, cost-heterogeneous configuration space with cold-start-corrected rewards and a posterior chance-constrained safety check. In the codex-py case study over Terminal-Bench 2, the harness exposes roughly 35 enhancement flags, with nine deep-agent flags selected for the paper’s main experiments. A single end-to-end HARBOR run returned a two-flag configuration—Tiered compressor plus cross-session memory—scoring 17/89 in 122 minutes, matching the best manual round with two flags instead of five and beating the eight-flag round D by five passes, while respecting the safety constraint (Sengupta et al., 22 Apr 2026).

A second approach is iterative self-improvement from execution traces. Self-Harness operationalizes this as a three-stage loop: Weakness Mining, Harness Proposal, and Proposal Validation. Weakness Mining clusters failure signatures from traces; Harness Proposal asks the same model to generate diverse yet minimal edits tied to those signatures; Proposal Validation accepts a candidate only if it improves or preserves both held-in and held-out pass rates. On Terminal-Bench-2.0, this loop improves held-out pass rates from 40.5% to 61.9% for MiniMax M2.5, from 23.8% to 38.1% for Qwen3.5-35B-A3B, and from 42.9% to 57.1% for GLM-5, with no reported regressions (Zhang et al., 8 Jun 2026).

A third approach is joint evolution of solutions and harnesses inside a reinforcement-learning loop. HASE places solution actions and harness-edit actions in a single multi-turn action space. Guidance-harness edits such as prompts or retrieval code receive proof-of-concept credit if they improve task reward, while evaluation-harness edits are only rewarded when they repair a mismatch between a local proxy evaluator and an immutable real-world evaluator. In text classification, a single Qwen3-8B model with the discovered HASE harness reaches 86.98%$\mathcal{C}$20.38 accuracy, matching the GPT-OSS-120B Meta-Harness baseline at 86.8%; in circle packing, HASE repairs an initially flawed validator and converges to a best score of 2.635983 (Luo et al., 4 Jul 2026).

A broader automation proposal pushes the same idea one level higher. “The Last Harness You’ll Ever Build” defines an inner Harness Evolution Loop that execute–evaluate–evolves a worker harness for a single task, and an outer Meta-Evolution Loop that optimizes the evolution protocol itself across tasks. Its stated objective is to shift manual harness engineering into automated harness engineering, and then to automate the design of the automation itself (Seong et al., 22 Apr 2026).

4. Budget allocation, control policies, and runtime scaling

Several papers study harness scaling as an inference-budget allocation problem rather than solely as architecture search. In Vesper, the key question is whether a fixed token budget $\mathcal{C}$3 should be spent on many shallow candidates or fewer, deeper candidates, with $\mathcal{C}$4 under average per-candidate spend $\mathcal{C}$5. On Circle Packing with a 40M-token budget, generating fewer algorithms while thinking more deeply about each one achieved higher scores than producing many brief attempts. The paper reports that Vesper with 452 algorithms at 89.6k tokens per algorithm reached a best score of 2.6360, while the OpenEvolve baseline with 1671 algorithms at 23.9k tokens per algorithm plateaued at 2.4185. The same work also finds that more capable models produced evaluation hacks at higher rates, making secondary hack detection increasingly necessary as models scale, and that Git worktree isolation yields an empirical parallelism ratio of approximately 3.2–3.9$\mathcal{C}$6 with four agents (Ishibashi et al., 13 May 2026).

A related but narrower control problem is solved by the metacognitive harness inspired by Nelson–Narens theory. That harness separates monitoring from reasoning: the model first emits a pre-solve feeling-of-knowing (FOK), then after each solve emits a judgment-of-learning (JOL), and an SVM-based controller decides whether to stop, retry with compact feedback, or aggregate multiple attempts. On HLE-Verified, LiveCodeBench v6, and R-Bench-V, this raises pooled accuracy from 48.3 to 56.9 for a fixed Claude Sonnet-4.6 base model. The average number of attempts is approximately 2.4, compared with 1.0 for pass@1 and 4.0 for a four-attempt aggregator, which positions the harness as an adaptive form of test-time scaling rather than a static retry rule (Cao et al., 13 May 2026).

Runtime scaling also appears in reinforcement-learning infrastructure. Polar treats the agent harness as a black box, proxies LLM API calls, records token-level interactions, reconstructs token-faithful trajectories, and exposes asynchronous rollout services. Its asynchronous staging separates INIT, RUNNING, and POSTRUN, while its prefix-merging trajectory builder combines multi-turn calls into longer traces without retokenization drift. In the reported reconstruction ablation, prefix_merging reduces wall-clock time from 189.5 to 35.2 minutes and increases average GPU utilization from 20.4% to 87.7%. Using GRPO on SWE-Bench Verified, Polar improves Qwen3.5-4B by 22.6 points on the Codex harness, 4.8 on Claude Code, 0.6 on Qwen Code, and 6.2 on Pi (Xu et al., 22 May 2026).

5. Empirical domains and representative systems

Harness scaling is not confined to a single application class. In general-purpose agent benchmarks, HarnessX reports an average pass@2 gain of +14.5 percentage points, with gains up to +44.0 points across ALFWorld, GAIA, WebShop, $\mathcal{C}$7-Bench, and SWE-bench Verified. A recurring pattern is inverse scaling: weaker models with lower initial performance gain more from harness evolution, as in Qwen 3.5 on ALFWorld with +44.0 points compared with Sonnet 4.6 at +11.2. The same work reports that model–harness co-evolution adds +4.7 points on GAIA and WebShop for Qwen 3.5, which is presented as evidence that harness adaptation and model adaptation can be complementary (Chen et al., 12 Jun 2026).

Program-analysis work provides a more domain-specific use of the term. HarnessAgent targets automatic fuzzing harness construction for arbitrary functions, especially internal functions, where static or incomplete context provisioning often causes failures. It introduces a rule-based strategy to identify and minimize various compilation errors, a hybrid tool pool for precise and robust symbol source-code retrieval, and an enhanced harness validation pipeline that detects fake definitions. On 243 target functions from OSS-Fuzz projects—65 C projects and 178 C++ projects—it improves the three-shot success rate by approximately 20% over prior techniques, reaching 87% for C and 81% for C++. In one-hour fuzzing, more than 75% of generated harnesses increase target-function coverage, and the hybrid retrieval system achieves a response rate of over 90%, outperforming Fuzz Introspector by more than 30% (Yang et al., 3 Dec 2025).

The survey literature broadens the application map further. “Code as Agent Harness” organizes representative methods and practical applications across coding assistants, GUI/OS automation, embodied agents, scientific discovery, personalization and recommendation, DevOps, and enterprise workflows, and argues that scaling the harness from single-agent systems to multi-agent settings requires shared code artifacts that support coordination, review, and verification (Ning et al., 18 May 2026). This suggests that the term “harness scaling” is becoming a unifying label for a family of problems in which runtime scaffolding, not only model weights, is treated as the primary object of intervention.

6. Evaluation disputes and open research questions

As automatic harness modification became more common, evaluation methodology itself became contested. “Rethinking the Evaluation of Harness Evolution for Agents” argues that harness evolution is an iterative search procedure and should therefore be compared with simple test-time scaling baselines under matched feedback and inference budgets. On Terminal-Bench 2.1 with GPT-5.4 and Claude Opus 4.6, the paper reports that automatic harness evolution does not consistently outperform simple test-time scaling methods and exhibits limited generalization. In the held-out train/validation/test split, average pass@1 rises only from 67.7 to 68.3, a gain of +0.6 that the paper treats as within run-to-run noise, and the authors recommend disjoint search and evaluation sets together with reporting both pass@1 and pass@k (Wang et al., 14 Jul 2026).

A related critique separates the capability to produce useful harness updates from the capability to benefit from those updates. “Harness Updating Is Not Harness Benefit” defines harness-updating capability $\mathcal{C}$8 for an evolver and harness-benefit capability $\mathcal{C}$9 for a solver. Across SWE-bench Verified, MCP-Atlas, and SkillsBench, it finds that harness-updating is essentially flat across model scale, while harness-benefit is non-monotonic: weak-tier models benefit little, mid-tier models benefit most, and strong-tier models benefit less than mid-tier models. The paper attributes weak-tier underperformance to Harness Activation Failure and Harness Adherence Failure, quantified through low skill-load rate and low harness-following rate, rather than to a lack of potentially useful updates (Lin et al., 28 May 2026).

The broader research agenda therefore shifts toward harness-level benchmarks and longitudinal evaluation. Proposed metrics include trajectory quality, memory hygiene, context efficiency, communication fidelity, verification cost, and safe evolution, together with explicit measurement of regression over time, reward hacking, and multi-agent conflict resolution. The survey literature adds shared-state consistency, transactional semantics for concurrent edits, and human oversight for safety-critical actions as outstanding problems. Taken together, these proposals indicate that harness scaling is becoming both an engineering discipline and an evaluation discipline: it concerns not only how to build or evolve harnesses, but how to determine when a harness improvement is real, safe, and transferable (Gu, 25 May 2026, Ning et al., 18 May 2026).