Harness-First Search Architecture

• Harness-first search architectures are defined by optimizing external harness elements—prompts, tool interfaces, orchestration logic, and evaluation criteria—to enhance agent performance.
• They utilize closed-loop evolution and meta-learning to systematically search, compose, and refine harness components for rapid adaptation across diverse tasks.
• Empirical results demonstrate significant performance gains, reduced iteration counts, and improved generalization in complex applications such as workflow automation and coding.

A harness-first search architecture is an agent system design paradigm in which the primary object of optimization is the agent’s external harness—the structured scaffolding that mediates its interaction with the environment—rather than its model parameters alone. In this framework, prompts, tool interfaces, orchestration logic, and evaluation criteria are systematically searched, composed, and evolved to maximize agentic performance, often via automated or meta-learning loops. This approach generalizes across domains, enables efficient adaptation to novel tasks, and supports rapid, robust generalization, as evidenced by empirical gains on challenges including workflow automation, algorithm discovery, retrieval, and coding (Seong et al., 22 Apr 2026, Chen et al., 12 Jun 2026, Chen et al., 1 Jun 2026, Ishibashi et al., 13 May 2026).

1. Formalization and Core Concepts

Harness-first architectures precisely delineate the “harness” as a formal object. For example, in “The Last Harness You’ll Ever Build,” a harness $\mathcal{H}$ for a foundation model is defined as: $\mathcal{H} = (P, T, O, C)$ where $P$ is the set of system/task-level prompts, $T$ the tool/skill interfaces, $O$ the orchestration logic (subagent sequencing, feedback, verification hooks), and $C$ the evaluation configuration (success criteria, thresholds). This tuple determines the agent’s observed context, permitted actions, orchestration, and measurement regime (Seong et al., 22 Apr 2026).

In HarnessX, the harness consists of typed “processors” categorized as prompts, tools, memory modules, and control operators, each with associated hooks and contractually enforced data signatures. The compositional space is formalized via a substitution algebra, ensuring that harness edits remain well-typed and that primitives can be safely exchanged (Chen et al., 12 Jun 2026).

In coding and discovery settings such as Vesper (Ishibashi et al., 13 May 2026), the harness tuple is operationalized as a pipeline: $$\mathcal{H} = \langle \text{Select}, \text{Sandbox}, \text{AgentLoop}, \text{Eval}, \text{Verify}, \text{Archive} \rangle$$ where each function mediates a critical phase (selection of parents, safe agent execution in isolated worktree, evaluation and verification, archival of viable candidates).

Harness-1 further externalizes all recoverable state—working memories, candidate pools, evidence graphs—into the harness, freeing the policy to focus on high-level semantic decisions (Jiang et al., 1 Jun 2026).

2. Harness Evolution: Automated Search Procedures

A central operational characteristic is the adoption of closed-loop search over the harness. In (Seong et al., 22 Apr 2026), the Harness Evolution Loop performs the following iterative process for a fixed task: - A Worker Agent $W_\mathcal{H}$ executes the task; the Evaluator Agent $V$ diagnoses using traces and scoring; the Evolution Agent $E$ produces harness modifications based on the cumulative history. - Updates are informed by the complete log of past failures and successes rather than isolated episodes.

HarnessX frames harness-editing as a symbolic MDP over configuration space. The AEGIS engine sequences extraction of failure summaries (Digester), planning of actionable edits (Planner), candidate generation (Evolver), and non-regressive gating and selection (DeterministicGate and Critic). All edits are type-checked via the substitution algebra (Chen et al., 12 Jun 2026).

Meta-Harness uses a coding agent as a harness proposer with full filesystem-level access to all prior harness code, execution traces, and evaluation scores. This agent synthesizes targeted edits grounded in error analysis, supporting search directly in the codebase rather than in brittle prompt or scalar spaces (Lee et al., 30 Mar 2026).

HARBOR recasts harness configuration as a constrained Bayesian optimization problem over a mixed-variable flag space, utilizing multi-fidelity cost-aware acquisition and trust-region exploration. Telemetry counters enable “silent-flag” detection to prune inert features, while non-additive feature interactions are modeled explicitly via a block-additive SAAS kernel (Sengupta et al., 22 Apr 2026).

3. Meta-Optimization and Co-Evolution Strategies

Harness-first architectures often leverage meta-evolution frameworks capable of automatic protocol search (Seong et al., 22 Apr 2026). The Meta-Evolution Loop in (Seong et al., 22 Apr 2026) optimizes the entire adaptation protocol $\mathcal{H} = (P, T, O, C)$0, seeking task-agnostic meta-procedures that induce rapid convergence of task-specific harnesses. The outer loop maximizes expected best score across a suite of tasks, using adversarial harness modifications and meta-level evolution agents without differentiable gradients.

HarnessX introduces harness–model co-evolution, employing shared off-policy buffers and cross-harness group-relative policy optimization (GRPO). The harness is evolved non-parametrically (via AEGIS), while the model is updated using trajectory-level, harness-conditioned advantages. The result is joint breaking of “scaffolding” and “signal” ceilings, preventing stagnation due to model–harness mismatch (Chen et al., 12 Jun 2026).

HarnessForge formalizes the agent system as a pair $\mathcal{H} = (P, T, O, C)$1 of harness and lightweight-adapted base reasoner, and enforces a “harness-first” process: the harness is Pareto-optimized for fault coverage and efficiency before policies are aligned through harness-conditioned supervised or RL objectives. Only then is policy adaptation performed, ensuring compatibility with the discovered interface (Chen et al., 1 Jun 2026).

4. Concrete Algorithms and Empirical Analyses

Harness-first search implementations exhibit diverse concrete algorithms:

• Evolutionary loop (Algorithm 1 in (Seong et al., 22 Apr 2026)): Each iteration accumulates diagnostic reports and verdicts, applies harness modifications, and logs outcomes, with convergence criteria based on performance plateaus or iteration budgets.
• Composition via edit algebra (Chen et al., 12 Jun 2026): Harness configurations are first-class algebraic objects; edits are type-safe and satisfy monoid laws, guaranteeing composability and variant isolation.
• Vesper (Ishibashi et al., 13 May 2026): Deep candidate generation is favored over high-throughput shallow sampling, with verification agents detecting reward hacking and Git worktree-based sandboxing enabling parallel agent execution.
• LEVI (Tanveer, 10 May 2026): Archives solutions using a CVT-MAP-Elites descriptor map for diversity, implements role-aware mutation routing (local refinement via small model, paradigm shifts via large model), and deploys rank-faithful proxy benchmarks to maximize efficiency versus budget.

Abridged complexity metrics are as follows:

• (Seong et al., 22 Apr 2026): Inner loop complexity $\mathcal{H} = (P, T, O, C)$2; outer $\mathcal{H} = (P, T, O, C)$3, with $\mathcal{H} = (P, T, O, C)$4, $\mathcal{H} = (P, T, O, C)$5 meta-iterations.
• (Chen et al., 12 Jun 2026): Each AEGIS step leverages log compression, type-safe candidate generation, and ensemble verification to defend against reward hacking, forgetting, and distributional drift.
• (Tanveer, 10 May 2026): Empirical sample efficiency is increased $\mathcal{H} = (P, T, O, C)$6–$\mathcal{H} = (P, T, O, C)$7 over model-first LLM evolutionary search via harness innovations.

5. Performance and Generalization

Empirical studies consistently demonstrate the practical efficacy of harness-first search:

• (Seong et al., 22 Apr 2026): Learned protocols achieve a $\mathcal{H} = (P, T, O, C)$8 reduction in iterations to reach $\mathcal{H} = (P, T, O, C)$9 pass rates on diverse workflow benchmarks; final task success after $P$0 iterations is $P$1 variance across held-out tasks.
• (Chen et al., 12 Jun 2026): HarnessX delivers $P$2 average performance gains (up to $P$3) across five benchmarks. Co-evolution adds another $P$4.
• (Chen et al., 1 Jun 2026): HarnessForge achieves up to $P$5 improvement over the strongest harness- or policy-only baselines, robustly outperforming in multi-hop tool use, QA, and API reasoning.
• (Jiang et al., 1 Jun 2026): Harness-1 attains $P$6 average curated recall, exceeding the next best open agent by $P$7 points and showing particularly strong transfer on unseen domains.

A general finding is that design and optimization of the harness, rather than model scaling alone, yield primary performance dividends—especially when the tasks require nontrivial state externalization, explicit tool composition, and rapid adaptation to novel domains.

6. Best Practices and Design Guidelines

Multiple works synthesize best practice guidelines for harness-first architectures:

• Prefer deep, multi-step candidate reasoning (high per-candidate budget) over shallow mass generation (Ishibashi et al., 13 May 2026).
• Enforce output verification via dedicated detection agents to block evaluation hacks; reward-hacking frequency increases with model scale.
• Decompose harnesses into modular primitives tied to explicit hooks, guaranteeing type safety and isolable edits.
• Frame harness search as a black-box optimization problem when feature interactions are non-additive or high-dimensional; leverage reward correction and telemetry.
• Separate harness design from model adaptation: adapt the harness first for structural and interface optimization, then align policy under strictly harness-conditioned rollouts (Chen et al., 1 Jun 2026).
• Provide consistent state rendering across all agent phases (training, evaluation, RL) to mitigate train–test drift (Jiang et al., 1 Jun 2026).
• Track program archives, scores, and diffs for causal analysis in agentic proposer frameworks (Lee et al., 30 Mar 2026).
• Use trusted regions and multi-fidelity evaluation to efficiently allocate computational cost in large flag or edit spaces (Sengupta et al., 22 Apr 2026).

7. Scope, Generalization, and Future Directions

Harness-first architectures are applicable across LLM-based agents in workflow automation, retrieval/curation, algorithmic discovery, and systems research. They support domain transfer, systematic recombination of runtime scaffolding, and integration with model-based RL and meta-learning.

Delineation of policy-environment boundaries, externalization of routine state, and auditable edit spaces render these systems tractable for formal optimization, automated debugging, and continual improvement. Empirical evidence suggests that, as environments and tasks proliferate, harness-first search supplants ad hoc or hand-crafted harness engineering as the dominant design paradigm for adaptive agent systems (Seong et al., 22 Apr 2026, Chen et al., 12 Jun 2026, Lee et al., 30 Mar 2026, Chen et al., 1 Jun 2026).