Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable

Abstract: The capability of a modern AI agent depends not only on its foundation model but also on its harness, which constructs prompts, manages state, invokes tools, and coordinates execution. As models, APIs, environments, and requirements evolve, the harness must be continually modified. Before such a change can be made, a developer or coding agent must identify all code locations that implement the target behavior. This is difficult because production harnesses are large, tightly coupled, and behaviorally distributed, while modification requests describe what the system should do and repositories are organized by files and modules. Code search, repository indexing, and long-context processing ease inspection, but still leave this behavior-to-code mapping to be recovered by hand. Behavior localization is therefore a central bottleneck in harness evolution. We introduce the Harness Handbook, a behavior-centric representation synthesized automatically from a harness codebase via static analysis and LLM-assisted structuring, linking each behavior to its corresponding source. We also introduce Behavior-Guided Progressive Disclosure (BGPD), which guides agents from high-level behaviors to relevant implementation details and verifies candidate locations against the current source. On diverse modification requests from two open-source harnesses, Handbook-Assisted planning improves behavior localization and edit-plan quality while using fewer planner tokens, with the largest gains on scattered sites, rarely executed paths, and cross-module interactions. Evolving complex agentic systems thus depends not only on generating edits, but also on determining where those edits should be made.

• The paper presents a behavior-centric repository that maps runtime behaviors to source code units for precise harness evolution.
• It employs a hybrid pipeline combining static program analysis with LLM-assisted behavioral mapping to construct a hierarchical Handbook.
• Experimental results show significant improvements in localization accuracy, planning efficiency, and reduced edit failures across agent harnesses.

Harness Handbook: Behavior-Centric Localization for Agent Harness Evolution

Motivation and Problem Statement

Modern agentic systems rely fundamentally on agent harnesses—software layers that orchestrate prompt construction, state management, tool invocation, and execution flow. The growing complexity and scale of these harnesses, alongside their distributed and tightly coupled behavioral logic, create a bottleneck for both human developers and coding agents when evolving system behavior: behavior localization. Given a natural language modification request, identifying all the code loci underlying the target behavior is a prerequisite, yet current repository-centric or code search-based approaches do not reconcile behavioral abstractions with highly distributed code implementations. This frequently results in incomplete or inefficient plans, especially when behaviors cross file, function, or execution stage boundaries.

The Harness Handbook Representation

To address this bottleneck, the paper introduces the Harness Handbook: a behavior-centric repository representation constructed automatically from source code and designed to directly answer the question, “where is this behavior implemented?” Unlike existing repository representations organized by syntactic structure (files, functions), the Handbook organizes implementation knowledge around runtime behaviors, linking high-level system and execution stages to their constituent source units.

The Handbook embodies a three-level hierarchy:

• L1 (System Overview): Architectural, execution, and data-flow summary of the entire harness.
• L2 (Stage-Level Overview): For each execution stage, outlines responsibilities, dependencies, and local state.
• L3 (Source-Backed Leaf Units): Canonical pointers into source code, defined at the granularity of either functions or files, depending on the characteristics of the harness.

A cross-stage state-register view complements this tree by mapping state dependencies and interactions across execution stages.

Figure 1: Harness Handbook’s hierarchical representation enables navigation from system-level behaviors down to concrete, source-anchored code units, with cross-stage traceability.

Automated Construction Pipeline

Harness Handbook instances are synthesized via a pipeline combining static program analysis with LLM-assisted behavioral structuring. The process has three core phases:

1. Static Fact Extraction: Language-specific adapters build program graphs anchored in concrete source locations, functions, call edges, and boundaries. Deterministic and LLM-free.
2. Behavioral Organization: Functions or files are assigned to execution stages, either via a seed skeleton (function-as-leaf) or inferred stage structure (file-as-leaf), using both program graph evidence and LLM-backed review loops.
3. Hierarchical Synthesis: The stage skeleton is converted into the L1-L3 document tree and the cross-stage state-register view, with persistent validation to ensure alignment with current source code.

   Figure 2: The construction pipeline fuses static code analysis and iterative LLM-based behavioral mapping to produce the three-level behavior-centric Handbook.

Two modes allow scaling to repository size: function-level leaves for systems with a tractable function graph, and file-level leaves for large monorepos.

Behavior-Guided Modification Workflow

For harness modification, the Handbook supports a behavior-guided progressive disclosure (BGPD) workflow. This process localizes all relevant code sites for a modification request via:

• Hierarchical navigation from behaviors and execution stages to concrete code units.
• Cross-stage reasoning via the state-register map.
• Call relation expansion for comprehensive candidate gathering.
• Verification of candidates against the live source repository.

This mechanism produces a set of source-grounded evidence supporting high-precision, contextually grounded edit plans.

Incremental edits to the harness also trigger automatic Handbook resynchronization, ensuring that the representation remains authoritative and up to date across repository lifecycles.

Experimental Evaluation

The evaluation focuses on two open-source agent harnesses—Terminus-2 (Python, function-as-leaf) and Codex (Rust, file-as-leaf)—and benchmarks 30 diverse behavior-driven modification requests per harness, spanning Query, Cross-file, and Search-Hostile classes with stratified localization difficulty.

Plan quality is assessed by independent LLM judges (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro) across three axes: Localization, Scope Control, and Reasoning. Localization accuracy is compared to gold-standard edit plans, and the planner’s token usage is measured to quantify efficiency.

Figure 3: Handbook-Assisted planning achieves higher win rates and lower planner token usage across harnesses and judges.

The results are robust:

• Plan quality: On Codex, Handbook-Assisted win rates increase from 28.3% to 38.3%; on Terminus-2, from 26.7% to 45.6%. Gains are distributed across all judges and all evaluated metrics.
• Efficiency: Planner token usage decreases by 12.7% (Codex) and 8.6% (Terminus-2) under Handbook guidance, contradicting any claim that quality improvements are due to larger context consumption.
• Localization accuracy: Recall, precision, and F1 improve at both file and symbol granularity for all harnesses and reference plans, with F1 gains up to 18.8 percentage points.
• Robustness across difficulty: Gains are consistent across request types (Query, Cross-file, Search-Hostile) and difficulty stratifications, notably with marked improvements for requests requiring cross-module or scattered localization.

  Figure 4: Consistent per-judge improvements for Localization, Scope Control, and Reasoning across both harnesses.

  

  Figure 5: Win rate improvements persist across request types and across localization difficulty; Search-Hostile and Cross-file requests particularly benefit from Handbook guidance.

Notable Claims

• Handbook guidance enables a weaker planner to match the implementation-site localization quality of substantially stronger models.
• Quality and efficiency improve simultaneously without increasing token budget.
• Handbook-Assisted planning reduces the rate of total localization failures (“Wrong”) by up to 25.9 points.

Implications and Future Directions

The formalization and deployment of behavior-centric repository representations address a central practical barrier for both human and AI-driven harness evolution. By shifting the locus of knowledge from source-centric to behavior-centric, the system exposes otherwise hidden distributed dependencies, enabling more accurate, minimal, and auditable change plans. This capability is critical for trusted, continuous agent system improvement and self-maintenance.

The Handbook framework, validated here for planning and localization in reactive agent harnesses, generalizes naturally to other tasks: automated behavior auditing, regression impact analysis, specification mapping, and scaffolding for harness self-evolution. The synchronization and progressive disclosure strategies also offer a template for integrating human and agent contributions in continuous codebases. The authors speculate, realistically, about using the Handbook as a persistent behavioral memory to close the self-improving harness loop—autonomously localizing, planning, executing, and auditing as agent capabilities and requirements evolve.

Conclusion

The Harness Handbook advances agent engineering by reifying the mapping from requested behaviors to their distributed implementations. Experimental evidence shows that Handbook-Assisted planning provides strong and consistent improvements in localization accuracy, plan quality, and efficiency across a spectrum of agent harnesses and modification types. The methodology counters previous implementation-centric limitations by instantiating an operational, source-grounded behavior-centric repository representation, with direct benefits for both human and coding agents tasked with evolving large, distributed agentic software systems. The framework opens avenues for rigorous, behavior-aware automation in harness evolution and a path toward self-maintaining agent platforms.

Reference: "Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable" (2607.13285).