An Empirical Study of Harness Design for Coding Agents

Abstract: Coding harnesses shape how autonomous coding agents translate model capabilities into long-horizon software-engineering performance, yet existing work typically evaluates harnesses as monolithic systems, leaving the effectiveness of individual components unclear. To enable component-level comparisons, we study this question with a lightweight coding harness whose execution loop is fixed while three components are varied: planning, action space, and context management. Across four models evaluated on SWE-Bench Verified and Terminal-Bench 2.1, we evaluate 176 matched settings spanning five context-management strategies, four context-window budgets, and targeted ablations of planning and action space. We find that: (1) Context management becomes increasingly valuable as the context-window budget tightens, with most of its benefit coming from preventing context-overflow failures. (2) Staging rule-based elision before LLM-based summarization provides the strongest overall efficiency among the context-management strategies, whereas making elided content recoverable adds machinery that models rarely use and yields no accuracy gain. (3) Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models, with little change in accuracy. (4) Predefined tools improve performance for models with weaker bash proficiency, whereas bash-capable models can operate effectively with a bash-only interface and achieve substantially lower cost, especially on command-line-centric tasks. Trajectory-level analysis explains these effects: context management extends execution trajectories without substantially altering agent behavior, planning changes where trajectories stop, and the action space changes the granularity at which code is written. These findings inform model- and budget-aware harness design and provide a modular framework for evaluating future harness components.

• The paper evaluates 176 matched harness configurations across four models and two coding benchmarks, showing that context management delivers its largest success gains under tight context limits by preventing overflow-related termination.
• The paper finds that planning improves weaker models’ success by sustaining trajectories, while reducing stronger models’ inference costs by limiting redundant verification; its effects vary by model capability and task type.
• The paper demonstrates a capability crossover in action interfaces: structured tools help models with weak shell control, whereas bash-only access can reduce costs and improve results for stronger, shell-proficient models.

Study objective and experimental design

The paper investigates coding harness design as a component-level systems problem. Its central premise is that coding-agent performance cannot be attributed to the underlying LLM alone: planning, action interfaces, and context management mediate how model capabilities are expressed during long-horizon software-engineering tasks. Existing evaluations commonly compare complete harnesses, thereby conflating these mechanisms. This study instead holds the ReAct execution loop, safety controls, diagnostics, stuck detection, task environments, and model fixed while varying three harness components independently (2609.20804).

The evaluation covers four models: Nemotron-3 at 30B, 120B, and 550B parameters, plus Mistral-Medium-3.5-128B. The models are evaluated on 500 SWE-Bench Verified tasks and 89 Terminal-Bench 2.1 tasks. The primary metrics are task success rate and model-inference cost. Context management is evaluated across five policies and four nominal context budgets—32k, 64k, 96k, and 128k tokens. Planning and action-space ablations are conducted at the T4/128k configuration. In total, the study reports 176 matched experimental settings.

The harness uses a ReAct loop in which each turn assembles the model input, executes tool calls in an isolated task container, and appends observations to the interaction history. Planning is implemented as a persistent task plan maintained through an update_plan tool. The structured action space exposes file, search, web, and shell tools, whereas the alternative interface provides only bash. Context management combines three mechanisms: elision of stale tool outputs, lossless external storage with recall_event, and LLM-generated summarization.

Figure 1: Modular harness architecture and the staged T4 context-management policy.

The five context-management tiers isolate the contribution of these mechanisms. T0 performs no compaction and terminates when the context window is exceeded. T1 applies irreversible elision, T2 adds recoverable external storage, T3 applies summarization without elision, and T4 stages elision before summarization while retaining recall capability. T4 therefore tests whether cheap, rule-based compression can defer or reduce expensive model-based summarization.

Context management and context-window pressure

The strongest aggregate result concerns the interaction between context management and available context capacity. Context management is most valuable when the nominal context window is small, and its primary benefit is preventing premature termination rather than changing the agent’s underlying workflow.

On SWE-Bench, the mean success-rate advantage of managed tiers over T0 is 35.7 percentage points at 32k tokens, 15.9 points at 64k, 5.5 points at 96k, and 2.7 points at 128k. On Terminal-Bench, the corresponding gains are 9.5, 7.5, 4.8, and 2.8 points. These differences track the frequency of context overflow in T0: the model-averaged overflow rate decreases from 78.7% to 8.7% on SWE-Bench and from 61.0% to 12.1% on Terminal-Bench as the budget expands from 32k to 128k. Every managed tier records zero overflow failures at every tested budget.

Figure 2: Managed context policies eliminate window-overflow failures and yield their largest success-rate gains at smaller context budgets.

This result supports a specific causal interpretation. Context management primarily increases the probability that a trajectory survives long enough to reach code modification and verification. It does not necessarily make the model’s local decisions more accurate. At 32k, unmanaged trajectories commonly terminate during localization, whereas managed trajectories preserve the phase ordering of the workflow and continue into fixing and verification. At 128k, the trajectory profiles and phase distributions of the tiers become substantially more similar. Thus, the principal function of context management in this setting is trajectory preservation under a binding context constraint.

The study also identifies T4 as the most favorable efficiency configuration. Its success rate is broadly comparable to T1–T3, but it has the lowest mean cost in seven of the eight model–benchmark combinations and the lowest average cost at every context budget. T4 also maintains the lowest average peak-context ratio across all budgets. The staged policy explains this result: early elision removes bulky stale observations before the harness invokes an additional LLM call for summarization.

Figure 3: Success-rate ranges, means, and costs across context-management tiers and context budgets.

Figure 4: T4 achieves lower peak-context usage and lower cost by reducing the frequency of summarization calls.

The comparison between T1 and T2 produces a particularly negative result for lossless recall. T2 outperforms T1 in 15 of 32 matched comparisons, underperforms in 14, and ties in three; its equal-weight mean difference is only -0.36 percentage points overall. Recall usage is extremely sparse: across 64 T2 and T4 settings, 56.3% of configurations never invoke recall_event, the median invocation rate is zero, and the mean rate declines from 0.540 calls per task at 32k to 0.007 at 128k. The strongest models almost never use recall, and even the heaviest-use configuration performs 3.37 percentage points worse than the corresponding elision-only condition. The implication is that recoverable storage adds implementation complexity without providing a measurable accuracy benefit in this harness. Models generally adapt to the compressed representation by rereading or rerunning operations rather than retrieving archived observations.

Planning as a capability-dependent intervention

Planning does not have a uniform effect across model capability. Its value shifts from improving trajectory survival for the weakest model to reducing redundant computation for stronger models.

For Nemotron-3 30B, enabling planning raises SWE-Bench success by 11.6 percentage points and Terminal-Bench success by 4.5 points, but increases cost on both benchmarks. The trajectory analysis shows why: without planning, the median SWE-Bench trajectory collapses from 40 turns to 5, while the fraction of runs terminating without an edit rises from 27.8% to 68.6%. The proportion stalled in localization rises from 10.4% to 58.4%. Planning therefore acts as an execution scaffold that sustains weak-model trajectories until they attempt an edit.

For Nemotron-3 550B and Mistral-Medium-3.5-128B, planning instead reduces cost while causing only small changes in accuracy. On SWE-Bench, the cost reductions are approximately 30% and 32%, respectively, accompanied by success-rate changes of -2.0 and -0.4 percentage points. Planning reduces median SWE-Bench trajectory lengths from 108 to 74 turns for Nemotron-3 550B and from 68 to 53 turns for Mistral. The removed actions are primarily post-edit verification rather than localization or repair. Planning consequently functions as a stopping-control mechanism for stronger models that otherwise continue verifying after producing an adequate patch.

Figure 5: Planning improves weak-model accuracy but primarily reduces strong-model execution cost.

The intermediate Nemotron-3 120B model exhibits no consistent planning effect. Planning increases cost on SWE-Bench but reduces it on Terminal-Bench, with little stable success-rate change. This dependence on benchmark type indicates that planning effectiveness is not a monotonic function of parameter count. It depends on the model’s failure mode and on the trajectory-length distribution induced by the task family. On Terminal-Bench, planning can simultaneously preserve trajectories that would terminate early and truncate excessively long trajectories; the net cost effect depends on which population dominates.

The paper’s interpretation is therefore narrower than a general claim that explicit planning is beneficial. The tested planning scaffold is useful when a model prematurely abandons a task and useful for strong models that over-verify, but the same mechanism can increase cost without improving accuracy when the model already maintains an adequate trajectory.

Action-space effects and the capability crossover

The action-space ablation compares a structured interface containing read_file, write_file, edit_file, search tools, bash, and auxiliary tools against a bash-only interface. This is not an isolated tool-count comparison. The intervention also changes interface instructions, file-state tracking, read-before-write enforcement, and automatic post-edit diagnostics. The results must therefore be interpreted as effects of complete action-interface designs.

Structured tools strongly benefit the weakest model. For Nemotron-3 30B, the full interface improves success by 15.0 percentage points on SWE-Bench and 10.1 points on Terminal-Bench relative to bash-only. Under bash-only, 66% of Terminal-Bench trajectories terminate after emitting calls incompatible with the available interface, reducing the average trajectory from 71 to 15 turns. The structured tools provide a model-compatible action vocabulary that the 30B model fails to reproduce reliably through shell commands.

The effect is much smaller for Nemotron-3 120B. The structured interface improves success by 1.6 points on SWE-Bench and 4.5 points on Terminal-Bench, while reducing average trajectory length and avoiding the large cost increase that would otherwise accompany longer execution. At 550B, the direction reverses: bash-only improves success by 3.6 points on SWE-Bench and 5.6 points on Terminal-Bench while reducing cost by 53% and 30%, respectively. The model issues 32% fewer calls on SWE-Bench and 24% fewer on Terminal-Bench under bash-only.

Figure 6: Structured tools help bash-weak models, whereas bash-only lowers cost and can improve accuracy for bash-capable models.

Trajectory-level statistics show that bash-only changes action granularity. For Nemotron-3 550B on Terminal-Bench, the median trajectory falls from 47 to 31 actions while the share of Write Code actions rises from 16% to 27%. The median largest edit on SWE-Bench increases from 18 to 54 lines, and repeated patching declines from 4.6 to 1.5 re-patches per task. On Terminal-Bench, the share of create-or-replace file-writing actions increases from 51% to 76%. Bash allows capable models to compose multiple low-level operations into fewer shell calls, reducing interaction overhead.

Mistral-Medium-3.5-128B demonstrates that the crossover also depends on task type. The structured interface improves SWE-Bench success by 23.2 points, while bash-only improves Terminal-Bench success by 6.7 points. Mistral routes 71.9% of Terminal-Bench workspace actions through bash under the full interface, compared with 40.4% on SWE-Bench, indicating that Terminal-Bench is more shell-centric. Conversely, on SWE-Bench, removing structured tools causes 32.8% of bash-only runs to terminate without editing any file, compared with 1.2% under the full interface. The preferred action space is therefore jointly determined by model-specific shell proficiency and the operational structure of the benchmark.

Trajectory-level synthesis

The trajectory analysis unifies the aggregate results by associating each harness component with a distinct failure mode.

Figure 7: SWE-Bench trajectories show context management preserving execution, planning changing termination behavior, and bash-only altering repair granularity.

Context management primarily increases trajectory survival. At tight context budgets, it preserves more active runs at later turns without substantially changing the proportions of localization, reproduction, fixing, and verification. At 128k, context-management tiers have nearly identical median trajectory lengths and similar re-patch counts, confirming that their principal effect is to prevent context overflow rather than to induce a different repair policy.

Planning changes the location at which trajectories stop. For weak models, it prevents premature termination before editing. For stronger models, it suppresses redundant post-edit verification and reduces the long tail of exploratory or repetitive execution.

Figure 8: Terminal-Bench trajectories show how planning reshapes trajectory length and bash-only concentrates actions into code-writing operations.

The action space changes the granularity of work. Structured tools decompose exploration and editing into typed, low-level operations, which benefits models that cannot reliably express those operations through shell syntax. Bash-only interfaces permit capable models to combine inspection, transformation, and execution into composite commands, reducing the number of interaction steps but increasing the risk of interface misuse for weaker models.

These mechanisms also explain the changing failure-stage distribution. For Nemotron-3 30B, more than half of unresolved SWE-Bench runs fail at file localization, and disabling planning or structured tools raises the localization-failure share to 73.8% and 76.6%, respectively. For Nemotron-3 550B and Mistral, most unresolved runs reach the correct file and lines but fail during patch implementation. The bottleneck therefore moves with capability: weaker models primarily fail to sustain or direct the trajectory, whereas stronger models more often produce an inadequate repair after successful localization.

Limitations and open questions

The results estimate the effects of particular implementations, not universal properties of planning, tools, or memory. Planning is instantiated through one prompt protocol and one persistent plan representation. Context management uses fixed thresholds and a single summarization prompt. The action-space comparison bundles tool availability with interface instructions, state tracking, and post-edit diagnostics, so it cannot isolate tool granularity from these supporting mechanisms.

The factorial design is also incomplete. Planning and action-space ablations are evaluated only under T4 with a 128k budget, leaving their interactions with smaller windows and alternative context tiers unresolved. Each task is run once per setting, and Terminal-Bench contains only 89 tasks; consequently, many Terminal-Bench contrasts do not reach significance under the paired McNemar analysis. The trajectory annotations are produced by LLM judges, although the reported validation is strong: aggregate judge–human agreement is approximately 94.2%, with a weighted mean Cohen’s $\kappa$ of 0.929.

External validity is bounded by four models and two benchmarks, with SWE-Bench Verified restricted to Python repositories. Model size is an imperfect capability proxy because training data, tool-interface exposure, and native shell proficiency may contribute to the observed effects. The paper consequently does not establish universal crossover points at which structured tools should be replaced by bash, nor does it determine whether the same context-management conclusions hold for other harnesses, models, programming languages, or task distributions.

Conclusion

The paper demonstrates that coding-harness components have conditional, model-dependent effects rather than uniformly positive contributions. Context management is most valuable when context capacity is scarce, primarily because it prevents overflow-induced trajectory termination. A staged policy that performs rule-based elision before selective summarization achieves the strongest cost profile, while recoverable recall is rarely used and provides no accuracy improvement. Planning improves weak-model success by sustaining execution but mainly reduces strong-model cost by limiting redundant verification. Structured tools scaffold models with weak shell control, whereas bash-only interfaces can improve both efficiency and accuracy for models capable of composing dense shell actions, especially on command-line-centric tasks.

The resulting design principle is empirical rather than architectural: harness components should be selected according to model capability, task structure, and context budget. The main unresolved question is whether these conditional effects persist under a full factorial evaluation in which planning, action space, and context management are jointly varied across models and context capacities.