How Harness Design Shapes Coding Agent Performance
This presentation examines a controlled empirical study that isolates three harness components—context management, planning scaffolds, and action-space design—to reveal how each mechanism interacts with model capability and task structure. By holding the execution loop and models fixed while varying these components independently across 176 experimental settings, the research demonstrates that harness effects are conditional rather than universal: weak models need trajectory scaffolding, strong models benefit from cost reduction, and the optimal interface depends on both shell proficiency and task type.Script
When a coding agent solves a software task, we usually credit the underlying language model, but three invisible components mediate everything the model does: how it manages limited memory, whether it maintains an explicit plan, and how it interacts with code. This study isolates those three harness mechanisms and reveals that their value depends entirely on which model you pair them with.
Context management does not make models smarter; it keeps them alive. At a 32 kilotoken budget, unmanaged agents overflow and terminate during localization in 79% of tasks, whereas managed agents eliminate every overflow failure and preserve the full repair workflow through fixing and verification.
Planning works differently depending on capability. For a 30 billion parameter model, enabling an explicit plan raises the median trajectory from 5 turns to 40 and cuts the rate of no-edit failures from 69% down to 28%. But for a 550 billion parameter model, planning instead trims 34 turns of redundant post-edit verification and saves 30% of inference cost with almost no accuracy change.
Structured file tools help the weakest model by 15 percentage points on software engineering tasks because it cannot reliably express read, write, and search through shell syntax. But bash-only improves the strongest model by 4 points and cuts cost in half, because it composes inspection and transformation into fewer, denser commands and raises the median edit size from 18 lines to 54.
The trajectory-level view unifies these results. Weak models fail because they cannot sustain or navigate the workflow, so more than half of their unresolved runs collapse during file localization. Strong models locate the right file and lines but produce inadequate patches, so their bottleneck shifts entirely to repair quality.
The study demonstrates that harness components have conditional, model-dependent effects rather than universal benefits. Design choices should match model capability, task structure, and memory constraints, and the factorial space remains open for future work. You can explore the full paper and generate your own video explainers at EmergentMind.com.
