Self-Evolving Coding Agents via Harness-Policy Optimization

Abstract: LLM agents have shown promising potential in coding tasks, and their performance is jointly determined by the policy model and the harness that mediates their interaction with the environment. Indeed, harness and policy are mutually reinforcing and evolving only one component is suboptimal---a harness tuned for one checkpoint may not target the failure modes of an updated policy, while policy learning remains limited to the trajectories elicited by a fixed harness. In light of this, we propose Self-HPO, a recursive self-improving framework for coding agents that alternates harness and policy optimization, using the evolving policy as its own harness proposer. Specifically, in each round, the current policy first analyzes its own failures and proposes new harness candidates (H-step). The policy is then updated through reinforcement learning (RL) under the selected harness (P-step), and hard tasks identified during training form the next harness search set, keeping harness optimization targeted to the policy's evolving failure modes. Across three coding benchmarks, Self-HPO consistently outperforms harness-only and policy-only baselines, improving Qwen3.5-35B-A3B's average resolve rate from 57.8% to 64.8%, with gains generalizing to out-of-distribution settings. Further analysis shows that both optimized components retain gains even when paired with their base counterparts, while their joint gain exceeds the sum of their individual gains. Self-HPO also improves reliability (Pass@1) and expands the capability frontier (Pass@k) over the base agent. Additionally, we show that with a stronger external model (e.g., Opus 4.8) as the harness proposer, the same framework yields a further performance boost.

• The paper presents Self-HOPO, which optimizes both harness and policy, proving usefulness of a fixed harness.
• Self-HPO results in a 7% increase in the average resolve rate across coding benchmarks compared to policy- and harness-only baselines.
• Evolution of harness search set based on policy failures significantly affects the performance, demonstrating the mechanism's effectiveness beyond simple prompting

The paper studies coding agents as coupled systems rather than as LLMs operating under a fixed execution scaffold. Its central claim is that policy optimization and harness optimization are mutually dependent: a harness determines which behaviors a policy can express, while policy training changes the failure distribution that the harness should address. The proposed method, Self-Harness-Policy Optimization (Self-HPO), therefore alternates between optimizing the harness and optimizing the policy, with the evolving policy itself serving as the harness proposer (2610.22791).

Problem formulation

A coding agent is represented as a pair $(H,P)$, where $P$ is the LLM policy and $H$ controls the interaction protocol: context construction, shell-tool execution, feedback injection, submission handling, and termination. For a task $x$, the pair generates a trajectory $\tau$ that is evaluated by an executable verifier. The optimization objective is the expected binary resolve rate over a training distribution.

This formulation distinguishes two mechanisms that are often conflated. Policy optimization changes the model parameters but is constrained by the trajectories produced by the current harness. Harness optimization can alter prompts, runtime logic, feedback, or termination conditions, but cannot expand the model's intrinsic behavior unless those changes expose capabilities already latent in the policy. Optimizing either component in isolation is therefore potentially misaligned with the other.

Self-HPO addresses this coupling with alternating H-steps and P-steps:

1. H-step: freeze the current policy, diagnose its failures, generate harness candidates, and retain the candidate with the highest resolve rate on a search set.
2. P-step: freeze the selected harness and update the policy using RL with executable outcome rewards.
3. Search-set update: identify tasks that remain unsolved across the P-step's rollouts and use them as the diagnostic set for the next H-step.

The method uses Qwen3.5-35B-A3B as the base policy and mini-SWE-agent as the seed harness. In the primary experiments, each H-step evaluates six candidates, each P-step performs 50 RL updates, and the full procedure runs for two alternating rounds.

Self-HPO algorithm

The H-step is failure-conditioned rather than a generic prompt search. The current policy receives trajectories from failed tasks and proposes a modification to the current best harness. Candidate modifications may change either language-level instructions or executable control logic. All candidates are evaluated on the same search set, and the best candidate is retained.

The search set is initially a random subset of 300 training tasks. After the first P-step, it is replaced by tasks that were never resolved in any rollout during policy training. This design makes the harness optimizer track the policy's evolving failure modes rather than repeatedly optimizing against a static diagnostic sample.

The P-step uses GRPO with Balanced Adaptive Rollout. BAR samples additional rollouts until it can construct reward-balanced groups, mitigating the zero-variance groups that arise when all sampled trajectories receive reward zero. The policy is trained under the optimized harness, and the resulting hard-task set becomes both a source of failure evidence and a targeting mechanism for the next H-step.

The recursive structure is important: after policy training, the updated policy is not merely a better solver; it is also the next harness proposer. Thus, policy optimization changes both the task-solving distribution and the mechanism used to modify the harness.

Figure 1: Self-HPO alternates failure-conditioned harness search with RL policy optimization and uses unresolved training tasks to define the next search set.

Experimental design

The evaluation uses 2,484 training tasks assembled from SWE-rebench, SWE-rebench V2, and Scale-SWE. Performance is measured by resolve rate on three benchmarks:

• SWE-Bench Verified: 500 human-validated tasks and the primary in-distribution evaluation.
• SWE-Bench Multilingual: 300 tasks spanning nine non-Python languages.
• SWE-Bench Pro: 642 more diverse repository-level tasks.

The self-improving comparison includes the base agent, RLVR, which optimizes only the policy, and Self-Harness, which optimizes only the harness. A second setup uses Opus 4.8 as an external harness proposer and compares SFT, Meta-Harness, and HPO. Results are averages over three evaluation runs.

Main results

Self-HPO consistently outperforms both single-component baselines. In the self-proposer setting, the average resolve rate rises from 57.8% for the base agent to 64.8% with Self-HPO. On SWE-Bench Verified, Self-HPO reaches 69.5%, compared with 67.5% for RLVR and 65.2% for Self-Harness.

Method	SWE-Verified	SWE-Multi	SWE-Pro	Average
Base	61.0%	54.2%	58.3%	57.8%
RLVR	67.5%	56.3%	61.8%	61.9%
Self-Harness	65.2%	54.3%	60.7%	60.1%
Self-HPO	69.5%	59.7%	65.2%	64.8%
HPO with Opus 4.8 proposer	71.6%	61.4%	66.0%	66.3%

The gains transfer beyond the Python-heavy training distribution. Relative to the base agent, Self-HPO improves SWE-Bench Multilingual by 5.5 percentage points and SWE-Bench Pro by 6.9 points. On SWE-Multi, Self-Harness improves over the base by only 0.1 point, whereas Self-HPO improves by 5.5 points. This contrast indicates that policy adaptation is necessary for the harness improvements to generalize across programming languages and repository distributions.

The external-proposer experiment establishes a stronger but more expensive variant. HPO with Opus 4.8 reaches 71.6% on SWE-Bench Verified and 66.3% on average, exceeding Self-HPO by 2.1 and 1.5 points, respectively. The result shows that external proposal quality remains relevant, but the self-proposer is not required for the alternating optimization principle to be effective.

Figure 2: Alternating harness and policy updates produce continued improvements, whereas harness-only or policy-only optimization saturates earlier.

The optimization trajectories support the proposed mechanism. Self-HPO and Self-Harness share the first harness update and reach the same intermediate performance. After Self-HPO updates the policy, its second harness search starts from a stronger policy and produces further gains. By contrast, an additional harness-only update can reduce performance; on SWE-Multi, Self-Harness falls from 55.0% to 54.3%. The implication is that harness search is not monotonically beneficial when its solver remains fixed.

Ablation results

The ablations isolate three design choices.

First, RL is more effective than supervised alternatives in the P-step. Replacing RL with SFT reduces the HPO result on SWE-Bench Verified from 71.6% to 67.2%, a 4.4-point decline. Replacing RL with rejection-sampling fine-tuning yields 70.5%, still 1.1 points below RL. The difference supports the use of online policy optimization under the harness rather than merely distilling successful trajectories.

Second, evolving the harness search set matters. Keeping the initial search set fixed reduces performance from 71.6% to 70.1%. The result directly supports the paper's assumption that the most informative failures change as the policy improves. It also indicates that failure-driven task selection contributes independently of the H-step/P-step alternation.

Third, the self-proposer incurs a measurable cost relative to Opus 4.8. Substituting the current policy for the external proposer lowers the Verified score by 2.1 points. Nevertheless, the self-proposer result remains above both single-component baselines, demonstrating that the method does not depend on a separately trained or proprietary proposer.

Complementarity of harness and policy improvements

The paper's factorized evaluation combines intermediate harnesses and policies in all pairings. The base pair $(H_0,P_0)$ obtains 61.0% on SWE-Bench Verified. Applying the optimized harness $H_2$ to the base policy raises performance to 64.5%, while applying the final policy $P_2$ to the seed harness raises it to 64.4%. The full pair $(H_2,P_2)$ reaches 69.5%.

Consequently, the joint gain is 8.5 points, whereas the isolated gains sum to only 6.9 points. The combined improvement exceeds the additive sum of the individual improvements by 1.6 points. This is evidence of interaction rather than simple accumulation: the policy benefits from behaviors elicited by the optimized harness, and the optimized harness remains effective because it is adapted to the policy's updated capabilities.

The result also weakens a possible interpretation of Self-HPO as merely policy RL with incidental prompt engineering. Both components retain transfer value when paired with their base counterparts, and their strongest performance occurs only when they are co-adapted.

Reliability and capability frontier

The analysis evaluates eight samples per task to distinguish first-sample reliability from the broader probability that at least one attempt succeeds. Self-HPO improves pass@1 at every reported sampling budget and raises pass@8 from 80.0% to 84.8%, a 4.8-point increase.

Figure 3: Self-HPO remains above the base agent across sampling budgets from pass@1 through pass@8.

The pass@1 breakdown is nonuniform. Relative to the base agent, Self-HPO improves Medium tasks by 17.2 points, Easy tasks by 12.9 points, and Hard tasks by 6.1 points. However, performance on tasks already resolved by the base agent declines by 2.2 points. The method therefore improves consistency on partially solved tasks and expands performance on difficult tasks, but does not dominate uniformly on every task stratum.

This distinction is important. The increase in pass@8 suggests that Self-HPO expands the set of tasks for which the policy can produce a successful solution under repeated sampling. The pass@1 improvements on Medium and Easy tasks indicate greater reliability. The decline on the Resolved category shows that the optimization may introduce regressions even while improving aggregate performance.

During the final P-step, the fraction of zero-success rollout groups falls to 6.5% over the last ten updates, compared with roughly 11–17% for the final 50 updates of RLVR. Under binary rewards, a zero-success group supplies no within-group contrast for GRPO. The optimized harness therefore appears to improve policy learning partly by making successful trajectories more observable to the RL procedure, not solely by directly increasing evaluation-time success.

Harness evolution and intervention mechanisms

The evolved harness does not replace the seed agent's core ReAct loop. It subclasses the seed implementation and overrides the step function with four targeted interventions:

• A budget valve warns the policy near the step or time limit and requests a focused best-effort submission.
• An empty-patch guard rejects a submission with no diff and gives the policy another turn.
• A contract-grounding intervention delays the first valid-looking submission to prompt verification of API names, signatures, and repository conventions.
• A patch fallback replaces a malformed submission with the sandbox's Git diff when the diff fails an apply check.

These interventions target failures in execution and submission rather than requiring new domain knowledge. In the qualitative case study, the final policy edits the correct Sphinx source file and passes the relevant tests under both harnesses. Under the seed harness, it submits a prose summary instead of a patch, producing no evaluated code change. Under the evolved harness, the empty submission is intercepted; the policy then generates and submits the Git diff, resolving the task.

Figure 4: Harness evolution improves the best-so-far score by adding targeted submission and execution controls around the existing agent loop.

The case study provides a concrete interpretation of the aggregate gains: the harness can convert an already successful internal trajectory into a valid benchmark submission. It does not establish that every improvement has this character, but it demonstrates how runtime intervention can recover failures that policy-only training may not reliably eliminate.

Limitations and open questions

The experiments use one base model, one seed harness, one primary coding-agent architecture, and two optimization rounds. The conclusions therefore do not establish that the same schedule, candidate budget, or RL configuration is optimal for other policies or environments. Harness search is also evaluated through a small candidate budget of six candidates per H-step, and each candidate is evaluated once on the search set, leaving sensitivity to search noise insufficiently characterized.

The method assumes that executable binary verifiers provide an adequate training signal. This assumption is reasonable for SWE-Bench-style patch validation but does not address tasks with incomplete tests, nondeterministic environments, or semantic regressions not captured by the verifier. The search-set rule also defines hardness as failure in every rollout during a P-step; this may conflate genuine capability gaps with sampling failures or harness-induced failures.

The external-proposer comparison shows that a stronger model improves results, so the self-proposer should not be interpreted as equivalent to external harness engineering. The paper also reports an aggregate improvement despite a 2.2-point regression on base-resolved tasks. Open questions include whether search-set selection can be made robust to noisy difficulty estimates, how co-evolution behaves over substantially more rounds, and whether the observed component complementarity persists under independent task distributions and stronger policy baselines.

Conclusion

Self-HPO presents harness and policy optimization as an alternating, failure-conditioned process. The evolving policy proposes harness revisions, learns through RL under the selected harness, and identifies the hard tasks that guide the next search round. On three coding benchmarks, the method raises average resolve rate from 57.8% to 64.8% and outperforms both harness-only and policy-only optimization, with gains transferring to multilingual and broader repository-level tasks. Its strongest evidence concerns complementarity: optimized harnesses and policies each retain independent value, while their combination produces a super-additive improvement. The results establish recursive harness-policy co-adaptation as a practical design axis for coding agents, subject to the stated dependence on verifiable rewards, task-selection quality, and the limited experimental scale.