Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

Abstract: Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.

• The paper introduces Mid-Harness, a test-time computation allocation method that enhances trajectory success by separating action-level sampling from trajectory scaling, optimising each action candidate selection prior to execution, and reducing intervention harm.
• Experiments demonstrate that action-level sampling significantly improves success when a reliable verifier is in place, primarily with long-horizon terminal tasks owing to substantial dependency on correct commands in an evolving environment.
• During test-time Mid-Harness, a distillated verifier consistently improved performance in both terminal agent tasks, leading to an average performance improvement of over 7%, and verifier training enhances trajectory success without changing the action generator.

Terminal agents fail not only because their models lack a useful action, but because they commit to one stochastic action before the environment can reveal whether an alternative would have been preferable. In long-horizon terminal tasks, this distinction is consequential: an incorrect edit, package installation, or process-management command changes the state on which all subsequent decisions depend. “Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents” studies test-time compute allocation at this model–environment interface (2609.39982). Its central claim is that action-level sampling can improve trajectory success, but only when the verifier can reliably distinguish useful candidates before execution.

Problem formulation and method

Mid-Harness inserts candidate generation and verification into the model-call wrapper while leaving both the action generator and execution harness unchanged. At state $h_t$, the generator samples $N$ candidate actions from the same interaction history. A verifier selects one candidate, which is then passed to the original harness for execution. The remaining candidates are discarded, and the next candidate set is generated only after observing the resulting environment state.

This placement separates action scaling from trajectory scaling. Best-of-$T$ and related methods generate multiple complete trajectories and select among them after execution; Mid-Harness performs selection before an action mutates the environment. The distinction is operationally important because it requires only one environment instance for each trajectory and can reject a potentially harmful command before it is run. It also creates a stricter verification problem: the verifier must infer command semantics, execution feasibility, and task progress without observing the candidate’s actual effects.

The study evaluates three verification mechanisms. Listwise verification presents all candidates in one prompt, pointwise verification scores each candidate independently, and pairwise verification compares candidates under a shared context and aggregates the resulting preferences. For eight candidates, the default pairwise procedure uses a ring followed by a pivot tournament, requiring approximately 22–25 comparisons rather than all 28 unordered pairs. The generator samples with temperature 0.8, while the TMAX-9B verifier operates without explicit reasoning in the primary pairwise configuration unless otherwise stated.

The main evaluation uses 98 valid tasks from TerminalBench-Lite, three runs per task, TMAX-9B as the generator, and the Vanillux2 harness. Pass@1 is the mean exact-success rate over runs; Pass@3 is the fraction of tasks solved by at least one of three runs. Additional experiments use TMAX-4B and TMAX-27B, Terminal-Bench 2.1, SWE-bench-Verified Mini, FeatureBench-Mini, Qwen3.5-9B, Nemotron3.5 Lightning, Nemotron3 Ultra, and the Terminus-2 harness.

Candidate coverage and the dependence on verification

The first empirical question is whether additional samples from the fixed generator contain useful alternatives. A frontier verifier, GPT-5.6 Sol, provides an indirect test because the paper has no gold action labels. With TMAX-9B generating candidates, frontier listwise verification raises Pass@1 from 50.00% for the base agent to 64.63% with four candidates and 68.03% with eight. Pass@3 increases from 69.39% to 80.61% at $N=8$.

Figure 1: Frontier verification extracts useful alternatives from TMAX-9B candidate sets, whereas zero-shot verification gains little from increasing candidate width.

This result supports the existence of exploitable candidate coverage: the generator already produces actions that can yield substantially better trajectories than the first sampled action. The improvement does not arise from modifying the generator or the harness. However, it is only an upper-bound-like diagnostic rather than a direct estimate of candidate quality, because trajectory success conflates action selection with all subsequent generation and execution decisions.

The stronger claim follows from the self-verification experiments. With TMAX-9B serving as both generator and verifier, increasing candidate width is ineffective under listwise verification. Pass@1 changes from 49.32% at $N=4$ to 51.02% at $N=8$, while Pass@3 changes from 66.33% to 67.35%. Thus, more candidates do not compensate for weak verification. The contrast with GPT-5.6 Sol indicates that candidate sampling is not an independent source of performance; its value is bottlenecked by the verifier’s ability to identify the correct action.

Verification structure matters even when the verifier model is fixed. At $N=8$, pointwise verification reaches 52.38% Pass@1 and 67.35% Pass@3, whereas pairwise verification reaches 54.76% and 71.43%, respectively. Pairwise comparison provides a common reference for alternatives that may have different purposes, such as inspecting evidence, repairing an artifact, or retrying a failed command. Independent pointwise scores must instead be comparable across such heterogeneous action types.

The result is important but not universal: pairwise verification is best among the evaluated mechanisms, not necessarily optimal under equalized cost. Its advantage is accompanied by substantially greater decoding expense. At $N=8$, zero-shot pairwise verification produces approximately 125.7k verifier output tokens per run, compared with 0.6k for listwise and 7.0k for pointwise verification.

Figure 2: Pairwise verification improves success but incurs much greater verifier decoding cost than listwise and pointwise alternatives.

Distilling frontier verification

The authors next transfer frontier-verifier behavior into a TMAX verifier using LoRA. The training corpus contains 117,631 pairwise supervision instances generated by GPT-5.6 Sol from trajectories on difficult TMAX-15k tasks. The action generator remains unchanged; only the verifier receives the adapter.

Distillation raises pairwise Mid-Harness performance at $N=8$ from 54.76% to 57.14% Pass@1 and from 71.43% to 75.51% Pass@3. At $N=4$, Pass@1 rises from 54.42% to 55.44%, and Pass@3 from 68.37% to 70.41%. The effect therefore demonstrates that verifier training can improve trajectory success without improving the action generator.

Offline diagnostics provide evidence that the behavioral transfer is not merely an artifact of end-to-end task variance. On 1,355 states with valid comparisons for both models, distillation reduces pairwise score MAE relative to the frontier verifier from 2.59 to 1.05. Pairwise preference agreement increases from 59.01% to 74.58%, and agreement on the top-ranked candidate increases from 38.52% to 57.79%.

Figure 3: Distillation substantially increases agreement with the frontier verifier, although later-trajectory states remain difficult.

Agreement is strongest early in trajectories: it reaches 68.13% for turns 1–4 but falls to 54.07% for turns 17–32. This degradation has a direct interpretation. Later states contain longer histories, more accumulated environmental changes, and more context-dependent action consequences, so improving verifier agreement does not eliminate the difficulty of reasoning about evolving execution state.

The residual failure analysis identifies candidate semantics and execution feasibility as 67.4% of the reviewed distilled-verifier failures. These categories include misinterpreting valid code, overlooking self-termination hazards, and preferring an action whose intended effect is plausible but whose execution cannot reach that effect. The analysis is necessarily relative to a teacher judged by GPT-5.6 Terra; these cases are not independently established gold errors. Nevertheless, the concentration of failures indicates that terminal-action verification requires more than recognizing generally plausible repairs. It requires predicting what a concrete command will do in a particular filesystem, process table, dependency environment, and task state.

Verification cost and decision-only inference

The paper examines whether explicit verifier reasoning is necessary. A decision-only pairwise verifier emits only an A/B preference, avoiding rationale and score generation. For TMAX-9B at $N$0, decision-only verification improves Pass@1 from 54.76% to 56.12% in the zero-shot setting and from 57.14% to 59.18% after distillation. Reference-priced cost falls by 20.9% and 24.1%, respectively.

This finding is contradictory across model scales. At TMAX-4B and TMAX-27B, decision-only verification lowers Pass@1 relative to reasoning-based verification. For distilled models, the changes are 43.88% to 41.50% at 4B and 76.19% to 74.15% at 27B. The paper therefore does not establish that shorter verifier outputs are generally superior. The benefit depends on model scale, training configuration, aggregation procedure, and candidate width.

The cost accounting also distinguishes parallelized output tokens from total verifier output. Pairwise comparisons can run largely in parallel, but the ring and pivot stages introduce sequential dependencies. Moreover, long candidate actions and histories make verifier input costs substantial: the distilled TMAX-9B pairwise verifier has a median input length of 6,495 tokens and a 90th percentile of 10,380 tokens.

The experiments further show that simply increasing verifier inference does not reliably improve outcomes. Sampling five verifier responses per pair lowers Pass@1 from 57.14% to 53.40% while multiplying verifier output by approximately 4.99. Rubric scaling yields only a small change, from 54.76% to 55.10% Pass@1, with more than twice the output. Thus, additional verifier computation is not interchangeable with better verifier judgment.

Composition with trajectory-level scaling

Mid-Harness complements both parallel and sequential trajectory scaling. With TMAX-9B, Best-of-$N$1 using three complete trajectories reaches 55.10% Pass@1. Adding zero-shot Mid-Harness raises this to 61.22%, while distilled Mid-Harness reaches 66.33%, an 11.23-percentage-point improvement over Best-of-$N$2 alone with the same number of environment executions.

Sequential Refine produces a similar pattern. One refinement round reaches 55.10% without Mid-Harness and 60.20% with distilled action verification. Pass@3 increases from 71.43% to 75.51%.

Figure 4: Action-level verification improves both parallel trajectory selection and sequential refinement across model scales.

The cost comparison is particularly strong for the TMAX-9B setting. Distilled Mid-Harness at $N$3 reaches 57.14% Pass@1, matching Best-of-$N$4 at $N$5 at approximately one-third of its reference-priced token cost. Combining distilled Mid-Harness with Best-of-$N$6 at $N$7 reaches 66.33%, exceeding Best-of-$N$8 at $N$9 at lower estimated cost. Combining it with one round of Sequential Refine reaches 60.20%, also above the performance of Best-of-$T$0 at $T$1.

These comparisons support a specific allocation principle: action-level and trajectory-level scaling address different failure modes and can be composed more efficiently than spending all additional compute on independent complete runs. The conclusion depends on the paper’s reference prices and idealized cost model, which count token usage but do not fully capture queueing, hardware utilization, environment startup, or the latency of sequential verifier stages.

Transfer across models, tasks, and harnesses

The action-scaling gains persist across the three TMAX sizes. On TerminalBench-Lite, distilled Mid-Harness raises Pass@1 from 38.78% to 43.88% at 4B, from 50.00% to 57.14% at 9B, and from 71.09% to 76.19% at 27B. The corresponding gains are not monotonic in absolute magnitude, but the direction is consistent in these aggregate results.

A broader transfer evaluation shows improvements on difficult task distributions:

Benchmark and setting	Base Pass@1	Zero-shot	Distilled
Terminal-Bench 2.1, TMAX-9B	21.72%	27.34%	26.59%
SWE-bench-Verified Mini, TMAX-9B	46.67%	48.00%	48.67%
FeatureBench-Mini, TMAX-9B	1.45%	5.80%	7.25%
FeatureBench-Mini, TMAX-27B	17.39%	17.39%	23.19%

The FeatureBench-Mini result is notable because the base TMAX-9B agent succeeds on only 1.45% of runs, while zero-shot and distilled verification raise this to 5.80% and 7.25%. At TMAX-27B, distillation increases Pass@1 by 5.80 percentage points and Pass@3 by 13.04 points. These results indicate that action verification can be useful in low-success regimes, although the absolute performance remains low.

Transfer also extends beyond TMAX models and Vanillux2. Zero-shot Mid-Harness improves Qwen3.5-9B and Nemotron3.5 Lightning under Terminus-2, as well as Nemotron3 Ultra on Terminal-Bench 2.1. Across the reported seven settings, zero-shot Mid-Harness improves Pass@3 and matches or improves Pass@1. Distillation is less uniformly beneficial: it improves some benchmarks while reducing Pass@3 on FeatureBench-Mini relative to zero-shot verification and failing to improve Terminal-Bench 2.1 over the zero-shot variant.

The stratified analysis qualifies the aggregate improvements. Distilled Mid-Harness helps software engineering, machine learning, debugging, and system-setup tasks across model sizes, but scientific-computing tasks include a 10.0-point regression at 27B. Longer base-agent executions often benefit, yet gains do not increase monotonically with observed trajectory length. The method therefore does not provide a universal improvement for every task class or execution horizon.

Limitations and open questions

The evaluation lacks gold action labels and cannot directly measure candidate coverage, verifier correctness, or the success attainable under perfect action selection. Because executing a candidate changes the environment, offline reuse of a fixed trajectory set cannot fully evaluate alternative verifiers online. Branching evaluation would require environment cloning, restoration, or a sufficiently accurate world model.

The primary comparisons also use only three runs per task, constrained by the cost of fresh Docker environments and long sequential executions. The bootstrap intervals show that the strongest evidence is concentrated in distilled TMAX-9B and TMAX-27B comparisons. Several other positive point estimates have intervals containing zero, so the paper does not establish statistically resolved improvement for every model and verifier configuration.

The compute allocation remains incomplete. The experiments keep generator reasoning settings fixed and do not compare candidate sampling against increased reasoning effort for a single candidate at matched cost. They also verify every action with a fixed candidate width, leaving adaptive policies for allocating verification effort unresolved. In addition, the frontier verifier remains substantially stronger: at $T$2, GPT-5.6 Sol reaches 68.03% Pass@1, compared with 57.14% for distilled TMAX-9B. The paper therefore establishes partial knowledge transfer, not parity with frontier verification.

The principal open technical question is how to train verifiers to model action effects and execution feasibility in later, heavily modified states without requiring exhaustive environment branching. A related question is whether generator and verifier should be jointly optimized so that the verifier’s training distribution tracks the generator’s evolving candidate distribution.

Conclusion

Mid-Harness isolates action-level test-time scaling at the boundary between a LLM and its execution harness. Its results show that TMAX-9B generates useful alternatives, but weak verification cannot exploit them; pairwise comparison and frontier-verifier distillation recover part of the available performance. Distilled verification raises TMAX-9B Pass@1 from 50.00% to 57.14% on TerminalBench-Lite without changing the generator, and composition with trajectory scaling reaches 66.33% at lower estimated token cost than generating substantially more complete trajectories alone. The remaining gap is concentrated in environment-dependent command semantics and execution feasibility, establishing action verification—not candidate generation—as the principal bottleneck identified by the study (2609.39982).