- The paper introduces a Manage-Execute-Audit framework in which a persistent manager tracks task state, fresh-context executors complete bounded contracts, and independent auditors verify results before records are marked complete.
- The framework raises WeaveBench performance from 51.8% to 80.7% and Terminal-Bench performance from 69.7% to 77.2%, with gains also demonstrated across OSWorld and multiple model backbones.
- The results show that harness design is a system-level capability multiplier for tasks with verifiable state, although independent auditing adds model-dependent cost and may not resolve hidden-threshold or convention-dependent correctness conditions.
Motivation and problem statement
Long-horizon agent execution—sustained reasoning, tool use, and revision across many interdependent steps—remains unreliable even for frontier models. The authors identify three recurring failure modes: compounding errors and goal drift, context rot as interaction histories grow, and task-state loss, where agents fail to retain an accurate record of requirements satisfied, artifacts produced, and facts discovered. They trace these failures to two structural properties of existing harnesses such as Claude Code and Codex CLI: task execution and task-state management share a single growing context, and completion assessment is performed by the same agent that executed the work, so incorrect self-assessments propagate into later decisions. LongHorizon-Harness (2608.01964) addresses both by reformulating long-horizon execution as a task-state management problem.
The Manage-Execute-Audit loop
The framework decomposes a task into a sequence of independently audited state transitions. A manager owns the persistent task state—a structured collection of requirements, artifacts, and facts, each marked completed, pending, blocked, or untrusted with references to supporting audit evidence—and has no direct access to the environment. Each round it emits one of four control decisions (execute, done, blocked, ask) and, when executing, constructs a bounded subtask contract specifying the immediate goal, acceptance criteria, boundary constraints, and relevant prior evidence. A fresh-context executor performs only this contract within a budget-bounded episode; its raw trajectory is discarded afterward, so only compact verified state persists across rounds. A read-only auditor then inspects the resulting environment through non-mutating tools, producing a report with three components: a completion status, an integrity status (clean/suspect/violation, with any detected mutation invalidating completed records), and evidence-backed state updates. Crucially, executor claims never directly advance the persistent state; records are marked complete only on clean audit evidence. The loop terminates when the audited state satisfies the original task, no permitted subtask can advance it, user input is required, or the round budget (Nmax=25) is exhausted.
A lightweight AgentAdapter preserves the native agent loops of interchangeable backends—Claude Code, Codex CLI, OpenClaw, Hermes Agent—across models including Qwen 3.7-Plus, Claude Opus 4.7, GPT-5.6 Luna, controlling only supplied context, tool exposure, permissions, budgets, and returned reports. GUI and CLI executor roles are separated by capability boundary, with the manager routing each contract to the appropriate interface.
Main results
The evaluation covers WeaveBench (114 hybrid GUI–CLI tasks), OSWorld 2.0 (108 desktop workflows with ~1.6-hour median human completion time), and Terminal-Bench 2.1. With matched backbones and execution backends:
| Benchmark |
Backbone |
Baseline |
LongHorizon-Harness |
| WeaveBench PassRate |
Qwen 3.7-Plus + Claude Code |
51.8% |
80.7% |
| Terminal-Bench 2.1 |
Qwen 3.7-Plus + Claude Code |
69.7% |
77.2% |
| OSWorld 2.0 Binary |
Qwen 3.7-Plus |
2.8% |
8.3% |
| OSWorld 2.0 subset (34 tasks) |
Claude Opus 4.7 |
20.6% |
35.3% |
On WeaveBench, the matched comparison isolates the contribution of the task-state layer itself; the authors note their result nearly doubles the strongest officially reported configuration (41.2% with Claude Opus 4.7), though official runs used a regular user account versus their root privileges, so those are reference points rather than matched comparisons. Gains hold across all eight WeaveBench domains, all six OSWorld capability tags, and—with Codex/GPT-5.6 Luna—reach 83.1% on Terminal-Bench 2.1. Because Terminal-Bench involves no visual perception or GUI–CLI routing, the improvement cannot be attributed to computer-use-specific mechanisms.
Two findings deserve emphasis. First, gains are complementary to backbone capability rather than compensatory: Opus 4.7 improves from 0.680 to 0.809 mean score on the WeaveBench Games subset while Qwen 3.7-Plus improves from 0.524 to 0.733, with the latter exceeding the former's baseline harness score. Second, agent capability is framed explicitly as a system property: the model determines per-round action quality, while the harness determines how reliably local capabilities are decomposed, verified, recovered, and accumulated into end-to-end completion.
Cost analysis
The overhead profile is asymmetric across roles: the manager consumes only 2.8–8.1% of total tokens, while the auditor accounts for 19.4–38.1%, making independent verification the principal additional investment. Total cost is not a fixed multiplier—it depends strongly on the underlying model. LongHorizon-Harness uses 2.3× baseline tokens on WeaveBench and 3.6× baseline output tokens on OSWorld 2.0 with Qwen 3.7-Plus, yet consumes 24% fewer tokens than baseline on Terminal-Bench 2.1 while improving success rate. On the Games subset, token consumption rises for Qwen (10.7M → 34.3M) but falls for Opus (16.5M → 11.1M), suggesting stronger models satisfy contracts in fewer audit–replan rounds. This implies harness design and model capability jointly determine both achievable performance and computational cost.
Task-dependent effectiveness
Gains concentrate where progress can be represented as verifiable environment state. On WeaveBench, Design (+60.0 pp PassRate) and Spatial/3D (+50.0 pp) improve most; on Terminal-Bench, system administration rises from 0.593 to 0.889 and hard tasks gain more (+0.122) than medium ones (+0.042). Conversely, several short analytical categories regress—notably mathematics (−0.167), data science (−0.125), and video-processing (−0.333)—and negative tags such as mteb expose cases where correctness conditions depend on hidden thresholds or conventions not recoverable from visible state, allowing a confidently verified wrong answer. The authors' diagnosis is precise: auditing can detect incorrect results and trigger recovery but cannot supply capabilities the model lacks, so benefit accrues when the bottleneck is execution reliability rather than individual-step difficulty. The Games subset illustrates the floor-raising effect starkly: all six tasks where the Qwen baseline scored at or below 0.04 recovered to between 0.30 and 0.92.
Case studies
Paired-trajectory analyses attribute the gains to specific mechanisms. In a WebRTC audit task, the baseline retries an unresponsive Wireshark dialog for over 400 steps because the failure remains embedded in its growing history, whereas the harness externalizes the stalled interaction as unresolved state and scores 0.92 versus 0.59. In a heading-normalization task, the baseline edits document XML directly and terminates on visual plausibility (score 0.00), while the harness's auditor parses the ODT XML to confirm all 15 headings carry the required style (score 0.89), preventing a plausible-but-noncompliant claim from entering persistent state. Additional cases show pre-repair evidence being preserved as pending requirements before state-modifying actions, and verified intermediate progress persisting outside execution history so later executors address only remaining deliverables.
Limitations and open questions
The paper concedes several constraints. The WeaveBench comparison against official results is confounded by differing privilege settings (root versus regular user), so cross-harness claims there rest partly on unmatched references. Token cost can be substantial for weaker backbones, and the cost profile is shown to be configuration-dependent rather than bounded. Regressions on hidden-threshold and convention-dependent tasks indicate that independent auditing amplifies whatever contract interpretation the manager holds; a misinterpreted acceptance criterion yields a confidently verified wrong answer. Human-in-the-loop handling relies on the benchmark's native user simulator rather than real users, and stale-evidence override failures are acknowledged. Open questions include how to specify acceptance criteria robustly when ground truth is not recoverable from visible environment state, and whether auditor reliability degrades on tasks lacking authoritative closure over environment state.
Conclusion
LongHorizon-Harness demonstrates that separating task-state management from execution—and admitting only independently verified facts into persistent state—yields consistent improvements across hybrid GUI–CLI, desktop, and pure CLI domains and across model backbones, at a role-decomposed and model-dependent computational cost. Its central empirical claim is that long-horizon agent capability is a property of the complete model–harness system, with the harness governing how reliably local model capability converts into end-to-end task completion.