LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

Abstract: Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the \textbf{Controller}: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the \textbf{Worker}, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is \textbf{24.69\%}, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages \textbf{64.4\%}, and Type II produces a similar ordering under the main Core criterion (Spearman's (ρ=\textbf{0.9747})). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .

• The paper introduces a benchmark LoopArena to isolate model Controllers-in runtime scenarios.

LoopArena isolates a component that is usually confounded with the coding agent itself: the model responsible for managing an ongoing execution loop. In conventional repository-level coding benchmarks, the evaluated object is an integrated model–harness system whose final repository state reflects the Worker model, prompt protocol, tool interface, state management, and termination policy. “LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering” (2608.28281) instead fixes the coding Worker and execution environment while varying a separate Controller. The benchmark therefore treats runtime loop control as an independently measurable capability.

Evaluation object and harness design

The benchmark uses a two-agent architecture. The Worker performs repository exploration, code modification, command execution, testing, and other tool-mediated operations within the inner loop. The Controller operates in an outer loop: after each Worker segment, it determines whether the Worker should advance, verify, or stop, and, when continuing, specifies a bounded next assignment.

The Controller does not access the repository directly. At every handoff, a temporary Reporter receives a copy of the Worker conversation and read-only workspace access. It produces a structured summary covering task context, completed work and current state, verification evidence, and unresolved issues. The harness converts this report and the cited Worker turns into an Evidence Packet. The Controller consumes the Packet and its own persistent decision history, then emits a validated Loop Contract. The harness renders a continuing Contract as the next Worker instruction without another model call.

This information boundary is central to the paper’s causal interpretation. The Controller cannot compensate for weak coding performance by editing files or running tests itself; its influence is restricted to selecting and specifying subsequent Worker segments. Conversely, the Worker, tools, evaluator, execution budget, Reporter configuration, and control schema remain fixed across Controller comparisons.

Figure 1: LoopArena separates the Worker’s repository interaction from the Controller’s runtime supervision through a Reporter-generated Evidence Packet and a structured Loop Contract.

The design distinguishes adaptive control from persistent goal repetition through two reference policies. No control gives the Worker the task once and lets it operate autonomously. Fixed control repeatedly instructs the Worker to continue toward the original goal without inspecting current evidence. The latter is particularly useful because it tests whether apparent benefits of supervision arise merely from repeated goal restatement rather than state-dependent intervention.

Three evaluation settings

LoopArena provides three evaluation granularities with different execution costs. Type I evaluates an isolated control decision. Type II evaluates repeated control over a selected task slice from an intermediate workspace. Type III evaluates the complete repository task from its original state.

Figure 2: Type III evaluates complete tasks, Type II evaluates paired intermediate task slices, and Type I evaluates frozen control decisions through replay-validated four-way selection.

Type I contains 90 questions derived from restorable control points in Controller-guided trajectories: 40 from SlopCodeBench and 50 from BeyondSWE. Each question presents an Evidence Packet and four complete candidate Loop Contracts. The candidates are replayed from the same restored state under two predeclared matched schedules. A candidate is accepted as correct only if it is the unique winner under both schedules, using downstream success and, when necessary, downstream control-cycle and Worker-turn counts. Items with unstable or non-unique winners are rejected.

This construction avoids treating the originally recorded Contract as an oracle. The recorded decision is merely one candidate; downstream executable consequences determine the label. Once the benchmark is frozen, evaluating a new Controller requires only one four-way response and no Worker execution. Type I consequently measures decision quality at low marginal cost, although it does not measure the Controller’s ability to maintain coherent behavior across a complete trajectory.

Type II begins from a prepared intermediate workspace and evaluates one coherent development stage. Its evaluator checks cumulative requirements through that stage. Type III evaluates the paired task from its original state, including initial investigation, implementation, verification, recovery, and termination. The pairing permits direct comparison of a lower-cost task-slice evaluation with its full-task counterpart.

Benchmark composition and metrics

The executable settings contain 27 paired task instances: 11 from SlopCodeBench and 16 from BeyondSWE. Every policy is evaluated with three independent runs per task. The primary executable metric is Strict Success Rate, which counts a run as successful only when it passes the frozen evaluator and satisfies the execution protocol. Type I uses Contract Accuracy, with malformed, missing, multiple, or otherwise unparsable responses counted as incorrect.

Inference cost is estimated from recorded input and output tokens using frozen public list prices and a no-cache assumption. Controller-guided costs include Worker, Reporter, and Controller calls; no-control and fixed-control costs include Worker calls. Repository execution, evaluator computation, storage, and other infrastructure costs are excluded. The reported dollar values should therefore be interpreted as standardized model-inference estimates rather than realized deployment expenditure.

The benchmark compares five Controllers—Qwen3.7-Plus, DeepSeek-V4-Flash-0731, GLM 5.2, GPT-5.5, and Claude Opus 4.8—while using Qwen3.7-Plus as the shared Worker and Reporter configuration. This setup makes the principal comparison a Controller comparison, but it also means that the findings are conditional on the selected Worker and harness interface.

Main results

The full-task results show that runtime loop control remains difficult. Type III Strict Success Rate ranges from 16.05% to 24.69% across the five Controllers. GPT-5.5 obtains the highest score at 24.69%, followed by Qwen3.7-Plus at 23.46%, Claude Opus 4.8 at 20.99%, DeepSeek-V4-Flash-0731 at 19.75%, and GLM 5.2 at 16.05%.

Policy	Type II SSR	Type II cost/run	Type III SSR	Type III cost/run
No control	39.51%	$1.04 | 18.52% |$2.01
Fixed control	46.91%	$1.08 | 18.52% |$5.58
Qwen3.7-Plus	48.15%	$4.30 | 23.46% |$6.89
DeepSeek-V4-Flash-0731	45.68%	$2.10 | 19.75% |$10.24
GLM 5.2	37.04%	$1.63 | 16.05% |$4.86
GPT-5.5	51.85%	$5.00 | 24.69% |$18.84
Claude Opus 4.8	48.15%	$5.87 | 20.99% |$16.82

The comparison between fixed and no control is especially informative. On Type II, fixed control improves success from 39.51% to 46.91%. On Type III, however, both policies achieve exactly 18.52%. Repeatedly restating the original goal helps on bounded task slices but provides no full-task benefit over unguided execution. The implication is that long-horizon control requires adaptation to changing execution evidence rather than persistent restatement of a static objective. The result should not be interpreted as showing that all fixed-control policies are ineffective: it concerns the deterministic policy and protocol specified by LoopArena.

Controller-guided policies generally improve over the references on Type III. Four of the five Controllers exceed the 18.52% reference score, although the absolute success rates remain low. The improvement is not attributable solely to additional compute: fixed control costs $5.58 per full-task run while no control costs$2.01, yet their success rates are identical. Controller-guided execution also incurs substantial overhead from longer Worker trajectories and Reporter and Controller calls. For example, GPT-5.5 reaches 288.90 Worker turns and 13.46 control cycles on average, with an estimated total cost of $18.84 per run. Its higher success rate is therefore accompanied by a large resource increase.

Type II substantially reduces evaluation cost. Across Controllers, the paired reduction relative to Type III averages 64.4%. The reduction preserves the closed-loop interaction and evaluator-based scoring; it arises from starting at a prepared intermediate state and evaluating only one task slice. Under the main SCBench Core criterion, the Type II and Type III Controller rankings have Spearman correlation $\rho=0.9747$$. No strictly ordered Controller pair reverses its ordering, although one pair is tied in one comparison. For this Controller panel and scoring criterion, the result supports Type II as a lower-cost proxy for comparative evaluation, while Type III remains necessary for measuring complete-task performance.</p> <p>The proxy claim is sensitive to the evaluator definition. When SCBench uses all checks rather than the Core checks, the Type II–Type III rank correlation falls to 0.1481; under an all-non-error criterion it becomes -0.2962. Thus, <strong>the strong 0.9747 correlation is not a criterion-independent property of task slicing</strong>. It depends materially on the main Core scoring rule and should not be generalized to arbitrary evaluator constructions.</p> <p>Type I produces a different view of Controller competence. Contract Accuracy ranges from 72.22 <p>The low-cost setting also reveals substantial differences in evaluation expense. For 90 questions, estimated cost ranges from$$0.31 for DeepSeek-V4-Flash-0731 to$13.68 for Claude Opus 4.8. GPT-5.5 reaches the highest accuracy at $9.43. Deterministic shortcuts perform poorly: majority answer position achieves 31.11%, uniform choice 25.00%, shortest candidate 23.33%, and lexical overlap 18.89%. Since all candidate contracts share the same action label within each question, action-only selection is equivalent to uniform choice. These results indicate that Type I requires integrating the Packet evidence with the candidate instruction rather than exploiting the tested surface statistics.

Execution behavior and failure modes

The resource measurements show that Controller-guided runs are not simply more frequent supervisory interventions layered onto otherwise identical trajectories. Type II Controller runs average between 51.38 and 80.12 Worker turns and between 3.04 and 4.19 control cycles. Type III runs average between 139.81 and 288.90 Worker turns and between 8.60 and 13.46 control cycles. Long-horizon operation therefore expands both the inner-loop execution and the number of outer-loop decisions.

The study also records protocol failures caused by excessively long Controller outputs. Among 1,614 Controller calls in the reported DeepSeek and GLM diagnostic subset, 162 reach the 20,480-token output limit. GLM 5.2 is particularly affected: 18.70% of its Type II Controller calls and 20.39% of its Type III calls reach the limit. At the evaluation level, 54.76% of GLM Type II evaluations and 75.93% of GLM Type III evaluations contain at least one such call. These events count as protocol failures, so the measured success rate reflects both runtime supervision quality and compliance with the structured Contract interface.

Run-to-run variability is also nontrivial. For GPT-5.5 on Type III, only three of 27 tasks succeed in all three repeats, while 17 succeed in none, four in one, and three in two. Qwen3.7-Plus succeeds on all three repeats for only three tasks as well. Consequently, aggregate Strict Success Rate is estimated from a small task panel with appreciable stochastic variation. The three-repeat design provides some indication of stability but does not eliminate uncertainty in task-level comparisons.

Source-specific results suggest that aggregate scores conceal heterogeneous behavior. GPT-5.5 achieves 24 of 33 successful Type II SCBench runs and 18 of 48 BeyondSWE runs, while GLM 5.2 achieves 8 of 33 and 22 of 48, respectively. On Type III, GLM 5.2 succeeds on 13 of 48 BeyondSWE runs but none of 33 SCBench runs. The Controller ranking is therefore dependent on the mixture of long-horizon iterative tasks and broader software-engineering tasks represented in the benchmark.

Relation to existing evaluation paradigms

LoopArena differs from final-state coding benchmarks such as SWE-bench and long-horizon coding evaluations because it does not score the complete coding stack as a single object. It also differs from process benchmarks that label individual tool-use steps or critiques without executing their downstream consequences. Type I resembles process-level decision evaluation, but its labels are established through replayed execution rather than annotation alone. Types II and III then reconnect those decisions to executable outcomes.

The benchmark is also distinct from evaluations that compare complete harnesses. In such settings, improvements may result from changes to context management, tool design, state representation, model assignment, termination logic, or loop structure. LoopArena fixes these factors and evaluates only the model occupying the Controller role. This controlled decomposition makes attribution clearer, but it narrows external validity: a Controller that performs well under LoopArena’s Evidence Packet and Contract schema may not perform similarly under a different harness.

The design is related to manager–worker and auditor–executor systems, as well as to process and outcome supervision. Its specific contribution is to make the runtime supervisor itself the controlled evaluation target while retaining a downstream coding executor and machine-verifiable task evaluator. This permits three linked measurements: isolated decision correctness, task-slice control, and full-task control.

Limitations and open questions

The benchmark is restricted to repository-level coding tasks and one Controller–Worker organization. The Worker is fixed to Qwen3.7-Plus, and the Reporter uses the same configuration, so the reported Controller rankings may depend on the compatibility between those models and the prescribed information interface. The paper does not establish whether the same rankings hold with weaker, stronger, heterogeneous, or multi-agent Workers.

The benchmark also contains only 27 paired executable tasks and three repeats per task. The strong Type II–Type III correlation is observed for five Controllers under the Core scoring criterion, but the sensitivity analysis demonstrates that the correlation can collapse or reverse under alternative SCBench criteria. The evidence therefore supports a conditional cost-reduction result rather than a general guarantee that task slices preserve full-task rankings.

Type I avoids marginal Worker execution but requires substantial one-time construction cost, including replaying all four candidates under two schedules. Its labels are operationally grounded in success and downstream resource usage, yet the winner rule imposes a particular preference ordering: success dominates, followed by control cycles and Worker turns. Different deployment objectives could select a different Contract among multiple successful candidates. The paper leaves open how robust Type I rankings are to alternative replay seeds, winner definitions, candidate-generation procedures, and Controller interfaces.

Finally, protocol compliance is part of the score. Output-limit exhaustion and invalid Contracts count as failures, which is defensible for an executable system but conflates supervisory judgment with interface reliability. Separating these failure modes could clarify whether a model failed to choose an appropriate next action, failed to express it in the required schema, or generated an excessively long response.

Conclusion

LoopArena establishes a benchmark in which runtime loop control is evaluated independently of the coding Worker. Its Type I, Type II, and Type III settings measure control decisions, task-slice supervision, and complete-task management at progressively higher execution cost. The results show high Type I decision accuracy but low full-task success, with the best Type III Strict Success Rate reaching only 24.69%. Fixed goal repetition improves bounded slices but does not improve complete-task success over no control, supporting the paper’s central distinction between persistent objectives and adaptive supervision.

The paired Type II setting reduces estimated inference cost by an average of 64.4% and reproduces Type III Controller ordering under the main Core criterion with $1.08 | 18.52% |$0, although this relationship is highly sensitive to scoring rules. The benchmark consequently provides a useful controlled framework for studying model-based runtime supervision while leaving open how its conclusions transfer across Workers, harness interfaces, evaluator criteria, and non-coding environments.