---
title: LoopArena
url: https://www.emergentmind.com/topics/looparena
type: topic
---

# LoopArena

LoopArena is a benchmark for evaluating models as runtime controllers of long-running coding-agent loops. It separates the **Worker**, which inspects repositories, edits files, runs tools, and executes tests, from the **Controller**, which receives structured evidence about progress and decides whether the Worker should advance, verify, or stop. By fixing the Worker, tools, task evaluator, runtime limits, and interface, LoopArena isolates outer-loop supervision from end-to-end coding ability. The benchmark evaluates this capability through isolated Contract selection, repeated control over task slices, and full-task execution [2608.28281].

## 1. Concept and motivation

LoopArena operationalizes **Loop Engineering**, in which development is organized around iterative coding-agent loops rather than manually authored prompts. A long-running software-engineering task may require repository exploration, implementation, focused verification, recovery from failed checks, integration, and termination. A loop can fail when it trusts stale progress reports, mistakes a narrow passing test for completion, allocates its budget inefficiently, repeats unproductive work, or stops before hidden requirements are satisfied.

Conventional coding-agent benchmarks generally score an entire system comprising the model, prompts, tools, memory, harness, and control policy. Consequently, a successful or failed run does not identify whether the decisive factor was the Worker’s coding ability or the surrounding loop’s guidance. LoopArena changes the evaluation object to the Controller’s ability to select effective next actions for a fixed Worker.

For executable task $i$ and control cycle $k$, the harness constructs an Evidence Packet $x_{i,k}$. Given its preceding decision history $h_{i,k}$, Controller $\pi$ produces a Loop Contract:

$$
c_{i,k}=\pi(x_{i,k},h_{i,k}).
$$

The Controller cannot inspect the repository directly, execute commands, or edit files. Its influence is mediated entirely through instructions given to the Worker. LoopArena also includes two nonadaptive baselines:

- **No control**: the Worker operates autonomously.
- **Fixed control**: the harness repeatedly restates the original task goal without adapting to evidence.

These baselines distinguish adaptive supervision from merely providing additional continuation prompts.

## 2. Roles, evidence, and control protocol

### Worker

The Worker is the only component allowed to inspect repository contents, run commands and tests, edit files, use coding tools, and follow its native ReAct-style coding loop. It maintains a persistent conversation throughout a run. Each assigned segment is bounded by a Controller-generated completion condition or by the relevant runtime budget.

At the beginning of a run, the Worker receives the task instruction and an orientation-only bootstrap assignment:

> Understand the task, locate the relevant repository area, and identify the current implementation state; do not implement yet.

This initial handoff requires the Controller to determine whether the Worker should investigate further, implement, verify, recover, or stop.

### Reporter

At each Controller-guided handoff, the harness creates a temporary Reporter using the same model configuration as the Worker. The Reporter receives a copy of the accumulated Worker conversation and read-only access to a static workspace view. It cannot edit files, run code or tests, modify the persistent Worker conversation, or decide the next Worker action.

The Reporter produces four factual sections:

1. Task context and constraints
2. Work history and current state
3. Verification and evidence
4. Open issues and uncertainty

Material claims cite relevant Worker turns. The Reporter’s interaction is temporary and is not appended to the Worker’s persistent history.

### Evidence Packet

The harness deterministically combines the Reporter output and the complete cited Worker turns into the Evidence Packet. The Controller sees the overall repository task, the four Reporter fields, cited Worker turns, and total, used, and remaining Worker-turn budget.

The Controller does not see the repository directly, coding tools, private evaluator output, hidden tests, or later events from the source trajectory. Additional provenance fields are logged for audit but are not exposed to the Controller.

### Loop Contract

The Controller returns a structured Loop Contract containing an action and rationale. The action space is:

$$
\{\texttt{advance},\texttt{verify},\texttt{stop}\}.
$$

For `advance` or `verify`, the Contract must additionally specify:

- a Worker instruction;
- goal;
- context;
- required outcomes;
- prohibited actions;
- completion condition;
- protected invariants;
- verification acceptance condition.

For `stop`, only the action and rationale are required. The harness validates the response and records the validated object as canonical. A continuing Contract is rendered as the next Worker message; a stopping Contract submits the current workspace to the evaluator. No additional model call is used to format a valid continuing Contract.

A controller-guided run proceeds by restoring the starting repository state, starting a fresh Worker conversation, issuing the bootstrap assignment, allowing a bounded Worker segment, pausing the conversation, running the Reporter, building the Evidence Packet, querying and validating the Controller, and either rendering a continuation or stopping for evaluation. The process repeats until stopping, budget exhaustion, protocol failure, or another terminal runtime condition.

## 3. Tasks and benchmark construction

LoopArena draws tasks from **SlopCodeBench (SCBench)**, which targets long-horizon iterative coding and degradation over repeated work, and **BeyondSWE**, which covers software-engineering tasks beyond ordinary single-repository bug fixing.

| Setting | Instances | SCBench / BeyondSWE | Worker execution during evaluation |
|---|---:|---:|---|
| Type I | 90 questions | 40 / 50 | None |
| Type II | 27 slices | 11 / 16 | Yes |
| Type III | 27 full tasks | 11 / 16 | Yes |

The 27 Type II slices and 27 Type III full tasks are paired. Each Type II slice is derived from one corresponding full task, and historical aliases do not create additional scored tasks.

### Type III full tasks

A Type III case preserves the source task’s original specification, repository state, development process, and evaluator. The Controller manages the entire trajectory, including initial investigation, implementation, verification, recovery, and final stopping.

For SCBench tasks with multiple ordered checkpoints, the official workspace persists across checkpoints, while Worker and Controller conversations restart at native checkpoint boundaries. The task is counted once.

### Type II slices

A Type II case begins from a prepared intermediate workspace and evaluates one coherent stage of the corresponding full task. A slice is retained only when the prepared state fails at least one requirement introduced by the selected stage and the source-provided completed state passes all requirements expected through that stage.

For SCBench, native checkpoints provide task slices. For BeyondSWE, tasks are divided into coherent development stages using the official repair and tests. The evaluator checks cumulative requirements through the selected stage rather than only a superficial local condition.

### Type I questions

Type I isolates one control decision. Each question contains a frozen Evidence Packet and four complete candidate Loop Contracts, one of which is correct according to execution-based construction.

The construction procedure selects a restorable non-bootstrap point from a Controller-guided trajectory, retains the recorded Contract as one candidate, authors three plausible alternatives, freezes candidate text and presentation order, restores the same state, and replays all four candidates under two matched schedules. The schedules use the same Worker, evaluator, continuation policy, stopping rules, budget, and randomization framework. For source seed $S$, the primary and confirmation schedules use $S+1{,}000{,}000$ and $S+2{,}000{,}000$.

A schedule first compares terminal success. If multiple candidates succeed, it prefers fewer downstream Controller cycles and then fewer downstream Worker turns. Token counts and repository calls are diagnostic only. An item is retained only when one candidate is the unique winner under both schedules. Items are rejected if all candidates fail, no unique winner exists, or the two schedules disagree. Type I therefore has low evaluation-time cost, but substantial execution cost is shifted into benchmark construction.

Executable success requires both passing the frozen task evaluator and obeying the machine-verifiable control protocol. Malformed Controller responses and model-caused protocol violations count as failures. For SCBench, the main criterion requires every check in the frozen **Core** scored set to pass. For BeyondSWE, success requires reward $1$ from the official Harbor evaluator.

## 4. Evaluation settings and metrics

### Type I: Contract selection

Type I evaluates isolated next-step decision quality without executing the Worker during evaluation. If $j_q^\star$ is the frozen correct candidate for question $q$ and $\widehat j_{\pi,q}$ is the Controller’s parsed selection, Contract Accuracy is:

$$
\operatorname{Acc}_{\mathrm I}(\pi)
=
\frac{1}{N_{\mathrm I}}
\sum_{q=1}^{N_{\mathrm I}}
\mathbf{1}\!\left[\widehat j_{\pi,q}=j_q^\star\right],
$$

where $N_{\mathrm I}=90$. Reported Contract Accuracy is $100\operatorname{Acc}_{\mathrm I}(\pi)$.

Invalid Rate is the fraction of responses that fail to identify exactly one valid candidate. Missing, multiple, out-of-range, and unparseable answers count as both incorrect and invalid. All five tested Controllers have a reported Invalid Rate of $0\%$.

### Type II: Condensed coding task

Type II evaluates repeated Controller–Worker interaction over one selected slice. It retains actual Worker consequences, repeated handoffs, evidence-based decisions, and executable evaluator feedback while reducing the cost and temporal scope of a full task.

### Type III: Full coding task

Type III evaluates complete runtime loop control from the original task state. It is the most deployment-like setting but also has the highest cost and variance. Failures are more difficult to attribute to individual decisions because multiple Controller choices and Worker actions interact over the full trajectory.

### Strict Success Rate

For executable setting $t\in\{\mathrm{II},\mathrm{III}\}$, with $N_t$ tasks, $K=3$ independent runs per task, and binary run outcome $Y^{(t)}_{p,i,r}$:

$$
\operatorname{SSR}_{t}(p)
=
\frac{1}{N_tK}
\sum_{i=1}^{N_t}
\sum_{r=1}^{K}
Y^{(t)}_{p,i,r}.
$$

Reported Strict Success Rate is $100\operatorname{SSR}_t(p)$. The term **Strict** indicates that a run must pass the frozen evaluator and satisfy the control protocol; partial credit is not awarded.

### Estimated inference cost

LoopArena estimates model-inference cost from input and output tokens for every model call. Repository execution, evaluator computation, storage, infrastructure, discounts, and caching effects are excluded. Costs are calculated using frozen public standard prices and therefore estimate inference expenditure rather than measuring actual spending.

The accounting includes Worker calls for no-control and fixed-control policies, and Worker, Reporter, and Controller calls for model-guided policies. Reasoning tokens are included once as output tokens when providers report them.

For a run:

$$
C^{(t)}_{p,i,r}
=
\sum_{j\in\mathcal{J}^{(t)}_{p,i,r}}
\frac{
I_jP^{\mathrm{in}}_j+O_jP^{\mathrm{out}}_j
}{10^6}.
$$

The paired Type II cost reduction relative to Type III averages **64.4% across Controllers**.

### Rank correlation

To test whether Type II preserves Type III Controller ordering, the benchmark computes Spearman correlation over the five evaluated Controllers, excluding no-control and fixed-control. Under the main Core criterion:

$$
\rho_{\mathrm{II},\mathrm{III}}=0.9747.
$$

None of the nine Controller pairs that were strictly ordered in both settings reversed order, while one of the ten possible pairs contained a tie. The paper does not report confidence intervals, hypothesis-test $p$-values, bootstrap intervals, or significance tests for this correlation or the success rates.

Auxiliary measurements include total model tokens, input and output tokens by component, Worker ReAct turns, Controller cycles, Reporter and Controller calls and costs, wall-clock time, protocol failures, output-limit exhaustion, invalid Contract events, task-level repeat stability, source-specific success counts, and Type I shortcut accuracy.

## 5. Experimental configuration and results

The evaluated Controller panel comprises Qwen3.7-Plus, DeepSeek-V4-Flash-0731, GLM 5.2, GPT-5.5, and Claude Opus 4.8. In Type II and Type III, the Worker is Qwen3.7-Plus and the Reporter uses the same model configuration. Tools, prompts, task environment, evaluator, runtime limits, and budgets are fixed across Controllers.

Qwen, DeepSeek, and GLM Controllers use temperature $0$, maximum output length $20{,}480$ tokens, provider-default reasoning, and disabled client-library retries. GPT-5.5 uses `gpt-5.5-0424-global`, while Claude uses `claude-opus-4-8`; both use provider-default thinking, no temperature or seed parameter, and a maximum output of $20{,}480$ tokens. Each executable policy is run three times per task, producing $27\times3=81$ scored runs per policy and setting.

Runtime limits include at most 600 Worker ReAct turns or 7,200 seconds of main Worker wall time, 128 Controller-guided cycles and 86,400 cumulative seconds, 50 Reporter turns per call, 20,480 Controller output tokens per request, and 8,192 Worker and Reporter output tokens per request.

| Method | Type I Contract Accuracy | Type II SSR | Type II cost/run | Type III SSR | Type III cost/run |
|---|---:|---:|---:|---:|---:|
| No control | — | 39.51% | \$1.04 | 18.52% | \$2.01 |
| Fixed control | — | 46.91% | \$1.08 | 18.52% | \$5.58 |
| Qwen3.7-Plus | 72.22% | 48.15% | \$4.30 | 23.46% | \$6.89 |
| DeepSeek-V4-Flash-0731 | 77.78% | 45.68% | \$2.10 | 19.75% | \$10.24 |
| GLM 5.2 | 74.44% | 37.04% | \$1.63 | 16.05% | \$4.86 |
| GPT-5.5 | **87.78%** | **51.85%** | \$5.00 | **24.69%** | \$18.84 |
| Claude Opus 4.8 | 76.67% | 48.15% | \$5.87 | 20.99% | \$16.82 |

GPT-5.5 achieves the highest observed Type III Strict Success Rate, at **24.69%**. Qwen3.7-Plus is second-best or tied near the top on executable success while costing substantially less than GPT-5.5 and Claude on Type III. GLM 5.2 is the least expensive Controller-guided policy but has the weakest full-task SSR and frequent output-limit failures. DeepSeek is relatively inexpensive in Type II but becomes more expensive in Type III.

Fixed control raises Type II SSR from 39.51% to 46.91%, but Type III SSR remains 18.52%, identical to no control. This indicates that repeated goal restatement can assist a bounded stage but does not replace evidence-sensitive supervision over a full development trajectory.

A significant protocol failure mode concerns Controller output limits. Among 167 analyzed Type II and Type III evaluations, 162 of 1,614 Controller calls reached the output limit, and 77 evaluations contained at least one such call. GLM was particularly affected: 23 of 123 Type II calls reached the limit, or 18.70%, and 125 of 613 Type III calls, or 20.39%.

Full-task outcomes also display repeat variability. For example, the number of Type III tasks with zero, one, two, or three successful runs was:

| Method | 0/3 | 1/3 | 2/3 | 3/3 |
|---|---:|---:|---:|---:|
| No control | 21 | 1 | 1 | 4 |
| Fixed control | 20 | 1 | 4 | 2 |
| Qwen3.7-Plus | 17 | 4 | 3 | 3 |
| DeepSeek | 19 | 4 | 0 | 4 |
| GLM 5.2 | 21 | 2 | 1 | 3 |
| GPT-5.5 | 17 | 3 | 4 | 3 |
| Claude Opus 4.8 | 20 | 1 | 2 | 4 |

These results limit interpretation of individual full-task runs and motivate repeated evaluation.

## 6. Interpretation, limitations, and reproducibility

The Type II–Type III Core correlation suggests that task slices can provide a lower-cost approximation to Controller ordering for the evaluated panel and criterion. However, this agreement depends materially on scoring definitions. On SCBench, the Spearman correlations are 0.1481 for all checks, -0.2962 for all non-error checks, and 0.9747 for Core checks. Type II therefore functions as a validated economical comparison setting in this experiment, not as a universal replacement for full-task evaluation.

LoopArena’s results are conditional on a fixed Worker and harness. Because the main experiments use Qwen3.7-Plus as Worker, the benchmark measures Controller quality with that Worker, Reporter, task environment, evaluator, and protocol rather than Controller quality independently of all Workers. Results may change with other Worker models, Reporter models, multiple Workers, manager–executor–auditor organizations, Contract schemas, or tool interfaces.

The benchmark contains only 27 full tasks from two source datasets. It covers long-horizon and beyond-single-repository work but is not a complete representation of software engineering. Executable success is evaluator-dependent, and the SCBench Core criterion yields different rankings from alternative check sets.

Type II begins from prepared intermediate workspaces and covers one coherent stage. It may omit early repository-understanding failures, planning errors that contaminate later work, poor transitions between phases, long-run drift, and cumulative stopping mistakes. Type I is similarly construction-dependent: its answer key reflects the chosen Worker, budget, continuation policy, evaluator, and replay procedure. It measures performance on frozen decisions rather than adaptive interaction.

The Reporter introduces an additional mediation layer. The Controller does not observe raw repository state and must rely on the Reporter and cited Worker turns. Reporter omissions or errors can therefore affect measured Controller performance. The Reporter’s use of the same model configuration as the Worker also means that summarization quality is part of the evaluated interface.

The benchmark’s cost comparisons are standardized no-cache estimates based on public list prices retrieved on August 16, 2026. They exclude repository computation, evaluator computation, storage, infrastructure, discounts, and caching effects. Model revisions and prices can change, so dollar values are tied to the frozen evaluation configuration.

The supplied results report point estimates, repeated-run summaries, pairwise rank comparisons, and Spearman correlation, but do not provide confidence intervals, standard errors, $p$-values, bootstrap procedures, or formal multiple-comparison corrections. The values 24.69% full-task SSR, 64.4% paired cost reduction, and 0.9747 Type II–Type III correlation should therefore be interpreted as benchmark observations rather than estimates accompanied by formal uncertainty intervals.

The release is available at `https://github.com/AMAP-ML/LoopArena`. It includes benchmark data, construction records, harness code, configuration manifests, evaluation records, and analysis code. Released artifacts include model-visible Type I questions without answer keys, Type II starting workspaces, Type II slice specifications, Type III task specifications, Worker–Reporter–Controller runners, Contract validation and rendering, evaluator adapters, prompts, message templates, inference parameters, seeds, runtime limits, evaluator versions, token records, and aggregation scripts. Reproduction requires restoring the exact task and workspace, model revisions, prompts, inference parameters, tool and runtime limits, evaluator versions, three repeats per executable task, frozen price schedule, and supplied aggregation procedures.

LoopArena’s central significance is its separation of coding execution from runtime supervision. Type I diagnoses individual control decisions, Type II measures repeated supervision over a condensed task, and Type III evaluates complete long-horizon management. The benchmark shows that runtime control remains difficult even with a fixed coding Worker: the best observed Controller succeeds on fewer than one quarter of full-task runs. It also demonstrates that fixed goal repetition is insufficient for full tasks, while lower-cost slices can preserve Controller ordering under a particular scoring criterion. Consequently, LoopArena treats effective coding-agent management as a distinct empirical capability involving evidence interpretation, phase-aware planning, verification scheduling, recovery, budget allocation, and calibrated stopping.

Source: https://www.emergentmind.com/topics/looparena