StaminaBench: Long-Horizon Coding Reliability

• StaminaBench is a benchmark for evaluating coding agent stamina by measuring how many consecutive interaction turns remain correct before failure.
• It implements an iterative REST API server task where incremental changes create a cumulatively evolving codebase, tested via programmatically generated HTTP suites.
• The benchmark shifts evaluation from one-shot correctness to persistent reliability, highlighting the critical role of feedback and harness choice in agent performance.

StaminaBench is a benchmark for evaluating the stamina of coding agents: not whether they can solve an isolated coding problem, but how many consecutive interaction turns they can handle before failing in a cumulatively evolving codebase. It was introduced to match “vibe-coding” workflows, in which an agent does not receive one neat, self-contained task and stop, but instead implements a system and then modifies it across dozens or hundreds of user-request turns. In the reported experiments, agents build a REST API server from a generated specification and then process 100 follow-up change turns, yielding codebases of up to roughly 6,000 lines of code; evaluation is fully black-box through HTTP, tests are generated programmatically without LLM involvement, and success is measured by the number of turns that remain correct before an unrecoverable failure occurs (Sobal et al., 17 Jun 2026).

1. Benchmark objective and conceptual scope

StaminaBench is explicitly framed as a benchmark for long-horizon coding reliability rather than isolated coding ability. Its central claim is that the prevailing fraction-of-tasks-solved metric, used by benchmarks such as SWE-Bench and similar single-shot tasks, says little about whether an agent can maintain coherence over a growing codebase, remember prior instructions, avoid regressions, handle renames consistently, preserve accumulated invariants, and debug itself over long conversations. The benchmark therefore shifts evaluation from one-shot correctness to persistent correctness under iterative change (Sobal et al., 17 Jun 2026).

The benchmark’s operational notion of stamina is simple: the number of consecutive turns an agent can keep the implementation correct before failing. This definition is meant to capture a setting in which the codebase is never reset between turns, complexity grows monotonically, and earlier decisions remain consequential. A scenario begins with an initial implementation request at turn 0 and then continues for up to 100 change turns, so a full run can contain up to 101 user-agent interactions.

The benchmark is built around a single but deliberately rich task family: iterative implementation of a REST API server. The initial request specifies entities, typed fields, CRUD behavior, validation logic, analytics endpoints, and sometimes business-logic action endpoints. Later prompts request changes such as adding entities, renaming fields, modifying constraints, extending enums, adding analytics, deleting endpoints, or introducing guarded state-transition actions. This concentrates the benchmark on one domain while preserving substantial structural variety.

A plausible implication is that StaminaBench occupies, within coding-agent evaluation, a role analogous to what dynamic long-horizon memory benchmarks attempt in other domains. StoryBench, for example, targets knowledge retention and sequential reasoning in branching narrative environments rather than software modification, but it shares the emphasis on multi-turn temporal dependence and delayed consequences (Wan et al., 16 Jun 2025).

2. Formal evaluation framework

StaminaBench presents a general formalism for long-horizon agent evaluation. The reference system is defined by a state space $S$, an initial state distribution $p_0(s)$, an action space $A(s)$, a transition function $T : S \times A \to S$, and an environment action-selection policy $T_{\text{env}}(a \mid s)$. These generate a reference trajectory

$s_0, a_0, s_1, a_1, \ldots, s_N.$

The agent maintains its own evolving state $\hat{s}_t$, but it does not directly observe the reference state. Instead, it receives natural-language renderings through three description functions:

• $\text{desc}_{\text{state}}(s_0)$: a natural-language description of the initial state
• $\text{desc}_{\text{action}}(a_t)$: a natural-language description of the current change request
• $\text{desc}_{\text{feedback}}(r_t)$: an optional natural-language rendering of test results

A test function compares the reference state and the agent’s implementation and returns a result $p_0(s)$0; a predicate $p_0(s)$1 determines whether the turn passed. In the REST instantiation, the initial state is an OpenAPI-like schema, the actions are schema changes, the transition function applies those changes to the schema, and the test function launches the agent’s server and executes an HTTP test suite against it (Sobal et al., 17 Jun 2026).

The paper’s Algorithm 1 instantiates the loop as follows. A schema is sampled from $p_0(s)$2; the agent reads the specification and writes the initial server; the benchmark launches the server and runs the test suite; if the result fails, the agent may retry up to the retry budget $p_0(s)$3 using feedback. If the initial implementation never passes, the scenario ends immediately. Otherwise, for each subsequent turn $p_0(s)$4, the environment samples a change, applies it to the reference specification, sends the natural-language change request to the agent, reruns the refreshed test suite, allows up to $p_0(s)$5 retries if feedback is enabled, and terminates the scenario at the first unrecovered failure. The returned score is the total number of passed turns.

This abstraction is broader than REST APIs. The paper states that the underlying framework could support alternative instantiations, and Appendix B mentions possibilities such as a CLI tool benchmark or an office-assistant environment. In StaminaBench itself, however, the concrete realization is the evolving REST server task family.

3. REST schema state, action space, and procedural generation

In StaminaBench, the state space $p_0(s)$6 consists of structured REST API schemas represented as a strongly typed data structure. These schemas include entities with fields of types such as string, integer, float, boolean, enum, and datetime; constraints such as required or optional, nullable or non-nullable, minimum and maximum numeric values, and string length limits; references and reference lists between entities; analytics endpoints such as count, sum, average, minimum, and maximum; and business logic through action endpoints that transition enum-valued state under guard conditions and trigger side effects (Sobal et al., 17 Jun 2026).

The action space $p_0(s)$7 is likewise structured rather than free-form. Supported change types include adding, removing, and renaming entities; adding and renaming fields; changing field constraints; adding and deleting analytics endpoints; adding and removing actions; changing action guards and effects; and extending enum value sets. The main text states that the system can “sample 5 individual changes and concatenate them into one bigger change at each interaction turn,” while Appendix D.2 describes an action as including 6 changes (configurable). The paper therefore records a mild specification discrepancy between the main text and appendix, but in both descriptions each user-facing turn is typically a bundle of several valid schema modifications rather than a single microscopic edit.

Scenario complexity grows over time. Early turns involve 2–3 entities and relatively simple CRUD and validation requirements; later turns can involve 10+ entities, relationships, cascade deletions, analytics, and more complex multi-step workflows. The codebase evolves cumulatively and is not reset between turns.

A typical initial specification consists of Markdown plus a README that fixes the interface contract and implementation rules. The README requires, among other things, that the agent create run_server.sh, accept a port argument, listen on 0.0.0.0, implement CRUD for every entity, provide POST /reset for test isolation, preserve exact enum values on renames, update endpoint paths when entities or fields are renamed, correctly distinguish required, default, and nullable behavior, update references after deletions, and use the client-provided id field as the primary key. Follow-up prompts are natural-language change requests, such as renaming a field, adding an analytics endpoint, or introducing a required non-nullable enum field.

A key design contribution is the procedural generation pipeline. Initial schemas are generated either by a seeded programmatic generator or by an LLM sampler, but the result must fit the same structured state representation. Change requests are generated in two modes:

• Hardcoded / programmatic sampler: draws uniformly or systematically from valid changes in the structured action space
• LLM-driven sampler: uses an LLM to propose coherent development narratives while remaining constrained by the same structured action schema

Because both samplers emit only structured, validity-checked changes, renames, references, enum edits, and analytics operations can be checked mechanically before use. The transition function that updates the reference schema is implemented deterministically in Python. The paper emphasizes that this separation between semantic prompting and typed state evolution is central to reproducibility and validity (Sobal et al., 17 Jun 2026).

4. Black-box architecture, testing protocol, and metrics

StaminaBench is designed as a fully black-box benchmark. The agent and the server run in an isolated Docker environment; the benchmark interacts with the resulting system only through HTTP. The agent writes code and a run_server.sh script, the benchmark starts the server on a chosen port, and the test harness issues HTTP requests and checks responses. Because the contract is defined at the level of URL paths, JSON bodies, and status codes, the benchmark is language-agnostic, even though the main experiments use a README that asks for Python. Appendix C argues that REST APIs are particularly suitable for this architecture, and the paper verifies the language-agnostic claim with an implementation-language ablation over Python, JavaScript, and Rust (Sobal et al., 17 Jun 2026).

Test generation is entirely programmatic and does not rely on LLM judgment. Given a schema state $p_0(s)$8, StaminaBench generates an HTTP test suite covering CRUD endpoints, validation behavior, analytics, relationships, deletions, cascade behavior, and action semantics. The tests include POST/GET/PATCH/DELETE for each entity, checks for required fields and nullability, type enforcement, range and length constraints, reference validation, aggregate correctness on empty and non-empty data, deletion verification and 404 behavior, and action happy paths and error paths with required status codes. This is one of the benchmark’s core reliability claims: tests are reproducible because they are derived mechanically from a formal schema rather than written or judged by an LLM.

A turn is considered passed if, after up to the allowed retries, the server passes the full generated test suite for the current reference schema. A scenario terminates at the first turn that remains failing after the retry budget is exhausted. The paper defines two principal metrics: average turns passed, which is the average number of passing turns per scenario before failure, and pass rate, which is the fraction of scenarios that pass all $p_0(s)$9 turns end-to-end. The notation in the extracted PDF is slightly garbled, but the prose states the intended meanings unambiguously.

The default experimental configuration uses:

Parameter	Default value	Meaning
$A(s)$0	20	scenarios
$A(s)$1	100	change turns per scenario
$A(s)$2	2	retries per turn when feedback is enabled

Under this setup, one configuration can produce up to $A(s)$3 user-agent interaction turns.

Feedback is configurable at three levels: Detailed, with specific failing assertions and request or response details; Medium, with information about which tests failed but not full details; and Minimal, which reports only that some tests failed. This makes it possible to evaluate agents either in a pure one-shot-per-turn regime or in an iterative debugging regime driven by test feedback. The same 20 scenarios are reused across conditions for paired comparisons, the paper uses paired Wilcoxon signed-rank tests scenario-by-scenario with Holm correction where appropriate, and five reruns of OpenCode + Qwen3.5-122B on the same scenarios are reported as stable within roughly 6–7 turns, indicating moderate but manageable stochasticity (Sobal et al., 17 Jun 2026).

5. Experimental results and empirical profile

The reported evaluation covers 7 open-source/open-weight LLMs—Devstral 2, Devstral Small 2, GLM-5, Kimi K2.5, Nemotron Super, Qwen3-Coder-Next, and Qwen3.5-122B—paired with 6 harnesses / agent frameworks: OpenCode, Mini-SWE, OpenHands, Mistral Vibe, Kimi CLI, and QwenCode. Not every model is paired with every provider harness, but the benchmark is explicitly presented as evaluating model–harness interaction, not model quality in isolation (Sobal et al., 17 Jun 2026).

The most striking result is the no-feedback condition. With $A(s)$4, all tested agents fail within about 5–6 turns. Table 2 reports the best no-feedback result as GLM-5 + Mini-SWE at 6.2 ± 0.7 turns, followed by GLM-5 + OpenCode at 4.5 ± 1.0 and Kimi K2.5 + Mini-SWE at 3.8 ± 1.0, while most weaker combinations fail in fewer than 2 turns. This is the benchmark’s strongest empirical confirmation of its premise: under long-horizon iterative modification, coding agents collapse quickly if they are not allowed to debug against test feedback.

Enabling detailed feedback with $A(s)$5 changes the scale of performance but does not make the benchmark easy. The strongest reported combinations are GLM-5 + OpenCode: 57.0 ± 8.6 turns, 25% pass rate, Qwen3.5-122B + OpenCode: 39.4 ± 7.1 turns, 10% pass rate, Qwen3.5-122B + Mini-SWE: 33.1 ± 9.0 turns, 15% pass rate, Kimi K2.5 + provider harness: 33.7 ± 6.4 turns, 0% pass rate, and Kimi K2.5 + OpenCode: 26.8 ± 5.5 turns. The paper states that all 26 tested model × harness cells improve significantly from $A(s)$6 to $A(s)$7.

Harness selection has a very large effect. OpenCode is reported as best or statistically indistinguishable from best for all 7 models, while OpenHands is worst for 6 of 7. Stronger models can show up to a 6× gap between their best and worst harness: GLM-5 ranges from 57.0 turns on OpenCode to 8.7 on OpenHands; Qwen3.5-122B ranges from 39.4 on OpenCode and 33.1 on Mini-SWE to 7.3 on OpenHands; Kimi K2.5 ranges from 4.9 on OpenHands to 33.7 on Kimi CLI. By contrast, weaker models often perform poorly on all harnesses.

The feedback ablation is equally strong. Under OpenCode with $A(s)$8, detailed feedback exceeds minimal feedback by 6× to 12× in turns completed. Representative examples include GLM-5: 57.0 with detailed feedback versus 10.7 with minimal feedback, and Qwen3.5-122B: 39.4 versus 2.8. The paper interprets this as evidence that current coding agents are heavily dependent on precise, local error information for recovery.

Increasing the retry budget further improves the best systems before performance plateaus. With OpenCode and up to 10 retries, the benchmark reports GLM-5 + OpenCode: 82.5 ± 7.3 turns, 65% pass rate, Qwen3.5-122B + OpenCode: 61.0 ± 7.9 turns, 35% pass rate, and Kimi K2.5 + OpenCode: 49.5 ± 6.9 turns, 10% pass rate. The paper states that most gains come in the first 3–5 attempts, after which curves flatten.

The language-agnostic architecture is supported empirically by an implementation-language ablation on OpenCode with $A(s)$9. Results vary by model: GLM-5 scores 57.0 in Python, 70.7 in JavaScript, and 50.1 in Rust; Kimi K2.5 scores 26.8, 43.6, and 13.7 respectively; Qwen3.5-122B scores 39.4, 26.8, and 8.6. The authors hypothesize that these differences reflect training distribution rather than benchmark design. They also compare LLM-driven sampling with programmatic sampling and find broadly similar performance for most models, with no significant difference except for Nemotron Super (Sobal et al., 17 Jun 2026).

6. Failure modes, limitations, and significance

The paper’s failure analysis shows that the dominant first-failure categories are implementation errors rather than total breakdown. Common failure modes include data validation errors, missing features, cascade deletion bugs, rename failures, regressions, wrong endpoint or wrong response format, type errors or precision errors, default-value handling errors, and server crashes. These are presented as failures despite an explicit README that specifies the expected behavior. The paper argues that the likely problem is not ambiguity in the instructions, but loss of careful attention over long sessions, degradation under growing context, or failure during context compression. It explicitly connects this to phenomena such as “lost in the middle” and context rot (Sobal et al., 17 Jun 2026).

As stronger agents repair more implementation bugs, infrastructure and harness failures become more visible bottlenecks. The paper identifies malformed tool calls, repetitive loops, harness loop-detection pathologies, context-compaction failures, and broad pkill commands that terminate the harness itself. OpenCode’s context compaction is reported as problematic for Qwen3-Coder-Next and Nemotron Super when they call tools during compaction; OpenHands is reported to have a strict loop detector that can poison the session after repeated messages. This reinforces the benchmark’s claim that it evaluates the reliability of the full system, not just the base model.

The benchmark is also computationally expensive. The appendix estimates that one 20-scenario sweep can consume billions of input tokens; one example reports GLM-5 + OpenCode using 4.5B input tokens and 7.5M output tokens for a single sweep. The paper estimates that reproducing comparable configurations on frontier closed-source models would cost roughly $T : S \times A \to S$022.7K for GPT-5.5 per configuration. A single scenario takes around 4–8 hours, although scenarios can be parallelized and the full 20-scenario sweep can finish in under a day on commodity infrastructure when fully parallelized.

The authors identify two principal limitations. First, multi-turn evaluation is expensive, so they use only 20 scenarios per configuration; additional scenarios would improve statistical power. Second, the benchmark centers on a single task family, iterative REST API server implementation. The paper argues that REST is broad and representative enough to capture many software-engineering issues and that the formal framework is domain-general, but it acknowledges that specific findings may not transfer perfectly to other domains.

The benchmark implementation and the generated tasks are publicly released at github.com/amazon-science/StaminaBench. The release includes the benchmark code and generated tasks, and the practical interface is intentionally minimal: the benchmark communicates only through HTTP against a server started with run_server.sh <port>, each scenario is seeded for deterministic regeneration, tests are fully black-box and programmatic, feedback granularity and retry budget are configurable, scenarios can be extended beyond 100 turns, and implementation language is unconstrained so long as the HTTP contract is satisfied (Sobal et al., 17 Jun 2026).

In the literature on agent evaluation, StaminaBench’s significance lies in making persistent correctness under iterative modification measurable in a reproducible way. Its results support a restrained but consequential conclusion: current coding agents remain brittle over long multi-turn sessions. Without detailed test feedback they usually fail in about 5–6 turns; with retries, detailed feedback, and strong harnesses they can survive dozens of turns, but are still far from robust across the full 100-turn horizon that the benchmark was designed to expose.