How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making

Abstract: Production deployments of LLM agents remain unreliable on long, multi-step workflows even as benchmark success rates climb steadily. We argue this gap is largely an artifact of task horizon: benchmarks are dominated by short-to-medium horizons where success remains high, while production workloads demand an order of magnitude more dependent steps. We measure the effect directly, characterizing the shape of agent degradation and disentangling its cause across a large controlled study spanning nine models, six open models from 1.2B to 671B parameters, and three deployed proprietary systems; four task families, including a genuinely agentic tool-use loop; five horizons; and three context regimes. Task success follows a geometric law governed by a single per-step reliability parameter, which rises with model scale but saturates well below 1 even for the strongest models, guaranteeing eventual collapse at sufficiently long horizons. The effect is sharpest on the agentic task, where every model tested, including widely deployed systems, falls from near-perfect success to near zero within sixteen steps of (n=10,664 analyzed trajectories. Degradation is driven by step count rather than context length: bounding the context window steepens decay rather than easing it (logit slope -0.69 vs. -0.44), p=3x10-6), contradicting a lost-in-the-middle explanation and warning against a common production shortcut. Projecting measured reliability onto representative benchmark horizons quantifies a substantial gap between benchmark and production conditions, from 0.42 at GAIA-length horizons to 0.24 at hundred-step production horizons. For teams responsible for agent orchestration and reliability at scale, these results argue for horizon-aware evaluation and reliability budgeting in place of aggregate pass-rate metrics. Code, prompts, seeds, and raw trajectories are released.

• The paper identifies geometric decay as the dominant degradation pattern in 28 of 36 model–task cells, highlighting that long-horizon reliability is severely impacted by response chain error accumulation.
• Longer and more dependent steps directly relate to model performance drop with the guiding hypothesis that as per-step reliability $(r)$ increases performance is typically defined as $r^H$.
• Per-step reliability increases with model scale, but even large models (eg proprietary models) cannot eliminate compounding error weill below their initial performance on longer tasks.

Research question and central thesis

“How Fast Do Agents Rot?” (2609.01660) investigates why LLM agents that perform well on conventional benchmarks remain unreliable on production workflows requiring many dependent actions. The paper’s central hypothesis is that task success is governed by compounding per-step reliability. If an agent must execute $H$ dependent steps and succeeds at each step with probability $r$, end-to-end success is approximately $r^H$. Even a high per-step reliability therefore produces severe degradation as the horizon increases.

The paper argues that standard benchmark scores obscure this phenomenon because many benchmarks evaluate relatively short horizons. Production workflows, by contrast, often require substantially more tool calls, state transitions, and dependent decisions. The resulting benchmark–production gap is presented not as an anomalous deployment failure but as a predictable consequence of evaluating the flat upper portion of a geometric decay curve.

The empirical study examines nine instruction-tuned models from six vendor families: six open models ranging from 1.2B to 671B parameters and three proprietary systems accessed through hosted interfaces. It analyzes 10,664 trajectories across four task families, five horizons, and three context regimes. The design emphasizes exact oracle verification, paired comparisons across context conditions, and model selection among geometric, threshold, and linear decay functions.

Experimental design

The four task families are synthetic but deliberately structured to isolate different forms of sequential dependence. Ledger requires maintaining account balances through a sequence of operations. Refchain tests retrieval and state tracking by requiring the model to follow assignments amid distractors. Cipher requires ordered string transformations, making early execution errors consequential for all later states. ToolQA is the most agentic condition: the model must traverse a hidden chain by issuing inspection calls, with each subsequent target revealed only after the preceding tool interaction. Consequently, the full path cannot be precomputed in advance.

This distinction is important. The first three tasks could be criticized as structured-output or symbolic-execution probes rather than representative agent workloads. ToolQA addresses that concern by using an unmodified ReAct-style loop in which the model selects tool calls while operating under partial information. The paper therefore evaluates both controlled sequential computation and self-directed tool use.

The context manipulation is designed to separate step count from context length. In the natural regime, the model receives one instruction per turn with the complete multi-turn history. In the compressed regime, the same number of operations and turns is retained while prior context is shortened using carried state and windowed history. In the padded regime, all operations for a horizon are presented in a single turn. Reusing the same underlying task instances across regimes permits paired rather than independent-sample comparisons.

The statistical procedure is appropriate to the study’s goals. Wilson intervals are used for success rates near zero or one, and AIC determines whether geometric, threshold, or linear functional forms better describe each model–task cell. Temperature is fixed, seeds are recorded, and the complete code, prompts, task generators, raw trajectories, and analysis scripts are released.

Geometric degradation across horizons

The primary result is that geometric decay is the best-fitting functional form in 28 of 36 model–task cells. The remaining cells are better described by threshold or cliff-like behavior, in which performance remains high until a critical horizon and then declines sharply. Thus, the paper does not claim that all agents exhibit a single universal curve; rather, it identifies geometric compounding as the dominant pattern under the tested conditions.

The task-specific ordering is also informative. Refchain produces the shallowest degradation for stronger models, Ledger shows intermediate degradation, and Cipher produces the steepest decline. Cipher’s behavior follows directly from its dependency structure: one incorrectly applied edit can invalidate the final answer even if subsequent reasoning is locally correct. This result implies that horizon alone is insufficient as a reliability descriptor. The dependency topology and error recoverability of the task materially affect the location and slope of the degradation curve.

The paper makes a strong claim that deserves careful qualification: because no model reaches per-step reliability of exactly one on a non-trivial task, eventual collapse is mathematically unavoidable under the independent-step approximation. This conclusion is valid as a statement about the fitted compounding model, but its empirical force depends on how well the approximation represents actual trajectories. The later finding that hazard increases over time suggests that the constant-$r$ geometric model is, if anything, optimistic at long horizons.

Collapse on the agentic tool-use task

ToolQA yields the paper’s most operationally consequential result. Eight of nine models perform at or near ceiling at the shortest horizon, indicating that the task is not intrinsically difficult when only a few steps are required. Nevertheless, by horizon 16 every model has fallen to a small fraction of its initial performance.

The Qwen2.5-72B trajectory illustrates the inadequacy of short-horizon pass rates. It achieves perfect performance at horizons 2 and 4 but succeeds in only approximately one in eight trials at horizon 16. A benchmark restricted to the shorter conditions would therefore classify the model as highly reliable while providing no evidence of the subsequent failure regime. The paper’s broader claim is that a high aggregate score at short horizons provides little information about long-horizon reliability unless the decay curve itself is measured.

The result is particularly significant because it holds for proprietary systems as well as open models. It also occurs in a self-directed tool-use loop rather than only in tasks where the sequence of operations is externally specified. However, the authors note that the ToolQA API budget was exhausted before longer conditions could be completed. The reported collapse at horizon 16 is therefore a lower bound on the severity of degradation, not an estimate of the eventual asymptote.

Scaling and per-step reliability

Per-step reliability increases with model scale across the open-model ladder, with a reported Pearson correlation of $r=0.36$ against the logarithm of parameter count. This is a positive but modest association, and it does not establish that parameter count is the primary determinant of reliability. The study also reports diminishing returns and observes that proprietary models cluster near the strongest open models rather than substantially exceeding them.

The operational interpretation is more important than the correlation itself. Scaling improves the per-step parameter that controls compounding, but it does not eliminate residual error. Consequently, even a model that appears highly capable on short tasks can remain unsuitable for sufficiently long dependent workflows. The result supports reliability budgeting based on measured step-level performance rather than model class, parameter count, or aggregate benchmark rank.

The paper’s phrasing that per-step reliability “saturates well below 1” should be understood as an empirical observation within the evaluated models, tasks, and decoding configuration. The study does not establish a universal scaling law or an irreducible upper bound on per-step reliability.

Increasing hazard within trajectories

The data provide evidence against a strictly constant per-step hazard. Mean per-step accuracy decreases from 0.58 in the first third of long trajectories to 0.44 in the final third. The first error occurs approximately one-third of the way through a trajectory, and recovery after an error is uncommon.

This pattern suggests that geometric decay is a useful aggregate approximation but not a complete mechanistic model. If later steps are less reliable than earlier steps, then a stationary-$r$ projection will overestimate long-horizon success. Errors appear to behave as absorbing states: once an invalid state, incorrect interpretation, or malformed tool action enters the trajectory, later steps generally propagate rather than repair it.

The paper also identifies format and tool-call drift in 21% of trajectories. This is an important decomposition because it shows that long-horizon degradation is not exclusively a reasoning failure. Interface-conformance errors, parsing failures, and invalid actions compound with state-tracking and decision-making errors. In production systems, step-level validation could therefore detect both semantic and interface failures before they become end-to-end task failures.

Step count versus context length

The context-regime experiment rejects the paper’s principal alternative explanation: that degradation is primarily caused by long contexts or lost-in-the-middle effects. The natural regime has a logit slope of $-0.44$ per horizon doubling. The compressed regime, which limits retained context, performs worse and has a steeper slope of $-0.69$; this difference is reported as statistically significant with $p=3 \times 10^{-6}$. The padded regime has a slope of $-0.40$, statistically indistinguishable from the natural regime with $r$0.

These results imply that the number of dependent operations, rather than the amount of context presented to the model, is the dominant driver under the experimental conditions. Collapsing all operations into one prompt does not remove the degradation, although it begins from a lower baseline. Similarly, truncating history does not improve reliability and appears to worsen it, presumably because the agent loses the scaffold of its prior state and reasoning.

The practical conclusion is direct: naive context truncation is not a general reliability intervention for long-horizon agents. It may reduce latency or token cost while simultaneously removing information required for state maintenance. The result does not show that context length is irrelevant in all agentic systems; it shows that, in this paired design, context reduction fails to explain the dominant degradation pattern.

The claimed benchmark–production gap

The paper projects measured per-step reliability onto representative benchmark horizons and reports success estimates of 0.42 at eight steps, 0.36 at 15 steps, 0.33 at 20 steps, 0.30 at 30 steps, and 0.24 at 100 steps. These figures are used to argue that common benchmarks sample horizons before the major collapse becomes visible.

There is, however, an internal quantitative inconsistency in the supplied manuscript. The paper states that the projection uses mean per-step reliability $r$1, but direct geometric compounding would yield $r$2, not 0.42, and would produce substantially smaller values at the longer horizons. The reported sequence is also not consistent with a single constant per-step reliability: the implied reliability varies considerably across the listed horizons. The benchmark-gap argument may still be valid, but these particular numerical projections require clarification, such as a definition of $r$3 different from per-step success, a nonstandard horizon mapping, or a transcription error.

This inconsistency is consequential because the benchmark-gap quantification is one of the paper’s headline results. The qualitative conclusion—that short-horizon evaluation can substantially overstate production reliability—is supported by the trajectory results and the observed decay curves. The exact projected values, however, should not be treated as reproducible until the underlying calculation is reconciled with the stated geometric model.

Limitations and open questions

The strongest limitation is ecological validity. The task families are synthetic, enabling exact verification and controlled horizon manipulation, but they do not fully represent open-ended software engineering, web navigation, organizational workflows, or multi-agent production systems. ToolQA improves the study’s agentic relevance, yet it remains a constrained hidden-chain environment.

The model sample is broad but limited to nine systems. Three additional small models were excluded because their hosted routes failed to follow the structured response protocol, which may introduce selection effects. The fixed, moderate decoding temperature also leaves sensitivity to temperature, sampling strategy, retries, verifier design, and inference-time search unresolved. These factors could alter both per-step reliability and recovery after errors.

The geometric form is selected in most, but not all, model–task cells. The increasing hazard observed within trajectories further indicates that a single stationary parameter is an approximation. A key open question is whether production agents with explicit state validation, rollback, branching, retries, or external memory exhibit the same decay law, or whether orchestration can convert absorbing errors into recoverable failures. Another unresolved issue is whether the compressed-regime penalty results from loss of state, loss of intermediate reasoning traces, or a particular implementation of context summarization.

Conclusion

The paper establishes a clear empirical relationship between task horizon and LLM-agent reliability. Across controlled tasks and nine models, success usually declines geometrically, while the agentic ToolQA condition shows collapse from near-ceiling performance to near failure within 16 steps. Larger models improve per-step reliability but do not eliminate compounding error, and later trajectory steps are less reliable than earlier ones.

Its most defensible practical conclusion is that agent evaluation should report success as a function of horizon and should measure step-level reliability directly. Context truncation alone is not supported as a remedy and may worsen performance. The qualitative evidence for horizon-driven degradation is strong, although the stated benchmark-gap projections contain a numerical inconsistency that must be corrected before those figures can support precise operational planning.