SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

Abstract: Software development is iterative, yet agentic coding benchmarks overwhelmingly evaluate single-shot solutions against complete specifications. Code can pass the test suite but become progressively harder to extend. Recent iterative benchmarks attempt to close this gap, but constrain the agent's design decisions too tightly to faithfully measure how code quality shapes future extensions. We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints, in which agents repeatedly extend their own prior solutions under evolving specifications that force architectural decisions without prescribing internal structure. We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion, the share of complexity mass concentrated in high-complexity functions. No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%. Quality degrades steadily: erosion rises in 80% of trajectories and verbosity in 89.8%. Against 48 open-source Python repositories, agent code is 2.2x more verbose and markedly more eroded. Tracking 20 of those repositories over time shows that human code stays flat, while agent code deteriorates with each iteration. A prompt-intervention study shows that initial quality can be improved, but it does not halt degradation. These results demonstrate that pass-rate benchmarks systematically undermeasure extension robustness, and that current agents lack the design discipline iterative software development demands.

• The paper introduces SlopCodeBench to evaluate iterative coding agents and reveals that no model achieves a complete end-to-end solution.
• It employs novel metrics like verbosity and structural erosion to quantify the accumulation of technical debt during multi-turn code extensions.
• Experimental results show that while prompt strategies lower initial slop, they fail to prevent long-horizon architectural degradation.

SlopCodeBench: Assessing Long-Horizon Degradation in Iterative Coding Agents

Motivation and Benchmark Design

The paper introduces "SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks" (2603.24755), a benchmark specifically devised to fill a pronounced methodological gap in the evaluation of code generation agents. Current agentic coding benchmarks focus predominantly on single-shot solutions, evaluating correctness against a complete specification but failing to capture the structural and design degradation that manifests when code is iteratively extended. This oversight ignores a central property of authentic software development: the accumulation of technical debt and architectural slop through incremental changes as requirements evolve—a dynamic rarely measurable via pass-rate-centric approaches.

SlopCodeBench (SCBench) targets this critical failure mode by tasking agents with repeatedly extending their own prior code across up to eight specification checkpoints per problem instance. Each problem is defined by observable behavioral contracts (CLI or API boundaries) and intentionally withholds the internal interface structure and explicit test suites, forcing the agent to commit to architectural choices whose implications are revealed only in subsequent iterations. The multi-turn iterative paradigm surfaces how early optimality can result in downstream complexity, redundancy, and code "slop."

Trajectory-Level Quality Metrics

To detect forms of quality decay not obvious in test-passing rates, the benchmark operationalizes two codebase-level structural metrics:

• Verbosity: Quantifies redundant and duplicated code through a composite score combining AST-Grep rule violations and clone detection, normalized by logical lines of code.
• Structural Erosion: Measures the fraction of complexity mass (a product of cyclomatic complexity and $\sqrt{\text{SLOC}}$ per function) concentrated in high-CC functions (CC $>$ 10), highlighting how new logic tends to accrete into large, unmanageable callables during iterative agent extension.

These orthogonal axes are crucial because codebases can remain functionally correct while becoming progressively less maintainable and less extensible, a reality that defies single-turn evaluation.

Experimental Results

The evaluation covers 20 language-agnostic programming problems, each featuring 3–8 checkpoints (93 checkpoints total), and includes 11 leading code-generation models such as Opus 4.6, several variants of GPT-5.x Codex, and Sonnet. The principal findings are as follows:

• Zero Complete Solution Trajectories: No model solves any problem end-to-end. The best strict solve rate per checkpoint remains at 17.2% for Opus 4.6.
• Monotonic Quality Decay: Verbosity increases in 89.8% and erosion in 80% of agent solution trajectories as checkpoints progress, independent of underlying correctness.

  Figure 2: Solve rates and cost growth over checkpoint progress; strict solve rates drop sharply by the final checkpoint, while mean cost per checkpoint triples.

• Disjunction from Pass-Rate Metrics: Pass rates for core and functionality tests are relatively preserved, but error-handling and regression pass rates collapse as tasks progress (core tests are up to 13x easier to pass than the complete suite by the final checkpoint).
• Exponential Cost Increase: Despite a 2.9x increase in mean cost per checkpoint from the first to the last progress bin, no corresponding gain in correctness is observed.

  Figure 4: Erosion and verbosity increase monotonically for all evaluated models, indicating consistent accumulation of technical debt across iterative progress.

• Human Repository Comparison: When contrasted with 48 mature open-source Python repositories, agent-generated code is on average 2.2x more verbose and exhibits substantially higher structural erosion. Critically, agent metrics on both axes grow consistently with each checkpoint, whereas human repositories plateau.

  Figure 6: Mean verbosity and structural erosion plotted for agent runs (rising) and human repository histories (flat), demonstrating divergence over development time.

Effects of Prompt Strategies

A prompt-side intervention study evaluated two quality-focused prompt strategies: "anti-slop" (explicitly forbidding verbose and overengineered patterns) and "plan-first" (requiring explicit architectural planning prior to code emission), in addition to a minimalist baseline.

• Lowered Initial Verbosity/Erosion but No Slope Change: Both interventions reduce initial codebase verbosity (by 33–35%) and erosion (by up to 50%), but the rate of quality degradation per checkpoint is statistically indistinguishable from baseline.

  Figure 1: Prompt strategies shift the starting point of verbosity and erosion (lower intercepts) but do not alter the trend; trajectories remain parallel across all prompt types.

• No Consistent Correctness Improvement: Substantial reductions in structural slop do not produce statistically significant improvements in any pass-rate subtype. In some cases, total monetary cost increases by up to 48% for only marginal code quality benefits.

Discussion and Implications

These results contradict a foundational assumption implicit in existing pass-rate benchmarks: that syntactic and architectural quality are meaningfully tethered to correctness. In SCBench, an agent can consistently pass core tests while its code becomes dramatically less maintainable and harder to extend—a failure mode hidden from traditional single-shot or test-passing evaluation. The outcome is robust to prompt-side instruction following: neither prescriptive code style nor enforced planning can halt the acceleration of architectural slop once the agent is required to chain its own outputs.

Practically, this signals that the current agent architectures—designed primarily for prompt completion rather than long-horizon planning—are grossly inadequate for iterative software engineering tasks. Techniques such as code health–aware action planning, explicit intermediate refactorization, or trajectory-level regularization (potentially at finetuning time) are necessary research directions. Furthermore, the fact that error-handling and regression test performance deteriorate most sharply exposes risk areas for safety-critical and evolving codebases under agentic maintenance.

From a theoretical perspective, the work formalizes a metric-based framework to quantify and track codebase decay dynamically, disentangled from correctness. This provides an empirical substrate for analyzing technical debt and architectural rot in agent-generated software, which will be vital for the development of long-term automated engineering agents.

Directions for Future Research

Key follow-up avenues emanating from this work include:

• Architectural Memory and Planning: Integrating explicit design reasoning or memory modules that allow agents to revisit and refactor architectural choices, thereby reducing the rate at which technical debt is acquired.
• Training Objective Augmentation: Incorporation of regularization terms tied directly to codebase verbosity and structural erosion during pretraining or finetuning.
• Continuous Evaluation Artifacts: Broader adoption of SlopCodeBench-like benchmarks, extended to other languages and real-world software lifecycles, to measure how interventions or new paradigms affect long-horizon codebase health.
• Automated Repair and Refactoring Loops: Embedding periodic health checks and automated repair interventions into agent action loops to proactively address metric drift.

Conclusion

SlopCodeBench exposes a significant limitation of current code-generation agents: their inability to sustain architectural discipline across long-horizon, iterative software development. Structural decay and verbosity accumulate monotonically in agent-generated code, independently of superficial test-passing correctness, and prompt-based mitigations are insufficient. The findings underscore the necessity of fundamental advances in agent architecture, objective design, and benchmarking if coding agents are to be deployed safely and effectively in real-world, evolving codebases.