Don't Blame the Large Language Model: How Scaffolding Evolution Shapes Coding Agent Quality

Abstract: Coding agents, autonomous systems that use LLMs to resolve software engineering tasks, rely on agentic scaffolding: a middleware layer in between a developer and a LLM that orchestrates system prompts, tool execution, context management, and iterative reasoning loops. While these scaffoldings evolve at extreme velocities, no study has examined how this evolution affects agent quality (i.e., effectiveness and efficiency) over time. Practitioners regularly report quality regressions after scaffolding updates, yet consistently attribute them to the underlying model rather than the scaffolding itself. In this paper, we address this gap by conducting the first controlled longitudinal study that isolates the scaffolding's contribution. Unlike prior work that fixes the scaffolding and varies the model, we fix the model and vary only the scaffolding, evaluating 35 sequential releases to measure their impact on agent effectiveness and efficiency. We first empirically study the development and release evolution of five major open-source scaffoldings (i.e., Codex, Qwen Code, Gemini, OpenCode, and OpenHands), revealing extreme release velocities exceeding two releases per day and thousands of issues within months. We then perform a controlled deep dive into 35 sequential releases of the Qwen Code CLI, evaluating each against 50 stratified SWE-bench Verified tasks while holding the underlying LLM constant. We trace the resulting quality fluctuations to specific development patterns and architectural components, and illustrate our findings with concrete qualitative evidence linking individual pull requests to measured quality shifts.

• The paper demonstrates that scaffolding evolution, rather than LLM changes, is the primary driver of coding agent quality fluctuations.
• A controlled longitudinal study on Qwen Code CLI reveals that increased release frequency leads to double resource consumption without a proportional improvement in task resolve rates.
• The analysis identifies specific development patterns and architectural adjustments, especially in LLM provider and context management layers, as key contributors to efficiency regression.

The Impact of Scaffolding Evolution on Coding Agent Quality: An Analysis from “Don't Blame the LLM” (2607.03691)

Introduction and Motivation

The proliferation of agentic coding systems—autonomous platforms built atop LLMs to automate software engineering workflows—depends critically on the agentic scaffolding: intermediary software that orchestrates LLM reasoning, tool invocation, prompt engineering, and system interaction. Despite escalating adoption and rapid iteration of scaffolding frameworks, practitioners frequently encounter quality regressions following scaffolding updates, attributing these to underlying LLM changes rather than the scaffolding itself. This paper provides the first rigorous, controlled longitudinal study isolating the distinct impact of scaffolding evolution on coding agent quality, using Qwen Code CLI as a central case, and systematically analyzes project- and architecture-level factors underlying observed quality shifts.

Agentic Scaffolding: Workflow and Scale of Evolution

Agentic scaffolding mediates complex, multi-step interactions where the LLM incrementally reasons, selects tools, observes results, and revises plans—moving well beyond static code generation paradigms.

Figure 1: An example interaction with Qwen Code v0.18.1 on a bug-fixing task, highlighting iterative tool use and reasoning cycles until all tests pass.

Figure 2: The agentic scaffolding’s control flow, orchestrating iterative LLM–environment loops.

A landscape analysis across five major open-source scaffoldings (Codex, Qwen Code, Gemini CLI, OpenCode, OpenHands CLI) and two mature baselines demonstrates “hyper-churn”—release velocities of 10–18 per week with up to 34 commits per day, starkly outpacing the mere 0.6–0.8 releases per week of baselines like VSCode. Issue backlogs balloon in agentic projects, in marked contrast to stable-trajectory baselines.

Figure 3: Methodological overview, contrasting the wide-scale landscape analysis with the controlled, deep longitudinal study of Qwen Code CLI.

Figure 4: Cumulative releases over time; agentic scaffolding projects vastly outpace mature baselines in release velocity.

Figure 5: Escalating issue backlogs for agentic scaffoldings, while non-agentic baselines maintain flat trajectories.

Longitudinal Quality Analysis: No Guaranteed Improvement

Thirty-five sequential releases of Qwen Code CLI, all evaluated on a fixed set of 50 stratified SWE-bench Verified tasks using Qwen3-Next-80B-A3B as a constant LLM, were analyzed for both effectiveness (resolve rate) and efficiency (token consumption, tool call frequency). The findings contravene the intuitive expectation that sustained, intensive scaffolding development will monotonically improve agent quality:

• No statistically significant improvement in task resolve rate across releases: Fluctuations are observed, with early versions achieving better or comparable functionality to the most recent versions.
• Substantial increase in computational resource usage: Newer scaffoldings require nearly double the tokens and more tool calls, incurring higher economic and computational costs without corresponding gains in correctness.

Figure 6: Task resolve rates across sequential scaffolding releases—no monotonic improvement is evident.

Figure 7: While patch generation rates are often high, correct resolution (test suite pass) rates lag significantly, revealing a persistent quality gap.

Figure 8: Tool calls per task are normalized to remove task-selection bias, confirming efficiency degradation over time.

Root Causes: Development and Architectural Patterns

Controlled analyses at both the project- and architectural-levels provide strong evidence that quality shifts are not random but align with discernible development and design patterns.

Project Level:

• Feature-heavy releases drive short-term resolve-rate increases but incur significant cost in tokens and tool calls—indicative of a trade-off between capability and efficiency.
• Fix-heavy or high-churn releases fail to improve outcomes, often increasing overhead without effecting correctness improvements.
• Large, consolidated PRs and code deletions are associated with efficiency improvements; scattered, small, or fix-dominant changes correlate with resource inflation.

  Figure 9: Quality tier distribution (Good, Neutral, Bad) per release; effectiveness and efficiency are often decoupled.

  

  Figure 10: Cliff’s delta effect sizes: only specific development patterns robustly separate Good/Bad releases.

Architectural Level:

Through a ten-component reference architecture synthesis, component-level risk is quantitatively mapped:

• LLM Provider and Context Management layers are “high-risk zones”: modifications here most often drive quality regressions due to their central role in prompt assembly, context window management, and model–tool interface.
• Changes in Extensibility and Security components are consistently safe, enabling the addition of features or hardening without adverse quality impact.

  Figure 11: Reference architecture for coding agent scaffolding: central communication backbone, core reasoning components, and cross-cutting infrastructure.

Underlying Mechanism: Absence of Agentic Non-functional QA

The primary underlying factor for persistent regressions is the lack of agentic quality assurance (Agentic QA). While functional and integration tests are present, exhaustive, automated agent-level benchmark testing (e.g., resolve rate, token and tool metrics) is prohibitively costly to run at high release cadence. Consequently, emergent behavioral regressions—such as increased token consumption—consistently escape detection by CI/CD pipelines, with all documented problematic releases passing pre-deployment checks.

Practical and Theoretical Implications

The findings mandate a rethinking of QA practices for agentic systems. Agentic scaffolding must be treated as quality-critical, on par with the LLMs they orchestrate. Reported LLM or agent benchmarks should always track/scaffold versions alongside model versions; failure to do so risks conflating infrastructure regressions with model limitations.

Practically, controlling for scaffolding version, enforcing constraints on non-functional properties (token/tool budgets), and prioritizing regression-safe development (targeted changes to low-risk architectural zones) become vital.

Theoretically, the results underscore the emergent nature of agentic behavior, where well-intentioned abstraction and capability growth can precipitate inefficiencies or task failures, highlighting the need for systematic methods of end-to-end behavioral testing and architectural risk analysis.

Looking forward, scalable forms of non-functional agentic evaluation, possibly via selective sampling, behavioral monitoring, or lighter-weight surrogates for full SWE-bench runs, will become essential for sustainable evolution of agentic CLIs and similar frameworks.

Conclusion

This study presents the first controlled longitudinal evidence that scaffolding evolution, rather than LLM change, primarily drives the trajectory of coding agent quality. Later scaffolding releases incur steep efficiency penalties without functional gain; only specific development and architectural practices associate robustly with positive outcomes. The absence of automated non-functional agentic QA enables such regressions to persist unnoticed, with direct implications for the operational cost and maintainability of real-world coding agents. Future work should advance QA protocols for agentic scaffolding and explore scalable, architecture-informed risk mitigation in the agentware development lifecycle.

Reference:

"Don't Blame the LLM: How Scaffolding Evolution Shapes Coding Agent Quality" (2607.03691)