SWE-Chain: Chained Package Upgrades Benchmark

• SWE-Chain is a benchmark for release-level package upgrades that models continuous software maintenance by requiring agents to carry forward modifications across successive versions.
• It employs the DecompSynth pipeline to align release notes with actual code diffs, generating grounded upgrade specifications for each version transition.
• Evaluated across 12 upgrade chains and 9 Python packages, SWE-Chain exposes long-horizon challenges through metrics like resolving rate, precision, and F1 score.

SWE-Chain is a benchmark for evaluating coding agents on chained release-level package upgrades rather than one-off bug fixes. Its defining property is that each version transition builds on the agent’s prior codebase, so an agent must carry its own previous modifications forward through a sequence of official releases while implementing the new upgrade requirements and avoiding regressions. The benchmark is intended to model continuous software maintenance at the granularity of package releases, where fixes, features, and compatibility changes are bundled, shipped, and inherited by subsequent versions. The released benchmark contains 12 upgrade chains across 9 real Python packages, with 155 version transitions and 1,660 grounded upgrade requirements (Lam et al., 14 May 2026).

1. Scope and problem setting

SWE-Chain asks whether an agent can take a real package forward through a sequence of official releases, making the right upgrade changes at each step while carrying its own previous modifications forward without breaking already-working behavior. This differs materially from benchmarks centered on isolated issue resolution, synthetic checkpoints, repository-level bug fixing, or CI-style milestones, because the unit of change is a release boundary rather than a single defect report.

The benchmark’s motivation is that real maintenance is often organized around consecutive versions. A maintainer ships one release, then the next version inherits that history. Under this regime, a patch that appears correct in isolation may still create downstream friction by introducing regressions, changing public APIs inconsistently, or failing to preserve compatibility with the next upgrade. SWE-Chain is therefore designed to expose long-horizon maintenance failure modes that are not visible when each task starts from a fixed pristine repository state.

A central implication is that release-level maintenance is not merely repeated single-step repair. The task is “do the fix, then continue evolving the same codebase through the next release, and the next,” which constitutes a stronger test of reliability, memory, and regression avoidance. This suggests that benchmark difficulty arises not only from the complexity of any individual upgrade, but from the accumulation of earlier agent decisions across the chain.

2. DecompSynth and grounded specification synthesis

To construct upgrade tasks, the authors introduce DecompSynth, a divide-and-conquer synthesis pipeline that aligns maintainer-authored release information with the concrete implementation changes in the gold diff (Lam et al., 14 May 2026). For each package version transition, the pipeline collects release notes plus any linked GitHub issues or pull requests, and extracts the actual code and test diffs between the old and new versions using git diff. The benchmark task is then derived by grounding the release narrative in the code changes that actually implemented it.

In the divide stage, the diff is decomposed through batch-level multi-label hunk matching. The diff is split into batches using greedy packing with explicit limits of at most 20 files, 150 hunks, and 3,000 diff lines per batch. If the release note is decomposed into task-like entries $T^{(1)}, \dots, T^{(n)}$, the matching agent assigns each hunk to one or more task labels, or to the special buckets doc and others. The paper formalizes the batch mapping as

$P_j : H(B_j) \to 2^{L},$

where $H(B_j)$ is the set of hunks in batch $B_j$, and $$L = \{1,\dots,n\} \cup \{\text{doc}, \text{others}\}$$. The batch assignments are merged into a global mapping $\Phi$, and the matched hunk set for task $k$ is reconstructed as

$M(k) = \{\, h \in H(D) : k \in \Phi(h)\,\}.$

In the conquer stage, the synthesis agent converts each matched hunk set into a structured, user-facing specification $S(k)$. The paper states that the resulting specification is intentionally grounded in code-diff evidence and uses a schema with fields such as problem statement, grounded and conceptual expectations and constraints, behaviors, and acceptance criteria. Hunks labeled others are revisited to detect additional upgrade tasks that were not explicitly stated in the release notes. The authors characterize this synthesis as “oracle-like”: high-quality and structured, but controlled so as not to leak implementation steps or raw code.

The reported stability study targets the largest diff in the dataset, a Pytest transition with 13k diff lines, 102 files, and 986 hunks. Across repeated runs, the divide step achieves 93.66% exact match, 94.54% Jaccard similarity, and 95.38% core agreement. In context, these figures are presented as evidence that the decomposition pipeline remains robust even on very large transitions.

3. Dataset composition and difficulty structure

The resulting benchmark contains 12 upgrade chains across 9 real Python packages: attrs, conan, Flask, Jinja2, poetry, PyJWT, pytest, urllib3, and xarray. Across these chains, SWE-Chain includes 155 version transitions and 1,660 grounded upgrade requirements. The chains vary substantially in scale, from relatively compact libraries to large codebases.

Aspect	Value
Upgrade chains	12
Packages	9 real Python packages
Version transitions	155
Grounded upgrade requirements	1,660

The paper reports that difficulty depends jointly on starting codebase size and per-upgrade diff burden, rather than on repository size alone. The appendix quantifies difficulty with a normalized combination of starting codebase size and per-upgrade diff burden. On that characterization, the hardest chains are the large xarray and conan trajectories, while smaller packages such as PyJWT are easier.

This heterogeneity is part of the benchmark design. The mixture of compact and large projects is meant to make SWE-Chain both broad and discriminative. A plausible implication is that the benchmark does not reduce to a single latent factor such as repository size; instead, it probes interactions among codebase scale, release churn, and long-horizon carry-over of earlier edits.

4. Evaluation protocol and metrics

Each upgrade chain is evaluated as an ordered sequence of version-upgrade steps (Lam et al., 14 May 2026). The agent begins from the initial codebase and, at each step, receives the synthesized specification for that version transition. It must then modify the current codebase accordingly. The crucial evaluation constraint is that the agent-produced codebase from one step becomes the starting point for the next step. The benchmark therefore measures continuous maintenance rather than isolated patching.

Evaluation is conducted under a Build+Fix regime. First, the agent attempts the upgrade (“Build”). If hidden test execution reveals non-behavioral setup or interface failures, the agent is allowed one controlled repair attempt (“Fix”) using an error report. The paper frames this as a way to avoid unfairly penalizing import or setup brittleness while preserving the underlying challenge of the upgrade itself.

Upgrade-related behavior is identified through a version-specific test subset. For each version transition $V_i$, the benchmark defines

$P_j : H(B_j) \to 2^{L},$0

that is, tests that fail or error on the old gold codebase but pass on the new gold codebase. These are interpreted as the behaviors the upgrade is supposed to fix.

The benchmark then classifies outcomes into categories including true positives $P_j : H(B_j) \to 2^{L},$1 for resolved upgrade-related tests, false negatives $P_j : H(B_j) \to 2^{L},$2 for unresolved upgrade-related tests, false positives $P_j : H(B_j) \to 2^{L},$3 for regressions on previously passing tests, and recovered versus unrecovered non-upgrade tests. From these, it computes three micro-averaged chain-level metrics:

$P_j : H(B_j) \to 2^{L},$4

and

$P_j : H(B_j) \to 2^{L},$5

The benchmark also reports Final Passing as an auxiliary measure of the final codebase health after the chain.

Conceptually, Resolving measures whether the required upgrade behavior was implemented, Precision measures maintenance safety by penalizing regressions on previously passing tests, and F1 balances the two. This metric design makes the benchmark explicitly two-sided: successful upgrading is insufficient if it damages previously correct functionality.

5. Empirical results and discriminative behavior

Across nine frontier model-agent configurations, SWE-Chain reports an average 44.8% resolving, 65.4% precision, and 50.2% F1 under the Build+Fix regime (Lam et al., 14 May 2026). The best overall configuration is Claude-Opus-4.7 with Claude Code, achieving 60.8% resolving, 80.6% precision, and 68.5% F1. The next strongest configuration is GPT-5.5 with Codex, at 57.5% resolving, 80.1% precision, and 64.8% F1.

These results are presented as evidence that the benchmark is feasible but challenging. Even the strongest systems remain substantially below perfect continuous maintenance, and the average performance indicates that chained release-level upgrading is not solved by current frontier agents.

The paper further reports that Build-only numbers are lower than Build+Fix, which it interprets as showing that the repair step mainly helps with brittle execution and interface issues rather than directly solving more upgrade requirements. It also observes that GPT-5.3-Codex can outperform GPT-5.4 in the Codex CLI, suggesting that agent CLI/model alignment matters, not just raw model size.

Chain-wise variation is pronounced. On easier chains such as PyJWT and Jinja2, agents still do not approach perfection, while on difficult chains such as conan and xarray they can drop below 30% resolving. No single model dominates across all chains. The paper treats this as positive evidence that SWE-Chain is neither trivial nor excessively noisy: it distinguishes among model strengths, package structure, and long-horizon maintenance capability.

A plausible interpretation is that SWE-Chain is discriminative in two senses simultaneously. First, it separates strong from weak agent-model configurations in aggregate. Second, it reveals non-uniform capability profiles across packages, indicating that release-level maintenance performance depends on more than generic coding competence.

6. Specification granularity, benchmark position, and implications

SWE-Chain includes a granularity study comparing specification styles. The paper reports that raw release notes plus GitHub issue or pull-request text are noisy and weaker as specifications, whereas adding conceptual expectations and constraints substantially improves precision and overall performance (Lam et al., 14 May 2026). The default configuration uses problem statements plus conceptual expectations and constraints, which the authors describe as a compromise between realism and reliability. More detailed grounded expectations, behaviors, and acceptance criteria help further, but are characterized as closer to oracle-style guidance and therefore more suitable for capability upper bounds than for simulating ordinary maintainer instructions.

This design addresses a common misconception about benchmark realism: more detail is not automatically more representative of real maintainer communication. In SWE-Chain, richer specifications can improve performance, but they also move the task away from ordinary release documentation toward a more guided setting.

Within the broader SWE benchmark landscape, SWE-Chain occupies a distinct position. SWE-Interact evaluates whether an agent can discover user intent, incorporate progressively revealed requirements, and refine its work through a multi-turn conversation; its emphasis is interactive goal discovery and iterative refinement with a user in the loop (Raghavendra et al., 29 Jun 2026). By contrast, SWE-Chain isolates continuous maintenance across consecutive releases. SWE-rebench V2 is designed as a large-scale training substrate with executable tasks, reproducible environments, and diagnostics across many languages and repositories, rather than as a chained release benchmark (Badertdinov et al., 27 Feb 2026). Kimi-Dev focuses on the relation between Agentless and multi-turn SWE-Agent training on SWE-bench Verified, arguing that structured skill priors such as localization, code editing, and self-reflection transfer into agentic settings (Yang et al., 27 Sep 2025). SWE-TRACE and SWE-AGILE target long-horizon agent optimization through trajectory reduction, process reward models, heuristic test-time scaling, and dynamic reasoning context management (Han et al., 16 Apr 2026, Lian et al., 13 Apr 2026). Taken together, these works suggest that long-horizon SWE evaluation now spans at least three axes: issue resolution, interactive requirement discovery, and chained release maintenance.

The benchmark’s main conclusion is that current coding agents are still not reliable enough for continuous package maintenance across chained releases. They can often make partially correct changes, especially with a repair step, but they still struggle to preserve existing behavior while tracking evolving requirements over long sequences of releases. For agent development, this points toward stronger state tracking, better regression awareness, and better handling of version-to-version continuity. For benchmark design, SWE-Chain demonstrates that release-level, chain-based evaluation is feasible, scalable, and highly discriminative.