---
title: 'SWE-Chain: Chained Package Upgrades Benchmark'
url: https://www.emergentmind.com/topics/swe-chain
type: topic
---

# SWE-Chain: Chained Package Upgrades Benchmark

SWE-Chain is a benchmark for evaluating coding agents on **chained release-level package upgrades** rather than one-off bug fixes. Its defining property is that each version transition builds on the agent’s prior codebase, so an agent must carry its own previous modifications forward through a sequence of official releases while implementing the new upgrade requirements and avoiding regressions. The benchmark is intended to model continuous software maintenance at the granularity of package releases, where fixes, features, and compatibility changes are bundled, shipped, and inherited by subsequent versions. The released benchmark contains **12 upgrade chains** across **9 real Python packages**, with **155 version transitions** and **1,660 grounded upgrade requirements** [2605.14415].

## 1. Scope and problem setting

SWE-Chain asks whether an agent can take a real package forward through a sequence of official releases, making the right upgrade changes at each step while **carrying its own previous modifications forward** without breaking already-working behavior. This differs materially from benchmarks centered on isolated issue resolution, synthetic checkpoints, repository-level bug fixing, or CI-style milestones, because the unit of change is a **release boundary** rather than a single defect report.

The benchmark’s motivation is that real maintenance is often organized around consecutive versions. A maintainer ships one release, then the next version inherits that history. Under this regime, a patch that appears correct in isolation may still create downstream friction by introducing regressions, changing public APIs inconsistently, or failing to preserve compatibility with the next upgrade. SWE-Chain is therefore designed to expose long-horizon maintenance failure modes that are not visible when each task starts from a fixed pristine repository state.

A central implication is that release-level maintenance is not merely repeated single-step repair. The task is “do the fix, then continue evolving the same codebase through the next release, and the next,” which constitutes a stronger test of reliability, memory, and regression avoidance. This suggests that benchmark difficulty arises not only from the complexity of any individual upgrade, but from the accumulation of earlier agent decisions across the chain.

## 2. DecompSynth and grounded specification synthesis

To construct upgrade tasks, the authors introduce **DecompSynth**, a divide-and-conquer synthesis pipeline that aligns maintainer-authored release information with the concrete implementation changes in the gold diff [2605.14415]. For each package version transition, the pipeline collects release notes plus any linked GitHub issues or pull requests, and extracts the **actual code and test diffs** between the old and new versions using `git diff`. The benchmark task is then derived by grounding the release narrative in the code changes that actually implemented it.

In the **divide** stage, the diff is decomposed through batch-level multi-label hunk matching. The diff is split into batches using greedy packing with explicit limits of at most **20 files, 150 hunks, and 3,000 diff lines** per batch. If the release note is decomposed into task-like entries \(T^{(1)}, \dots, T^{(n)}\), the matching agent assigns each hunk to one or more task labels, or to the special buckets `doc` and `others`. The paper formalizes the batch mapping as
$$
P_j : H(B_j) \to 2^{L},
$$
where \(H(B_j)\) is the set of hunks in batch \(B_j\), and \(L = \{1,\dots,n\} \cup \{\text{doc}, \text{others}\}\). The batch assignments are merged into a global mapping \(\Phi\), and the matched hunk set for task \(k\) is reconstructed as
$$
M(k) = \{\, h \in H(D) : k \in \Phi(h)\,\}.
$$

In the **conquer** stage, the synthesis agent converts each matched hunk set into a structured, user-facing specification \(S(k)\). The paper states that the resulting specification is intentionally grounded in code-diff evidence and uses a schema with fields such as problem statement, grounded and conceptual expectations and constraints, behaviors, and acceptance criteria. Hunks labeled `others` are revisited to detect additional upgrade tasks that were not explicitly stated in the release notes. The authors characterize this synthesis as “oracle-like”: high-quality and structured, but controlled so as not to leak implementation steps or raw code.

The reported stability study targets the largest diff in the dataset, a Pytest transition with **13k diff lines, 102 files, and 986 hunks**. Across repeated runs, the divide step achieves **93.66% exact match**, **94.54% Jaccard similarity**, and **95.38% core agreement**. In context, these figures are presented as evidence that the decomposition pipeline remains robust even on very large transitions.

## 3. Dataset composition and difficulty structure

The resulting benchmark contains **12 upgrade chains** across **9 real Python packages**: `attrs`, `conan`, `Flask`, `Jinja2`, `poetry`, `PyJWT`, `pytest`, `urllib3`, and `xarray`. Across these chains, SWE-Chain includes **155 version transitions** and **1,660 grounded upgrade requirements**. The chains vary substantially in scale, from relatively compact libraries to large codebases.

| Aspect | Value |
|---|---|
| Upgrade chains | 12 |
| Packages | 9 real Python packages |
| Version transitions | 155 |
| Grounded upgrade requirements | 1,660 |

The paper reports that difficulty depends jointly on starting codebase size and per-upgrade diff burden, rather than on repository size alone. The appendix quantifies difficulty with a normalized combination of starting codebase size and per-upgrade diff burden. On that characterization, the hardest chains are the large `xarray` and `conan` trajectories, while smaller packages such as `PyJWT` are easier.

This heterogeneity is part of the benchmark design. The mixture of compact and large projects is meant to make SWE-Chain both broad and discriminative. A plausible implication is that the benchmark does not reduce to a single latent factor such as repository size; instead, it probes interactions among codebase scale, release churn, and long-horizon carry-over of earlier edits.

## 4. Evaluation protocol and metrics

Each upgrade chain is evaluated as an ordered sequence of version-upgrade steps [2605.14415]. The agent begins from the initial codebase and, at each step, receives the synthesized specification for that version transition. It must then modify the **current** codebase accordingly. The crucial evaluation constraint is that the agent-produced codebase from one step becomes the starting point for the next step. The benchmark therefore measures continuous maintenance rather than isolated patching.

Evaluation is conducted under a **Build+Fix** regime. First, the agent attempts the upgrade (“Build”). If hidden test execution reveals non-behavioral setup or interface failures, the agent is allowed **one controlled repair attempt** (“Fix”) using an error report. The paper frames this as a way to avoid unfairly penalizing import or setup brittleness while preserving the underlying challenge of the upgrade itself.

Upgrade-related behavior is identified through a version-specific test subset. For each version transition \(V_i\), the benchmark defines
$$
U_i = \{ q \in Q_{v_i} \mid r(q, c_{v_{i-1}}) \in \{\text{fail}, \text{error}\} \land r(q, c_{v_i}) = \text{pass} \},
$$
that is, tests that fail or error on the old gold codebase but pass on the new gold codebase. These are interpreted as the behaviors the upgrade is supposed to fix.

The benchmark then classifies outcomes into categories including true positives \(TP_i\) for resolved upgrade-related tests, false negatives \(FN_i\) for unresolved upgrade-related tests, false positives \(FP_i\) for regressions on previously passing tests, and recovered versus unrecovered non-upgrade tests. From these, it computes three micro-averaged chain-level metrics:
$$
\text{Resolving} = \frac{\sum_i TP_i}{\sum_i (TP_i + FN_i)},
\qquad
\text{Precision} = \frac{\sum_i TP_i}{\sum_i (TP_i + FP_i)},
$$
and
$$
F1 = \frac{2 \sum_i TP_i}{\sum_i (2TP_i + FP_i + FN_i)}.
$$
The benchmark also reports **Final Passing** as an auxiliary measure of the final codebase health after the chain.

Conceptually, **Resolving** measures whether the required upgrade behavior was implemented, **Precision** measures maintenance safety by penalizing regressions on previously passing tests, and **F1** balances the two. This metric design makes the benchmark explicitly two-sided: successful upgrading is insufficient if it damages previously correct functionality.

## 5. Empirical results and discriminative behavior

Across **nine frontier model-agent configurations**, SWE-Chain reports an average **44.8% resolving**, **65.4% precision**, and **50.2% F1** under the **Build+Fix** regime [2605.14415]. The best overall configuration is **Claude-Opus-4.7 with Claude Code**, achieving **60.8% resolving**, **80.6% precision**, and **68.5% F1**. The next strongest configuration is **GPT-5.5 with Codex**, at **57.5% resolving**, **80.1% precision**, and **64.8% F1**.

These results are presented as evidence that the benchmark is feasible but challenging. Even the strongest systems remain substantially below perfect continuous maintenance, and the average performance indicates that chained release-level upgrading is not solved by current frontier agents.

The paper further reports that Build-only numbers are lower than Build+Fix, which it interprets as showing that the repair step mainly helps with brittle execution and interface issues rather than directly solving more upgrade requirements. It also observes that **GPT-5.3-Codex can outperform GPT-5.4 in the Codex CLI**, suggesting that agent CLI/model alignment matters, not just raw model size.

Chain-wise variation is pronounced. On easier chains such as `PyJWT` and `Jinja2`, agents still do not approach perfection, while on difficult chains such as `conan` and `xarray` they can drop below **30% resolving**. No single model dominates across all chains. The paper treats this as positive evidence that SWE-Chain is neither trivial nor excessively noisy: it distinguishes among model strengths, package structure, and long-horizon maintenance capability.

A plausible interpretation is that SWE-Chain is discriminative in two senses simultaneously. First, it separates strong from weak agent-model configurations in aggregate. Second, it reveals non-uniform capability profiles across packages, indicating that release-level maintenance performance depends on more than generic coding competence.

## 6. Specification granularity, benchmark position, and implications

SWE-Chain includes a granularity study comparing specification styles. The paper reports that raw release notes plus GitHub issue or pull-request text are noisy and weaker as specifications, whereas adding conceptual expectations and constraints substantially improves precision and overall performance [2605.14415]. The default configuration uses **problem statements plus conceptual expectations and constraints**, which the authors describe as a compromise between realism and reliability. More detailed grounded expectations, behaviors, and acceptance criteria help further, but are characterized as closer to oracle-style guidance and therefore more suitable for capability upper bounds than for simulating ordinary maintainer instructions.

This design addresses a common misconception about benchmark realism: more detail is not automatically more representative of real maintainer communication. In SWE-Chain, richer specifications can improve performance, but they also move the task away from ordinary release documentation toward a more guided setting.

Within the broader SWE benchmark landscape, SWE-Chain occupies a distinct position. **SWE-Interact** evaluates whether an agent can discover user intent, incorporate progressively revealed requirements, and refine its work through a multi-turn conversation; its emphasis is interactive goal discovery and iterative refinement with a user in the loop [2606.30573]. By contrast, SWE-Chain isolates continuous maintenance across consecutive releases. **SWE-rebench V2** is designed as a large-scale training substrate with executable tasks, reproducible environments, and diagnostics across many languages and repositories, rather than as a chained release benchmark [2602.23866]. **Kimi-Dev** focuses on the relation between Agentless and multi-turn SWE-Agent training on SWE-bench Verified, arguing that structured skill priors such as localization, code editing, and self-reflection transfer into agentic settings [2509.23045]. **SWE-TRACE** and **SWE-AGILE** target long-horizon agent optimization through trajectory reduction, process reward models, heuristic test-time scaling, and dynamic reasoning context management [2604.14820; 2604.11716]. Taken together, these works suggest that long-horizon SWE evaluation now spans at least three axes: issue resolution, interactive requirement discovery, and chained release maintenance.

The benchmark’s main conclusion is that **current coding agents are still not reliable enough for continuous package maintenance across chained releases**. They can often make partially correct changes, especially with a repair step, but they still struggle to preserve existing behavior while tracking evolving requirements over long sequences of releases. For agent development, this points toward stronger state tracking, better regression awareness, and better handling of version-to-version continuity. For benchmark design, SWE-Chain demonstrates that release-level, chain-based evaluation is feasible, scalable, and highly discriminative.

Source: https://www.emergentmind.com/topics/swe-chain