SWE-ContextBench: Cumulative Repository Benchmark

• SWE-ContextBench is a benchmark that measures cumulative repository-level competence by enabling agents to reuse prior coding experience from related GitHub issues and pull requests.
• It augments 300 base SWE-Bench Lite tasks with 99 dependency-mined related tasks, capturing inter-task context within Python repositories.
• The benchmark evaluates experience reuse by comparing no-experience, free, and oracle settings based on prediction accuracy, runtime, and cost efficiency.

SWE-ContextBench is a benchmark for repository-level coding agents that evaluates whether an agent can accumulate, retrieve, and apply prior experience across related software engineering tasks rather than solving each issue independently. Built on SWE-Bench Lite, it augments 300 base tasks with 99 related tasks mined from real dependency and reference relationships among GitHub issues and pull requests, thereby creating task sequences with shared context. Its evaluation protocol treats experience reuse as a first-class capability and measures it along three dimensions: prediction accuracy, time efficiency, and cost efficiency (Zhu et al., 9 Feb 2026).

1. Motivation and conceptual scope

SWE-ContextBench was introduced to address a limitation in repository-level software engineering benchmarks such as SWE-Bench and SWE-Bench Lite: those benchmarks evaluate whether agents can resolve realistic GitHub issues in real repositories, but they treat tasks as independent. This omits a capability that is central to practical programming agents, namely reuse of experience accumulated on prior, related tasks. In actual development workflows, issues and pull requests are often interdependent: one pull request can resolve multiple issues, issues can reference each other, and later fixes often depend on earlier debugging or implementation work. SWE-ContextBench operationalizes this dependency structure by constructing related task sequences within the same repository and measuring whether memory of earlier work improves subsequent performance (Zhu et al., 9 Feb 2026).

The benchmark’s central claim is methodological rather than purely outcome-oriented. It does not ask only whether an agent can generate a correct patch. It asks whether prior trajectories or summaries of prior trajectories can improve resolution accuracy, reduce runtime, and lower inference cost. This framing positions experience reuse as an observable systems property of coding agents, not merely an informal desideratum.

A key implication is that SWE-ContextBench shifts attention from single-task competence to cumulative competence. This suggests a closer alignment with agent architectures that include memory modules, retrieval mechanisms, trajectory summarization, or repository-aware indexing. At the same time, the benchmark is still repository-scoped rather than multi-system or observability-centric; its context sharing arises from issue and pull request relationships inside a repository history rather than from external services or live operational artifacts.

2. Benchmark construction and related-task mining

SWE-ContextBench is built from two layers of data. The first layer is an experience pool of 300 SWE-Bench Lite instances. The second layer is a set of 99 related tasks mined from real dependency and reference relationships among GitHub issues and pull requests. These related tasks are always drawn from the same repository as their corresponding experience tasks, and cross-repository links are not considered (Zhu et al., 9 Feb 2026).

The 99 related tasks are constructed by systematically analyzing issue and pull request metadata. The paper identifies several relationship subtypes:

• Multi-issue resolution: one pull request resolves multiple issues; issues not already present in SWE-Bench Lite become related tasks paired with the same pull request.
• PR-to-issue references: the pull request of a SWE-Bench Lite instance references other issues; these are paired either with the same pull request or with their own resolving pull requests.
• PR-only references: a pull request references another pull request; the resolved issue associated with the referenced pull request is used as a related task.
• Issue-to-issue references: an issue references other issues; each referenced issue is paired with its resolving pull request.
• Issue-to-PR references: an issue references pull requests; the resolved issues associated with those pull requests become related tasks.
• Multi-reference cases: issues or pull requests reference other pull requests with sufficiently detailed descriptions but no explicit issue.
• Recursive context expansion: the same rules are applied to newly discovered artifacts, yielding second-order related tasks.

In total, the construction yields 89 first-order plus 10 recursively discovered tasks, for 99 related tasks overall. All candidates were manually verified to ensure clear problem statements and identifiable resolutions. The benchmark discusses sequences and second-order dependencies, but it does not formalize a DAG or report sequence-length statistics (Zhu et al., 9 Feb 2026).

The repository coverage is entirely Python and spans 12 repositories. The experience-task and related-task counts are as follows:

Repository	Experience tasks	Related tasks
django	114	36
sympy	77	26
scikit-learn	23	10
matplotlib	23	12
pytest	17	2
sphinx	16	2
pylint	6	1
requests	6	3
astropy	6	2
xarray	5	2
seaborn	4	2
flask	3	1

This composition makes the benchmark repository-level and language-specific. A plausible implication is that the reported effects of experience reuse are grounded in realistic issue/PR linkage, but within a relatively narrow ecological range.

3. Experience pool, representations, and reuse settings

For each of the 300 SWE-Bench Lite instances, the authors run a base agent, Claude Code with Claude Sonnet 4.5, to attempt issue resolution. They record the full interaction trajectory, including tool calls, file navigation, intermediate reasoning, and a final summary. These logs become the reusable experience pool. Two representations are then exposed to downstream runs: the full trajectory and the final summary. Full trajectories average 24,765 words, whereas final summaries average 204.5 words (Zhu et al., 9 Feb 2026).

SWE-ContextBench evaluates five experience-reuse settings:

1. No-Experience (Baseline): the agent receives only repository name, base commit, and problem statement.
2. Free Experience Reuse: the agent has access to all 300 full trajectories and may autonomously retrieve and use any of them.
3. Oracle Experience Reuse: the agent is given the most relevant full trajectory, identified using factual issue/PR relationships.
4. Free Summary Reuse: the agent has access to all 300 summaries and may autonomously retrieve and use any of them.
5. Oracle Summary Reuse: the agent is given the most relevant summary, identified using factual relationships.

The distinction between “oracle” and “free” is fundamental. Oracle selection uses ground-truth relatedness derived from repository history. Free selection delegates relevance judgment to the agent. The paper does not specify embedding models, indices, chunking strategies, scoring functions, or top-$k$ policies for the free setting. This omission is substantive: the benchmark isolates the effect of correct versus incorrect or noisy reuse, but it does not standardize a retrieval stack (Zhu et al., 9 Feb 2026).

The two representations also encode different hypotheses about useful memory. Full trajectories preserve detailed exploration traces and procedural context. Summaries preserve only the distilled result of that process. The benchmark’s empirical results show that this distinction is not cosmetic; it materially changes both correctness and efficiency.

4. Evaluation protocol, artifacts, and metrics

Each related task provides a base_commit, a problem statement, and ground-truth patches split into test_patch and solution_patch. Python version is detected from repository configuration files or inferred from pull request date. The evaluation harness builds a conda environment from base_commit, installs dependencies, and runs tests before and after applying patches to identify FAIL_TO_PASS and PASS_TO_PASS sets. On average, each related task contains 5.09 FAIL_TO_PASS tests and 128.32 PASS_TO_PASS tests (Zhu et al., 9 Feb 2026).

Correctness is defined at the task level. A task is considered resolved if all FAIL_TO_PASS tests pass after applying the agent’s generated patch. PASS_TO_PASS tests are used to check regressions but do not define resolution. Patch applicability is tracked as Patch N/A and excluded from the pass-rate metrics. This produces a hierarchy of measurements:

• FAIL_TO_PASS Tests: percentage of bug-fixing tests passed across tasks with applicable patches.
• PASS_TO_PASS Tests: percentage of regression tests passed across tasks with applicable patches.
• FAIL_TO_PASS Tasks: percentage of tasks for which all bug-fixing tests pass.
• PASS_TO_PASS Tasks: percentage of tasks for which all regression tests pass.
• Resolved: percentage of tasks for which all FAIL_TO_PASS tests pass.

Wall-clock time is measured per task from environment setup to patch generation. The benchmark reports average runtimes and relative speedups; median runtimes are also discussed because averages are affected by heavy tails. Cost efficiency is measured as average dollar cost per instance using provider costs for Claude with cache usage. Tokens are dominated by cache reads, accounting for more than 97% of total tokens. The paper reports average dollar cost per instance rather than a normalized abstract cost metric (Zhu et al., 9 Feb 2026).

The experimental matrix contains 5 settings across 99 related tasks, yielding 495 runs. Each run is executed in an isolated environment, and experience reuse occurs only via explicit access to the stored pool rather than hidden conversational carryover. Sampling parameters, temperatures, retries, and hardware specifics are not provided.

5. Empirical results and process-level interpretation

The benchmark’s central result is that correctly selected summaries outperform both the no-experience baseline and full-trajectory reuse. Oracle Summary Reuse reaches a Resolved rate of 34.34%, compared with 26.26% for No-Experience, 27.27% for Oracle Experience Reuse, 26.26% for Free Experience Reuse, and 22.22% for Free Summary Reuse. The same pattern appears in runtime and cost (Zhu et al., 9 Feb 2026).

Setting	Resolved	Average runtime per task	Average cost per instance
No-Experience	26.26%	381.95 s	\$0.79
Free Experience Reuse	26.26%	406.77 s	\$0.98
Oracle Experience Reuse	27.27%	399.43 s	slightly above baseline
Free Summary Reuse	22.22%	376.77 s	\$0.85
Oracle Summary Reuse	34.34%	356.95 s	\$0.77

The test-level results reinforce the same ordering. Oracle Summary Reuse achieves 27.48% on FAIL_TO_PASS Tests, compared with 19.84% for No-Experience, 21.19% for Oracle Experience Reuse, 22.36% for Free Experience Reuse, and 20.00% for Free Summary Reuse. PASS_TO_PASS test pass rates remain high across settings, in the 97–99% range, and task-level regression-free rates are 85–90%, indicating that differences in resolution are not driven mainly by regression behavior (Zhu et al., 9 Feb 2026).

The efficiency story is equally important. Oracle Summary Reuse is the fastest setting at 356.95 seconds per task, whereas Free Experience Reuse is the slowest at 406.77 seconds. Median runtimes are close, from 331 to 356 seconds, so the main differences arise from heavy-tail cases. Free Experience Reuse has the highest variance and worst-case runtime exceeding 2,100 seconds, suggesting prolonged and unproductive exploration when the agent is allowed to search over large, unfiltered experience logs. On harder tasks, grouped by longer baseline runtime, Oracle Summary Reuse saves over 60% of runtime on the most time-consuming tasks (Zhu et al., 9 Feb 2026).

The cost results parallel the runtime results. Oracle Summary Reuse has the lowest average dollar cost per instance at \$0.77. No-Experience is \$0.79. Free Summary Reuse rises to \$0.85, and Free Experience Reuse reaches \$0.98, a 27.3% increase relative to Oracle Summary Reuse. Because cache reads dominate token consumption, these differences indicate that poor reuse can inflate repository exploration rather than reducing it (Zhu et al., 9 Feb 2026).

The benchmark interprets these results in terms of representation quality and selection quality. Summaries are approximately 120 times shorter than full trajectories, yet they deliver better accuracy when correctly selected. The paper argues that compact representations steer the agent toward essential insights without overwhelming it with long, noisy traces. By contrast, long trajectories increase cognitive load and encourage re-navigation of repository context. The gap between Oracle Summary Reuse and Free Summary Reuse—34.34% versus 22.22%, a difference of 12.12 percentage points—shows that short but irrelevant context can be actively harmful. This suggests that retrieval precision, not memory size alone, is the principal bottleneck.

6. Position in the benchmark landscape, limitations, and research implications

Within the software engineering benchmark ecosystem, SWE-ContextBench occupies a specific niche. Relative to SWE-Bench and SWE-Bench Lite, it adds inter-task context via real issue/PR relationships and emphasizes efficiency gains from memory reuse rather than treating each instance as independent (Zhu et al., 9 Feb 2026). It is therefore a benchmark of cumulative repository-level competence, not merely end-to-end patch generation.

It should also be distinguished from the separate benchmark “ContextBench,” which evaluates context retrieval inside single issue-resolution trajectories using human-verified gold contexts at file-, block-, and line-level granularity and measures recall, precision, F1, efficiency, redundancy, and usage drop (Li et al., 5 Feb 2026). ContextBench studies whether agents retrieve the right code evidence during one task; SWE-ContextBench studies whether agents can reuse experience from earlier tasks to improve later ones. The two benchmarks are complementary rather than interchangeable.

A further contrast appears in APEX-SWE, which explicitly positions itself as broader than context-centric repository benchmarks such as SWE-ContextBench. APEX-SWE expands “context” beyond codebases and issue threads to include live service APIs, infrastructure-as-code surfaces, credentials, dashboards, and logs in a tool-enabled containerized environment (Kottamasu et al., 13 Jan 2026). That comparison locates SWE-ContextBench as repository-centric even when it is sequence-aware.

The benchmark’s limitations are explicit. It contains 99 related tasks across 12 Python repositories, which is a modest scale and language-specific. It demonstrates first- and second-order relationships but does not formalize DAGs or report sequence-length distributions. Oracle selection depends on ground-truth relatedness, whereas the free settings rely on the agent’s ad hoc selection without a formal retrieval model. Results are shown for one agent, Claude Code with Claude Sonnet 4.5, under one tool-use protocol. Sampling parameters, temperatures, retries, and hardware are not specified (Zhu et al., 9 Feb 2026).

These constraints shape the benchmark’s research implications. The main empirical conclusion is that correctly selected, compact summaries improve accuracy and efficiency, whereas unfiltered or misselected experience is neutral or harmful. This motivates future work on high-precision retrieval, structured memory representations, repository-aware ranking features, conservative context injection, and deeper task graphs. The paper explicitly recommends evaluation of retrieval algorithms, richer memory stores, structured summaries or automated distillations, and expansion to more repositories, languages, and deeper dependency chains. In that sense, SWE-ContextBench serves less as a final benchmark than as a controlled instrument for studying memory representation and retrieval quality in repository-level coding agents (Zhu et al., 9 Feb 2026).