SWE Context Bench: A Benchmark for Context Learning in Coding

Abstract: LLMs are increasingly used as programming agents for repository level software engineering tasks. While recent benchmarks evaluate correctness in realistic codebases, they largely treat tasks as independent and do not assess whether agents can reuse experience across related problems. As a result, the ability of agents to accumulate, retrieve, and apply prior experience, as well as the efficiency gains from such reuse, remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate experience reuse in programming agents. Built on SWE-Bench Lite, SWE-ContextBench augments 300 base tasks with 99 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests, forming task sequences with shared context. The benchmark evaluates agents along three complementary dimensions: prediction accuracy, time efficiency, and cost efficiency. Using SWE-ContextBench, we study multiple experience reuse settings, including oracle guided and autonomous retrieval, as well as full execution trajectories and compact summaries. Our results show that correctly selected summarized experience improves resolution accuracy and substantially reduces runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected experience provides limited or negative benefits. These findings highlight the importance of experience representation and retrieval quality, and position SWE-ContextBench as a principled benchmark for studying experience reuse in programming agents.

• The paper introduces SWE-ContextBench, evaluating programming agents on experience reuse across interconnected coding tasks.
• It leverages 300 tasks from SWE-Bench Lite and 99 GitHub-based tasks to measure performance metrics like accuracy, time efficiency, and cost savings.
• Results show that precise experience selection, particularly through Oracle Summary Reuse, significantly boosts task resolution and runtime performance.

SWE Context Bench: A Benchmark for Context Learning in Coding

Introduction

The paper "SWE Context Bench: A Benchmark for Context Learning in Coding" (2602.08316) introduces SWE-ContextBench, a groundbreaking approach to evaluating experience reuse in programming agents. With the rise of LLMs as essential tools in software engineering tasks, this benchmark aims to assess an agent's ability to accumulate, retrieve, and apply past experiences in repository-level scenarios. Through SWE-ContextBench, the authors attempt to bridge the gap in existing benchmarks by incorporating context reuse into performance evaluation. The benchmark leverages 300 base tasks from SWE-Bench Lite, augmented by 99 related tasks derived from real-world GitHub interactions. This paper provides insights into the advantages and limitations of experience reuse, emphasizing its significant effects on prediction accuracy, runtime efficiency, and computational cost.

Figure 1: Comparison with existing agent benchmark and long context benchmark.

Benchmark Design

Motivation

The core motivation driving the design of SWE-ContextBench is the observed deficiency in existing code benchmarks which fail to measure sequential learning and experience reuse capabilities of programming agents. Despite advances in repository-level evaluations, benchmarks have historically focused on isolated tasks rather than interconnected task sequences. SWE-ContextBench is therefore designed to evaluate how repositories can provide context that aids in solving subsequent tasks efficiently, a practice synonymous with human engineering processes where past successes and failures influence future problem solving.

Desiderata: Benchmark Evaluation Criteria

A robust benchmark should evaluate task accuracy, time efficiency, and cost efficiency:

1. Accuracy: Correct solutions must be produced, typically verified through human-validated fixes within GitHub repositories.
2. Time Efficiency: Experience reuse should enable quicker task resolution by leveraging previously identified solutions.
3. Cost Efficiency: Experience reuse should minimize redundant computational steps, ultimately reflected in lower token usage.

These dimensions are integral to advancing benchmarks beyond merely correctly resolving problems, acknowledging the significance of collective learning.

Data Construction

Base Dataset and Experience Trajectories

SWE-Bench Lite comprises 300 instances drawn from GitHub repositories, each encompassing an issue description and resolving pull requests. The experience trajectory for each task, capturing reasoning processes and tool interactions, forms an accessible pool for subsequent task evaluations. To accurately gauge experience reuse, related tasks are identified through references among issues and pull requests, constructing sequences where context is shared.

Related Task Identification and Context Expansion

Related tasks in SWE-ContextBench emerge from explicit analysis of repository interactions. The benchmark encapsulates varied interdependencies, including multi-issue resolutions and interconnected pull request discussions. Recursive context expansion uncovers task relationships that extend beyond immediate references, enhancing the realism of task sequences.

Experimental Setup and Results

Experience Reuse Settings

The benchmark evaluates different experience reuse settings ranging from no experience to oracle-guided selections. Notably, "Oracle Summary Reuse" demonstrates the highest improvement in task resolution rates, underscoring the importance of concise and correctly chosen prior experiences. Conversely, free experience access fails to consistently enhance performance, implicating the necessity for precise retrieval mechanisms.

Efficiency and Cost Analysis

Oracle Summary Reuse markedly reduces both runtime and cost, particularly for complex tasks, showcasing its practical advantages. This setting minimizes token usage by efficiently leveraging cached experiences, highlighting that compact summaries of prior solutions are more beneficial than comprehensive but potentially irrelevant traces.

Conclusion

SWE-ContextBench represents a pivotal advancement in evaluating programming agents by incorporating experience reuse metrics across well-defined evaluation dimensions. The empirical studies affirm that precise experience selection and representation are vital for improving task resolution rates, runtime efficiency, and token consumption. This benchmark lays the groundwork for future exploration into memory-augmented coding systems, guiding the development of agents that learn cumulatively and adaptively across complex software engineering landscapes.