ContextBench: Evaluating Repository Context Retrieval

• ContextBench is a benchmark for context retrieval in coding agents, evaluating which code segments are inspected against expert-annotated gold contexts.
• It comprises 1,136 issue-resolution tasks from 66 repositories across 8 programming languages, enabling granular analysis at file, block, and line levels.
• The framework measures retrieval effectiveness using metrics such as recall, precision, F1, efficiency, redundancy, and evidence drop to diagnose agent performance.

ContextBench is a benchmark and evaluation framework for context retrieval in coding agents. It is designed to measure not only whether an agent resolves a repository-level software issue, but also what code it actually inspects, how that retrieved context aligns with a human-annotated gold context, and how much of the explored context is ultimately used in patch generation. The benchmark consists of 1,136 issue-resolution tasks from 66 repositories across 8 programming languages, each paired with expert-annotated gold contexts at file, block, and line granularity, and an instrumentation framework that computes context recall, precision, F1, efficiency, redundancy, and evidence drop during issue resolution (Li et al., 5 Feb 2026).

1. Definition and motivation

ContextBench addresses a process-level question in repository-scale software engineering: when an LLM-based coding agent attempts to fix a real issue, does it find the right code context, and how closely does that context match what expert developers consider necessary? Existing evaluations such as SWE-bench, SWE-bench Pro, Multi-SWE-bench, and SWE-PolyBench primarily emphasize end-to-end success, typically whether a generated patch passes tests. ContextBench is instead process-oriented: it “opens the black box” of agent behavior by evaluating file reads, code inspection actions, and final patch-context declarations against a gold reference (Li et al., 5 Feb 2026).

The benchmark formalizes the idea that context retrieval is a core bottleneck in repository-level issue resolution. Repositories are large, model context windows are finite, and fixes often depend on dispersed functions, classes, and files. An agent may fail because it never opens the relevant file, inspects the wrong function, or retrieves a large amount of irrelevant code. ContextBench therefore augments patch-level evaluation with intermediate signals that expose whether an agent knows where to look and how to use what it found (Li et al., 5 Feb 2026).

A broader implication is that ContextBench sits at the intersection of coding-agent evaluation and context-learning evaluation. Other benchmarks study context learning in general domains, software engineering, long-context reasoning, or real-life archives, but ContextBench isolates repository-context retrieval as a distinct object of measurement. This suggests that repository-level code understanding is not reducible to final patch success alone, because success can otherwise obscure retrieval failures, trial-and-error behavior, or over-reliance on tests (Li et al., 5 Feb 2026).

2. Benchmark composition and gold-context annotation

ContextBench is constructed from 4,497 tasks aggregated from four existing GitHub-issue benchmarks: SWE-bench Verified (500 tasks), Multi-SWE-bench (1,632 tasks), SWE-PolyBench PB500 (500 tasks), and SWE-bench Pro (1,865 tasks). The curation pipeline first performs ID-based deduplication and then semantic near-duplicate removal using embedding similarity, reducing the pool to 3,100 unique tasks. From these, the benchmark selects challenging tasks using three criteria: agent solvability, edit scope, and edit dispersion. After manual removal of semantically trivial but noisy issues such as bulk renames or formatting-only changes, the final benchmark contains 1,136 tasks (Li et al., 5 Feb 2026).

The repository distribution spans 66 repositories across 8 languages: Python (20 repos, 512 tasks), Java (6, 57), JavaScript (9, 153), TypeScript (8, 119), Go (7, 104), Rust (9, 63), C (3, 68), and C++ (4, 60). The total annotated gold context comprises 4,548 files, 23,116 blocks, and 522,115 lines. A Lite subset of 500 tasks is also provided for faster experimentation (Li et al., 5 Feb 2026).

A gold context is defined as a compact but sufficient set of code regions that expert annotators identify as necessary to resolve the issue and generate a test-passing patch. The benchmark does not claim global minimality, but it enforces sufficiency and compactness through a multi-round annotation procedure. Annotators begin from the real patch, trace semantic dependencies through calls, inheritance, interfaces, control flow, data flow, and nearby code, and then verify sufficiency using GPT-5 in a controlled setting where only the annotated context is provided. If at least one generated patch passes the official test suite, the context is treated as sufficient. Compactness is then checked by a second annotator, and failed cases enter additional rounds of independent re-annotation and reconciliation (Li et al., 5 Feb 2026).

Gold contexts are recorded at three granularities. File-level context uses absolute paths. Block-level context uses definition-level AST nodes extracted with tree-sitter, such as functions, methods, and classes. Line-level context uses explicit start–end spans. This three-level representation enables evaluation of whether an agent merely opened the right file, inspected the right semantic unit, or actually focused on the critical lines (Li et al., 5 Feb 2026).

3. Evaluation framework and metrics

ContextBench instruments agent trajectories by logging every code segment inspected during issue resolution, including file path, line range, and content. These are normalized into machine-parseable context blocks such as <PATCH_CONTEXT> or <EXPLORE_CONTEXT>. The framework records a sequence of intermediate context snapshots, denoted $C^A_1, C^A_2, \dots, C^A_T$, and requires the agent, before submitting a patch, to emit a final declared context $C^A$: the code it claims to have used in generating the patch (Li et al., 5 Feb 2026).

The benchmark then compares agent context against gold context $C^G$ at file, block, and line level. The core retrieval metrics are:

$$\mathrm{Recall}(C^{A}, C^{G}) = \frac{|C^{A} \cap C^{G}|}{|C^{G}|}$$

$$\mathrm{Precision}(C^{A}, C^{G}) = \frac{|C^{A} \cap C^{G}|}{|C^{A}|}$$

$$F_1 = \frac{2 \times \mathrm{Precision} \times \mathrm{Recall}}{\mathrm{Precision} + \mathrm{Recall}}$$

Recall measures how much of the necessary gold context the agent retrieved. Precision measures how much of the retrieved context was truly necessary. F1 balances the two. The benchmark reports that coding agents and LLMs tend to favor recall over precision, retrieving broadly but including substantial irrelevant code (Li et al., 5 Feb 2026).

ContextBench also measures retrieval dynamics over time. If $A^{(t)} = \bigcup_{i=1}^{t} C^A_i$ is the cumulative explored context up to step $t$, then efficiency is defined as average cumulative recall over steps:

$\mathrm{AUC\mbox{-}Cov} = \frac{1}{T} \sum_{t=1}^{T} \mathrm{Recall}(A^{(t)}, C^{G})$

This rewards agents that find the essential gold context early. Redundancy measures how much a step revisits already explored context:

$$\mathrm{Redun}_{t} = \frac{\left| C^{A}_{t} \cap \left(\bigcup_{i=1}^{t-1} C^{A}_{i}\right) \right|}{|C^{A}_t|}$$

$C^A$0

High redundancy indicates repeated rereading rather than productive exploration (Li et al., 5 Feb 2026).

A distinctive metric is evidence drop, which captures the gap between explored and utilized context. Let the gold elements ever seen during exploration be

$C^A$1

Then the kept fraction is

$C^A$2

and evidence drop is

$C^A$3

High drop means the agent saw relevant evidence during exploration but did not retain it in final patch reasoning. ContextBench identifies this as a major bottleneck, especially for some model backbones (Li et al., 5 Feb 2026).

4. Models, agents, and empirical findings

The benchmark evaluates both LLM backbones and agent scaffolds. The four LLM backbones are GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro, and Devstral 2, all used within the same mini-SWE-agent scaffold. The five coding agents are mini-SWE-agent, Agentless, SWE-agent, OpenHands, and Prometheus, all evaluated with GPT-5 as the underlying model (Li et al., 5 Feb 2026).

A headline empirical result is described as “The Bitter Lesson” of coding agents: sophisticated scaffolding yields only marginal gains in context retrieval. On the Lite subset, mini-SWE-agent attains file-level recall 0.682, precision 0.709, and F1 0.634, with Pass@1 = 0.472. More complex agents often improve recall at the cost of much lower precision. For example, OpenHands achieves the best file-level recall 0.733, but its precision drops to 0.400 and file-level F1 to 0.463. Prometheus attains the best Pass@1 = 0.512, but with low precision at block level (0.258) and low block-level F1 (0.285) (Li et al., 5 Feb 2026).

Across model backbones, ContextBench finds that LLMs are consistently recall-biased. File-level recall is relatively high—GPT-5: 0.682, Claude Sonnet 4.5: 0.720, Gemini 2.5 Pro: 0.587, Devstral 2: 0.660—but block- and line-level F1 remain low, with block F1 below 0.45 and line F1 below 0.35 for all models. Claude Sonnet 4.5 provides the best balance and attains the highest Pass@1 = 0.530 with line-level F1 0.344. Gemini 2.5 Pro shows the highest block- and line-level precision but lower recall. GPT-5 exhibits high recall with weaker precision (Li et al., 5 Feb 2026).

Evidence consolidation emerges as another critical differentiator. The reported evidence drop values are GPT-5: 0.179, Claude Sonnet 4.5: 0.196, Gemini 2.5 Pro: 0.431, and Devstral 2: 0.435. This indicates that Gemini and Devstral often inspect relevant gold context during exploration but fail to carry it into final patch-context declarations. ContextBench therefore separates finding relevant evidence from retaining and using it (Li et al., 5 Feb 2026).

The framework also exposes differences in retrieval style. GPT-5 takes relatively few steps (5.87) but reads many lines per step (119.29). Devstral 2 takes many steps (22.16) but reads few lines per step (11.98). Claude Sonnet 4.5 lies between them (14.38 steps, 29.74 lines/step) and achieves the best trade-off between performance and cost. Gemini 2.5 Pro uses 7.57 steps, 26.29 lines per step, and the lowest cost (\$0.38 per instance) (Li et al., 5 Feb 2026).

A robustness analysis over 82 tasks with multiple semantically equivalent patches shows that patch-conditioned gold contexts are highly stable: the average Jaccard similarity between gold contexts is 0.9518. This supports the claim that the annotated contexts are not brittle artifacts of one specific patch formulation (Li et al., 5 Feb 2026).

5. Relation to the broader context-benchmark landscape

ContextBench is one member of a wider family of benchmarks centered on context use, but its unit of analysis is unusually specific: repository context retrieval in coding agents. Other benchmarks isolate related but distinct capabilities. CL-bench studies whether models can learn new domain knowledge from long, complex contexts and solve tasks grounded in that context; it contains 500 contexts, 1,899 tasks, and 31,607 verification rubrics, and reports an average task solving rate of 17.2% across ten frontier LMs (Dou et al., 3 Feb 2026). CL4SE adapts the context-learning paradigm to software engineering tasks, introducing four SE-specific context types and reporting that context learning yields an average performance improvement of 24.7% across its tasks (Hu et al., 26 Feb 2026).

Benchmarks such as CL-bench Life shift the focus to messy everyday contexts including group chats, personal archives, and behavioral traces. It contains 405 context-task pairs and 5,348 verification rubrics, and shows that even the best model achieves only 19.3% task solving rate (Dou et al., 29 Apr 2026). LOCA-bench instead studies language agents under dynamically growing context, using controllable environment states to stress context rot as trajectories lengthen (Zeng et al., 8 Feb 2026). LongCodeBench evaluates long-context coding ability up to 1M tokens, but it emphasizes repository-scale comprehension and repair rather than explicit gold-context retrieval dynamics (Rando et al., 12 May 2025).

A different axis is represented by CUB, which evaluates context utilisation techniques in retrieval-augmented generation across gold, conflicting, and irrelevant contexts (Hagström et al., 22 May 2025). Its focus is on whether a model follows supplied evidence or parametric memory, whereas ContextBench compares explored and used code context against a human reference set. This suggests that “context” is not a single benchmarkable phenomenon but a family of capabilities: context learning, context selection, context faithfulness, context robustness, and context retention (Hagström et al., 22 May 2025).

Within this broader landscape, ContextBench is distinctive because it operationalizes context as a compact but sufficient code subgraph attached to a real issue-resolution task, then measures overlap between that subgraph and actual agent behavior. It therefore complements end-to-end coding benchmarks: patch success answers whether the bug was fixed, while ContextBench asks whether the agent retrieved and used the right repository evidence in the process (Li et al., 5 Feb 2026).

6. Limitations, uses, and implications

ContextBench is deliberately focused. Its repository and language coverage is substantial but not exhaustive, and domains such as mobile, embedded systems, notebooks, or very large monorepos are not the emphasis. The benchmark is built around standardized harnesses similar to SWE-bench environments, so it does not capture all features of real developer workflows such as IDE assistance, live documentation search, or human feedback (Li et al., 5 Feb 2026).

Gold context remains an expert annotation, not an absolute minimal proof object. The benchmark explicitly frames precision relative to a compact-and-verified reference set, not to a globally unique minimal context. This controlled subjectivity is mitigated by multi-round annotation, consensus refinement, and LLM-based sufficiency checks, but it remains a design assumption (Li et al., 5 Feb 2026).

For research use, ContextBench offers several concrete functions. It can evaluate new coding agents by instrumenting <PATCH_CONTEXT> declarations and comparing them against gold context. It can separate retrieval from reasoning by testing whether a method fails because it never found the right code or because it could not use it once found. It can guide tool and scaffold design by revealing low recall, low precision, high redundancy, or high evidence drop. It can also complement end-to-end benchmarks by supplying intermediate process metrics that are otherwise invisible (Li et al., 5 Feb 2026).

A plausible implication is that future coding-agent progress may depend less on adding increasingly elaborate scaffolds and more on improving retrieval balance and evidence consolidation. ContextBench’s results suggest that broad retrieval without precision, or exploration without retention, is a recurring failure mode. The benchmark therefore functions not only as an evaluation suite but also as a diagnostic framework for repository-scale software agents (Li et al., 5 Feb 2026).