SWE-Explore Benchmark

• SWE-Explore is a benchmark for repository exploration in coding agents that isolates exploration as a distinct capability and evaluates it on line-level evidence retrieval and ranking efficiency.
• It leverages a dataset of 848 issues from 203 repositories across 10 programming languages, using trajectory-grounded supervision to define core code regions.
• The benchmark’s metrics—coverage, ranking quality, and context efficiency—offer actionable insights into the trade-offs in evidence acquisition for automated code repair.

Searching arXiv for the cited SWE-Explore paper and closely related repository-exploration work. SWE-Explore is a benchmark for repository exploration in coding agents. It isolates exploration as a distinct capability rather than treating software repair as a single binary outcome. Given a repository and an issue, an explorer must return a ranked list of relevant code regions under a fixed line budget. The benchmark covers 848 issues across 10 programming languages and 203 open-source repositories, derives line-level ground truth from independent agent trajectories that successfully solved the same issue, and evaluates explorers along coverage, ranking, and context-efficiency dimensions. Its central finding is that these exploration metrics strongly track downstream repair behavior: agentic explorers form a clear tier above classical retrieval, while file-level localization is already strong for modern methods and line-level coverage and efficient ranking remain the main differentiators (Zhang et al., 5 Jun 2026).

1. Formal task definition

SWE-Explore takes as input an issue description $q$ in natural language and a repository snapshot $\mathcal{R}$. The required output is a ranked list of $K$ code regions,

$P=(r_1,\dots,r_K),$

where each region $r_i=(p_i,s_i,e_i)$ consists of a file path $p_i$ and a closed line range $[s_i,e_i]$. Each region covers a set of lines $L(r_i)\subseteq \{(p,\ell)\}$. For a fixed budget $B$, evaluation uses only the prefix $P_{\le B}$ whose total covered lines satisfy the line-budget constraint $\mathcal{R}$0. The set of predicted lines is $\mathcal{R}$1, and the ground-truth core lines are denoted $\mathcal{R}$2 (Zhang et al., 5 Jun 2026).

This formulation makes repository exploration a line-level ranking problem rather than a file classification problem or an end-to-end patch-generation problem. A plausible implication is that it becomes possible to study repository understanding, context retrieval, code localization, and bug diagnosis independently of patch synthesis.

2. Dataset composition and trajectory-grounded supervision

The benchmark is built from SWE-bench Verified, SWE-bench-Pro, and SWE-bench Multilingual. After filtering, it retains 848 issues from 203 open-source repositories. The language mix is 10 languages: Python $\mathcal{R}$3, Go $\mathcal{R}$4, JavaScript $\mathcal{R}$5, with Rust, Java, PHP, TypeScript, Ruby, C, and C++ comprising the remainder. Per-instance averages are 4.3 ground-truth files, 4.7 ground-truth regions, 1,578 ground-truth lines, and repository size of 759 files with 179.6 K non-test lines (Zhang et al., 5 Jun 2026).

Ground truth is explicitly “trajectory-grounded.” For each instance, the construction procedure is: collect at least two successful trajectories $\mathcal{R}$6 from strong LLM agents such as GPT-5.4 and Gemini-3-Pro; extract all explicit read actions, including editor views and shell commands such as cat, head, sed, and grep, to obtain regions $\mathcal{R}$7 for each trajectory $\mathcal{R}$8; form the raw core as the line-level intersection

$\mathcal{R}$9

form raw optional as the union minus the intersection; then refine $K$0 via an LLM-based “load-bearing” classifier plus manual audit to obtain the final $K$1 (Zhang et al., 5 Jun 2026).

This trajectory-grounded construction differs from supervision based only on final edited files. It ties exploration targets to the specific code regions that successful solution paths actually consulted. This suggests a notion of relevance defined by evidential necessity during problem solving rather than by patch overlap alone.

3. Metric design

SWE-Explore defines three metric families. The first measures coverage and line-level accuracy. Precision is

$K$2

line-level recall is

$K$3

and $K$4 is the harmonic mean of the two. It also reports file and region hit rates: $K$5 measures the fraction of ground-truth files touched by the prediction, and $K$6 measures the fraction of core regions with any overlap (Zhang et al., 5 Jun 2026).

The second family evaluates ranking quality under the budget $K$7. It defines a gain $K$8 as the number of ground-truth lines newly covered by region $K$9, and uses budgeted ranking metrics including $P=(r_1,\dots,r_K),$0 and First Useful Hit (FUH), where FUH is $P=(r_1,\dots,r_K),$1 and $P=(r_1,\dots,r_K),$2 is the rank of the first region covering any $P=(r_1,\dots,r_K),$3, with 0 if none. The third family measures context efficiency and noise. Optional context lines are $P=(r_1,\dots,r_K),$4, and

$P=(r_1,\dots,r_K),$5

Noise rate is the fraction of predicted regions overlapping neither core nor optional (Zhang et al., 5 Jun 2026).

These choices separate three often conflated properties: whether an explorer finds the necessary evidence, whether it ranks that evidence early under budget, and whether it avoids spending context on irrelevant spans. Downstream restricted-context validation further shows that Context-efficiency has Pearson $P=(r_1,\dots,r_K),$6 and early coverage measured by Rec@100 has Spearman $P=(r_1,\dots,r_K),$7 with actual patch success, making these not merely descriptive metrics but effective proxies for repair-relevant exploration behavior (Zhang et al., 5 Jun 2026).

4. Empirical results

The benchmark compares Oracle, Random, classical retrievers, general coding agents, and specialized localizers. At $P=(r_1,\dots,r_K),$8 and $P=(r_1,\dots,r_K),$9, Random is near zero on all metrics, while BM25, TF-IDF, and Potion cluster near Random. BM25 records HitReg $r_i=(p_i,s_i,e_i)$0, Recall$r_i=(p_i,s_i,e_i)$1 $r_i=(p_i,s_i,e_i)$2, nDCG@500 $r_i=(p_i,s_i,e_i)$3, and CtxEff $r_i=(p_i,s_i,e_i)$4; TF-IDF improves to HitReg $r_i=(p_i,s_i,e_i)$5, Recall$r_i=(p_i,s_i,e_i)$6 $r_i=(p_i,s_i,e_i)$7, nDCG@500 $r_i=(p_i,s_i,e_i)$8, and CtxEff $r_i=(p_i,s_i,e_i)$9; Potion gives HitReg $p_i$0, Recall$p_i$1 $p_i$2, nDCG@500 $p_i$3, and CtxEff $p_i$4 (Zhang et al., 5 Jun 2026).

General coding agents form a tight tier. OpenHands has HitReg $p_i$5, Recall$p_i$6 $p_i$7, nDCG@500 $p_i$8, and CtxEff $p_i$9. Mini-SWE-Agent has HitReg $[s_i,e_i]$0, Recall$[s_i,e_i]$1 $[s_i,e_i]$2, nDCG@500 $[s_i,e_i]$3, and CtxEff $[s_i,e_i]$4. AweAgent records HitReg $[s_i,e_i]$5, nDCG@500 $[s_i,e_i]$6, and CtxEff $[s_i,e_i]$7, while Claude Code has HitReg $[s_i,e_i]$8, Recall$[s_i,e_i]$9 $L(r_i)\subseteq \{(p,\ell)\}$0, nDCG@500 $L(r_i)\subseteq \{(p,\ell)\}$1, and CtxEff $L(r_i)\subseteq \{(p,\ell)\}$2. Codex reaches Recall$L(r_i)\subseteq \{(p,\ell)\}$3 $L(r_i)\subseteq \{(p,\ell)\}$4, nDCG@500 $L(r_i)\subseteq \{(p,\ell)\}$5, and CtxEff $L(r_i)\subseteq \{(p,\ell)\}$6. Specialized localizers are more heterogeneous. AutoCodeRover achieves Recall$L(r_i)\subseteq \{(p,\ell)\}$7 $L(r_i)\subseteq \{(p,\ell)\}$8 and nDCG@500 $L(r_i)\subseteq \{(p,\ell)\}$9; LocAgent reaches HitReg $B$0, Recall$B$1 $B$2, and nDCG@500 $B$3; OrcaLoca under-covers with HitReg $B$4 and Recall$B$5 $B$6; CoSIL achieves the highest non-oracle Recall$B$7 $B$8, with HitReg $B$9, nDCG@500 $P_{\le B}$0, and CtxEff $P_{\le B}$1, by emitting large regions (Zhang et al., 5 Jun 2026).

The aggregate pattern is precise. Agentic explorers clearly outperform one-shot lexical and embedding retrieval. File-level localization is already strong, with HitFile approximately $P_{\le B}$2, but line-level recall remains a bottleneck, with Rec$P_{\le B}$3 approximately $P_{\le B}$4 for many strong agents. This means that many modern systems can identify the right files while still failing to surface enough of the load-bearing lines within a constrained budget (Zhang et al., 5 Jun 2026).

5. Interpretation and methodological significance

SWE-Explore argues that repository exploration should be treated as a first-class subproblem in coding-agent evaluation. Its experiments show that file-level success can conceal a line-level recall bottleneck, and that efficient ranking matters because only a bounded prefix of the prediction is evaluated. The benchmark therefore shifts attention from coarse localization to budgeted evidence acquisition (Zhang et al., 5 Jun 2026).

A central methodological result is that downstream patchers are more sensitive to missing core evidence than to moderate excess context. The benchmark’s analysis states that the main avenue for improvement is high-recall line-level exploration, for example iterative code-graph search as in CoSIL, rather than only better reranking or precision filtering. This suggests that improvements in repository exploration may require broader search strategies before they require better final ranking models (Zhang et al., 5 Jun 2026).

The benchmark also sharpens what “good exploration” means. High $P_{\le B}$5 without sufficient line recall can still leave a patcher starved of evidence; high recall achieved by emitting large regions may improve repair prospects at the cost of tighter context efficiency. SWE-Explore makes these trade-offs explicit by jointly measuring coverage, ranking, and context efficiency rather than collapsing them into a single resolved-or-unresolved score (Zhang et al., 5 Jun 2026).

6. Relation to adjacent work on repository exploration and analysis

Several contemporaneous studies illuminate mechanisms that SWE-Explore measures. “Exploration Structure in LLM Agents for Multi-File Change Localization” compares linear sequential exploration with non-linear domain-scoped parallel exploration. On ansible-centered evaluations, domain-scoped parallel agent spawning improves the recall–precision trade-off for moderate-sized models, while also surfacing three limitations: documentation evolution is a latent dependency, naive file-system access can degrade localization through test-file over-prediction, and forced multi-agent consultation does not measurably help while substantially increasing token cost (Fattha et al., 10 Jun 2026).

Other work addresses observability and process analysis rather than benchmark design. “SeaView: Software Engineering Agent Visual Interface for Enhanced Workflow” provides a full-stack visualization and analysis framework for long SWE-agent trajectories, including experiment health, comparison, and summarization views, with windowed rendering for trajectories exceeding 128 K tokens and a user study reporting a reduction in median analysis time from 25 minutes to under 5 minutes (Bula et al., 11 Apr 2025). “Projecting the Emerging Mindset of SWE Agent by Launching a Wild Code Understanding Journey” introduces Ada, a bounded tool-mediated apparatus for repository-level code understanding, and defines five observation lenses over 408 trajectories to characterize navigation, evidence selection, synthesis, grounding, and stopping (Zhuo et al., 7 Jun 2026).

Taken together, these adjacent results situate SWE-Explore within a broader research program. The benchmark specifies what repository exploration should be evaluated on; domain-scoped exploration studies propose alternative exploration structures; and trajectory-analysis systems provide the instrumentation needed to understand why an explorer attains a particular profile on coverage, ranking, and context-efficiency axes. In that sense, SWE-Explore functions not only as an evaluation suite but also as a focal task for “explorer” modules in future coding-agent architectures (Zhang et al., 5 Jun 2026).