---
title: 'SHERLOC: Code Repair Localization'
url: https://www.emergentmind.com/topics/sherloc
type: topic
---

# SHERLOC: Code Repair Localization

Searching arXiv for the specified SHERLOC paper and closely related work to ground the article in current literature.
SHERLOC, short for “Structured Hypothesis-driven Exploration and Reasoning for Localization,” is a training-free framework for repository-level code repair agents that reframes bug localization as actionable diagnosis rather than file retrieval [2606.24820]. It pairs a single reasoning LLM with a compact repository toolset and lightweight self-recovery, and returns both predicted fault locations and structured diagnostic findings. In the reported evaluation, SHERLOC reaches 84.33% accuracy@1 on SWE-Bench Lite and 81.27% recall@1 on SWE-Bench Verified; when its outputs are injected into repair agents, the average resolve rate on SWE-Bench Verified increases by +5.95 percentage points while localization and total tokens are reduced by 36.7% and 23.1%, respectively [2606.24820].

## 1. Problem setting and conceptual scope

SHERLOC is motivated by the observation that repository-level code repair agents must find and fix faults in large, unfamiliar codebases under tight interaction budgets. Empirically, general-purpose repair agents spend nearly half of their interaction budget just to locate faults, with an average 18.5 turns, 48% of turns, and over 320k tokens per instance before proposing the first patch [2606.24820]. Within this setting, pure “file retrieval” localization is treated as insufficient because a ranked path list does not supply the diagnostic context required for patch construction.

The framework therefore recasts localization as a joint problem of identifying where to edit and why the location is implicated. Its outputs are not limited to files: SHERLOC predicts precise line-number spans and attaches five diagnostic fields for each predicted location, namely location explanation, root cause, solution idea, dependencies, and testing impact [2606.24820]. This design is intended to reduce wasted search and to provide a compact scaffold that downstream repair agents can use to orient, inspect evidence, and plan their first edits.

Formally, with issue description $I$ and repository snapshot $R$, the localizer computes a set of line spans
$$
L = \{(f_j, s_j, e_j)\}_{j=1..m}
$$
and a set of diagnostic findings
$$
D = \{d_j\}_{j=1..m},
$$
where each $d_j$ contains the five structured fields above [2606.24820]. SHERLOC does not optimize an explicit parametric objective; instead, it relies on structured exploration and reasoning to accumulate evidence and synthesize $(L, D)$.

## 2. Architecture, tools, and control logic

SHERLOC is a training-free framework pairing a single reasoning LLM with a compact, deterministic tool executor and lightweight self-recovery; there is no fine-tuning, reinforcement learning, or multi-agent orchestration [2606.24820]. The reasoning backbone is a “thinking-mode” model such as Qwen3-235B-A22B-Thinking, with additional evaluations on Qwen3-30B, DeepSeek-V3, and DeepSeek-R1. The deterministic executor mediates between the LLM and the repository through narrow schemas and bounded, model-readable outputs, while enforcing safety and consistency through a no-shell interface.

The repository toolset is intentionally small:

| Tool | Function |
|---|---|
| `view_file(path, [start, end])` | Return file content, optionally sliced, with import metadata |
| `codebase_search(query)` | Repository-wide case-insensitive literal search with $\pm 20$ lines of context and line numbers |
| `repo_tree()` | Show filtered tree with paths and line counts |
| `connected_tree([path])` | Summarize direct and reverse imports |

The small tool vocabulary is part of the design rationale. The paper states that the suite is intentionally small to keep the exploration legible and the LLM’s actions stable [2606.24820]. The hypothesis manager is implicit rather than external: issue text, repo tree, tool calls, and tool observations accumulate as conversation state, and no external vector store or graph index is required.

Self-recovery is a first-class component. It includes first-and-recent truncation for context management, loop detection for repeated unproductive calls with the same parameters, implicit tool-call recovery for malformed but unambiguous requests, response-length management near output limits, and forced final-turn synthesis when the step budget is nearly exhausted [2606.24820]. This control layer is lightweight but consequential in ablations.

## 3. Iterative exploration and actionable diagnosis

The operating protocol is a bounded multi-turn loop. Initialization builds a filtered repository view by pruning docs, build artifacts, and VCS metadata, then composes the initial context from the issue description, tool schemas, and the filtered repo tree [2606.24820]. The first response must be a tool call, never a final answer. Thereafter, at each turn, the LLM either acquires more evidence by selecting one tool and arguments or terminates with final outputs.

Execution proceeds through tool-call parsing, validation, and observation appending, followed by self-recovery checks for loop detection, implicit-call repair, context truncation, and response-length guards [2606.24820]. Localization is bounded to at most 20 turns, with inference settings temperature $=0.99$, top-p $=0.95$, and generation cap $81{,}920$ tokens per call. If no well-formed final answer is produced before budget exhaustion, a final-turn prompt forces synthesis into `<findings>` and `<locations>` blocks.

The final output structure is central to SHERLOC’s definition of localization. Each predicted location is paired with five diagnostic fields:

1. **Location explanation**: the rationale for implicating the file or line span.  
2. **Root cause**: the suspected mechanism of failure.  
3. **Solution idea**: actionable edit direction without code.  
4. **Dependencies**: impacted modules, imports, or assumptions.  
5. **Testing impact**: how to validate and what tests are affected.  

This transforms localization from ranked retrieval into actionable diagnosis [2606.24820]. The paper explicitly argues that merely naming a file is insufficient, whereas root cause and solution idea materially change downstream agent behavior.

Two reported case studies illustrate the protocol. In `requests/sessions.py`, SHERLOC searches for normalization logic, inspects `prepare_request`, observes that `builtin_str(method)` mishandles Python 3 byte strings, and produces a finding whose solution idea is to replace `builtin_str` with `to_native_string` before `upper()` [2606.24820]. In `xarray/core/computation.py`, it identifies that `where()` drops attributes because the `apply_ufunc` call lacks `keep_attrs=True`, and it notes tests in `tests/test_computation.py` [2606.24820].

## 4. Evaluation methodology and metrics

SHERLOC is evaluated on SWE-Bench Lite and SWE-Bench Verified with both file-level retrieval metrics and structure-agnostic chunk-level metrics over predicted line spans [2606.24820]. The file-level measures are standard:
$$
\mathrm{Accuracy@k} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\mathrm{hit}_i \le k]
$$
and, for multi-file bugs,
$$
\mathrm{Recall@k} = \frac{1}{N}\sum_{i=1}^{N}\frac{|correct_i \cap top\_k_i|}{|correct_i|}.
$$

For chunk-level evaluation, ground truth is represented as $G_i = \{(f,s,e)\}$ and predictions as $P_i = \{(f,s',e')\}$, with “cover” defined by strict containment of a gold span inside a predicted span [2606.24820]. The reported metrics are coverage recall, chunk precision, and average tightness. Average tightness measures span compactness on covered pairs:
$$
t(g,p)=\frac{length(g)}{length(p)}=\frac{e(g)-s(g)+1}{e(p)-s(p)+1}.
$$
This evaluation is explicitly structure-agnostic, which is important when correct edits do not align cleanly with syntactic units.

The framework also accounts for trajectory cost. If $T_{loc}$ denotes localization-phase tokens, $T_{edit}$ post-first-edit tokens, and $T_{total}$ total tokens, then
$$
T_{total}=T_{loc}+T_{edit}.
$$
This token accounting is central to the downstream transfer analysis because SHERLOC is intended not only to improve localization accuracy but also to reduce search overhead [2606.24820].

The benchmark protocol includes three seeds for main localization runs, identical prompts and tools across backbones, deterministic execution, and downstream transfer experiments across two repair frameworks, SWE-Agent and OpenHands, and five repair-model backbones [2606.24820]. Baselines include retrieval rankers such as SWERank, multi-agent and graph methods such as SWE-Debate, OrcaLoca, and LocAgent, and RL tool-train methods such as RepoSearcher and ToolTrain.

## 5. Empirical performance and downstream transfer

On SWE-Bench Lite, SHERLOC achieves 84.33% $\pm$ 0.72% Accuracy@1 with Qwen3-235B [2606.24820]. On SWE-Bench Verified, it achieves 81.27% $\pm$ 1.16% Recall@1 with Qwen3-235B, 79.53% $\pm$ 0.47% with DeepSeek-V3, and 75.07% $\pm$ 1.24% at $\sim 30$B scale with Qwen3-30B, outperforming prior 32B agentic baselines such as ToolTrain-32B-FT at 68.03% [2606.24820]. For chunk-level span quality on Verified with Qwen3-235B, the reported values are coverage recall 41.28%, chunk precision 53.47%, and tightness 36.36%.

Trajectory efficiency is a major reported result. SHERLOC’s standalone localization uses 4.7 turns on average, with approximately 28.6k tokens per instance, and turns increase only modestly with difficulty [2606.24820]. This profile contrasts with the much larger pre-edit localization cost reported for general-purpose repair agents and underwrites the downstream token savings.

Transfer to repair is performed by injecting `<locations>` and `<findings>` into the initial prompt or context of repair frameworks [2606.24820]. Across 10 backbone$\times$framework cells on SWE-Bench Verified, this yields an average resolve-rate gain of +5.95 percentage points, together with reductions of 36.7% in localization tokens and 23.1% in total tokens. At $\sim 30$B scale, SHERLOC improves resolve by +8–12 percentage points while also reducing search cost [2606.24820].

The quality of the diagnostic findings is itself predictive of downstream repair. Using LLM-as-judge scoring with GPT‑5.2 on a 1–5 scale for root-cause correctness, location accuracy, and solution actionability, the composite score is
$$
score(d)=\frac{root\_cause + location\_accuracy + solution\_actionability}{3}.
$$
Very-high-quality findings with score $>4.0$ resolve at 75.9%, whereas low-quality findings with score $\le 2.0$ resolve at 20.0%, with Pearson $r=0.45$ between composite score and binary resolution [2606.24820]. A threshold of 4.0 retains 63% of cases and is reported to balance reliability and coverage.

## 6. Ablations, validity controls, limitations, and naming

Ablation studies identify the main contributors to SHERLOC’s performance [2606.24820]. On 100 SWE-Gym dev issues, removing `view_file` causes the largest tool drop at $-7.0$ percentage points F1; removing `codebase_search` or `connected_tree` causes drops of $-4.5$ to $-5.4$ percentage points F1, whereas removing `repo_tree` has a small effect. In the self-recovery ablation, removing final-turn synthesis causes the largest drop at $-5.0$ percentage points F1, while removing context management or implicit tool detection causes drops of $-3.3$ to $-4.0$ percentage points F1. A particularly strong result concerns model mode: at 30B scale, thinking-mode versus instruct-only produces a recall collapse from approximately 74–77% to approximately 10–12%, and 87% of samples fail to produce a valid output under instruct-only prompting.

The paper also devotes substantial attention to implicit knowledge and benchmark validity [2606.24820]. In a text-only setting with tools off, repo tree off, and file paths masked, performance is still 57.86% Recall@1 on Verified, indicating strong familiarity for popular repositories. Full SHERLOC with masked issue paths but tools retained reaches 79.96% Recall@1, only 1.3 percentage points below the unmasked system. Implicit recall is reported to concentrate on popular repositories such as scikit-learn at 85.3% and requests at 87.5%, which the authors use to support the contention that benchmark contamination inflates some metrics. The masked-issue control is presented as a bound on, rather than a removal of, this confound.

Failure analysis indicates that most zero-recall@1 cases are near misses rather than navigational collapse [2606.24820]. The reported breakdown is 40% reasoning error after seeing the right file, 27% close miss in the correct directory but wrong file, 25% wrong module entirely, and 4% true multi-file cases with more than three files. This suggests that final candidate selection, not repository traversal, is the dominant remaining bottleneck.

The stated limitations are that the implicit-knowledge confound persists, the benchmark scope is Python repositories in SWE-Bench, and the best configuration uses a large reasoning model at 235B, although matched-scale 30B results are also reported [2606.24820]. Future work includes broader tooling while keeping legibility, improved diagnostics and evidence coverage scoring, uncertainty calibration without judge-based filtering at test time, cross-repository and multilingual generalization, and adaptive test-time scaling policies.

The acronym is also used by unrelated systems in other domains. “SherlockLLM” denotes a dialogue-driven retrieval framework that learns to ask yes/no questions for ambiguous-query retrieval [2510.18659], while “SHeRLoc” denotes “Synchronized Heterogeneous Radar Place Recognition for Cross-Modal Localization” in robotics [2506.15175]. In the present usage, SHERLOC refers specifically to the code-repair localization framework introduced in 2026 [2606.24820].

Source: https://www.emergentmind.com/topics/sherloc