SHERLOC: Code Repair Localization

• SHERLOC is a training-free framework that reframes bug localization as actionable diagnosis by predicting fault locations with structured diagnostic findings.
• It couples a reasoning LLM with a minimal repository toolset and lightweight self-recovery to efficiently navigate large codebases.
• Evaluations show SHERLOC achieves up to 84.33% accuracy and improves repair resolve rates while significantly reducing token usage.

Searching arXiv for the specified SHERLOC paper and closely related work to ground the article in current literature. SHERLOC, short for “Structured Hypothesis-driven Exploration and Reasoning for Localization,” is a training-free framework for repository-level code repair agents that reframes bug localization as actionable diagnosis rather than file retrieval (Tamoyan et al., 23 Jun 2026). It pairs a single reasoning LLM with a compact repository toolset and lightweight self-recovery, and returns both predicted fault locations and structured diagnostic findings. In the reported evaluation, SHERLOC reaches 84.33% accuracy@1 on SWE-Bench Lite and 81.27% recall@1 on SWE-Bench Verified; when its outputs are injected into repair agents, the average resolve rate on SWE-Bench Verified increases by +5.95 percentage points while localization and total tokens are reduced by 36.7% and 23.1%, respectively (Tamoyan et al., 23 Jun 2026).

1. Problem setting and conceptual scope

SHERLOC is motivated by the observation that repository-level code repair agents must find and fix faults in large, unfamiliar codebases under tight interaction budgets. Empirically, general-purpose repair agents spend nearly half of their interaction budget just to locate faults, with an average 18.5 turns, 48% of turns, and over 320k tokens per instance before proposing the first patch (Tamoyan et al., 23 Jun 2026). Within this setting, pure “file retrieval” localization is treated as insufficient because a ranked path list does not supply the diagnostic context required for patch construction.

The framework therefore recasts localization as a joint problem of identifying where to edit and why the location is implicated. Its outputs are not limited to files: SHERLOC predicts precise line-number spans and attaches five diagnostic fields for each predicted location, namely location explanation, root cause, solution idea, dependencies, and testing impact (Tamoyan et al., 23 Jun 2026). This design is intended to reduce wasted search and to provide a compact scaffold that downstream repair agents can use to orient, inspect evidence, and plan their first edits.

Formally, with issue description $I$ and repository snapshot $R$, the localizer computes a set of line spans

$L = \{(f_j, s_j, e_j)\}_{j=1..m}$

and a set of diagnostic findings

$D = \{d_j\}_{j=1..m},$

where each $d_j$ contains the five structured fields above (Tamoyan et al., 23 Jun 2026). SHERLOC does not optimize an explicit parametric objective; instead, it relies on structured exploration and reasoning to accumulate evidence and synthesize $(L, D)$.

2. Architecture, tools, and control logic

SHERLOC is a training-free framework pairing a single reasoning LLM with a compact, deterministic tool executor and lightweight self-recovery; there is no fine-tuning, reinforcement learning, or multi-agent orchestration (Tamoyan et al., 23 Jun 2026). The reasoning backbone is a “thinking-mode” model such as Qwen3-235B-A22B-Thinking, with additional evaluations on Qwen3-30B, DeepSeek-V3, and DeepSeek-R1. The deterministic executor mediates between the LLM and the repository through narrow schemas and bounded, model-readable outputs, while enforcing safety and consistency through a no-shell interface.

The repository toolset is intentionally small:

Tool	Function
view_file(path, [start, end])	Return file content, optionally sliced, with import metadata
codebase_search(query)	Repository-wide case-insensitive literal search with $\pm 20$ lines of context and line numbers
repo_tree()	Show filtered tree with paths and line counts
connected_tree([path])	Summarize direct and reverse imports

The small tool vocabulary is part of the design rationale. The paper states that the suite is intentionally small to keep the exploration legible and the LLM’s actions stable (Tamoyan et al., 23 Jun 2026). The hypothesis manager is implicit rather than external: issue text, repo tree, tool calls, and tool observations accumulate as conversation state, and no external vector store or graph index is required.

Self-recovery is a first-class component. It includes first-and-recent truncation for context management, loop detection for repeated unproductive calls with the same parameters, implicit tool-call recovery for malformed but unambiguous requests, response-length management near output limits, and forced final-turn synthesis when the step budget is nearly exhausted (Tamoyan et al., 23 Jun 2026). This control layer is lightweight but consequential in ablations.

3. Iterative exploration and actionable diagnosis

The operating protocol is a bounded multi-turn loop. Initialization builds a filtered repository view by pruning docs, build artifacts, and VCS metadata, then composes the initial context from the issue description, tool schemas, and the filtered repo tree (Tamoyan et al., 23 Jun 2026). The first response must be a tool call, never a final answer. Thereafter, at each turn, the LLM either acquires more evidence by selecting one tool and arguments or terminates with final outputs.

Execution proceeds through tool-call parsing, validation, and observation appending, followed by self-recovery checks for loop detection, implicit-call repair, context truncation, and response-length guards (Tamoyan et al., 23 Jun 2026). Localization is bounded to at most 20 turns, with inference settings temperature $=0.99$, top-p $=0.95$, and generation cap $81{,}920$ tokens per call. If no well-formed final answer is produced before budget exhaustion, a final-turn prompt forces synthesis into <findings> and <locations> blocks.

The final output structure is central to SHERLOC’s definition of localization. Each predicted location is paired with five diagnostic fields:

1. Location explanation: the rationale for implicating the file or line span.
2. Root cause: the suspected mechanism of failure.
3. Solution idea: actionable edit direction without code.
4. Dependencies: impacted modules, imports, or assumptions.
5. Testing impact: how to validate and what tests are affected.

This transforms localization from ranked retrieval into actionable diagnosis (Tamoyan et al., 23 Jun 2026). The paper explicitly argues that merely naming a file is insufficient, whereas root cause and solution idea materially change downstream agent behavior.

Two reported case studies illustrate the protocol. In requests/sessions.py, SHERLOC searches for normalization logic, inspects prepare_request, observes that builtin_str(method) mishandles Python 3 byte strings, and produces a finding whose solution idea is to replace builtin_str with to_native_string before upper() (Tamoyan et al., 23 Jun 2026). In xarray/core/computation.py, it identifies that where() drops attributes because the apply_ufunc call lacks keep_attrs=True, and it notes tests in tests/test_computation.py (Tamoyan et al., 23 Jun 2026).

4. Evaluation methodology and metrics

SHERLOC is evaluated on SWE-Bench Lite and SWE-Bench Verified with both file-level retrieval metrics and structure-agnostic chunk-level metrics over predicted line spans (Tamoyan et al., 23 Jun 2026). The file-level measures are standard:

$R$0

and, for multi-file bugs,

$R$1

For chunk-level evaluation, ground truth is represented as $R$2 and predictions as $R$3, with “cover” defined by strict containment of a gold span inside a predicted span (Tamoyan et al., 23 Jun 2026). The reported metrics are coverage recall, chunk precision, and average tightness. Average tightness measures span compactness on covered pairs:

$R$4

This evaluation is explicitly structure-agnostic, which is important when correct edits do not align cleanly with syntactic units.

The framework also accounts for trajectory cost. If $R$5 denotes localization-phase tokens, $R$6 post-first-edit tokens, and $R$7 total tokens, then

$R$8

This token accounting is central to the downstream transfer analysis because SHERLOC is intended not only to improve localization accuracy but also to reduce search overhead (Tamoyan et al., 23 Jun 2026).

The benchmark protocol includes three seeds for main localization runs, identical prompts and tools across backbones, deterministic execution, and downstream transfer experiments across two repair frameworks, SWE-Agent and OpenHands, and five repair-model backbones (Tamoyan et al., 23 Jun 2026). Baselines include retrieval rankers such as SWERank, multi-agent and graph methods such as SWE-Debate, OrcaLoca, and LocAgent, and RL tool-train methods such as RepoSearcher and ToolTrain.

5. Empirical performance and downstream transfer

On SWE-Bench Lite, SHERLOC achieves 84.33% $R$9 0.72% Accuracy@1 with Qwen3-235B (Tamoyan et al., 23 Jun 2026). On SWE-Bench Verified, it achieves 81.27% $L = \{(f_j, s_j, e_j)\}_{j=1..m}$0 1.16% Recall@1 with Qwen3-235B, 79.53% $L = \{(f_j, s_j, e_j)\}_{j=1..m}$1 0.47% with DeepSeek-V3, and 75.07% $L = \{(f_j, s_j, e_j)\}_{j=1..m}$2 1.24% at $L = \{(f_j, s_j, e_j)\}_{j=1..m}$3B scale with Qwen3-30B, outperforming prior 32B agentic baselines such as ToolTrain-32B-FT at 68.03% (Tamoyan et al., 23 Jun 2026). For chunk-level span quality on Verified with Qwen3-235B, the reported values are coverage recall 41.28%, chunk precision 53.47%, and tightness 36.36%.

Trajectory efficiency is a major reported result. SHERLOC’s standalone localization uses 4.7 turns on average, with approximately 28.6k tokens per instance, and turns increase only modestly with difficulty (Tamoyan et al., 23 Jun 2026). This profile contrasts with the much larger pre-edit localization cost reported for general-purpose repair agents and underwrites the downstream token savings.

Transfer to repair is performed by injecting <locations> and <findings> into the initial prompt or context of repair frameworks (Tamoyan et al., 23 Jun 2026). Across 10 backbone$L = \{(f_j, s_j, e_j)\}_{j=1..m}$4framework cells on SWE-Bench Verified, this yields an average resolve-rate gain of +5.95 percentage points, together with reductions of 36.7% in localization tokens and 23.1% in total tokens. At $L = \{(f_j, s_j, e_j)\}_{j=1..m}$5B scale, SHERLOC improves resolve by +8–12 percentage points while also reducing search cost (Tamoyan et al., 23 Jun 2026).

The quality of the diagnostic findings is itself predictive of downstream repair. Using LLM-as-judge scoring with GPT‑5.2 on a 1–5 scale for root-cause correctness, location accuracy, and solution actionability, the composite score is

$L = \{(f_j, s_j, e_j)\}_{j=1..m}$6

Very-high-quality findings with score $L = \{(f_j, s_j, e_j)\}_{j=1..m}$7 resolve at 75.9%, whereas low-quality findings with score $L = \{(f_j, s_j, e_j)\}_{j=1..m}$8 resolve at 20.0%, with Pearson $L = \{(f_j, s_j, e_j)\}_{j=1..m}$9 between composite score and binary resolution (Tamoyan et al., 23 Jun 2026). A threshold of 4.0 retains 63% of cases and is reported to balance reliability and coverage.

6. Ablations, validity controls, limitations, and naming

Ablation studies identify the main contributors to SHERLOC’s performance (Tamoyan et al., 23 Jun 2026). On 100 SWE-Gym dev issues, removing view_file causes the largest tool drop at $D = \{d_j\}_{j=1..m},$0 percentage points F1; removing codebase_search or connected_tree causes drops of $D = \{d_j\}_{j=1..m},$1 to $D = \{d_j\}_{j=1..m},$2 percentage points F1, whereas removing repo_tree has a small effect. In the self-recovery ablation, removing final-turn synthesis causes the largest drop at $D = \{d_j\}_{j=1..m},$3 percentage points F1, while removing context management or implicit tool detection causes drops of $D = \{d_j\}_{j=1..m},$4 to $D = \{d_j\}_{j=1..m},$5 percentage points F1. A particularly strong result concerns model mode: at 30B scale, thinking-mode versus instruct-only produces a recall collapse from approximately 74–77% to approximately 10–12%, and 87% of samples fail to produce a valid output under instruct-only prompting.

The paper also devotes substantial attention to implicit knowledge and benchmark validity (Tamoyan et al., 23 Jun 2026). In a text-only setting with tools off, repo tree off, and file paths masked, performance is still 57.86% Recall@1 on Verified, indicating strong familiarity for popular repositories. Full SHERLOC with masked issue paths but tools retained reaches 79.96% Recall@1, only 1.3 percentage points below the unmasked system. Implicit recall is reported to concentrate on popular repositories such as scikit-learn at 85.3% and requests at 87.5%, which the authors use to support the contention that benchmark contamination inflates some metrics. The masked-issue control is presented as a bound on, rather than a removal of, this confound.

Failure analysis indicates that most zero-recall@1 cases are near misses rather than navigational collapse (Tamoyan et al., 23 Jun 2026). The reported breakdown is 40% reasoning error after seeing the right file, 27% close miss in the correct directory but wrong file, 25% wrong module entirely, and 4% true multi-file cases with more than three files. This suggests that final candidate selection, not repository traversal, is the dominant remaining bottleneck.

The stated limitations are that the implicit-knowledge confound persists, the benchmark scope is Python repositories in SWE-Bench, and the best configuration uses a large reasoning model at 235B, although matched-scale 30B results are also reported (Tamoyan et al., 23 Jun 2026). Future work includes broader tooling while keeping legibility, improved diagnostics and evidence coverage scoring, uncertainty calibration without judge-based filtering at test time, cross-repository and multilingual generalization, and adaptive test-time scaling policies.

The acronym is also used by unrelated systems in other domains. “SherlockLLM” denotes a dialogue-driven retrieval framework that learns to ask yes/no questions for ambiguous-query retrieval (Yun et al., 21 Oct 2025), while “SHeRLoc” denotes “Synchronized Heterogeneous Radar Place Recognition for Cross-Modal Localization” in robotics (Kim et al., 18 Jun 2025). In the present usage, SHERLOC refers specifically to the code-repair localization framework introduced in 2026 (Tamoyan et al., 23 Jun 2026).