Precise Debugging Benchmark (PDB)

• The paper demonstrates that PDB distinguishes precise, localized edits from broad regenerations by enforcing edit-level and bug-level metrics.
• PDB transforms coding datasets into debugging benchmarks with verified buggy programs, ground-truth edit scripts, and precision-aware evaluations.
• The framework emphasizes fault localization and targeted repairs, underscoring the need for diff-aware decoding and edit-aware training in automated debugging.

Searching arXiv for the benchmark paper and a few related benchmarks mentioned in the provided data. Precise Debugging Benchmark (PDB) is a framework for evaluating whether an automated debugging system performs localized, minimal repair or instead regenerates broad portions of a program while still passing unit tests (Zhu et al., 19 Apr 2026). Unlike code completion, debugging requires a model to localize faults in an existing program and apply targeted edits that preserve the program’s broader structure and intent. PDB was introduced to make that distinction measurable: it automatically converts a coding dataset with ground-truth solutions and tests into a debugging benchmark with verified buggy programs, edit-level ground truth, and precision-aware metrics. The framework releases two evaluation benchmarks, PDB-Single-Hard and PDB-Multi, and reports that frontier models can attain unit-test pass rates above $76\%$ while still exhibiting edit-level precision below $45\%$, even under explicit minimal-debugging instructions (Zhu et al., 19 Apr 2026).

1. Motivation and problem definition

PDB is motivated by the claim that debugging is not the same problem as code completion. Code completion asks a model to synthesize a working program from a prompt; debugging requires fault localization and targeted correction within an existing artifact. The benchmark paper argues that frontier LLMs often default to regeneration during debugging: they rewrite large portions, or even the entirety, of a program to pass unit tests. In real-world software maintenance, such broad rewrites are costly to review, risk new regressions, and obscure intent (Zhu et al., 19 Apr 2026).

A central criticism of pass-rate–only evaluation is that it conflates three distinct behaviors: a minimal, localized fix; a whole-program rewrite; and hard-coded outputs that merely satisfy the tests. It also suppresses partial progress. Under a binary pass/fail regime, fixing one of three bugs is scored identically to fixing none if the final program still fails. PDB therefore reframes evaluation from final program correctness alone to edit granularity and bug granularity. This suggests a shift from asking whether a model can produce a passing program to asking whether it can repair the injected faults and only those faults (Zhu et al., 19 Apr 2026).

The framework’s scope is automated debugging systems that receive a buggy program $C_b$ and problem description $x$, optionally with tests and execution feedback, and return a revision $\hat{S} = M(C_b, x)$. A binary test function $F_{\mathcal{U}}(C) \to \{0,1\}$, defined over a test suite $\mathcal{U}$, determines whether the revised program passes all tests (Zhu et al., 19 Apr 2026).

2. Formalization and precision-aware metrics

PDB defines minimal corrections as the smallest set of line edits that makes a buggy program pass the test suite:

$$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$

A line edit on line $l$ is denoted $e_l$, and a bug is atomic when a minimal correction consists of edits on a contiguous sequence of lines. For single-line bugs, atomicity is trivial because $45\%$0. For composed multi-bug programs, PDB further uses an independence definition: two bugs are independent if fixing one neither introduces nor removes edits required to fix the other (Zhu et al., 19 Apr 2026).

For evaluation, the ground-truth edit set is decomposed as

$45\%$1

where each $45\%$2 is contiguous and non-overlapping. A mapping function $45\%$3 pairs each ground-truth edit block with the closest predicted edits in $45\%$4. For each bug $45\%$5, PDB constructs a pseudo-revision $45\%$6 by combining all other ground-truth fixes with the model’s mapped edits for bug $45\%$7; if $45\%$8, the candidate is treated as semantically correct for that bug (Zhu et al., 19 Apr 2026).

The two central metrics are edit-level precision and bug-level recall:

$45\%$9

The intuition is explicit in the benchmark design. Precision weights each semantically correct bug by its ground-truth minimal edit size $C_b$0, normalized by the model’s total edits $C_b$1. Recall measures the fraction of injected bugs that are actually resolved. PDB then introduces an $C_b$2-relaxed essential-edit variant to avoid rewarding wholesale regeneration while not enforcing exact minimality. It allows up to $C_b$3 edited lines per bug and trims the predicted edits to an essential subset via unit tests:

$C_b$4

$C_b$5

In the reported experiments, $C_b$6 for single-line bugs and $C_b$7 for multi-line blocks. The paper denotes $C_b$8 as edit-level precision $C_b$9 and recall as bug-level recall $x$0. Unit-test pass rate, reported as Pass@1, is retained but is not part of the composite edit-aware metrics (Zhu et al., 19 Apr 2026).

A worked contrast in the benchmark paper illustrates the difference. For a single off-by-one bug in binary search, a one-line exact fix yields $x$1, $x$2, and Pass@1$x$3. A regeneration-heavy solution that rewrites ten lines and still passes tests can have $x$4 but $x$5. Both pass the tests, but only the former is counted as a precise debug (Zhu et al., 19 Apr 2026).

3. Benchmark construction pipeline

PDB starts from a coding benchmark that provides a task description $x$6, a ground-truth program $x$7, and tests $x$8. It first retains only verified programs that pass all unit tests. From those seeds, it synthesizes atomic bugs and then composes them into multi-bug programs (Zhu et al., 19 Apr 2026).

Single-line atomic bug synthesis uses one of three line-level operations: insertion, deletion, and substitution. The pipeline applies a rule-based filter to avoid unsafe deletions, such as indentation errors, and non-editable lines, such as function headers. It then randomly selects one operation type, one defect category from an ODC-aligned taxonomy, and one compatible editable line. An LLM from a generator pool consisting of GPT-5.1-Codex, Claude-4.5-Sonnet, and Gemini-2.5-Pro is prompted to modify exactly one line. Only programs that fail unit tests are retained, which enforces bug validity and reproducibility (Zhu et al., 19 Apr 2026).

For multi-line atomic bugs, the framework selects a contiguous block size $x$9, one primary and two auxiliary defect categories, and a valid contiguous block of lines. Because atomicity is not guaranteed by multi-line editing, PDB enforces it by enumerating all partial fixes that revert strict subsets of the modified lines back to the ground truth and keeping the bug only if all such partial fixes still fail unit tests. The full block is therefore treated as an atomic unit of correction (Zhu et al., 19 Apr 2026).

Bug composition produces more difficult multi-bug programs. For each task and target bug count $\hat{S} = M(C_b, x)$0, PDB samples $\hat{S} = M(C_b, x)$1 distinct block edits from the generated bug pool and imposes a stride constraint $\hat{S} = M(C_b, x)$2, defined as a minimum line distance between blocks, to encourage independence and avoid degenerate interactions. For each $\hat{S} = M(C_b, x)$3, it repeats the composition $\hat{S} = M(C_b, x)$4 times and records only composed programs that satisfy independence constraints. It then subsamples at most $\hat{S} = M(C_b, x)$5 buggy programs per bug count per task to prevent over-representation of easy or prolific tasks (Zhu et al., 19 Apr 2026).

The injected bugs are organized with an ODC-style taxonomy spanning Assignment, Checking, Algorithm, Build/Package/Merge, and Timing/Serialization. Subcategories listed in the benchmark description include Off-by-One Error, Negation Error, Missing/Incomplete Checks, Wrong Math Expression, Modifying While Iterating, Invalid API Call, and Serialization Issue. These categories guide bug injection prompts and are intended to ensure diversity. Manual analysis reports that interaction violations are rare, at approximately $\hat{S} = M(C_b, x)$6 (Zhu et al., 19 Apr 2026).

4. Released benchmarks and benchmark parameters

PDB releases two benchmarks that share the same precision-aware evaluation logic but differ in bug granularity and composition constraints (Zhu et al., 19 Apr 2026).

Benchmark	Key configuration	Dataset details
PDB-Single-Hard	Single-line bugs; up to $\hat{S} = M(C_b, x)$7; $\hat{S} = M(C_b, x)$8; $\hat{S} = M(C_b, x)$9, $F_{\mathcal{U}}(C) \to \{0,1\}$0, $F_{\mathcal{U}}(C) \to \{0,1\}$1; stride $F_{\mathcal{U}}(C) \to \{0,1\}$2	Initial size 7,591; final size 5,751 after filtering “easy” instances; sources: BigCodeBench and LiveCodeBench; Python
PDB-Multi	Contiguous multi-line bug blocks; up to $F_{\mathcal{U}}(C) \to \{0,1\}$3; $F_{\mathcal{U}}(C) \to \{0,1\}$4; $F_{\mathcal{U}}(C) \to \{0,1\}$5; same $F_{\mathcal{U}}(C) \to \{0,1\}$6, $F_{\mathcal{U}}(C) \to \{0,1\}$7, $F_{\mathcal{U}}(C) \to \{0,1\}$8; stride $F_{\mathcal{U}}(C) \to \{0,1\}$9	256 examples; programs exceeding 35 lines; Python

PDB-Single-Hard is designed as the main challenging benchmark for single-line bugs. Its final version is obtained after model-based filtering of “easy” instances defined as cases with perfect precision, recall, and unit score for at least 7 of 9 evaluated models under $\mathcal{U}$0. PDB-Multi is substantially smaller and restricted to programs exceeding 35 lines so that block independence can be maintained more reliably (Zhu et al., 19 Apr 2026).

The evaluation protocol supports both single-pass LLMs and agentic systems. Systems receive the task description and buggy program, optionally the tests and execution feedback, and produce a patch $\mathcal{U}$1 and revised program $\mathcal{U}$2. PDB aligns predicted edits with ground-truth blocks via $\mathcal{U}$3, constructs bug-specific pseudo-revisions, evaluates semantic correctness via $\mathcal{U}$4, computes $\mathcal{U}$5 and $\mathcal{U}$6, and reports Pass@1. The metrics are micro-averaged across bug counts (Zhu et al., 19 Apr 2026).

5. Empirical findings, reranking, and failure modes

On PDB-Single-Hard with $\mathcal{U}$7, edit-level precision and bug-level recall rerank frontier models relative to unit-test pass rates. Claude-4.5-Sonnet reports Precision $\mathcal{U}$8, Recall $\mathcal{U}$9, Unit $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$0; Gemini-2.5-Pro reports Precision $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$1, Recall $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$2, Unit $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$3; Qwen3-Coder-480B reports Precision $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$4, Recall $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$5, Unit $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$6. By contrast, DeepSeek-V3.2-Thinking reaches Unit $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$7 with Precision $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$8, and GPT-5.1-Codex reaches Unit $$\mathcal{E}_{C_b} = \arg\min_{E} |E| \ \text{s.t.} \ F_{\mathcal{U}}(\operatorname{apply}(E, C_b)) = 1.$$9 with Precision $l$0. The paper interprets this as regeneration-heavy behavior: high unit scores can coexist with low edit-level precision, whereas Qwen3-Coder-480B exhibits lower unit score but more targeted editing. Precision also generally shows a negative correlation with unit-test score as bug count increases (Zhu et al., 19 Apr 2026).

The same pattern persists on PDB-Multi with $l$1. Claude-4.5-Sonnet reports Precision $l$2, Recall $l$3, Unit $l$4; Gemini-2.5-Pro reports Precision $l$5, Recall $l$6, Unit $l$7; GPT-5.1-Codex reports Precision $l$8, Recall $l$9, Unit $e_l$0. Multi-line contexts are harder overall, but the misalignment between unit-test ranking and edit precision remains: GPT-5.1-Codex has the highest unit score among the three reported models yet less than half the precision of Claude-4.5-Sonnet (Zhu et al., 19 Apr 2026).

The paper also evaluates iterative and agentic debugging on 500 BigCodeBench-derived instances. Iterative debugging allows up to three revision attempts per problem with access to previous failed outputs; agentic debugging allows up to three attempts with unit tests and execution feedback each step. Both settings improve unit-test pass rates and $e_l$1 relative to single-shot, but edit-level precision does not improve and sometimes degrades. Even a stronger agentic system, exemplified by Claude-Code in the figure, achieves only approximately $e_l$2 precision with feedback, which the paper interprets as evidence that feedback is often treated as a coarse success signal that triggers broader rewrites rather than localized edits (Zhu et al., 19 Apr 2026).

Error analysis is central to the benchmark’s interpretation. Among passing cases, $e_l$3 have recall $e_l$4; when precision is imperfect, the reported causes are $e_l$5 redundant guard checks, $e_l$6 modifications to correct code blocks, $e_l$7 correct but non-minimal edits, $e_l$8 complete rewrites, and $e_l$9 fixes to bugs missing in ground-truth references. Among failing cases with partially correct edits, $45\%$00 are under-repair $45\%$01, $45\%$02 are imprecise repair $45\%$03, and $45\%$04 are regressive repair $45\%$05, meaning new bugs cause failures. Freeform debugging prompts, without a minimal-edit constraint, substantially reduce precision and recall across all models; Gemini-2.5-Pro shows an approximately $45\%$06 absolute drop in precision, and GPT-5.1-Codex fails to reach $45\%$07 precision (Zhu et al., 19 Apr 2026).

6. Position within debugging evaluation, implications, and limitations

PDB is positioned against prior debugging and program-repair benchmarks such as Defects4J, DebugBench, MDEval, LiveCodeBench, and CodeEditorBench, which are described as relying primarily on unit-test pass/fail or coarse metrics. The stated point of departure is not bug realism alone, but verified edit structure: PDB synthesizes atomic bugs with edit scripts, composes multi-bug programs under independence constraints, and evaluates patches with $45\%$08 and $45\%$09. The reported effect is that model rankings can invert relative to unit tests, exposing regeneration and over-editing that would otherwise remain hidden (Zhu et al., 19 Apr 2026).

The implications for training and inference are explicit. The benchmark paper recommends incorporating edit-aware training signals, optimizing for fault localization and minimizing effective edit sets, and using diff-level supervision rather than only whole-program correctness. It also recommends diff-aware and AST-guided decoding to constrain proposals to targeted hunks, and integrating essential-edit discovery at inference to prune superfluous edits while preserving semantic correctness. A plausible implication is that PDB is less a replacement for Pass@1 than a correction to it: unit tests remain necessary, but they are insufficient as a sole measure of debugging capability (Zhu et al., 19 Apr 2026).

The framework also has stated limitations. The independence assumption can be violated in broader software systems, although observed violations are approximately $45\%$10 with the reported stride constraints and filtering. The current pipeline targets Python; adapting it to other languages is said to require modifications to in-context prompts and language-specific subcategories rather than to the core design. Edit-aware metrics can also miss semantically equivalent fixes, and the benchmark identifies robust semantic edit evaluation as an open problem. About $45\%$11 of recall $45\%$12 cases arise because a single bug admits multiple minimal correct fixes, which the paper’s $45\%$13 and $45\%$14 procedures mitigate but do not eliminate (Zhu et al., 19 Apr 2026).

A terminological ambiguity is worth noting. The acronym “PDB” also appears in a distinct benchmark specification for compiler-generated debugging information, where the primary score is program-level variable value coverage $45\%$15 over source lines rather than edit-aware code-repair quality. That separate usage is grounded in source-line coverage metrics for DWARF debugging information and addresses optimized binaries, not LLM debugging behavior (Stinnett et al., 2024).