--- title: 'CanItEdit: Instruction-Based Code Editing' url: https://www.emergentmind.com/topics/canitedit type: topic --- # CanItEdit: Instruction-Based Code Editing CanItEdit is a benchmark for **instruction-based code editing** in which a model is given existing code \(c\) and a natural-language instruction \(I\), and must produce edited code \(c'\), formalized as \(M(c, I) \rightarrow c'\). It was introduced to evaluate a capability that differs from de novo code synthesis, bug-fix-only repair, and fill-in-the-middle infilling: the model must understand what should remain unchanged, infer where to edit, and follow either detailed or under-specified instructions while preserving program behavior outside the requested change. The benchmark consists of **105 hand-written Python code-editing tasks**, each with **before code**, **after code**, **two instruction styles**, and a **hidden test suite** [2312.12450]. Subsequent work has treated CanItEdit as a standard evaluation target for code-editing models, data-generation pipelines, decoding methods, and benchmark audits; an empirical audit later identified it as one of only **two peer-reviewed benchmarks** directly targeting instructed code editing with **human-authored natural-language instructions** and **test-based evaluation** [2604.05100]. ## 1. Definition and task model CanItEdit targets a setting in which the model edits an existing program rather than synthesizing a new one from a natural-language specification. The benchmark paper distinguishes this setting from natural-language-to-code synthesis benchmarks such as HumanEval and MBPP, from narrow bug-fixing benchmarks such as HumanEvalFix, and from fill-in-the-middle tasks in which edit locations are specified in advance. In CanItEdit, the model must infer where to modify the code, coordinate edits across possibly multiple locations, and avoid irrelevant modifications [2312.12450]. The benchmark also separates two instruction regimes. **Descriptive instructions** are detailed and explicit, whereas **lazy instructions** are shorter and under-specified, intended to resemble real user requests. This distinction is central to the benchmark’s construct: descriptive prompts probe execution under precise guidance, while lazy prompts probe whether the model can infer implied intent from minimal instructions. The original paper motivates this focus by noting that, in the LMsys in-the-wild chat dataset, among **4,188 conversations containing code**, **831 (19%)** involve code-editing requests [2312.12450]. A later audit situates that motivation in a broader deployment context, stating that instructed code editing accounts for roughly **19%** of real-world coding-assistant conversations. That audit treats CanItEdit as measuring a real but restricted construct: primarily **single-file**, **Python**, **human-instructed**, **test-verified** editing, rather than general software-engineering competence across languages and domains [2604.05100]. ## 2. Benchmark construction and composition CanItEdit is a **handcrafted** benchmark rather than a repository-mined corpus. Its construction involved a team of **eight experienced Python programmers** with a designated lead. Contributors first produced a brief problem description and intended changes for review, then wrote the before code, hidden tests, after code, and both instruction variants; the lead reviewed all in-progress problems, and the full team reviewed tasks in weekly meetings [2312.12450]. Each benchmark item contains a **before** code segment, an **after** code segment, a **descriptive** instruction, a **lazy** instruction, and a **hidden test suite**. Because every task has two instruction variants, the benchmark contains **105 tasks** and effectively **210 problems**. The tasks are evenly balanced across software-maintenance categories: **35 corrective**, **35 perfective**, and **35 adaptive** [2312.12450]. The benchmark is **Python-only**. Its topic distribution is reported as **39** tasks in data structures and algorithms, **21** in language processing, **25** in mathematics, **10** in data science, and **10** miscellaneous tasks. Problems involving external libraries number **22** in the main text; the appendix notes **21 problems import external libraries**, including **NumPy: 13**, **Pandas: 6**, **SciPy: 3**, **scikit-learn: 3**, **PyTorch: 3**, **Z3: 2**, **autograd: 2**, **Flask: 1**, and **vLLM: 1** [2312.12450]. The benchmark is materially larger than toy single-function editing problems. Reported code statistics are: **mean lines, before: 42.5 ± 33.9**; **mean lines, after: 49.8 ± 36.6**; **combined mean lines: 92.3 ± 69.9**; **combined mean tokens: 865.3 ± 639.7**; **combined max tokens: 3,583**; and **mean Levenshtein distance between before and after: 302.1 ± 339.6**. Instruction lengths also differ substantially: **descriptive: 81.7 ± 50.4 tokens** and **lazy: 35.6 ± 30.6 tokens** [2312.12450]. ## 3. Evaluation protocol and metrics CanItEdit uses **hidden tests** as the primary evaluation oracle. The benchmark paper reports test suites built from **unit tests**, **property-based testing**, **mocking**, **fuzzing**, and **integration tests**. An automated verification pipeline enforces **100% line coverage**, requires that the suite **passes** on the after code, and requires that it **fails at least one test** on the before code. This fail-before/pass-after validation is a defining property of the benchmark’s oracle design [2312.12450]. The principal metric is **pass@k**, with the main results reported at **pass@1**. A completion is successful if the generated after-code passes the hidden test suite, so evaluation is functional rather than exact-match based. The benchmark also introduces **ExcessCode**, defined as the fraction of changed lines not covered by the test suite, to quantify unnecessary edits in otherwise correct solutions [2312.12450]. Main evaluation settings are **max new tokens: 2048**, **temperature: 0.2**, **top-\(p\): 0.95**, and **20 samples per problem**, with all tests executed in a **Docker container**. The paper also reports a higher-sampling regime with **temperature: 0.8**, **top-\(p\): 0.9**, and **100 samples** to examine search effects and verifier-assisted performance [2312.12450]. A later audit emphasizes that these testing choices are unusually strong by benchmark standards. It reports **median 13 tests** per CanItEdit problem and **median 100% whole-file coverage**, and argues that CanItEdit compensates for modest test counts with near-complete coverage and fail-before/pass-after validation, making its correctness oracles materially stronger than those of EDIT-Bench [2604.05100]. ## 4. Empirical findings in the original study The original CanItEdit study reports a substantial performance gap between closed and open models. On **descriptive** instructions, **GPT-4** achieves **63.33** pass@1 and **GPT-3.5-Turbo** **48.14**; among untuned open models, **DeepSeekCoder-Instruct 33b** achieves **49.78** and **CodeLlama-Instruct 70b** **45.05**. On **lazy** instructions, **GPT-4** scores **51.95**, **GPT-3.5-Turbo** **42.71**, and **DeepSeekCoder-Instruct 33b** **38.94**. The paper characterizes descriptive prompts as materially easier, reporting an **8.68 absolute** average increase in pass@1 for descriptive relative to lazy prompts [2312.12450]. The paper also shows that code-editing ability scales with model size and that exposure to commit-style data matters. Within DeepSeekCoder-Base, performance rises from **17.90 / 11.76** at **1.3b**, to **32.62 / 27.76** at **6.7b**, to **47.71 / 34.71** at **33b** on descriptive/lazy prompts. It further notes that models trained on commit-like data, such as StarCoder, can outperform similarly sized models that are stronger at code synthesis but weaker at editing [2312.12450]. A major contribution of the original study is a fine-tuning recipe based on new training datasets. **EditPackFT** is derived from the Python split of CommitPackFT and filtered from **56,025** to **22,602** examples; **Commits2023FT** is filtered to **24,129** Python file changes; and the combined deduplicated dataset has **46,274 items** and **74M tokens**. Fine-tuned DeepSeekCoder-Base models, denoted **E**, improve substantially: **E-33b** reaches **55.90** descriptive and **42.33** lazy pass@1, **E-6.7b** reaches **48.33 / 39.29**, and **E-1.3b** reaches **26.67 / 21.43** [2312.12450]. These gains materially narrow the open/closed gap. **E-33b** exceeds **GPT-3.5-Turbo** on descriptive prompts (**55.90** vs **48.14**) and nearly matches it on lazy prompts (**42.33** vs **42.71**), although **GPT-4** remains ahead at **63.33 / 51.95**. Under high sampling, **E-33b** reaches **81.90** pass@100 on descriptive prompts, slightly above **GPT-4** at **80.00**. The paper also reports that **corrective** edits are easiest and **perfective** edits hardest, with **E-33b** especially strong on corrective and adaptive changes [2312.12450]. ## 5. Role in subsequent code-editing research Later work treats CanItEdit as a standard benchmark rather than a method. The OpenCodeEdit paper states this explicitly, defining **CanItEdit** as an existing benchmark for instruction-guided code editing and distinguishing it from **OpenCodeEdit**, the paper’s own synthetic data-generation pipeline. Using **OCEDataFT**, a **20,000-sample** filtered fine-tuning dataset balanced as **10,000 descriptive + 10,000 lazy**, that work fine-tunes three open models and reports **OpenCodeEdit-Qwen3-8B** at **54.10** overall pass@1 on CanItEdit, leaving a **3.54** point gap to **GPT-4** at **57.64** [2509.25203]. CanItEdit has also been used to support architectural arguments about edit-oriented modeling. **Stable-DiffCoder** includes CanItEdit as a direct editing benchmark and reports **Stable-DiffCoder-8B-Instruct** at **60.0** pass@1, above **Seed-Coder-8B-Instruct** at **50.5**, **Qwen2.5-Coder-7B-Instruct** at **49.5**, and **Qwen2.5-Coder-14B-Instruct** at **52.9**. That paper interprets the result as evidence that diffusion-based any-order modeling improves structured code modeling for editing and reasoning [2601.15892]. Systems work has used CanItEdit to study inference efficiency rather than raw model quality. **Reuse or Generate? Accelerating Code Editing via Edit-Oriented Speculative Decoding** evaluates **FastEditor** on CanItEdit and reports up to **10.38×** speedup over autoregressive decoding while keeping pass@1 close to greedy AR; for example, with DeepSeek-Coder on lazy instructions, **AR** runs at **11.8 token/s** with pass@1 **49.5**, whereas **FastEditor** runs at **122.5 token/s** with pass@1 **48.5** [2506.02780]. A later format-oriented study, **“To Diff or Not to Diff?”**, uses CanItEdit to compare full-code generation with structure-aware diffs. For **Qwen2.5-Coder-7B**, **FullCode** reaches **53.17** pass@1 on CanItEdit, while **FuncDiff + AdaEdit** reaches **52.67**; on the CanItEdit subset with input code length **> 300 tokens**, **FuncDiff + AdaEdit** attains **40.69** pass@1 with **481.63** output tokens, compared with **39.75** and **648.30** for **FullCode** [2604.27296]. CanItEdit has also served as a testbed for prompting and decomposition strategies. **Applying the Chinese Wall Reverse Engineering Technique to Large Language Model Code Editing** uses CanItEdit to evaluate a two-stage pipeline in which **Gemini 2.5 Pro** writes inline `EDIT:` comments and a weaker model executes them. On descriptive prompts, **Comma v0.1 1T** improves from **9.14** to **21.24** pass@1, **starcoder2:instruct** from **35.10** to **42.05**, and **phi4** from **54.71** to **68.67**, with corresponding reductions in **ExcessCode** for all three models [2507.15599]. ## 6. Scope, validity, and critical assessment An empirical audit of instructed code-editing benchmarks argues that CanItEdit measures a narrower construct than deployment decisions require, but is comparatively strong as an evaluation oracle. Across a survey of **over 150** code-related benchmarks, only **CanItEdit** and **EDIT-Bench** were found to target instructed code editing with **human-authored instructions** and **test-based evaluation** under the audit’s filtering criteria [2604.05100]. The audit identifies several representational limitations. CanItEdit is **exclusively Python**; **85.7%** of its problems are concentrated in **feat: 47.6%** and **fix: 38.1%**; it has only **two test-related problems (1.9%)**; and it has **zero representation** for `docs`, `chore`, `build`, and `ci`. In application-domain terms, **68.6%** of CanItEdit is classified as **Algorithm Design and Problem Solving**, while **backend** and **frontend development**, which together account for **46%** of real-world editing activity in the audit’s comparison sources, are absent [2604.05100]. At the same time, the audit treats CanItEdit as comparatively reliable. It reports **median 13** tests per problem, **median 100% whole-file coverage**, and fail-before/pass-after validation. It also finds low internal duplication: **6%** of CanItEdit problems share a codebase with at least one other problem, corresponding to **103 distinct contexts out of 105**. On this reading, high performance on CanItEdit is meaningful evidence of competence at **Python instructed editing under strong test oracles**, but not sufficient evidence that a model is broadly ready for real-world coding-assistant deployment across languages, domains, and edit intents [2604.05100]. A plausible implication is that CanItEdit’s enduring value lies less in breadth than in the precision of its construct. It has functioned as a benchmark for instruction following, as a target for dataset curation, as a workload for inference-acceleration systems, and as an oracle-rich substrate for auditing evaluation methodology. That combination has made it a central reference point in the study of code-editing LLMs, even as later work has emphasized the need for broader language coverage, richer application domains, and more deployment-aligned edit distributions [2312.12450]. Source: https://www.emergentmind.com/topics/canitedit