CanItEdit: Instruction-Based Code Editing

• CanItEdit is a benchmark for instruction-based code editing that evaluates a model’s ability to modify existing Python code based on natural language instructions.
• It comprises 105 handcrafted tasks featuring both descriptive and lazy instructions, emphasizing user intent inference and precise test-based validation.
• The benchmark supports research on code-editing models, data-generation pipelines, and inference strategies by providing detailed metrics like pass@1 and ExcessCode.

CanItEdit is a benchmark for instruction-based code editing in which a model is given existing code $c$ and a natural-language instruction $I$, and must produce edited code $c'$, formalized as $M(c, I) \rightarrow c'$. It was introduced to evaluate a capability that differs from de novo code synthesis, bug-fix-only repair, and fill-in-the-middle infilling: the model must understand what should remain unchanged, infer where to edit, and follow either detailed or under-specified instructions while preserving program behavior outside the requested change. The benchmark consists of 105 hand-written Python code-editing tasks, each with before code, after code, two instruction styles, and a hidden test suite (Cassano et al., 2023). Subsequent work has treated CanItEdit as a standard evaluation target for code-editing models, data-generation pipelines, decoding methods, and benchmark audits; an empirical audit later identified it as one of only two peer-reviewed benchmarks directly targeting instructed code editing with human-authored natural-language instructions and test-based evaluation (Ebrahimi et al., 6 Apr 2026).

1. Definition and task model

CanItEdit targets a setting in which the model edits an existing program rather than synthesizing a new one from a natural-language specification. The benchmark paper distinguishes this setting from natural-language-to-code synthesis benchmarks such as HumanEval and MBPP, from narrow bug-fixing benchmarks such as HumanEvalFix, and from fill-in-the-middle tasks in which edit locations are specified in advance. In CanItEdit, the model must infer where to modify the code, coordinate edits across possibly multiple locations, and avoid irrelevant modifications (Cassano et al., 2023).

The benchmark also separates two instruction regimes. Descriptive instructions are detailed and explicit, whereas lazy instructions are shorter and under-specified, intended to resemble real user requests. This distinction is central to the benchmark’s construct: descriptive prompts probe execution under precise guidance, while lazy prompts probe whether the model can infer implied intent from minimal instructions. The original paper motivates this focus by noting that, in the LMsys in-the-wild chat dataset, among 4,188 conversations containing code, 831 (19%) involve code-editing requests (Cassano et al., 2023).

A later audit situates that motivation in a broader deployment context, stating that instructed code editing accounts for roughly 19% of real-world coding-assistant conversations. That audit treats CanItEdit as measuring a real but restricted construct: primarily single-file, Python, human-instructed, test-verified editing, rather than general software-engineering competence across languages and domains (Ebrahimi et al., 6 Apr 2026).

2. Benchmark construction and composition

CanItEdit is a handcrafted benchmark rather than a repository-mined corpus. Its construction involved a team of eight experienced Python programmers with a designated lead. Contributors first produced a brief problem description and intended changes for review, then wrote the before code, hidden tests, after code, and both instruction variants; the lead reviewed all in-progress problems, and the full team reviewed tasks in weekly meetings (Cassano et al., 2023).

Each benchmark item contains a before code segment, an after code segment, a descriptive instruction, a lazy instruction, and a hidden test suite. Because every task has two instruction variants, the benchmark contains 105 tasks and effectively 210 problems. The tasks are evenly balanced across software-maintenance categories: 35 corrective, 35 perfective, and 35 adaptive (Cassano et al., 2023).

The benchmark is Python-only. Its topic distribution is reported as 39 tasks in data structures and algorithms, 21 in language processing, 25 in mathematics, 10 in data science, and 10 miscellaneous tasks. Problems involving external libraries number 22 in the main text; the appendix notes 21 problems import external libraries, including NumPy: 13, Pandas: 6, SciPy: 3, scikit-learn: 3, PyTorch: 3, Z3: 2, autograd: 2, Flask: 1, and vLLM: 1 (Cassano et al., 2023).

The benchmark is materially larger than toy single-function editing problems. Reported code statistics are: mean lines, before: 42.5 ± 33.9; mean lines, after: 49.8 ± 36.6; combined mean lines: 92.3 ± 69.9; combined mean tokens: 865.3 ± 639.7; combined max tokens: 3,583; and mean Levenshtein distance between before and after: 302.1 ± 339.6. Instruction lengths also differ substantially: descriptive: 81.7 ± 50.4 tokens and lazy: 35.6 ± 30.6 tokens (Cassano et al., 2023).

3. Evaluation protocol and metrics

CanItEdit uses hidden tests as the primary evaluation oracle. The benchmark paper reports test suites built from unit tests, property-based testing, mocking, fuzzing, and integration tests. An automated verification pipeline enforces 100% line coverage, requires that the suite passes on the after code, and requires that it fails at least one test on the before code. This fail-before/pass-after validation is a defining property of the benchmark’s oracle design (Cassano et al., 2023).

The principal metric is pass@k, with the main results reported at pass@1. A completion is successful if the generated after-code passes the hidden test suite, so evaluation is functional rather than exact-match based. The benchmark also introduces ExcessCode, defined as the fraction of changed lines not covered by the test suite, to quantify unnecessary edits in otherwise correct solutions (Cassano et al., 2023).

Main evaluation settings are max new tokens: 2048, temperature: 0.2, top-$p$: 0.95, and 20 samples per problem, with all tests executed in a Docker container. The paper also reports a higher-sampling regime with temperature: 0.8, top-$p$: 0.9, and 100 samples to examine search effects and verifier-assisted performance (Cassano et al., 2023).

A later audit emphasizes that these testing choices are unusually strong by benchmark standards. It reports median 13 tests per CanItEdit problem and median 100% whole-file coverage, and argues that CanItEdit compensates for modest test counts with near-complete coverage and fail-before/pass-after validation, making its correctness oracles materially stronger than those of EDIT-Bench (Ebrahimi et al., 6 Apr 2026).

4. Empirical findings in the original study

The original CanItEdit study reports a substantial performance gap between closed and open models. On descriptive instructions, GPT-4 achieves 63.33 pass@1 and GPT-3.5-Turbo 48.14; among untuned open models, DeepSeekCoder-Instruct 33b achieves 49.78 and CodeLlama-Instruct 70b 45.05. On lazy instructions, GPT-4 scores 51.95, GPT-3.5-Turbo 42.71, and DeepSeekCoder-Instruct 33b 38.94. The paper characterizes descriptive prompts as materially easier, reporting an 8.68 absolute average increase in pass@1 for descriptive relative to lazy prompts (Cassano et al., 2023).

The paper also shows that code-editing ability scales with model size and that exposure to commit-style data matters. Within DeepSeekCoder-Base, performance rises from 17.90 / 11.76 at 1.3b, to 32.62 / 27.76 at 6.7b, to 47.71 / 34.71 at 33b on descriptive/lazy prompts. It further notes that models trained on commit-like data, such as StarCoder, can outperform similarly sized models that are stronger at code synthesis but weaker at editing (Cassano et al., 2023).

A major contribution of the original study is a fine-tuning recipe based on new training datasets. EditPackFT is derived from the Python split of CommitPackFT and filtered from 56,025 to 22,602 examples; Commits2023FT is filtered to 24,129 Python file changes; and the combined deduplicated dataset has 46,274 items and 74M tokens. Fine-tuned DeepSeekCoder-Base models, denoted E, improve substantially: E-33b reaches 55.90 descriptive and 42.33 lazy pass@1, E-6.7b reaches 48.33 / 39.29, and E-1.3b reaches 26.67 / 21.43 (Cassano et al., 2023).

These gains materially narrow the open/closed gap. E-33b exceeds GPT-3.5-Turbo on descriptive prompts (55.90 vs 48.14) and nearly matches it on lazy prompts (42.33 vs 42.71), although GPT-4 remains ahead at 63.33 / 51.95. Under high sampling, E-33b reaches 81.90 pass@100 on descriptive prompts, slightly above GPT-4 at 80.00. The paper also reports that corrective edits are easiest and perfective edits hardest, with E-33b especially strong on corrective and adaptive changes (Cassano et al., 2023).

5. Role in subsequent code-editing research

Later work treats CanItEdit as a standard benchmark rather than a method. The OpenCodeEdit paper states this explicitly, defining CanItEdit as an existing benchmark for instruction-guided code editing and distinguishing it from OpenCodeEdit, the paper’s own synthetic data-generation pipeline. Using OCEDataFT, a 20,000-sample filtered fine-tuning dataset balanced as 10,000 descriptive + 10,000 lazy, that work fine-tunes three open models and reports OpenCodeEdit-Qwen3-8B at 54.10 overall pass@1 on CanItEdit, leaving a 3.54 point gap to GPT-4 at 57.64 (Zhang et al., 19 Sep 2025).

CanItEdit has also been used to support architectural arguments about edit-oriented modeling. Stable-DiffCoder includes CanItEdit as a direct editing benchmark and reports Stable-DiffCoder-8B-Instruct at 60.0 pass@1, above Seed-Coder-8B-Instruct at 50.5, Qwen2.5-Coder-7B-Instruct at 49.5, and Qwen2.5-Coder-14B-Instruct at 52.9. That paper interprets the result as evidence that diffusion-based any-order modeling improves structured code modeling for editing and reasoning (Fan et al., 22 Jan 2026).

Systems work has used CanItEdit to study inference efficiency rather than raw model quality. Reuse or Generate? Accelerating Code Editing via Edit-Oriented Speculative Decoding evaluates FastEditor on CanItEdit and reports up to 10.38× speedup over autoregressive decoding while keeping pass@1 close to greedy AR; for example, with DeepSeek-Coder on lazy instructions, AR runs at 11.8 token/s with pass@1 49.5, whereas FastEditor runs at 122.5 token/s with pass@1 48.5 (Wang et al., 3 Jun 2025). A later format-oriented study, “To Diff or Not to Diff?”, uses CanItEdit to compare full-code generation with structure-aware diffs. For Qwen2.5-Coder-7B, FullCode reaches 53.17 pass@1 on CanItEdit, while FuncDiff + AdaEdit reaches 52.67; on the CanItEdit subset with input code length > 300 tokens, FuncDiff + AdaEdit attains 40.69 pass@1 with 481.63 output tokens, compared with 39.75 and 648.30 for FullCode (Cheng et al., 30 Apr 2026).

CanItEdit has also served as a testbed for prompting and decomposition strategies. Applying the Chinese Wall Reverse Engineering Technique to LLM Code Editing uses CanItEdit to evaluate a two-stage pipeline in which Gemini 2.5 Pro writes inline [EDIT](https://www.emergentmind.com/topics/layer-aligned-encoder-decoder-edit): comments and a weaker model executes them. On descriptive prompts, Comma v0.1 1T improves from 9.14 to 21.24 pass@1, starcoder2:instruct from 35.10 to 42.05, and phi4 from 54.71 to 68.67, with corresponding reductions in ExcessCode for all three models (Hanmongkolchai, 21 Jul 2025).

6. Scope, validity, and critical assessment

An empirical audit of instructed code-editing benchmarks argues that CanItEdit measures a narrower construct than deployment decisions require, but is comparatively strong as an evaluation oracle. Across a survey of over 150 code-related benchmarks, only CanItEdit and EDIT-Bench were found to target instructed code editing with human-authored instructions and test-based evaluation under the audit’s filtering criteria (Ebrahimi et al., 6 Apr 2026).

The audit identifies several representational limitations. CanItEdit is exclusively Python; 85.7% of its problems are concentrated in feat: 47.6% and fix: 38.1%; it has only two test-related problems (1.9%); and it has zero representation for docs, chore, build, and ci. In application-domain terms, 68.6% of CanItEdit is classified as Algorithm Design and Problem Solving, while backend and frontend development, which together account for 46% of real-world editing activity in the audit’s comparison sources, are absent (Ebrahimi et al., 6 Apr 2026).

At the same time, the audit treats CanItEdit as comparatively reliable. It reports median 13 tests per problem, median 100% whole-file coverage, and fail-before/pass-after validation. It also finds low internal duplication: 6% of CanItEdit problems share a codebase with at least one other problem, corresponding to 103 distinct contexts out of 105. On this reading, high performance on CanItEdit is meaningful evidence of competence at Python instructed editing under strong test oracles, but not sufficient evidence that a model is broadly ready for real-world coding-assistant deployment across languages, domains, and edit intents (Ebrahimi et al., 6 Apr 2026).

A plausible implication is that CanItEdit’s enduring value lies less in breadth than in the precision of its construct. It has functioned as a benchmark for instruction following, as a target for dataset curation, as a workload for inference-acceleration systems, and as an oracle-rich substrate for auditing evaluation methodology. That combination has made it a central reference point in the study of code-editing LLMs, even as later work has emphasized the need for broader language coverage, richer application domains, and more deployment-aligned edit distributions (Cassano et al., 2023).