EDIT-Bench: Real-World Code Editing Benchmark

• EDIT-Bench is a real-world code editing benchmark that systematically evaluates LLMs using in-place edits with contextual cues such as highlights and cursor positions.
• It incorporates multilingual instructions and diverse edit types, including bug fixing, feature modifications, and optimizations, to reflect authentic coding scenarios.
• Empirical findings show that explicit context like code highlights boosts performance, while additional cursor data can sometimes introduce confusion in model evaluations.

EDIT-Bench refers to multiple distinct benchmarks in AI evaluation, unified by a focus on systematically assessing model-driven edits under real, user-driven or instruction-based scenarios. The term appears in the context of both code editing—where it is a canonical benchmark for evaluating LLM capabilities in real-world code editing settings (Chi et al., 6 Nov 2025)—as well as image and video editing (and understanding), either as a proper name of a benchmark or as a metonym for instruction-based evaluation paradigms. This article catalogs and analyzes the origin, methodology, and impact of EDIT-Bench in code (the primary and authoritative usage), as well as clarifies its relationship to similarly named or conceptually parallel editing benchmarks across other domains.

1. Definition and Unique Motivation

EDIT-Bench ("Edits of Developer-Instructed Tasks Benchmark") is a real-world code editing benchmark, designed to rigorously measure the ability of LLMs to perform in-place edits of source code in response to developer instructions, mirroring workflows in practical IDE-based usage. It was introduced to address fundamental gaps in prior code editing evaluations, which typically relied on artificially generated or pedagogical prompts, omitted contextual cues like region highlights and cursor positions, and rarely captured the ambiguity and heterogeneity of human-written, multilingual instructions (Chi et al., 6 Nov 2025).

EDIT-Bench instances are constructed to pose "context-dependent" editing challenges: brief and often ambiguous user requests (e.g., "fix this") only become actionable in conjunction with the full file context, the precise code region highlighted, and the cursor location. The dataset includes multilingual instructions (English, Spanish, Russian, Chinese, Portuguese) and source code in Python or JavaScript, accurately reflecting the diversity of edits encountered by real programmers.

2. Dataset Construction and Structure

The EDIT-Bench dataset is sourced from over 2,600 accepted, in-the-wild edit suggestions, contributed by nearly 500 programmers via an unmodified VSCode extension. After stringent filtering—removing duplicates, trivial or style-only edits, non-testable cases—the benchmark core comprises 109 unique problems. Each is translated into five natural languages using GPT-4o, yielding 545 distinct evaluation instances. Programming language distribution is highly skewed toward Python (104/109 problems), with remainder in JavaScript (including React).

For each problem, the data representation includes:

• The full file text, preserving prefix and suffix around the edit (frequently exceeding 5,000 tokens, mirroring large real codebases).
• The exact sequence of code tokens highlighted by the user (median 138 tokens).
• The explicit position of the cursor at the time of edit.
• The raw, unprocessed user-written instruction, which often embeds ambiguity, linguistic variation, and references to error traces or context not otherwise provided.

The edit use-case taxonomy is as follows:

• Feature addition: 43%
• Feature modification: 27%
• Bug fixing (error resolution): 22%
• Code optimization: 8%

This structure compels models to reason about multimodal context, as many tasks are only meaningful when all cues—highlight, cursor, and instruction—are present.

3. Evaluation Methodology

EDIT-Bench deployment centers on task success rate (pass@1), defined as the fraction of problems where a model's single proposed edit causes the code to pass a human-written unit test harness. The formal metric is: $$\text{SuccessRate} = \frac{\#\{\text{problems whose edit passes all tests}\}}{\text{Total \# of problems}} \times 100\%$$ Ablations are performed to quantify the role of context:

• Code-Only: Model receives file and instruction, but no highlight or cursor annotation.
• +Highlight: Model is explicitly shown the highlighted region.
• +Highlight+Cursor: Model additionally receives the precise cursor position. Performance drops when ablated features are omitted are denoted as: $$\Delta_{\text{X}} = \mathrm{SuccessRate}_{\text{full}} - \mathrm{SuccessRate}_{\text{ablated-X}}$$ Problems are partitioned into "Easy" and "Hard" subsets, with "Hard" defined as those unsolved by at least half the evaluated models (solved by <20 of 40).

4. Empirical Findings and Analysis

EDIT-Bench’s empirical evaluation includes 40 LLMs (both open and closed-weight), exposing substantial difficulty in the task. Key results:

Top Model Performance (pass@1, full context):

Model	Success Rate
claude-sonnet-4	66.7%
gpt-o3-mini	63.9%
claude-3.5-sonnet	61.1%
gpt-4o	60.2%
gpt-5	60.1%

Only 5 models surpass 60%, underscoring the challenge. Performance varies by edit category: | Category | Avg. Success Rate | |-------------------|------------------| | Bug fixing | 52.2% | | Optimization | 44.6% | | Feature mod. | 42.8% | | Feature addition | 39.6% |

Context ablation demonstrates up to an 11% gain in success rate from simply providing the highlight; including cursor data exerts inconsistent and sometimes negative impact, apparently due to model confusion regarding attention allocation.

Model	Code Only	+Highlight	+Highlight+Cursor
claude-sonnet-4	60.2%	66.7% (+6.5%)	64.8% (–1.9%)
gpt-o3-mini	56.5%	63.9% (+7.4%)	52.8% (–11.1%)
gemini-2.5-pro	49.5%	55.7% (+6.2%)	55.6% (–0.1%)

EDIT-Bench exhibits low correlation with existing code-editing benchmarks (Pearson $r\approx0.24-0.32$ versus Aider Polyglot, SWE-Bench), suggesting that it uniquely captures real-world, context-dependent edit behaviors.

5. Comparative and Broader Context

Relative to prior code-editing benchmarks (e.g., CanItEdit, EditEval, Aider Polyglot), EDIT-Bench is distinguished by:

• Use of authentic, in-situ developer edit data, not synthetic or educationally constrained snippets.
• Explicit modeling of IDE-intrinsic context: highlights and cursor positions.
• Multilingual annotation reflecting natural international usage.
• Inclusion of ambiguous, under-specified, and terse instructions necessitating context grounding.

Other "EDIT-Bench" or related-named benchmarks in the literature (e.g., EditBench in image inpainting (Wang et al., 2022), text editing as in EditEval (Dwivedi-Yu et al., 2022), or GEdit-Bench for general image editing (Liu et al., 24 Apr 2025)) use similar paradigms but serve distinct domains and modeling tasks. The code-centric EDIT-Bench (Chi et al., 6 Nov 2025) is unique in its scale, realism, and multi-signal context representation for LLM code editing.

6. Implications, Limitations, and Future Directions

EDIT-Bench demonstrates that current state-of-the-art LLMs, even with advanced multimodal and contextual capabilities, exhibit significant gaps in real-world code editing, especially in multilingual and context-heavy scenarios. Precise provision of visual context (e.g., highlighted region) drastically improves performance, implying that IDE-to-model interface design is critical to future progress in coding assistants.

The persistent performance delta between closed-source and open-source models highlights an ongoing need for community-accessible, competitive LLMs dedicated to robust code editing. The benchmark’s reliance on manually built unit tests presents an obstacle to large-scale expansion, motivating research into automated test-harness construction.

Recommended future research includes:

• Enriching model pretraining with real-world IDE-based edit traces, explicitly leveraging highlight and cursor signals.
• Continual expansion of the benchmark to new languages, frameworks, and update-resistant data.
• Development of advanced evaluation protocols—potentially model-agnostic test generation or dynamic contextual augmentation.

By posing routine yet inherently ambiguous edit challenges, EDIT-Bench steers both model training and interface development toward genuinely useful, context-native AI tools for real-world developer workflows (Chi et al., 6 Nov 2025).