---
title: Controllable Code Completion Benchmark
url: https://www.emergentmind.com/topics/controllable-code-completion-benchmark-c3-bench
type: topic
---

# Controllable Code Completion Benchmark

Searching arXiv for the benchmark paper and closely related repository-level code completion benchmarks to ground the article in the primary literature.
Controllable Code Completion Benchmark (C3-Bench) is an instruction-guided code completion benchmark designed to evaluate whether a model can complete code not only correctly but also in the specific manner requested by a natural-language instruction. It formalizes a shift from conventional completion, which evaluates only functional correctness under code context, to **Controllable Code Completion (CCC)**, where the missing middle code must satisfy both executable requirements and instruction adherence. In its initial release, C3-Bench comprises 2,195 Python instances and is presented as the first benchmark explicitly targeting this capability in infilling-style code completion [2601.15879].

## 1. Conceptual definition and task formulation

The benchmark defines a conventional completion instance as a tuple \((P, S, G, T)\), where \(P\) is the prefix code, \(S\) the suffix code, \(G\) the ground-truth middle code, and \(T\) the unit tests. C3-Bench extends this to \((P, S, G, T, I)\), adding an instruction \(I\) that constrains how the completion should be written. The learning objective is stated as
\[
M(P_i, S_i, I_i) \rightarrow G_i
\]
for a model \(M\) over a dataset of controllable completion instances [2601.15879].

This formulation separates two dimensions that conventional completion benchmarks typically conflate. The first is **functional correctness**, evaluated by whether the completed program satisfies tests. The second is **instruction adherence**, evaluated by whether the implementation follows the requested method, structure, or scope. The benchmark is motivated by the observation that many implementations can satisfy the same tests while differing substantially in implementation strategy or completion granularity. This makes correctness alone insufficient when code completion is used in interactive systems such as Copilot Chat or Cursor, where developers often request that the missing code be implemented in a particular way [2601.15879].

A central implication is that C3-Bench evaluates infilling under explicit behavioral constraints rather than generic hole filling. This distinguishes it from repository-level executable benchmarks that improve realism through multi-file context and execution, but do not evaluate whether the model obeys user instructions during completion [2412.11990].

## 2. Benchmark composition and task taxonomy

C3-Bench contains **2,195 high-quality Python CCC instances**. It is divided into two task families: **Implementation-Control Completion (ICC)** with **1,286** instances, and **Scale-Control Completion (SCC)** with **909** instances. ICC evaluates whether a model can implement the missing code according to a specified implementation requirement. SCC evaluates whether a model can generate code of the requested scope or granularity; because SCC targets structural scope rather than full functional behavior, it does not use unit tests [2601.15879].

| Task family | Category | Instances |
|---|---|---:|
| ICC | Structural Specification Requirements | 111 |
| ICC | Algorithmic Implementation Requirements | 502 |
| ICC | Control Flow Requirements | 547 |
| ICC | Critical Parameter Requirements | 126 |
| SCC | Line Span Completion | 97 |
| SCC | Multi-line Completion | 467 |
| SCC | Statement Block Completion | 345 |

The ICC categories cover four implementation requirement types. **Structural Specification Requirements** include data structure definitions, composite type design, class or interface structure, and data model design. **Algorithmic Implementation Requirements** cover specific algorithmic approaches, computational logic, transformation procedures, and optimization strategies. **Control Flow Requirements** concern execution flow, branch logic, loop structure, and exception handling. **Critical Parameter Requirements** concern variable definitions, parameter passing, state variable management, and configuration settings [2601.15879].

The SCC categories cover three scope-control modes. **Line Span Completion** targets partial code-line completion. **Multi-line Completion** requires generation of a specified number of complete lines. **Statement Block Completion** requires a specific control structure block, explicitly including `IF STATEMENT BLOCK`, `FOR STATEMENT BLOCK`, and `WHILE STATEMENT BLOCK` [2601.15879].

Each instance consists of prefix code \(P\), suffix code \(S\), ground-truth middle code \(G\), instruction \(I\), and unit tests \(T\) for ICC only. Token statistics reported in the benchmark indicate that the tasks are not limited to trivial one-line infills. For ICC overall, instruction, prefix, middle, and suffix tokens are reported as **4 / 27 / 10**, **27 / 1447 / 413**, **5 / 709 / 73**, and **1 / 1455 / 57** in min/max/mean form; for SCC overall, the corresponding values are **6 / 11 / 8**, **133 / 2717 / 655**, **2 / 1083 / 66**, and **3 / 1825 / 77** [2601.15879].

## 3. Construction methodology

C3-Bench is constructed from **HumanEval** and **SAFIM**, which are used as executable source datasets for deriving controllable infilling tasks. The construction process begins with **AST-based extraction** using `tree-sitter-languages` to create more substantial middle-code spans than those present in the original datasets. The extraction traverses and manipulates ASTs to mask nodes at multiple levels and extract “logically complete code blocks that maintain semantic coherence.” For **30%** of instances, the pipeline additionally masks **3–5 consecutive lines**, specifically to support SCC tasks [2601.15879].

For ICC, the benchmark then generates **functionally equivalent but implementation-diverse** versions of the extracted middle code. The models used for this stage are **GPT-4o-2024-11-20**, **Claude3.5-Sonnet-20241022**, **DeepSeek-V3**, and **Qwen2.5-Coder-32B-Instruct**. Multiple CCC instances may therefore share the same prefix and suffix while differing in both instruction and target implementation. Generated implementations are retained only if they pass all unit tests, and the paper reports that **over 50% of ICC cases contain three distinct implementations for the same code context** [2601.15879].

Further filtering imposes **PEP8**-consistent readability, a **length constraint** requiring the middle code to be at most **30%** of total context, **significant implementation diversity**, and **algorithmic efficiency**. Instruction creation differs by task family: **ICC instructions are manually crafted implementation specifications**, whereas **SCC instructions are generated by Claude3.5-Sonnet**. Instruction quality is then validated through **expert review by five senior Python developers** and **automated consistency checking with Claude3.5-Sonnet** [2601.15879].

The benchmark examples make the intended form of controllability concrete. One example uses the same SPFA code context with alternative instructions requiring either **Small Label First (SLF)** optimization using a **deque** or **Large Label Last (LLL)** optimization using a **queue** and queue statistics. Another appendix example uses the same labyrinth path-finding context with alternative instructions for iterative DFS with a stack, recursive DFS with parent pointers, or BFS with a queue. An SCC example requires generating **only a single `for` statement block** and nothing else. These examples are intended to show that multiple semantically plausible completions may exist for the same code context, while only some satisfy the requested control [2601.15879].

## 4. Evaluation protocol and metrics

C3-Bench evaluates models under two prompting modes: **FIM special tokens** for models that natively support fill-in-the-middle completion, and a **ChatML-style prompt format** for other, mostly chat-oriented models. For open-source models, inference uses **vLLM**, **greedy sampling**, and an output limit of **1024 tokens**. The study evaluates **40+ models**, including families such as Qwen2.5-Coder, DeepSeek-Coder, DeepSeek-V3, StarCoder2, CodeLlama, Yi-Coder, Codestral, OpenCoder, GPT-4/GPT-4o, Claude 3.5, Gemini, and the o1 series [2601.15879].

Three metrics are used: **Pass@1**, **Instruction-Following Rate (IF)**, and **Edit Similarity (ES)**. **Pass@1** is used for ICC only and measures whether the generated completion passes the associated unit tests. **IF** is the benchmark’s central new metric. For ICC, IF is evaluated **only among functionally correct cases**, so a completion must first pass tests and then be judged for compliance with the instruction. For SCC, IF is measured by fully automatic structural checks: **AST-based node type matching** for statement or block requirements, and **length-based verification** for line-count requirements. **ES** is used as a supplementary static-analysis metric [2601.15879].

For ICC instruction adherence, the benchmark uses **LLM-as-judge semantic validation**. The primary judge is **Claude3.5-Sonnet** and an alternative cheaper judge is **Qwen2.5-32B-Instruct**. The judgment prompt requests a binary assessment based on instruction adherence and alignment with ground-truth implementation aspects such as function definitions, data structures, algorithm steps, and control flow. The paper reports that the judge achieved **98% agreement with senior Python developers across 10 independent assessment rounds** [2601.15879].

Methodologically, this evaluation protocol departs from prior completion benchmarks that typically report only exact match, edit similarity, or unit-test pass rate. The key claim is that controllable completion requires an explicit measure of whether the requested implementation or scope was followed. This is also where C3-Bench differs most sharply from adjacent realism-focused benchmarks such as R²C²-Bench, which varies context perturbation and retrieval conditions but does not benchmark user-specified constraints or instruction-following behavior [2406.01359].

## 5. Empirical findings and the Qwen2.5-Coder-C3 model

The main empirical finding is that strong performance on conventional completion benchmarks does not imply strong performance on controllable completion. The paper reports that several open-source code LLMs remain competitive or superior on traditional benchmarks such as CrossCodeEval, RepoEval, CrossCodeLongEval, and SAFIM, yet perform poorly on C3-Bench, especially on SCC. For example, **Qwen2.5-Coder-32B** achieves **ICC Pass@1 58.1**, **ICC IF 38.7**, but **SCC IF 5.2**; **DeepSeek-Coder-33B-Base** achieves **ICC Pass@1 48.1**, **ICC IF 32.0**, and **SCC IF 5.2** [2601.15879].

The benchmark’s strongest open model is **Qwen2.5-Coder-32B-C\(^3\)**, which is fine-tuned specifically for controllable completion. Its reported scores are **ICC ES 49.3**, **ICC Pass@1 62.0**, **ICC IF 52.5**, **SCC ES 44.2**, **SCC IF 80.7**, and **Average IF 66.6**. Among the reported proprietary systems, the highest average IF values are lower: **o1-2024-12-17** reaches **61.3**, **Claude3.5-Sonnet-20241022** reaches **55.8**, and **o1-preview** reaches **53.0** [2601.15879].

To obtain this model, the authors construct a synthetic supervised fine-tuning pipeline on Python GitHub code from **The Stack v2 / StarCoder2 data source**. Phase 1 uses **Claude3.5-Sonnet** to generate **1,000 high-quality instruction-completion pairs** as seed examples. Phase 2 uses **Qwen2.5-Coder-32B-Instruct** with those seeds as few-shot demonstrations to produce a large synthetic corpus. The final training set contains **200,000 synthetic instruction-completion pairs**. Training uses **64 NVIDIA A100-80GB GPUs**, **Adam**, learning rate **\(3 \times 10^{-5}\)**, **50 warmup steps**, **global batch size 1024**, **tensor parallel size 2**, and **4K token** truncation, with **10-gram decontamination** against C3-Bench [2601.15879].

The gains from this specialization are large. For **Qwen2.5-Coder-32B-Instruct** versus **Qwen2.5-Coder-32B-C\(^3\)**, ICC Pass@1 improves from **49.8** to **62.0**, ICC IF from **28.8** to **52.5**, SCC IF from **16.9** to **80.7**, and Average IF from **22.8** to **66.6**. For **Qwen2.5-Coder-1.5B-Instruct** versus **Qwen2.5-Coder-1.5B-C\(^3\)**, ICC Pass@1 improves from **22.9** to **39.7**, ICC IF from **0.7** to **29.6**, SCC IF from **8.0** to **66.8**, and Average IF from **4.3** to **48.2** [2601.15879].

The paper also reports measurable trade-offs on conventional benchmarks. For **Qwen2.5-Coder-32B-C\(^3\)** relative to the base **Qwen2.5-Coder-32B**, average EM/ES on **CrossCodeEval** changes from **57.1 / 86.8** to **55.9 / 86.2**; on **RepoEval**, average EM changes from **51.6** to **51.8** while ES drops from **78.5** to **77.0**; on **CrossCodeLongEval**, average EM/ES drops from **36.9 / 66.4** to **29.1 / 60.5**; and on **SAFIM**, average pass drops from **71.2** to **67.5**. This suggests that training for controllable completion is not equivalent to optimizing general completion performance, and may redistribute capability across evaluation regimes [2601.15879].

## 6. Position within the benchmark landscape and stated limitations

C3-Bench occupies a distinct position relative to neighboring completion benchmarks. Conventional completion benchmarks such as **HumanEval**, **CrossCodeEval**, **RepoEval**, and **SAFIM** primarily evaluate whether code is functionally correct given context. Repository-level benchmarks such as **ExecRepoBench** strengthen realism by using multi-file context, AST-aligned masking granularity, and repository-level executable checking, but they do not evaluate whether the model can follow explicit user instructions during completion [2412.11990]. Developer-centric benchmarks such as **Codev-Bench** move toward realistic cursor-centric and boundary-sensitive completion behavior, yet still operationalize intent largely through scenario structure rather than explicit instruction-conditioned control [2410.01353]. C3-Bench’s distinctive contribution is therefore not repository realism alone, but the addition of explicit instruction adherence as a first-class evaluation target [2601.15879].

The paper states several limitations. First, C3-Bench is currently **Python-only**. Second, it is presently **in-file** rather than repository-level. Third, broader context scenarios remain future work, including multi-language and repository-level controllable completion. The paper also notes that ICC performance remains constrained by the base model’s coding ability, so instruction tuning cannot fully compensate for weak functional competence [2601.15879].

Other limitations are partly explicit and partly methodological. ICC instruction adherence depends on an **LLM judge**, even though the reported agreement with senior Python developers is high. SCC is evaluated through **structural checks** rather than semantic tests, so it measures granularity compliance rather than executable correctness. Benchmark construction also relies partly on **synthesized implementations and instructions**, though these are filtered by unit tests, expert review, and automated consistency checks [2601.15879].

In benchmark-design terms, C3-Bench proposes an evaluation philosophy rather than only a dataset. The benchmark argues that future code-completion evaluation should measure at least **functional correctness**, **instruction adherence**, and **structural or granularity control when appropriate**. A plausible implication is that controllable completion benchmarking will eventually need to combine C3-Bench’s explicit instruction layer with the repository-level execution realism developed in adjacent work, yielding benchmarks that test whether a model can complete code correctly, in context, and under specified developer controls [2601.15879].

Source: https://www.emergentmind.com/topics/controllable-code-completion-benchmark-c3-bench