---
title: Fill-in-the-Middle Code Completion
url: https://www.emergentmind.com/topics/fill-in-the-middle-code-completion
type: topic
---

# Fill-in-the-Middle Code Completion

Fill-in-the-middle (FIM) code completion is a paradigm in code intelligence where a model is required to generate a plausible, contextually coherent code fragment given both a left (prefix) and right (suffix) context. FIM explicitly models infilling, rather than only left-to-right (L2R) prediction, and is now foundational in code large language model (LLM) pretraining, inference, evaluation, and alignment. The dominance of FIM in modern code completion systems is supported by both empirical performance advantages and increased alignment with real-world developer workflows.

## 1. Definition, Motivation, and Task Formulations

FIM completion is formally defined over a code sequence $x = (x_0, \ldots, x_L)$, partitioned into a prefix, a contiguous middle span, and a suffix:

- $\mathrm{prefix} = (x_0, ..., x_{i-1})$
- $\mathrm{middle} = (x_i, ..., x_j)$
- $\mathrm{suffix} = (x_{j+1}, ..., x_L)$

The completion task is to generate $\mathrm{middle}$ given both $\mathrm{prefix}$ and $\mathrm{suffix}$, yielding code $\mathrm{prefix} \| \mathrm{middle} \| \mathrm{suffix}$. This setting supports editing, insertion, and completion “in the middle” of code, which aligns with how developers interact with IDEs—editing partial functions, refactoring, or filling in code blocks [2403.04814].

FIM input formatting typically employs special sentinel or boundary tokens (e.g., $\langle\text{PRE}\rangle$, $\langle\text{SUF}\rangle$, $\langle\text{MID}\rangle$) and may use multiple orderings:
- **Prefix-Suffix-Middle (PSM)**: $[\mathrm{prefix}; \mathrm{suffix}; \mathrm{middle}]$
- **Suffix-Prefix-Middle (SPM)**: $[\mathrm{suffix}; \mathrm{prefix}; \mathrm{middle}]$

The model is trained with an autoregressive causal objective to maximize $p(\mathrm{middle} \mid \mathrm{prefix}, \mathrm{suffix})$ [2410.13187].


## 2. Syntax-Aware and Structure-Guided Span Selection

A central limitation of standard (random-span) FIM is that it often yields incoherent or unrepresentative training examples, as the masked span may cut across arbitrary token or character boundaries. Syntax-aware Fill-in-the-Middle addresses this by selecting the middle span via abstract syntax tree (AST) analysis to ensure that the infill region is a complete, well-formed syntactic construct (e.g., function body, loop, conditional expression).

**Methods:**
- **AST-FIM**: Masking complete AST nodes or aligned multi-node spans—a mixture of node- and range-based selection ensures that all infill regions are strictly localizable subtrees [2506.00204].
- **SFIM (Structured FIM in aiXcoder-7B)**: Select internal, non-leaf AST nodes (never root, never leaf), and further align span boundaries to line endings, ensuring the infill region corresponds to a complete code snippet [2410.13187].
- **SAFIM Benchmark**: All evaluation spans are derived from AST-based, semantically critical code elements–algorithmic block, control-flow condition, API call–rather than random masking [2403.04814].

This structure-aware span selection is empirically shown to drive improvements in syntax validity and exact-match metrics, particularly on benchmarks targeting semantic code blocks [2410.13187, 2506.00204].


## 3. Training Objectives and Multi-Objective Pretraining

FIM is implemented as a multi-mode, multi-objective pretraining strategy:
- **Next-Token Prediction (NTP):** Standard L2R causal objective $L_{\mathrm{NTP}} = -\sum_{t=1}^{L} \log p(x_t \mid x_{<t})$.
- **FIM Random-Span Objective:** Prefix, random middle, and suffix; mask chosen uniformly at random.
- **Structured FIM (SFIM):** Syntax-guided selection of the middle span with both PSM and SPM input orderings; dominant sampling weight for code.
- **AST-FIM Pretraining:** Mixture of single-node and aligned-span AST masking, combined with L2R data [2506.00204].

Sampling probabilities are typically skewed to favor syntax-aware FIM for code (e.g., 70% SFIM in aiXcoder-7B), ensuring models see mostly structurally valid infilling tasks [2410.13187].

**Recent Enhancements:**
- **Horizon-Length Prediction (HLP):** Attach an auxiliary head to regress the fraction of the remaining middle at each generation step, providing planning and stop-point awareness. This helps eliminate reliance on dataset-specific truncation rules and improves suffix boundary alignment [2410.03103].
- **Curriculum and Context FIM:** Oversample “hard” AST node types where acceptance is low, and inject relevant in-file or cross-file context via static analysis or retrieval, boosting acceptance rates of small and medium models [2412.16589].


## 4. Inference Mechanisms and Post-Processing

At inference time, several classes of decoding and post-processing are employed:
- **Greedy or temperature-driven decoding** of the target span, conditioned on observed context.
- **Self-infilling with non-monotonic decoding:** Introduce interruption (switch to suffix when confidence drops) and looping (alternate suffix/middle-left2right predictions) to enhance regularity and boundary control [2311.17972].
- **Syntax-Aware Constrained Decoding:** Incrementally parse the generated snippet (with Earley’s algorithm extended for right/left quotienting), rejecting token extensions that cannot produce a legal combined AST with the prefix and suffix [2402.17988].
- **Overlap-removal and syntax-aware truncation:** Especially for random-span or instruction-tuned outputs, strip maximal overlapping prefix and/or suffix fragments from the raw output to remove over-generation [2505.18789, 2403.04814].
- **Instruction-Aware Infilling (IFIM):** Incorporate explicit developer intent (instructions/comments) in the infilling prompt, training models to condition explicitly on a natural language instruction together with prefix and suffix [2509.24637].
- **Edit-Oriented FIM (SRI):** Perform code search-and-replace tasks grounded in a single-pass inference, overcoming the optimal-context assumption and enabling edit-oriented infilling robust to buggy context [2601.13384].


## 5. Empirical Performance and Benchmarking

Comprehensive benchmarking frameworks such as SAFIM and Real-FIM-Eval provide large-scale, syntax-aware, multilingual, and decontaminated testbeds for FIM evaluation [2403.04814, 2506.00204]. Key points:

| Model / Method          | FIM Task Type         | Syntax-Aware Infills | Avg. Pass@1 / EM |
|------------------------|----------------------|----------------------|------------------|
| aiXcoder-7B            | Random & SFIM        | Yes                  | 79.3% (SantaCoder FIM) |
| AST-FIM (8B, 1–2T)     | AST, Block, Control  | Yes                  | +5–8 pts over Rand-FIM (SAFIM, Real-FIM-Eval) |
| SynthCoder-8B          | AST/Heuristic, DPO   | Yes                  | 49.8% aiXcoder EM, +17.8% over base |
| StructureCoder         | Granular AST/DPO     | Yes                  | +1–2% pass@1 gains        |
| HLP-FIM                | Planning FIM         | No (aux head)        | +5–24% rel. on FIM tasks  |
| IFIM                   | Instruction-aware    | Yes (when provided)  | 84.6% $\to$ 93.6% on HumanEval-infilling |
| SRI                    | Edit+FIM             | Yes (SRI process)    | +26–46 pts on CrossCodeEval, SAFIM vs. FIM |

Performance gains from FIM pretraining are robust to parameter scaling and paradigm; models with structure-aware masking, horizon planning, DPO alignment, or instruction/context augmentation consistently outperform random-FIM or L2R-only baselines, often by 5–25% relative depending on the task [2506.00204, 2410.03103, 2508.15495, 2412.16589, 2508.19532, 2509.24637].


## 6. Operational Trade-Offs, Limitations, and Routing

FIM-based completion presents several practical trade-offs:
- **Latency versus quality:** Large-base models yield better accuracy but are impractical for real-time IDE completion; curriculum/context and MoE architectures allow small models to approach larger-model performance at low latency [2412.16589].
- **Boundary placement:** Standard FIM training lacks explicit horizon awareness, requiring rule-based truncation; HLP and syntax-aware post-processing alleviate this but add complexity [2410.03103, 2403.04814].
- **Syntax errors:** Unconstrained generation yields 25–30% more syntax-invalid infills; constrained decoding recovers nearly all of this gap at modest runtime cost [2402.17988].
- **Model routing:** Syntax-aware routing (SynConfRoute) combines token-level confidence and online syntax validation, escalating only syntax-invalid or low-confidence completions to larger models. This reduces accelerator usage by 58% while increasing pass@1 by 6–31% over confidence-only routing, and never rejects a correct local completion [2605.04894].
- **Alignment and security:** Classic FIM “base” models are vulnerable to prompt injection; edit-oriented variants (SRI) preserve instruction-following priors and maintain high accuracy and safety post-alignment, with minimal inference overhead [2601.13384].
- **Generalization and transfer:** Gains from structure-aware and instruction-augmented FIM persist across languages and repository-level settings, but require adequate AST support and high-quality training data [2506.00204, 2403.04814, 2509.24637].


## 7. Key Insights and Recommendations

- Syntax-aware and AST-based masking strategies should replace random-span FIM training for both pretraining and benchmark construction, as they yield more coherent infill instances and align with actual developer edits [2410.13187, 2506.00204].
- Curriculum learning targeting difficult AST node types and context retrieval for cross-file completion significantly benefit low-capacity models, reducing the gap to much larger models with minimal latency impacts [2412.16589].
- Horizon prediction or planned-length auxiliary losses provide explicit lookahead, resolving the open task of boundary alignment in open-domain FIM; these additions have negligible training and inference cost [2410.03103].
- Instruction-aware FIM manages the trade-off between developer intent-following and pure code infilling, enabling models to leverage comments/instructions without degrading core FIM capabilities [2509.24637].
- Direct Preference Optimization with granular, structure-aligned infill regions, and adversarial negative sampling (especially suppression of prefix/suffix repetition) ensures better alignment and reduces pathological completion behaviors [2508.15495, 2508.19532].
- Syntax-aware routing of completions in distributed pipelines enables cost-efficient, local-first deployments without accuracy trade-offs or risk of silent rejection of correct outputs [2605.04894].

These characteristics position FIM as the definitive paradigm for context-aware code completion, and continued advances in syntax-awareness, planning, and alignment will determine the trajectory of next-generation code LLM capabilities.

Source: https://www.emergentmind.com/topics/fill-in-the-middle-code-completion