---
title: Fill-In-the-Middle (FIM) Modeling
url: https://www.emergentmind.com/topics/fill-in-the-middle-fim-7b568ef0-b0a7-4698-aa61-7201c531f679
type: topic
---

# Fill-In-the-Middle (FIM) Modeling

Fill-In-the-Middle (FIM) is a neural language modeling paradigm that generalizes classic left-to-right generative objectives by training a model to predict and generate a contiguous span removed from the interior of a sequence, conditioned on both its left (“prefix”) and right (“suffix”) contexts. In code, natural language, and even protein sequence modeling, FIM provides a direct mechanism to solve infilling tasks—such as code completion, document editing, or reasoning step augmentation—where future and past context are simultaneously salient. FIM is now integral to state-of-the-art code LLMs and is implemented at scale for both synthetic and real-world workflows.

## 1. Formal Definition and Training Objective

Let a sequence $X = (x_1, \dots, x_n)$ be partitioned by indices $1\leq a < b \leq n$ into three spans:
- Prefix $P = (x_1, ..., x_a)$
- Middle $M = (x_{a+1}, ..., x_b)$ (the “hole” to infill)
- Suffix $S = (x_{b+1}, ..., x_n)$

The FIM modeling task is to learn the conditional distribution
\[
P_\theta(M \mid P, S)
\]
where $\theta$ are model parameters. The typical data transformation for a decoder-only transformer appends sentinel tokens to demarcate spans, yielding the prompt:
\[
\langle\mathrm{PRE}\rangle\,P\,\langle\mathrm{SUF}\rangle\,S\,\langle\mathrm{MID}\rangle
\]
and seeks to autoregressively generate $M$ token by token. The cross-entropy loss is minimized over $M$, formally:
\[
\max_\theta\,\, \mathbb{E}_{(P, M, S) \sim \mathcal{D}} \,\, \left[\log P_\theta(M \mid P, S)\right]
\]
where $\mathcal{D}$ is the corpus of $(P, M, S)$ triplets. Interleaving FIM-structured and ordinary left-to-right (L2R) training retains both autoregressive sequence modeling and infilling capabilities [2207.14255, 2401.14196].

## 2. Architectures, Prompt Formats, and Operational Regimes

FIM is natively implemented in decoder-only transformer architectures. Prompt engineering is critical for span demarcation and cache management:
- **Tokenization/Delimiters:** FIM utilizes special tokens (e.g., `<PRE>`, `<SUF>`, `<MID>`) to mark prefix, suffix, and the start of the middle span [2401.14196, 2207.14255].
- **Prompt Rearrangement:** The dominant format is Prefix-Suffix-Middle (PSM), but Suffix-Prefix-Middle (SPM) is also used for inference/serving efficiency [2207.14255, 2505.21889]. A 50/50 PSM+SPM mix provides broad compatibility.
- **KV-cache Reuse:** The EFIM prompt rearrangement enables maximal reuse of key-value (KV) cache by placing only user-updated increments after static contexts. Simultaneously, fragment-tokenization retraining resolves subtoken-generation at arbitrary boundaries, improving latency by up to 52% and throughput by 98% without loss of infilling performance [2505.21889].
- **Instruction Augmentation:** The Instruction-Aware FIM (IFIM) framework extends the input with a structured instruction (quadruple $(P, I, S, M)$), resulting in
\[
\langle\mathrm{PRE}\rangle P \langle\mathrm{SUF}\rangle S \langle\mathrm{INS}\rangle I \langle\mathrm{MID}\rangle
\]
and trains the model to incorporate developer intent [2509.24637].

## 3. Specialized FIM Strategies and Domain Adaptations

FIM has evolved with structural and contextual enhancements across multiple tasks:
- **Structure-Aware FIM:** Masking entire Abstract Syntax Tree (AST) subtrees (as opposed to random tokens/chars) aligns masked spans with semantically meaningful code constructs. This structurally coherent masking (AST-FIM) delivers up to +7 Pass@1 gain over random-character FIM on standard code infilling benchmarks, and matches human editing patterns [2506.00204].
- **Curriculum and Code Context:** Incorporating context and hard-to-complete code patterns (curriculum learning) enhances FIM performance, especially for smaller models. Statistics from fine-tuning on curriculum and context-rich datasets report improvements in Pass@1, Prefix Match, and edit similarity on multi-line infilling and CCEval [2412.16589].
- **Instruction-Conditioned FIM:** IFIM achieves double-digit Pass@1 gains (e.g., Deepseek-Coder: 84.6% to 93.6% on IHumanEval) on instruction-guided infilling, with no loss (even improvement) of core FIM capabilities when instructions are absent. Physically separated instruction tokens (not comments) are critical for accurate instruction following [2509.24637].
- **Horizon Planning:** By augmenting the next-token loss with a horizon-length regression objective (HLP), models internalize the “distance-to-suffix” at each infilling step, boosting alignment with infilling boundaries and improving repository-level and file-level pass rates by up to 24% relative, obviating the need for heuristic post-processing [2410.03103].
- **Byte-Level Decoding:** Precise handling of mid-token boundaries in random-span infilling is resolved by exact byte-level marginalization over all tokenizations, yielding absolute pass rate gains of ~18% over token-level decoding [2410.09303].

## 4. Evaluation Protocols and Benchmarks

FIM evaluation metrics center on syntax, semantics, and boundary control:
- **Pass@k:** Fraction of generated fills that pass all reference unit tests (code) [2403.04814].
- **Exact Match (EM):** Token-wise or character-wise exact equality with ground truth [2505.18789].
- **Perplexity:** Exponential average negative log-likelihood over ground-truth $M$[2403.04814, 2506.00204].
- **Specialized benchmarks:**
  - SAFIM: Syntax-aware, execution-based code infilling, including block, control-flow, and API call completion [2403.04814].
  - Real-FIM-Eval: Derived from >30,000 GitHub commits across 12 languages, assessing real-world code editing [2506.00204].
  - HumanEval-infilling and RepoMasterEval: Single/multi-line and real-world repo infilling for code [2509.24637].
  - SEIFER: Secondary structure infilling for protein engineering [2303.16452].
  - Others: CCEval (acceptance and persistence in IDEs), CrossCodeEval (context-aware completion), Multi-line Infilling from SWE-bench [2412.16589, 2601.13384].

## 5. Empirical Findings and Best Practices

A cross-paper synthesis yields these high-level insights:
- **Infilling does not degrade L2R:** FIM pretraining at moderate rates ($\leq$0.5) does not harm left-to-right performance and is “free” in terms of perplexity and sample quality on L2R tasks [2207.14255, 2401.14196, 2506.00204].
- **Boundary Awareness Is Central:** Post-processing of generated output (to remove extraneous lines or ensure alignment with suffix) is necessary for random-span infilling, but superfluous for line-aligned tasks when FIM is trained with explicit span boundaries [2505.18789, 2410.03103].
- **FIM is critical for context-sensitive code completion:** Models lacking FIM objectives underperform even when scaling up, and data quality in pretraining (syntax/alignment, AST-aware masks) outweighs raw parameter count [2403.04814, 2506.00204].
- **AST-based masking and curriculum:** Realistic structure masking converges faster and achieves higher accuracy than random spans; curriculum and context add synergy [2412.16589, 2508.19532].
- **Instruction integration:** IFIM, with explicit special-token instructions, closes the gap between code LLMs and natural developer workflows, far outperforming comment-based or inline instruction schemes [2509.24637].
- **Domain transfer:** FIM supports protein design (recovering mid-chain amino acids; ProtFIM matches or outperforms larger CLM/PLM baselines [2303.16452]), math reasoning step expansion (MathFimer consistently lifts benchmark scores by up to 8 percentage points; [2502.11684]), and general text tasks (FiLM; [2310.09930]).

| Model/Paper                | Domain      | FIM Variant                  | Notable Result(s)              |
|----------------------------|------------|------------------------------|-------------------------------|
| DeepSeek-Coder [2401.14196]| Code        | PSM (50%)                    | SOTA open-source infilling     |
| IFIM [2509.24637]          | Code        | Instruction-aware FIM        | +9 to +12 pp Pass@1           |
| AST-FIM [2506.00204]       | Code        | AST-structure masking        | +4–7 pts pass@1 over Rand-FIM |
| ProtFIM [2303.16452]       | Protein     | [PRE]/[SUF]/[MID] FIM        | Outperforms 2–30x CLMs        |
| FiLM [2310.09930]          | Text        | Any-order masked infilling   | +5–14 ROUGE-PPL gap vs AR     |
| EFIM [2505.21889]          | Code        | KV-cache-optimized FIM       | –52% latency, +98% throughput |
| MathFimer [2502.11684]     | Math Reason | Step-infill in solution chain| Up to +8pp on GSM8K/MATH      |

## 6. Limitations, Extensions, and Future Directions

FIM is robust but has known boundaries and open research directions:
- **Contextual Repair:** Standard FIM cannot correct errors in the conditioning context (prefix/suffix). Methods like SRI (Search-and-Replace Infilling) internalize editing/verification cycles, enabling bug-fixing in the context at FIM-level latency [2601.13384].
- **Syntax Guarantee:** Unconstrained decoders still admit syntax errors. Left/right quotient-based constrained decoding using context-sensitive grammars can boost syntactic correctness from 65%→89.5% in Python FIM, with minor inference overhead [2402.17988].
- **Subtoken and Byte Handling:** Fragment-tokenization and byte-level marginalization remove pitfalls near token boundaries, markedly improving random-span fill [2505.21889, 2410.09303].
- **Post-processing:** Needed only for random/partial-line tasks; high-quality FIM + supervised fine-tuning yields models that learn exact output boundaries [2505.18789].
- **Scaling Laws:** FiLM and AST-FIM show that the infilling–autoreg gap shrinks at scale and with code-structure alignment, suggesting further gains with increased compute or bidirectional generation [2310.09930, 2506.00204].
- **Expanded Curriculum, Context, and Instructions:** Combining structural, context-aware, and instruction-rich examples is essential for high infilling accuracy, persistence, and human alignment [2412.16589, 2509.24637, 2508.19532].

## 7. Impact, Applications, and Best Practices

FIM is now standard in foundation code LLMs, code assistants, and editing tools. Key best-practices distilled from the literature include:
- Use moderate FIM rates ($\sim$50%), character-level span selection, and context-level masking [2207.14255, 2401.14196].
- For enhanced realism and efficiency, mask AST subtrees rather than random tokens [2506.00204].
- For performance-critical applications, apply EFIM for cache reuse and fragment tokenization for subtoken robustness [2505.21889].
- For instruction-guided flows, employ explicit structured instruction tokens (IFIM) rather than in-line comments [2509.24637].
- Use HLP loss for robust boundary planning, especially as post-processing is phased out in evaluation [2410.03103].
- To guarantee syntax, incorporate constrained decoding using left/right grammar quotients [2402.17988].
- For domain transfer (proteins, math, text), adapt prompt and masking formats to respect semantic units—secondary structure, reasoning steps, or paragraphs [2303.16452, 2502.11684, 2310.09930].

Fill-In-the-Middle thus provides a general, extensible, and empirically validated paradigm for sequence infilling across domains, with structural, efficiency, and instruction-following enhancements emerging as the main determinants of state-of-the-art performance [2509.24637, 2410.03103, 2506.00204, 2405.17103, 2412.16589, 2601.13384, 2508.19532, 2403.04814, 2505.21889, 2505.18789, 2410.09303, 2402.17988, 2303.16452, 2310.09930, 2502.11684].

Source: https://www.emergentmind.com/topics/fill-in-the-middle-fim-7b568ef0-b0a7-4698-aa61-7201c531f679