--- title: Fill-in-the-Middle Tokens url: https://www.emergentmind.com/topics/fill-in-the-middle-fim-tokens type: topic --- # Fill-in-the-Middle Tokens Fill-in-the-Middle (FIM) tokens constitute a paradigm and tokenization strategy that enables autoregressive language models—particularly large language models (LLMs) for code—to generate an arbitrary “middle” span of text given both preceding (prefix) and subsequent (suffix) context. FIM tokenization introduces dedicated control tokens that mark the segments of the input and output, decoupling standard left-to-right text continuation from bidirectional infilling. FIM has become the de facto standard for modern code completion LLMs due to its simplicity, architectural compatibility, and empirical gains in infilling accuracy and developer workflow alignment [2509.24637], [2410.03103], [2412.16589], [2403.04814], [2207.14255]. ## 1. Formal Definition and Tokenization Schemes At its core, FIM takes a contiguous sequence of tokens \( x = (x_1, ..., x_n) \), randomly selects a middle subspan \( (x_{i}, ..., x_{j-1}) \), and partitions the sequence into: - Prefix \( P = (x_1, ..., x_{i-1}) \) - Middle \( M = (x_{i}, ..., x_{j-1}) \) - Suffix \( S = (x_{j}, ..., x_n) \) Special sentinel tokens such as `
`, ``, and `` (notation varies: e.g., `<|fim_prefix|>`, etc.) are concatenated to organize the prompt:
- Prefix–Suffix–Middle (PSM): `
  P    S  `
- Suffix–Prefix–Middle (SPM): `  S  
  P  `

The model is then autoregressively trained to generate the middle segment \( M \) given the concatenated (prefix, suffix) context. The canonical FIM objective is:
\[
\max_\theta \mathbb{E}_{(P, M, S) \sim D}\; [\; \log P_\theta(M \mid P, S) \;]
\]
where \( D \) denotes the pretraining dataset [2509.24637], [2207.14255], [2401.14196].

Variants such as FIM-SE and AST-FIM introduce further constraints (e.g., line-level, AST-node masking) or additional tokens for character- or structure-aware infilling [2405.17103], [2506.00204].

## 2. Training Paradigms, Objective Functions, and Practical Encoding

FIM pretraining is primarily implemented as a data transformation rather than an architectural change. Models interleave FIM-formatted examples with standard next-token autoregressive (L2R) samples—typically mixing at rates \( p \in [0.5, 0.9] \) [2207.14255], [2401.14196]. Each example retains the identical cross-entropy loss structure, with the target restricted to the middle span after sentinel tokens are injected:
\[
\mathcal{L}_{\text{FIM}} = - \sum_{t=1}^{|M|} \log P_\theta(m_t | P, S, m_{` token marking a natural language developer comment to be leveraged alongside context [2509.24637].

Empirical findings consistently show that FIM pretraining does not degrade standard L2R performance up to high FIM rates (even 90%), and that syntax-aware or curriculum-augmented span selection further boosts infilling proficiency, especially for challenging code constructs [2412.16589], [2506.00204].

## 3. Evaluation Benchmarks and Syntax-Awareness

FIM-based models are evaluated on code infilling tasks that reflect practical use-cases: single- or multi-line insertion, random-span masking, control flow completion, or API call recovery. Notable public benchmarks include:
- HumanEval-infilling: Python code tasks with masked spans, measured by pass@1 (reference solution match/unit test success) [2509.24637].
- SAFIM: Syntax-Aware FIM, segmenting completions into algorithmic block, control-flow, and API call splits with 17,720 multi-language examples [2403.04814].
- Real-FIM-Eval: Derived from real-world git commits, measuring perplexity on real insertion and edit tasks across 12 languages [2506.00204].

Syntax-aware post-processing methods, such as AST-guided truncation or incremental parsing with context-sensitive left/right quotients, meaningfully reduce compile errors and ensure syntactic validity of completions. Earley-style incremental parsing with quotient grammars supports syntactic-constrained decoding, boosting valid completion rates from ~65% to nearly 90% on challenging corpora [2402.17988]. AST-based FIM (AST-FIM) masking aligns infill spans with code block or function constructs, rather than arbitrary token ranges, yielding up to +7.4 pp improvements on SAFIM benchmarks [2506.00204].

## 4. Practical Model Engineering, Inference, and Efficiency

FIM tokenization is compatible with standard decoder-only transformers. No changes to attention masks or transformer block structure are necessary—sentinel tokens are simply embedded alongside the regular vocabulary [2207.14255], [2401.14196]. For long-context infilling (e.g., 16k token windows in DeepSeek-Coder), rotary position embeddings are extended, attention optimizations (e.g., FlashAttention v2), and group-query attention applied [2401.14196].

KV-cache reuse in interactive serving is an efficiency bottleneck due to prompt structure: changes to prefix/suffix typically invalidate each other's cached keys/values. The EFIM prompt transformation freezes the shared prefix and suffix portions, appends new tokens at the end, and incorporates fragment tokenization at training time to ensure accurate subtoken generation. This yields a 52% average reduction in latency and doubles throughput in multi-user settings without infilling degradation [2505.21889].

At the byte level, tokenization bias (i.e., zero-probability holes induced by prompts ending mid-subword/unit) degrades SPM-style FIM infilling under standard token-level autoregressive LMs. Recent methods recover exact byte-level sampling via the Byte-Token Representation Lemma, marginalizing over all possible token extensions of a prefix, and eliminating bias without model retraining. This correction yields an 18% pass@1 gain on open-coded FIM benchmarks in SPM mode [2410.09303].

## 5. Extensions: Instruction Awareness, Reasoning, and Robust Infilling

Base FIM models perform suboptimally when the developer's intent is underspecified by code context alone. Pure FIM-pretrained LLMs tend to ignore embedded natural-language comments, while unstructured instruction tuning destroys FIM alignment. The Instruction-aware FIM (IFIM) paradigm introduces a fourth instruction segment with a dedicated delimiter, synthesizing training data via LLM-generated comments. IFIM fine-tuning (without further model modification) delivers dramatic pass@1 gains on instruction-aware infilling tasks, e.g., +9.0pp on HumanEval-infilling for Deepseek-Coder (84.6%→93.6%) [2509.24637], and critically, does not diminish vanilla FIM code completion in instruction-absent settings.

FIM tokenization has also been extended outside code, e.g., for mathematical reasoning. MathFimer applies FIM to chain-of-thought solutions—randomly hiding intermediate reasoning steps and directing models to interpolate them between visible context. This methodology improves reasoning accuracy across datasets (e.g., +7.43pp on GSM8K, +4.16pp on MATH) and is composable: multi-round expansion of solution chains yields compounding improvements [2502.11684].

Character-level infilling with sub-token boundary management (FIM-SE) or correct factorization over mask sets in MARIA (Masked and Autoregressive Infilling Architecture) further expand FIM's utility. FIM-SE constrains the boundary alignment to avoid broken tokens at insert points, realizing up to +11.5pp gains on single-line infilling without architectural cost [2405.17103]. MARIA fuses AR and MLM hidden states, with a learned linear head, outperforming discrete diffusion baselines across all mask rates [2502.06901].

## 6. Quantitative Effects, Ablation Findings, and Best Practices

Empirical studies converged on several best practices for FIM tokenization:
- Training with a 50–90% FIM example rate yields no L2R degradation, enables robust context modeling, and boosts infilling up to 0.9 pass@100 on single-line code infill [2207.14255], [2401.14196].
- Mixing PSM and SPM formats gives maximal flexibility; PSM guards against tokenization bias, whereas SPM provides seamless insertions when byte-level sampling is properly handled [2410.09303].
- Syntax- or curriculum-aware masking via ASTs or hard span mining closes the gap between pretraining and real code editing, improving performance most for small-parameter models [2506.00204], [2412.16589].
- Explicit instruction tokens must be delimited by a standalone marker; inline-comment instructions degrade performance [2509.24637].
- For character-level infilling, always eliminate mid-token boundary prediction by aligning prompts to line/word boundaries [2405.17103].
- Tiny architectural changes (e.g., adding regression heads for horizon prediction or concatenating MLM-AR embeddings) bring significant boundary-awareness and mask-invariant infilling [2410.03103], [2502.06901].

## 7. Impact, Open Problems, and Research Frontiers

FIM tokenization underpins the rapid progress in code-generating LLMs, supporting real-world IDE workflows, automated patching, multi-line and reasoning-heavy tasks. It enables bidirectional context utilization with minimal model overhead. Its influence extends into curriculum design for preference optimization (DPO) [2508.19532], grammar-constrained decoding [2402.17988], and efficiency-optimized serving infrastructure [2505.21889].

Open challenges include scaling FIM to complex semantic edits, integrating deeper type/data/control-flow signals, developing adaptive span selection for diverse codebases, and parameter-efficient adaptation to new programming languages or mathematical domains. The increasing sophistication of FIM—through grammar, tokenization, curriculum, and auxiliary loss refinements—continues to drive gains well beyond the basic data transformation, making FIM tokens a foundational concept in the contemporary LLM research toolkit.

Source: https://www.emergentmind.com/topics/fill-in-the-middle-fim-tokens