--- title: Fill-in-the-Middle (FIM) Capabilities Overview url: https://www.emergentmind.com/topics/fill-in-the-middle-fim-capabilities type: topic --- # Fill-in-the-Middle (FIM) Capabilities Overview Fill-in-the-Middle (FIM) refers to the capability of language models, particularly in code and structured sequence generation domains, to generate or infill arbitrary contiguous spans given both prefix (left context) and suffix (right context). Unlike traditional left-to-right or causal autoregressive models, FIM-trained models can condition on both preceding and succeeding context, allowing for powerful applications in code completion, editing, and structured content infilling across several domains. ## 1. Formalization and Core Objectives Let a sequence $x = (x_1, x_2, \ldots, x_T)$ be partitioned into prefix $P = x_{j}$. The FIM task is defined as inferring $M$ conditioned on $P$ and $S$, with the training objective: \[ \mathcal{L}_{\mathrm{FIM}}(\theta) = -\sum_{(P, S, M)\sim \mathcal{D}} \log p_\theta(M \mid P, S) \] Special tokens (e.g., $\langle$PRE$\rangle$, $\langle$SUF$\rangle$, $\langle$MID$\rangle$) are inserted to demarcate these regions. The primary variant, Prefix–Suffix–Middle (PSM), presents the model with prefix and suffix as context and supervises generation of the middle [2207.14255][2509.24637][2401.14196][2506.00204]. FIM examples are mixed with left-to-right (L2R) autoregressive examples during pretraining, preserving both infilling and generative capabilities [2207.14255]. ## 2. Training Strategies and Masking Schemes ### Random vs Structure-Aware FIM Traditionally, FIM “splits” are randomly sampled at the character or token level, yielding robust but sometimes unstructured or semantically broken spans [2207.14255]. More recent work proposes AST-FIM, where middle spans are aligned with abstract syntax tree (AST) nodes, preserving complete syntactic units (function, block, or expression subtrees). Masking approaches include: - **Random Span (Rand-FIM):** Uniformly samples start/end positions without regard to syntax. - **Single-Node Masking:** Samples entire AST subtrees, proportional to span length. - **Aligned-Span Masking:** Random interval is expanded to the minimal set of adjacent AST children covering it [2506.00204]. AST-FIM yields training examples aligned with real edit operations, improving FIM pass rates by up to 5 points on syntax-aware code infilling benchmarks [2506.00204]. ### Curriculum and Context Augmentation Curriculum learning strategies identify and upsample “hard” infill cases, such as symbol-rich or structurally complex spans (measured via the number of unique identifiers or low historical acceptance rate). Context augmentation further prepends semantically relevant code definitions, often extracted via static analysis or code search, to assist infilling [2412.16589]. ## 3. Prompting, Decoding, and Post-Processing Prompt templates encode the FIM task as sequence infilling marked by sentinel tokens. Common formats include: - **PSM (Prefix–Suffix–Middle):** $[\langle\text{PRE}\rangle\;P\;\langle\text{SUF}\rangle\;S\;\langle\text{MID}\rangle]$ - **SPM (Suffix–Prefix–Middle):** Alternative ordering for left-to-right execution [2207.14255][2403.04814]. Syntax-aware post-processing is critical for accurately extracting the infilled segment at inference time. Techniques include AST-based truncation, line-based cutoffs, and handling cases where models overshoot/undershoot segment boundaries [2403.04814][2505.18789]. Supervised FIM-specific fine-tuning substantially reduces the necessity of post-processing for line-aligned spans but not for arbitrary token spans [2505.18789]. ## 4. Architectural, Algorithmic, and Decoding Advances ### Constrained Decoding with Grammar Quotienting To guarantee syntactic validity, efficient constrained decoding is achieved by quotienting context-free (and sensitive) grammars: - **Left/Right Quotients:** Given $L$ and right context $R$, form $L / R = \{ \ell: \ell \cdot r \in L,\ r \in R \}$. - **Incremental Parsing:** An Earley-style recognizer is extended to efficiently check if the middle span, combined with prefix and suffix, admits a valid parse [2402.17988]. - **Context-sensitive quotienting:** Handles indentation (Python), leftmost-longest lexing, and parenthesis-checking, allowing interactive FIM decoding with $O(1)$ amortized complexity per token for regular grammars. Constrained FIM decoding reduced syntax errors from 65.0% (unconstrained) to 89.5% pass rate on Python code with constraints, dramatically improving over unconstrained models [2402.17988]. ### Inferential Planning: Horizon-Length Prediction Horizon-Length Prediction (HLP) introduces a lookahead planning head that estimates the number of tokens left to infill. The auxiliary regression loss over the normalized remaining-length \[ \mathcal{L}_{\mathrm{HLP}} = \sum_{j=1}^{M} |\hat y_j - y_j| \] imbues the model with explicit infilling horizon-awareness, yielding up to 24% relative gains on real-world evaluation without rule-based output truncation [2410.03103]. ### Tokenization and Byte-Level Decoding Standard tokenized models exhibit sharp degradation (“tokenization bias”) when FIM prefixes end in mid-token, causing invalid tokenizations. Phan et al. introduce exact byte-level marginalization—zero-shot, inference-time conversion of tokenized LMs to byte-level decoders—eliminating this bias and yielding an 18.9 percentage point gain in pass rate for SPM prompts, especially on mid-token-aligned FIM settings [2410.09303]. ## 5. Alignment, Instruction-Tuning, and Extensions ### Instruction-Aware FIM Vanilla FIM training fails to leverage inline developer intent (e.g., comments/instructions). Standard instruction tuning—prevalent in chat-style LLMs—improves instruction following but degrades infilling accuracy by up to 10–20 points. The IFIM protocol injects an explicit instruction section $I$ so that the model learns on $(P,I,S) \rightarrow M$ quadruples, maintaining infilling performance while recovering instruction sensitivity. With 100% IFIM data, pass@1 rises from 84.6% (baseline) to 93.6% with instructions, with no regression on zero-instruction tasks [2509.24637]. ### Editing Paradigms: Search-and-Replace Infilling FIM is inherently context-invariant and cannot correct erroneous prefix or suffix. The Search-and-Replace Infilling (SRI) framework models explicit identification (search) and targeted replacement (replace) in one pass, harmonizing edit-centric capabilities with FIM’s low-latency dynamic [2601.13384]. SRI dramatically reduces adversarial code injection vulnerabilities (>80–90% reduction), robustifies completion in the presence of noisy context (EM: +33 points), and preserves general code generation competency without latency cost. ### FIM in New Domains: Protein and Mathematical Sequences - **Protein Design:** ProtFIM, trained with FIM transformations on protein sequences, yields improved Recovery@5 from 0.73 to 0.78 over non-FIM LMs for secondary structure-preserving infilling [2303.16452]. - **Mathematical Reasoning:** MathFimer applies FIM to insert intermediate CoT steps, improving GSM8K accuracy from 67.78% to 75.21% (Meta-Llama-3.1-8B) and supports multi-round expansion without reliance on stronger external generators [2502.11684]. ## 6. Evaluation, Benchmarks, and Empirical Findings Comprehensive FIM evaluation requires syntax-aware and functional correctness benchmarks. SAFIM [2403.04814] defines three AST-based categories (Algorithmic block, Control-flow, API-call), spanning 17,720 multilingual examples. AST-FIM [2506.00204] and curriculum/context-based FIM [2412.16589] demonstrate that structural masking and context retrieval improve pass@1 by several points (up to 5 on SAFIM splits), with curriculum/context-augmented fine-tuning yielding especially strong gains for small models in low-latency scenarios. Empirical highlights (pass@1 unless otherwise noted): | Model / Method | SAFIM-Alg | HumanEval-FIM | Multiline (SWE-bench) | |-----------------------------|-----------|---------------|----------------------| | Rand-FIM-1B | 28.2 | | | | AST-FIM-1B | 33.5 | | | | StarCoder2-7B + curriculum | | | 13.4 | | DeepSeek-1B + curriculum | | | 8.5 | | Chat LLMs (GPT-4o, Claude) | | | 19–23 | | SRI-Coder-32B (editing) | | | 61.6 | Syntax- and context-aware decoding, with output boundary supervision and targeted fine-tuning, consistently yields improvements across code and mathematical FIM tasks [2402.17988][2505.18789][2506.00204][2412.16589]. ## 7. Limitations and Future Directions Current FIM approaches exhibit several limitations: - Models not specifically tuned with FIM data or horizon prediction fail to respect segment boundaries or context, requiring post-processing, especially for random token splits. - Some FIM decoding frameworks rely on EBNF grammars, which may not fully capture PEG or tab/space semantics [2402.17988]. - AST-FIM’s performance depends on the availability and accuracy of AST parses. Directions for further research include: - Integration of richer static analysis (e.g., type systems, undefined variable checks) via quotient grammars [2402.17988]. - Expansion of instruction-augmented and editing-centric FIM to cross-lingual, multi-domain, and large-model settings [2509.24637][2601.13384]. - Broader benchmarks in real-world, multi-file codebases and stepwise reasoning domains [2502.11684][2506.00204]. - Guaranteeing output boundaries for arbitrary (non-line) spans via hybrid planning/decoding objectives [2410.03103][2505.18789]. Fill-in-the-Middle is now the methodological standard for robust, context-aware sequence generation in code and beyond, with ongoing advances unifying structural, instructional, and editing capabilities across parameter scales and domains. Source: https://www.emergentmind.com/topics/fill-in-the-middle-fim-capabilities