--- title: Fill-In-the-Middle Objective url: https://www.emergentmind.com/topics/fill-in-the-middle-fim-objective type: topic --- # Fill-In-the-Middle Objective Fill-In-the-Middle (FIM) is a span-based generation objective for language models that extends standard left-to-right (autoregressive) pretraining by teaching the model to reconstruct an arbitrary masked span (“middle”) given both the preceding (“prefix”) and succeeding (“suffix”) context. Originally motivated by the demands of program synthesis and code completion, FIM-style objectives have enabled a new class of language and code models with strong bidirectional infilling capabilities, robust context conditioning, and practical advantages in tasks ranging from code completion, block/sentence infilling, step expansion in mathematical reasoning, protein sequence design, and beyond. ## 1. Formal Definition and Mathematical Objective The canonical FIM objective decomposes a sequence into three contiguous segments: prefix $P$, infill (mask/middle) $M$, and suffix $S$. The model is explicitly trained to reconstruct $M$ given both $P$ and $S$: \[ \mathcal{L}_{\text{FIM}}(\theta) = -\mathbb{E}_{(P, M, S)\sim\mathcal{D}}\left[ \sum_{t=1}^{|M|} \log p_\theta(m_t \mid P, S, m_{$, $<$fim\_hole$>$, $<$fim\_end$>$) [2401.14196]. The loss is always computed only over the infill span. The FIM setting belongs to the family of span corruption/infill objectives, generalizing left-to-right (which only conditions on past context), and subsuming classical token-level Masked Language Modeling (MLM), span-infilling for denoising (T5, BART), and permutation-based models [2310.09930]. ## 2. Motivation and Core Properties Standard autoregressive (next-token) language models only learn to predict the next token given the previous context, which biases them towards rightward generation and makes them inherently left-contextual. This unidirectionality limits the capacity to produce coherent infillings for arbitrary gaps within a sequence unless strong architectural and pretraining changes are made. FIM modeling directly addresses several critical requirements: - **Bidirectional Conditioning:** FIM allows tokens in the infilled region to attend to both previous and future context, leading to more semantically consistent and structurally valid infills [2403.04814][2412.16589]. - **Span Generality:** By supporting reconstruction of arbitrary-length masked spans (instead of only single-token or rightmost continuation), FIM empowers models to handle variable-length completions and structural edits (code blocks, mathematical steps, sequence segments) [2506.00204][2303.16452]. - **Data Augmentation and Robustness:** FIM objectives do not require architectural changes; rather, the dataset is transformed or augmented so that models see a mix of standard next-token and FIM-formatted sequences, achieving “FIM-for-free” bidirectionality with no perceptible loss in autoregressive performance [2207.14255][2401.14196]. In mathematical reasoning, FIM uniquely enables “step expansion”: interpolating more granular intermediate steps into a human-verified chain-of-thought (CoT) without requiring outputs to be regenerated from scratch, thereby enriching solution quality and improving downstream task performance [2502.11684]. ## 3. Construction of FIM Training Data Three main design choices govern practical FIM data construction: 1. **Span Selection:** Random contiguous spans may be sampled at the character, token, or line/block level. Random character-level splits yield generalization across arbitrary substrings, but structure-aware masking—e.g., masking spans aligned to Abstract Syntax Tree (AST) subtrees [2506.00204], or homologous protein regions [2303.16452]—can better match real-world editing operations and improve infilling accuracy. 2. **Prompt Formatting:** FIM relies on special delimiters, with popular formats including: - PSM: $<$pre$>$ prefix $<$suf$>$ suffix $<$mid$>$ middle - SPM: $<$suf$>$ suffix $<$mid$>$ prefix middle - Context addition: Prepended semantic or repository context may be included, especially in code (e.g., C₁…Cₖ before prefix) [2412.16589]. 3. **Mixing Ratios:** Typically, 50–90% of the dataset is processed into FIM format (using the above transformations), with the remainder used for standard left-to-right next-token prediction. This preserves left-to-right capabilities (“for free”) and achieves optimal infilling performance without trade-off [2207.14255][2401.14196]. ## 4. Extensions, Variants, and Benchmarking ### 4.1 Structured- and Context-Aware FIM - **AST-FIM:** Masks are chosen to align with AST subtrees, ensuring that the infill spans maintain code structure and reflect authentic developer editing patterns; AST-FIM outperforms random-span FIM by up to 5 points on code benchmarks [2506.00204]. - **Context Augmentation:** Augmenting FIM prompts with symbol definitions or repository-level background via static analysis tools (e.g., TypeScript compiler, Tree-sitter) improves infilling, especially on difficult cross-file dependencies and for small models [2412.16589]. - **Curriculum FIM:** By oversampling difficult code structures (e.g., function/class definitions, call expressions, control flow), curriculum-based span sampling can target model weaknesses and yields the largest gains in low-compute regimes [2412.16589]. ### 4.2 Task-Specific Adaptations - **Instruction-aware FIM (IFIM):** Extends FIM to support an explicit natural language instruction segment between prefix and middle, enabling the model to condition infilling on developer intent while preserving standard FIM capabilities [2509.24637]. - **Horizon-Length Prediction (HLP):** Augments standard FIM with an auxiliary regression task that at each infill step estimates the normalized distance to the reconnection point (i.e., number of tokens left before suffix), addressing the lack of lookahead and improving both output boundary accuracy and success rates in open-domain code completion [2410.03103]. ### 4.3 Benchmarks - **SAFIM:** Syntax-Aware FIM with ∼18K examples across languages, measuring algorithmic, control-flow, and API-aware block completion; establishes that FIM-pretrained models achieve higher infilling and also improved left-to-right performance [2403.04814]. - **Real-FIM-Eval:** >30K real-world edits from GitHub commits across 12 languages, evaluating both “add” (insertion) and “edit” (replace) forms of FIM; AST-FIM achieves best results [2506.00204]. - **SEIFER:** Middle-span protein editing with structure-aware constraints; ProtFIM (FIM-trained) surpasses left-to-right and permutation-based baselines [2303.16452]. ## 5. Empirical Impact and Model Comparisons FIM training confers substantial empirical benefits: - **Mathematical Reasoning:** MathFimer-style FIM augmentation increases GSM8K accuracy by +5.61 points, MATH by +3.52, and yields up to +15.54 on repeated expansion [2502.11684]. - **Code Completion:** On HumanEval infilling and multi-language single-line FIM, DeepSeek-Coder and similar FIM models consistently outperform left-to-right baselines, with mean accuracies of 70%–78% (Python/Java/JS), exceeding comparably sized causal models [2401.14196]. FIM pretraining universally improves both infilling and left-to-right code generation benchmarks [2403.04814]. - **Protein Design:** ProtFIM achieves retrieval@5 = 0.73 versus 0.70 for best left-to-right baselines, and superior property prediction on the FLIP benchmark [2303.16452]. - **Tokenization Correction:** Byte-level FIM decoding eliminates tokenization bias at span boundaries, raising SPM random-span FIM pass@1 scores from 45.0% (tokenized) to 63.9% (byte-corrected), a ≈18% absolute gain [2410.09303]. - **Syntactic Validity:** Constrained FIM decoding via quotient parsing halves syntax error rates for Python FIM code tasks, with 89.5% syntax-valid completions versus 65% unconstrained [2402.17988]. Ablations indicate that most FIM advantages cannot be recovered by fine-tuning standard LMs on infilling tasks alone; pretraining or retraining with FIM objectives is crucial [2207.14255]. Further, FIM's empirical gains are robust to model size but especially critical for small, latency-sensitive models (e.g., 1–2B parameters) [2412.16589]. ## 6. Limitations, Practical Considerations, and Future Directions Despite broad success, FIM models have open challenges: - **Boundary Awareness:** Next-token objectives cannot by themselves teach models to align infill boundaries precisely, often leading to overwriting or duplication of suffix context. Solutions include post-processing (e.g., syntax-aware truncation [2403.04814]), horizon-length prediction architectures [2410.03103], or constrained decoding [2402.17988]. - **Tokenization Effects:** Conventional byte-pair or subword tokenization leads to bias near span boundaries (“tokenization bias”); byte-level sampling resolves but incurs compute overhead [2410.09303]. - **Instruction Integration:** Naive insertion of NL instructions as code comments degrades infilling capability; explicit instruction channels (IFIM) are necessary to reconcile instruction-following with accurate FIM [2509.24637]. - **Generalization and Transfer:** While FIM does not reduce next-token generation performance, its absolute effectiveness depends on span-choice alignment (random vs. block- or function-aware) and model exposure to real edit distributions (e.g., via AST or commit benchmarks) [2506.00204][2403.04814]. - **Evaluation Complexity:** Strict evaluation necessitates bespoke post-processing for each language and span type, as free-form model outputs may include extraneous line breaks, comments, or prefix/suffix duplicates [2505.18789]. Emerging directions include combinatorial FIM objectives (multi-span or recursive infilling), explicit modeling of output horizon (HLP), and grammar-constrained decoding spanning not just syntax but also semantic properties. There is also ongoing work on unifying FIM with any-order generation and scalable variable-rate masking for general-purpose infilling [2310.09930]. The technique is being extended to new domains—biosequence editing, instructional code infilling, mathematical proof refinement—and is central to the next generation of tool-oriented, context-sensitive LLMs. Source: https://www.emergentmind.com/topics/fill-in-the-middle-fim-objective