---
title: Fill-in-the-Middle (FIM) Modeling
url: https://www.emergentmind.com/topics/fill-in-the-middle-fim-993a87a4-2524-4748-943d-0b0ae4448bd1
type: topic
---

# Fill-in-the-Middle (FIM) Modeling

Fill-in-the-Middle (FIM) is a sequence modeling paradigm that extends conventional left-to-right (autoregressive) language modeling by enabling models to generate a missing span (“middle”) given both a prefix and a suffix (the “left” and “right” contexts). Originally developed for code and text infilling, FIM now underlies state-of-the-art methods in code completion, code reasoning, mathematical chain-of-thought, type prediction, protein design, and multilingual sequence modeling. Its central mechanism is the reordering of training data and the introduction of special markers to teach models to condition on both sides of an infilling site—an ability essential for robust program repair, document editing, and reasoning with intermediate steps.

## 1. Canonical FIM Paradigm and Variants

Given a sequence $[x_1, x_2, \ldots, x_T]$, FIM randomly selects two cutpoints $0 < p < p+m < T$, yielding three contiguous spans:
- **Prefix:** $x_{1 \ldots p}$
- **Middle:** $x_{p+1 \ldots p+m}$
- **Suffix:** $x_{p+m+1 \ldots T}$

Training transforms the input sequence (using format variants such as PSM/PSM: Prefix-Suffix-Middle/Suffix-Prefix-Middle) by concatenating the prefix and suffix with sentinel tokens, asking the model to infill the middle segment:
- Input: `<pre> prefix <suf> suffix <mid> middle <eoi>`

The standard loss is autoregressive cross-entropy only on the middle tokens, while preserving positional information through sentinels. The model is optimized to maximize
\[
\mathcal{L}_{\mathrm{FIM}} = -\sum_{t=1}^{m} \log P_\theta(x_{p+t} \mid x_{1:p}, x_{p+m+1:T}, x_{p+1:p+t-1}).
\]
In practical training setups, FIM is typically mixed with conventional left-to-right objectives to retain next-token generation abilities [2207.14255, 2401.14196, 2506.00204].

### Format Choices and Best Practices

- **Transformation Rate:** Data is FIM-transformed with probability $p$ (default $0.5$–$0.9$) [2207.14255].
- **Format Mixing:** Alternating between PSM and SPM templates improves flexibility [2207.14255].
- **Span Selection:** Character-level random splits maximize transfer across inference settings, outperforming line- or token-level splits [2207.14255].
- **Context-Level Transform:** Applying FIM after packing maximizes infill performance [2207.14255].

## 2. Methodological Advances and Variants

### Structural and Semantic Extensions

- **AST-FIM:** Masks entire Abstract Syntax Tree subtrees to create syntactically aligned infilling spans, outperforming random character-level masking by 5–7 points on real-world infilling tasks at both 1B and 8B scale [2506.00204].
- **FIM-SE:** Imposes character-level constraints and introduces line-level markers to eliminate sub-token boundary errors, substantially improving random-span and single-line infilling (by 8–12 points) on benchmarks such as HumanEval [2405.17103].
- **Instruction-aware FIM (IFIM):** Incorporates explicit developer-provided instructions into FIM prompts via special `<INS>` delimiters, preserving general infilling capacity and dramatically enhancing instruction-following in code completion [2509.24637].
- **Horizon-Length Prediction (HLP):** Supplementary loss ensures models explicitly predict the length (horizon) of the middle segment at each generation step. HLP increases alignment with input boundaries, eliminating the need for dataset-specific truncation heuristics and improving both file-level and repository-level FIM by up to 24% relative [2410.03103].
- **Search-and-Replace Infilling (SRI):** Replaces FIM’s rigid context assumption with a patch-based editing approach that first recalls the target context (SEARCH phase) and then applies replacements (REPLACE phase), preserving latency and improving context-aware code repair [2601.13384].
- **Direct Preference Optimization (DPO) with AST granularity:** Pairs FIM splits with DPO for fine-grained feedback and curriculum schemes based on code block type and difficulty, yielding consistent gains in pass@1 metrics [2508.19532].

### Broad Application Domains

- **Mathematical Reasoning:** MathFimer applies FIM to chain-of-thought expansion, teaching LLMs to insert missing intermediate steps in mathematical solution chains, reliably improving accuracy on GSM8K, MathInstruct, and MATH datasets by 2–5 points [2502.11684].
- **Protein Design:** ProtFIM applies FIM to mask and recover segments in amino-acid sequences, outperforming both standard AR models and even 30$\times$ larger models on structure recovery metrics [2303.16452].
- **Type Prediction:** FIM fine-tuning for TypeScript/Python type annotation prediction (“fill-in-the-type”) achieves 14.5 points higher type-check success than standard FIM, especially when paired with program decomposition and search [2305.17145].

## 3. Model Architectures and Training Protocols

FIM training, whether on code or text, is architecturally straightforward:
- Base models are standard decoder-only Transformers (e.g., GPT-style), with added sentinels for context separation [2207.14255, 2401.14196, 2506.00204].
- Structural extensions such as AST-FIM operate entirely at the data pre-processing stage; the model architecture remains unchanged [2506.00204].
- Auxiliary heads (e.g., HLP’s linear horizon head) add negligible (<0.01%) parameter overhead and are discarded during inference [2410.03103].
- FIM is compatible with bidirectional architectures: FiLM enables truly arbitrary-order (non-causal) infilling with global attention, and exhibits competitive perplexity and ROUGE with autoregressive benchmarks [2310.09930].

Optimizer, learning rate schedules, context window, and tokenizer selection are typically held constant between FIM and standard AR pre-training runs to ensure comparability [2207.14255, 2401.14196].

## 4. Evaluation Methodologies and Benchmarks

FIM evaluation requires benchmarks designed to probe infilling capabilities:

- **HumanEval-Infilling, InCoder tasks, MBPP:** Sample (prefix, missing line[s], suffix) tuples from real code function bodies. Metrics: pass@1 (unit test execution), line exact match (EM), CodeBLEU [2401.14196, 2207.14255, 2506.00204].
- **SAFIM:** Syntax-aware benchmark masking whole AST nodes (blocks, control-flow, API calls). Reports pass@1 (all tests pass), exact match, token F1, and character-level perplexity [2403.04814].
- **RepoEval, CrossCodeEval:** Repository-level, multi-file FIM to measure long-horizon and cross-file infilling [2412.16589, 2410.03103].
- **MathFimer, SEIFER:** Domain-adapted FIM for stepwise mathematical reasoning or secondary-structure-preserving protein infilling [2502.11684, 2303.16452].

Post-processing is often required, especially for evaluating extraneous code tokens in raw outputs:
- Complete-line truncation for line-based tasks;
- Overlap removal for random spans (ensuring no duplication of context) [2505.18789, 2403.04814].

## 5. Limitations, Pitfalls, and Remedies

**Boundary Unawareness:** Vanilla FIM-trained models frequently overrun or underrun the target span boundary, particularly when the middle’s size or exact boundaries are unspecified. Heuristic truncation—by line count or syntax-aware AST truncation—has been the default, but this is unreliable in open-domain or non-dataset-aligned settings, leading to 5–14% relative loss in pass@1 if omitted [2410.03103, 2505.18789].

**Tokenization Bias:** Standard FIM models degenerate on mid-token cutpoints due to sub-token fragmentation, resulting in invalid completions and low pass rates (e.g., 45% vs 64% for SPM prompts). Exact byte-level sampling algorithms, which marginalize over all aligned tokenizations, restore correct next-byte distributions and boost pass@1 by as much as 18 points [2410.09303].

**Context-Only Rigidness:** The “optimal context” assumption in FIM—treating context as ground truth—makes FIM unable to correct contextual errors. SRI (Search-and-Replace Infilling) and IFIM (Instruction-aware FIM) generalize the paradigm to context-aware patching and mixed instruction following without degrading infilling [2601.13384, 2509.24637].

**Scaling and Transfer:** Gains from FIM pretraining saturate at moderate model sizes; data quality, pretraining signals, and prompt engineering (including structural and curriculum design) have higher leverage than raw parameter scaling on practical infilling tasks [2403.04814].

**Architectural Compatibility:** FIM is maximally effective in architectures with unrestricted self-attention, but adaptation to strict causal models (Code Llama, SantaCoder, StarCoder) is routine via sentinel markers [2207.14255, 2401.14196].

## 6. Key Results and Quantitative Impact

| Model or Technique     | Evaluation           | Pass@1/EM gain | Other Impact                                                 |
|-----------------------|----------------------|---------------|--------------------------------------------------------------|
| HLP (Horizon-Length)  | Repo-level infilling | +24% rel.     | Eliminates truncation, improves code reasoning [2410.03103]  |
| AST-FIM               | SAFIM/Real-FIM-Eval  | +5–7 pts      | Generalizes to 100+ languages via Tree-sitter [2506.00204]   |
| FIM-SE                | Random/single-line   | +8–12 pts     | Addresses sub-token boundary errors in character infilling    |
| IFIM                  | HumanEval-infilling  | +9–10 pp      | Preserves/increases performance without instruction [2509.24637] |
| MathFimer             | GSM8K, MATH          | +2–5 pp       | Step expansion in math chain-of-thought [2502.11684]         |
| ProtFIM               | SEIFER Benchmark     | +0.03 R@K     | Outperforms 2.7B ProGen2 on structure infilling [2303.16452] |
| DPO+AST+Curriculum    | HumanEval/MBPP/BigCodeBench | +1–2 pts | Granular pair alignment for code infill feedback [2508.19532] |

## 7. Future Directions and Theoretical Significance

- **Long-Horizon Planning as Objective:** Direct supervision of horizon length (e.g., HLP) encourages internalization of not just language syntax, but also end-of-segment planning—this is empirically linked to improved multi-step reasoning [2410.03103].
- **Structural and Context-aware Prompting:** Structure-aligned masking (AST, curriculum mining) and explicit instruction injection (IFIM) show that model-agnostic improvements arise from input formatting, not fundamental changes in architecture [2506.00204, 2509.24637].
- **Generalization to Non-Code Domains:** Step-expansion in math and protein domains demonstrates that FIM is a paradigm for nonmonotonic, bidirectional, and intent-aware generation beyond just code infilling [2502.11684, 2303.16452].
- **Integration with Parsing and Formal Methods:** Grammar-constrained decoding and right-quotient parsing further constrain FIM completions to ensure syntactic and, eventually, semantic correctness, with negligible inference cost [2402.17988].
- **Post-processing Redundancy:** As models internalize boundary planning (via objectives like HLP) or character-level constraints (FIM-SE), reliance on output truncation and heuristic boundary defenses is expected to diminish [2410.03103, 2405.17103].

The FIM paradigm, through progressive data-centric innovations, has reshaped generation objectives in code, reasoning, and sequence modeling, supporting a shift from strictly monotonic (L2R) generation to fully context-aware, structure-sensitive completion with robust stopping criteria and composable, task-driven format specialization.

Source: https://www.emergentmind.com/topics/fill-in-the-middle-fim-993a87a4-2524-4748-943d-0b0ae4448bd1