Inducing-Input Infilling

• Inducing-input infilling is a technique that reconstructs missing or masked sequences by using both preceding and following contextual cues.
• It employs methods like autoregressive reparameterizations and inference-time optimization to generate fluent and semantically coherent completions.
• Diffusion and dynamic anchor techniques further enable adaptive length calibration and efficient context repair in both text and code generation.

Inducing-input infilling denotes, in broad terms, the reconstruction of missing or masked portions of a model’s input so that the completed sequence is coherent with both left and right context and scores highly under the underlying model. In the sequence-generation literature, the concept appears in several closely related forms: general text infilling with arbitrary blanks and unknown span lengths, fill-in-the-middle code completion with prefix and suffix conditioning, instruction-aware and frame-guided infilling, search-and-replace editing, and diffusion-based infilling with dynamic length or anchor estimation. Across these settings, the common pattern is to treat the visible context, placeholders, and auxiliary control signals as conditioning inputs that induce a distribution over the missing span or spans (Liu et al., 2019, Zhu et al., 2019, Donahue et al., 2020).

1. Problem formulation and conceptual scope

The foundational text-infilling formulation treats a template as a sequence in which one or more contiguous spans have been deleted and replaced by placeholders. The missing spans may be arbitrary in number, located at arbitrary positions, and of arbitrary unknown length; the objective is to fill them so that the final sequence is fluent, semantically coherent, and consistent with the visible context (Zhu et al., 2019). In “Enabling LLMs to Fill in the Blanks,” this is reparameterized as learning a conditional distribution over the missing material, $p(y \mid x)$, where $x$ is the incomplete text containing blank markers and $y$ is the concatenation of the missing spans separated by answer tokens (Donahue et al., 2020).

A complementary formulation appears in TIGS, which casts infilling as direct optimization under a pretrained sequence model. Given conditioning input $x$, a template $y^{\mathbb{B}}$, and missing tokens $\hat y$, the method seeks a completion minimizing the negative log-likelihood of the completed sequence: $$\hat{y} = \argmin_{\hat{y}_j \in \mathcal{V}} \mathcal{L}_{NLL}(x, y^*), \qquad \mathcal{L}_{NLL}(x, y^*) = -\log P(y^* \mid x).$$ This makes infilling an inference problem over latent discrete symbols rather than only a supervised generation task (Liu et al., 2019).

The phrase “inducing input” also has an established and distinct meaning in sparse Gaussian processes, where inducing inputs are auxiliary support locations in a variational approximation (Izmailov et al., 2016). In the infilling literature discussed here, the phrase is used in a broader sense: the partially observed sequence, blank markers, and optional structured controls act as conditioning inputs that induce the model’s completion behavior. This suggests a family resemblance rather than a single unified formalism.

2. Autoregressive reparameterizations of infilling

A major line of work shows that infilling can be implemented without abandoning standard autoregressive language modeling. ILM constructs training sequences of the form $[x, \text{sep}, y]$, where $x$ is masked text and $y$ is the ordered list of missing spans. Because all observed context appears before $x$0, a unidirectional GPT-2 can condition on both left and right context when generating the missing content, while preserving standard next-token training and using only a small set of added tokens such as blank, ans, and sep (Donahue et al., 2020). On sentence infilling, ILM matches the perplexity of a longer “LM-All” formulation on Stories and Abstracts while keeping relative sequence length at $x$1 rather than $x$2 (Donahue et al., 2020).

Code infilling generalizes the same principle through explicit sequence transformations. InCoder uses a causal-masking objective in which a masked span is replaced in situ by <Mask:k> and appended at the end with <EOM>, so that a decoder-only Transformer learns to reconstruct arbitrary regions using both left and right context while remaining purely autoregressive (Fried et al., 2022). Fill-in-the-middle training in code models similarly rearranges prefix $x$3, middle $x$4, and suffix $x$5 into prompts such as

$x$6

thereby learning $x$7 under a causal decoder (Sun et al., 29 Sep 2025).

A common misconception is that infilling necessarily requires a bidirectional encoder or a specialized masked-LM architecture. The cited work shows otherwise: ILM, InCoder, and FIM-based code models all recover bidirectional conditioning by reordering the sequence so that right context appears in the autoregressive prefix (Donahue et al., 2020, Fried et al., 2022, Sun et al., 29 Sep 2025).

3. Inference-time optimization and non-monotonic decoding

Another branch treats inducing-input infilling as an inference procedure applied to an already trained generator. TIGS is the clearest example. It relaxes each unknown token into a continuous embedding vector, minimizes the completed sequence’s negative log-likelihood with respect to those embedding variables, and then projects back to discrete tokens by nearest-neighbor search in embedding space followed by local NLL-based selection. Coordinate-style updates over blanks are used to reduce incompatible joint discretizations, and the method is described as broadly applicable to any differentiable neural sequence generative model (Liu et al., 2019).

Self-infilling code generation pushes this inference view further by allowing the model to construct its own future context. A FIM-trained decoder first generates a suffix from a prefix, optionally guided by a suffix prompt such as return, and then infills the skipped middle conditioned on both prefix and self-generated suffix. Interruptions are triggered when $x$8, and looping alternates self-infilling with left-to-right regeneration so that suffix and middle can be refined across iterations (Zheng et al., 2023). This converts ordinary autoregressive decoding into a non-monotonic process in which future context becomes an induced control signal.

Search-and-Replace Infilling extends the same intuition from completion to editing. Instead of generating only a middle span, SRI asks a chat-aligned model to emit a structured patch: $\hat y$0 `` with a required marker/* MIDDLE CODE TO COMPLETE */` in the search region and edits restricted to a 10-line window around that marker (Zhang et al., 19 Jan 2026). Because the model must first restate the region it will modify and then produce the replacement, the method internalizes a locate-and-edit procedure within single-pass autoregressive generation. This suggests that inducing-input infilling can be generalized from blank completion to localized context repair.

4. Structured inducing signals: frames, instructions, and edit schemas

Beyond raw prefix and suffix context, several systems introduce explicit symbolic or natural-language controls. InFillmore augments bidirectional-context infilling with FrameNet frame tokens, yielding a model of

$x$9

where $y$0 is a frame sequence such as [Food] or [Commerce_buy] (Ou et al., 2021). Frames can be injected during fine-tuning or enforced at decoding time through disjunctive lexical constraints built from frame lexical units. Human evaluation shows that fine-tuned frame-guided generation retains distinguishability comparable to ILM while improving semantic fidelity to the specified frames (Ou et al., 2021).

Instruction-aware Fill-in-the-Middle introduces an explicit instruction segment $y$1 and trains

$y$2

The instruction is placed in a dedicated slot marked by <INS>, producing formats such as PIMS or PSIM while preserving the original relative ordering of $y$3, $y$4, and $y$5 from base FIM training (Sun et al., 29 Sep 2025). On HumanEval-infilling, DeepSeek-Coder improves from $y$6 to $y$7 Pass@1 when instructions are provided, and the paper reports that this does not compromise FIM performance when no instructions are present (Sun et al., 29 Sep 2025). A recurrent finding is that structured placement matters: comment-style inline instructions are less effective than a separate instruction channel, and placing the instruction immediately before <MID> yields the strongest gains (Sun et al., 29 Sep 2025).

SRI makes a related point from the chat-model side. Natural-language FIM prompting degrades completion performance relative to base FIM models, whereas a search-and-replace instruction format aligns better with chat priors and diff-like data seen during pretraining (Zhang et al., 19 Jan 2026). On CrossCodeEval-Flex, where context around the gap is deliberately perturbed, base FIM models achieve $y$8 exact match, while SRI-Coder-32B reaches $y$9 EM and $x$0 edit similarity, indicating that structured edit schemas can induce not just completion but local correction of erroneous context (Zhang et al., 19 Jan 2026).

5. Diffusion models, dynamic lengths, and anchor-based infilling

Diffusion LLMs make inducing-input infilling explicit because they operate on partially masked sequences with bidirectional attention. Their main limitation is that the mask length must usually be fixed in advance. CAL addresses this by probing candidate lengths $x$1 using the first-step denoising confidence

$x$2

and then calibrating a systematic Length Bias via a fitted function $x$3 to obtain $x$4 (Liu et al., 31 Jan 2026). The paper identifies a local “Oracle Peak” near the ground-truth infill length and reports that CAL improves Pass@1 by up to $x$5 over fixed-length baselines and $x$6 over chat-based adaptive methods in code infilling, while also improving BLEU-2 and ROUGE-L in text infilling (Liu et al., 31 Jan 2026). This suggests that the visible context can induce not only content but also the appropriate canvas length.

Dynamic Infilling Anchors applies a related idea to format-constrained generation in diffusion LLMs. Begin anchors such as > and <answer> are fixed, but end anchors such as `and</answer>` are placed dynamically: a short masked block is initialized, one-step diffusion predictions are scanned for confident end anchors, and the block expands by $x$7 masks until an acceptable end position is found; only then does full iterative infilling proceed (Han et al., 3 Jun 2026). The method is training-free and improves GSM8K format correctness from $x$8 to $x$9 and accuracy from $y^{\mathbb{B}}$0 to $y^{\mathbb{B}}$1, while on a WikiBio-derived JSON task it raises strict valid-JSON rate from $y^{\mathbb{B}}$2 with fixed infilling to $y^{\mathbb{B}}$3 with minimal hallucination score $y^{\mathbb{B}}$4 (Han et al., 3 Jun 2026). A common misconception is that fixed anchors alone suffice for structured outputs; the reported failures of rigid spans, truncated reasoning, and redundant content show that dynamic boundary induction can be decisive.

6. Serving, robustness, and security implications

Inducing-input infilling is also a systems problem. EFIM observes that standard FIM prompt orderings hinder cross-request KV cache reuse in interactive editing because prefix or suffix growth invalidates cached context. It therefore transforms prompts so that stable prefix and suffix tokens remain before <M> and the changing increment appears after it, allowing substantially higher reuse of precomputed attention state (Guo et al., 28 May 2025). Because this exposes partial-word generation failures, the paper adds fragment tokenization training, which randomly splits raw text into short fragments before tokenization so that the model sees more partial-word continuations during continued pretraining (Guo et al., 28 May 2025). On two representative LLMs, EFIM lowers latency by $y^{\mathbb{B}}$5 and improves throughput by $y^{\mathbb{B}}$6 while maintaining original infilling capability (Guo et al., 28 May 2025).

The same bidirectional conditioning that improves utility also broadens extraction risk. “Extracting Training Data from Diffusion LLMs via Infilling” formalizes infilling extraction by a binary mask $y^{\mathbb{B}}$7 and a decoding function $y^{\mathbb{B}}$8, defining verbatim extraction as

$y^{\mathbb{B}}$9

The paper shows that mask geometry governs extractability: edge-conditioned masks can extract up to three times more verbatim sequences than prefix-conditioned ones, and targeted masks that reveal all surrounding context can recover redacted email addresses with higher recall from a DLM than from a scale-matched autoregressive model in the reported Enron setting (Wang et al., 22 May 2026). It also finds that denoising steps, block size, temperature, and remasking strategy measurably affect extraction performance, while subsequent supervised fine-tuning does not eliminate prior memorization (Wang et al., 22 May 2026).

Across the literature, several recurrent limitations appear. TIGS requires white-box gradient access and faces non-convex optimization and embedding-space mismatch (Liu et al., 2019). Sequential blank filling in general text infilling does not jointly decode all gaps (Zhu et al., 2019). SRI shows smaller gains for small models and has only offline evaluation (Zhang et al., 19 Jan 2026). CAL depends on an offline-fitted Length Bias curve (Liu et al., 31 Jan 2026), and DIA requires manual anchor design and adds Stage-1 inference overhead (Han et al., 3 Jun 2026). These constraints indicate that inducing-input infilling is not a single method but a design space spanning representation, inference, training objective, serving format, and safety analysis.