InverFill: Enhanced Few-Step Diffusion Inpainting

• InverFill is a diffusion inpainting technique that replaces random Gaussian noise with semantically aligned noise to improve harmony and visual consistency.
• The method employs a one-step inversion network within a blended sampling pipeline, preserving visible context and enabling effective few-step generation.
• Empirical evaluations reveal reduced runtime and improved metrics like IR and CLIP similarity, although minor issues such as color inconsistency can occur.

Searching arXiv for the cited InverFill paper and closely related inpainting work. InverFill is a one-step inversion method for diffusion-based image inpainting that targets the failure mode of naive few-step text-to-image sampling under masking constraints. It was introduced as “InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting” (Vu et al., 24 Mar 2026). The method begins from the diagnosis that few-step inpainting degrades primarily because random Gaussian noise initialization provides no semantic information about the visible context, and that, under low numbers of function evaluations (NFEs), the denoising process has too little capacity to repair this mismatch. InverFill therefore injects semantic information from the input masked image into the initial noise, and uses this semantically aligned noise inside a blended sampling pipeline. The reported result is improved harmonization, fidelity, and text coherence for few-step inpainting, without training specialized inpainting models, without real-image supervision, and with minimal inference overhead (Vu et al., 24 Mar 2026).

1. Problem formulation and failure mode

InverFill addresses masked image completion in the standard latent-diffusion setting. The inputs are a masked image $I_m \in \mathbb{R}^{H \times W \times C}$, a binary mask $M \in \{0,1\}^{H \times W \times C}$ where $1$ indicates missing pixels, and a text prompt $c$. The final composite is written as

$$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$

The stated objective is to produce a result that is semantically aligned with the prompt, visually consistent with the visible context, and harmonized in style, lighting, color, and texture (Vu et al., 24 Mar 2026).

The central diagnosis is specific to few-step diffusion. Standard blended sampling can preserve known regions while denoising the unknown region, but it relies on a denoising trajectory that starts from pure Gaussian noise. In many-step samplers this mismatch can be gradually corrected. In few-step models, however, each denoising update is large, and the model has only $2$–$4$ steps in which to reconcile the latent with the unmasked background. The paper therefore attributes visible seams, inconsistent style, reduced harmonization, blur, and prompt drift to the combination of random Gaussian initialization and low NFEs (Vu et al., 24 Mar 2026).

This diagnosis narrows the problem from generic inpainting difficulty to what the paper presents as a starting-point problem. Rather than treating harmonization errors as a purely architectural deficiency of few-step generators, InverFill treats them as a consequence of initializing the reverse process from a latent that is semantically unrelated to the observed image content.

2. One-step inversion and semantically aligned initialization

The core mechanism is a one-step inversion network that maps the masked image into an initial latent better aligned with the visible image context. Let $z_0 = \mathcal{E}(I_{gt})$ be the latent of the ground-truth image, $z_0^m = \mathcal{E}(I_m)$ the latent of the masked image, $\epsilon \sim \mathcal{N}(0,I)$ Gaussian noise, $M \in \{0,1\}^{H \times W \times C}$0 the inversion network, and $M \in \{0,1\}^{H \times W \times C}$1 a pretrained one-step text-to-image generator. The inversion network predicts

$M \in \{0,1\}^{H \times W \times C}$2

and the generator reconstructs

$M \in \{0,1\}^{H \times W \times C}$3

The paper states that the goal is not exact pixel recovery but rather a latent whose denoising trajectory preserves the visible background and yields a plausible completion of the masked region (Vu et al., 24 Mar 2026).

InverFill is embedded into a blended sampling procedure. Vanilla blended sampling encodes the masked image, diffuses the known region forward at each timestep, predicts a denoised latent, and blends the known and unknown regions. InverFill changes the initialization: instead of starting from random Gaussian noise, it uses the inverted latent and then applies Re-Blending,

$M \in \{0,1\}^{H \times W \times C}$4

before feeding the result into the few-step generator. In the paper’s description, this preserves semantic information where context matters while restoring Gaussian randomness inside the masked region (Vu et al., 24 Mar 2026).

The method is explicitly framed as a one-step alternative to iterative inversion procedures such as DDIM inversion. This gives it a different computational profile: one forward pass predicts the initial latent, avoiding multi-step latent optimization. The paper characterizes the additional runtime as almost negligible relative to the overall few-step sampling pipeline (Vu et al., 24 Mar 2026).

3. Mathematical objective and training protocol

The diffusion background is stated in standard latent form. The forward process is

$M \in \{0,1\}^{H \times W \times C}$5

with reparameterization

$M \in \{0,1\}^{H \times W \times C}$6

and the standard denoising objective

$M \in \{0,1\}^{H \times W \times C}$7

InverFill augments this with reconstruction, regularization, and adversarial distillation terms (Vu et al., 24 Mar 2026).

Because the inversion input is masked, the noise-space reconstruction is applied only on the unmasked region:

$M \in \{0,1\}^{H \times W \times C}$8

The image-latent reconstruction term is

$M \in \{0,1\}^{H \times W \times C}$9

and these are combined as

$1$0

To counter bias away from a valid Gaussian prior, the paper adds a moment-matching Gaussian regularizer. For the $1$1-th moment,

$1$2

where $1$3, and the regularizer is

$1$4

The model also adopts an adversarial distillation loss following LADD, using a frozen teacher $1$5 (Vu et al., 24 Mar 2026).

The final objective is

$1$6

with the reported weights $1$7, $1$8, $1$9, and $c$0 (Vu et al., 24 Mar 2026).

The training setup is also specified. InverFill was trained on SANA-Sprint 0.6B and SDXL-Turbo, covering both DiT-based and UNet-based architectures, on 4 NVIDIA A100 40GB GPUs for 8–10 hours, with batch size 32, AdamW, learning rate $c$1, prompts sampled from BrushData and MSCOCO, and resolution $c$2 (Vu et al., 24 Mar 2026).

4. Empirical evaluation and reported results

The evaluation uses BrushBench with 600 inpainting images and annotated masks, MagicBrush with an adapted 535-image test set for inpainting, and additional tests on FFHQ with 10K images and DIV2K with 900 images. The reported metrics are ImageReward (IR), HPS v2, Aesthetic Score (AS), and CLIP similarity, with LPIPS, SSIM, and FID used in additional experiments on FFHQ (Vu et al., 24 Mar 2026).

Representative quantitative gains reported for baseline few-step models are summarized below.

Setting	Metric changes	Result
BrushBench, SANA-Sprint 0.6B, 2 NFEs	IR, HPS, AS, CLIP	$c$3, $c$4, $c$5, $c$6
MagicBrush, SANA-Sprint 0.6B, 2 NFEs	IR, HPS, AS, CLIP	$c$7, $c$8, $c$9, $$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$0
SDXL-Turbo, 4 NFEs, BrushBench	IR, HPS, AS, CLIP	$$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$1, $$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$2, $$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$3, $$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$4

The paper further reports that SDXL-Turbo + BrushNet also improves slightly with InverFill, which is used to argue that the method can strengthen specialized inpainting pipelines as well as vanilla few-step text-to-image models (Vu et al., 24 Mar 2026).

A particularly prominent comparison is against iterative inversion. The runtime for DDIM inversion (50 steps) + SDXL-Turbo is reported as approximately 4.32 s, whereas InverFill + SDXL-Turbo is reported at approximately 0.74 s. The paper states that InverFill also scores higher across all metrics in that comparison (Vu et al., 24 Mar 2026).

The ablation on SANA-Sprint 0.6B (2 NFEs) assigns measurable value to each component. The baseline with masked-image inversion only gives IR 7.93; adding Re-Blending yields IR 11.11; adding Gaussian Reg yields IR 11.40; and adding LADD yields IR 11.65. The paper also states that the Gaussian regularizer outperforms SDS-style regularization and avoids over-regularizing away semantic structure (Vu et al., 24 Mar 2026).

5. Practical role, strengths, and limitations

InverFill is positioned as a method that does not require training inpainting models, does not require real-image supervision, does not need curated image-mask-text triplets, and avoids complex multi-stage training. Instead, it is described as a plug-in enhancement for existing few-step text-to-image models, with the principal change occurring in the initialization of the latent trajectory rather than in the denoiser architecture itself (Vu et al., 24 Mar 2026).

The reported inference overhead is small: about 0.06 s extra on SANA-Sprint and about 0.04 s extra on SDXL-Turbo. The paper therefore presents the method as suitable for low-latency applications such as photo restoration, object removal, and fast creative editing (Vu et al., 24 Mar 2026).

The main limitation explicitly identified is color inconsistency between the inpainted region and the background in some cases. This suggests that semantically aligned initialization does not fully solve global photometric matching. A plausible implication is that InverFill primarily improves the early latent trajectory and boundary harmonization, while some scene-level appearance mismatches remain dependent on the underlying few-step generator and prompt-conditioned sampling dynamics.

6. Terminological scope and related names

The term InverFill most directly denotes the 2026 diffusion-inpainting method described above (Vu et al., 24 Mar 2026). However, adjacent literature uses similar strings for substantially different problems, and the distinction is important.

Name	Domain	Paper
InverFill	Few-step diffusion inpainting	(Vu et al., 24 Mar 2026)
InvertFill	GAN inversion for image inpainting	(Yu et al., 2022)
infilling	Visual narrative generation	(Chandu et al., 2020)

“InvertFill” (Yu et al., 2022) is an encoder-based GAN inversion approach built around StyleGAN2, a pre-modulation module, the $$I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .$$5 latent space, and a Soft-update Mean Latent module. It addresses image inpainting, but its technical substrate is GAN inversion rather than few-step diffusion. By contrast, InverFill (Vu et al., 24 Mar 2026) is explicitly designed for few-step diffusion sampling and focuses on semantically aligned initial noise.

The broader term infilling in “Reading Between the Lines: Exploring Infilling in Visual Narratives” (Chandu et al., 2020) refers to predicting a missing step in a visual procedure or story from surrounding left and right context. That work concerns image–text narrative interpolation rather than image-space masked-region synthesis. Other similarly named works in the provided literature refer to tabular missing-data imputation, astronomical image mask filling, and structural or additive-manufacturing infill; these are terminologically adjacent but methodologically unrelated to diffusion inpainting (Teague, 2022, Dokkum et al., 2023, Jensen et al., 2023, Li et al., 2020, Wang et al., 2021).

Within this landscape, InverFill is best understood as a specialized diffusion-inversion technique for low-NFE inpainting: a method that replaces random initialization with semantically aligned noise so that few-step generators can preserve background context and produce more coherent masked-region synthesis (Vu et al., 24 Mar 2026).