---
title: 'InverFill: Enhanced Few-Step Diffusion Inpainting'
url: https://www.emergentmind.com/topics/inverfill
type: topic
---

# InverFill: Enhanced Few-Step Diffusion Inpainting

Searching arXiv for the cited InverFill paper and closely related inpainting work.
InverFill is a one-step inversion method for diffusion-based image inpainting that targets the failure mode of naive few-step text-to-image sampling under masking constraints. It was introduced as “InverFill: One-Step Inversion for Enhanced Few-Step Diffusion Inpainting” [2603.23463]. The method begins from the diagnosis that few-step inpainting degrades primarily because random Gaussian noise initialization provides no semantic information about the visible context, and that, under low numbers of function evaluations (NFEs), the denoising process has too little capacity to repair this mismatch. InverFill therefore injects semantic information from the input masked image into the initial noise, and uses this semantically aligned noise inside a blended sampling pipeline. The reported result is improved harmonization, fidelity, and text coherence for few-step inpainting, without training specialized inpainting models, without real-image supervision, and with minimal inference overhead [2603.23463].

## 1. Problem formulation and failure mode

InverFill addresses masked image completion in the standard latent-diffusion setting. The inputs are a masked image $I_m \in \mathbb{R}^{H \times W \times C}$, a binary mask $M \in \{0,1\}^{H \times W \times C}$ where $1$ indicates missing pixels, and a text prompt $c$. The final composite is written as
$$
I = I_m \odot (1 - M) + I_{\text{inpaint}} \odot M .
$$
The stated objective is to produce a result that is semantically aligned with the prompt, visually consistent with the visible context, and harmonized in style, lighting, color, and texture [2603.23463].

The central diagnosis is specific to few-step diffusion. Standard blended sampling can preserve known regions while denoising the unknown region, but it relies on a denoising trajectory that starts from pure Gaussian noise. In many-step samplers this mismatch can be gradually corrected. In few-step models, however, each denoising update is large, and the model has only $2$–$4$ steps in which to reconcile the latent with the unmasked background. The paper therefore attributes visible seams, inconsistent style, reduced harmonization, blur, and prompt drift to the combination of random Gaussian initialization and low NFEs [2603.23463].

This diagnosis narrows the problem from generic inpainting difficulty to what the paper presents as a starting-point problem. Rather than treating harmonization errors as a purely architectural deficiency of few-step generators, InverFill treats them as a consequence of initializing the reverse process from a latent that is semantically unrelated to the observed image content.

## 2. One-step inversion and semantically aligned initialization

The core mechanism is a one-step inversion network that maps the masked image into an initial latent better aligned with the visible image context. Let $z_0 = \mathcal{E}(I_{gt})$ be the latent of the ground-truth image, $z_0^m = \mathcal{E}(I_m)$ the latent of the masked image, $\epsilon \sim \mathcal{N}(0,I)$ Gaussian noise, $F_\theta$ the inversion network, and $G$ a pretrained one-step text-to-image generator. The inversion network predicts
$$
\hat{z}_T = F_\theta(z_0^m, c),
$$
and the generator reconstructs
$$
\hat{z}_0 = G(\hat{z}_T, c).
$$
The paper states that the goal is not exact pixel recovery but rather a latent whose denoising trajectory preserves the visible background and yields a plausible completion of the masked region [2603.23463].

InverFill is embedded into a blended sampling procedure. Vanilla blended sampling encodes the masked image, diffuses the known region forward at each timestep, predicts a denoised latent, and blends the known and unknown regions. InverFill changes the initialization: instead of starting from random Gaussian noise, it uses the inverted latent and then applies **Re-Blending**,
$$
\hat{z}_T^{blend} = \hat{z}_T \odot (1-m) + \epsilon' \odot m,\quad \epsilon' \sim \mathcal{N}(0,I),
$$
before feeding the result into the few-step generator. In the paper’s description, this preserves semantic information where context matters while restoring Gaussian randomness inside the masked region [2603.23463].

The method is explicitly framed as a one-step alternative to iterative inversion procedures such as DDIM inversion. This gives it a different computational profile: one forward pass predicts the initial latent, avoiding multi-step latent optimization. The paper characterizes the additional runtime as almost negligible relative to the overall few-step sampling pipeline [2603.23463].

## 3. Mathematical objective and training protocol

The diffusion background is stated in standard latent form. The forward process is
$$
q(z_t \mid z_{t-1}) = \mathcal{N}(z_t; \sqrt{1-\beta_t}z_{t-1}, \beta_t I),
$$
with reparameterization
$$
z_t = \sqrt{1-\beta_t}z_{t-1} + \sqrt{\beta_t}\epsilon,\quad \epsilon \sim \mathcal{N}(0, I),
$$
and the standard denoising objective
$$
\min_\theta \mathbb{E}\left[\|\epsilon - \epsilon_\theta(z_t, t, c)\|_2^2\right].
$$
InverFill augments this with reconstruction, regularization, and adversarial distillation terms [2603.23463].

Because the inversion input is masked, the noise-space reconstruction is applied only on the unmasked region:
$$
\mathcal{L}_{\text{noise}} = \left\|(1-m)\odot \hat{z}_T - (1-m)\odot \epsilon\right\|_2^2 .
$$
The image-latent reconstruction term is
$$
\mathcal{L}_{\text{image}} = \|\hat{z}_0 - z_0\|_2^2 ,
$$
and these are combined as
$$
\mathcal{L}_{\text{recons}} = \lambda_{\text{noise}}\mathcal{L}_{\text{noise}} + \lambda_{\text{image}}\mathcal{L}_{\text{image}} .
$$

To counter bias away from a valid Gaussian prior, the paper adds a moment-matching Gaussian regularizer. For the $n$-th moment,
$$
\mathcal{L}_n = \left\| \left| \frac{1}{D}\sum_{k=1}^{D} (\hat{z}_T^{blend})^n \right|^{\frac{1}{n}} - \mu_n^{\frac{1}{n}} \right\|,
$$
where $D = c \times h \times w$, and the regularizer is
$$
\mathcal{L}_{\text{reg}} = \sum_{n \in \{1,2\}} \mathcal{L}_n .
$$
The model also adopts an adversarial distillation loss following LADD, using a frozen teacher $G_{\text{pre}}$ [2603.23463].

The final objective is
$$
\mathcal{L}_{\text{final}} = \lambda_{\text{recons}}\mathcal{L}_{\text{recons}} + \lambda_{\text{reg}}\mathcal{L}_{\text{reg}} + \lambda_{\text{adv}}\mathcal{L}_{\text{adv}} ,
$$
with the reported weights
$\lambda_{\text{noise}} = 1.0$,
$\lambda_{\text{image}} = 1.0$,
$\lambda_{\text{reg}} = 0.5$,
and
$\lambda_{\text{adv}} = 0.5$ [2603.23463].

The training setup is also specified. InverFill was trained on **SANA-Sprint 0.6B** and **SDXL-Turbo**, covering both **DiT-based** and **UNet-based** architectures, on **4 NVIDIA A100 40GB GPUs** for **8–10 hours**, with **batch size 32**, **AdamW**, learning rate **$1 \times 10^{-5}$**, prompts sampled from **BrushData** and **MSCOCO**, and resolution **$1024^2$** [2603.23463].

## 4. Empirical evaluation and reported results

The evaluation uses **BrushBench** with 600 inpainting images and annotated masks, **MagicBrush** with an adapted 535-image test set for inpainting, and additional tests on **FFHQ** with 10K images and **DIV2K** with 900 images. The reported metrics are **ImageReward (IR)**, **HPS v2**, **Aesthetic Score (AS)**, and **CLIP similarity**, with **LPIPS**, **SSIM**, and **FID** used in additional experiments on FFHQ [2603.23463].

Representative quantitative gains reported for baseline few-step models are summarized below.

| Setting | Metric changes | Result |
|---|---|---|
| BrushBench, SANA-Sprint 0.6B, 2 NFEs | IR, HPS, AS, CLIP | $11.02 \rightarrow 11.65$, $26.21 \rightarrow 27.93$, $6.05 \rightarrow 6.15$, $27.12 \rightarrow 27.17$ |
| MagicBrush, SANA-Sprint 0.6B, 2 NFEs | IR, HPS, AS, CLIP | $2.55 \rightarrow 3.04$, $25.07 \rightarrow 25.37$, $5.32 \rightarrow 5.42$, $25.67 \rightarrow 25.71$ |
| SDXL-Turbo, 4 NFEs, BrushBench | IR, HPS, AS, CLIP | $11.42 \rightarrow 12.38$, $28.20 \rightarrow 28.44$, $6.06 \rightarrow 6.08$, $27.26 \rightarrow 27.67$ |

The paper further reports that **SDXL-Turbo + BrushNet** also improves slightly with InverFill, which is used to argue that the method can strengthen specialized inpainting pipelines as well as vanilla few-step text-to-image models [2603.23463].

A particularly prominent comparison is against iterative inversion. The runtime for **DDIM inversion (50 steps) + SDXL-Turbo** is reported as approximately **4.32 s**, whereas **InverFill + SDXL-Turbo** is reported at approximately **0.74 s**. The paper states that InverFill also scores higher across all metrics in that comparison [2603.23463].

The ablation on **SANA-Sprint 0.6B (2 NFEs)** assigns measurable value to each component. The baseline with masked-image inversion only gives **IR 7.93**; adding **Re-Blending** yields **IR 11.11**; adding **Gaussian Reg** yields **IR 11.40**; and adding **LADD** yields **IR 11.65**. The paper also states that the Gaussian regularizer outperforms SDS-style regularization and avoids over-regularizing away semantic structure [2603.23463].

## 5. Practical role, strengths, and limitations

InverFill is positioned as a method that does **not** require training inpainting models, does **not** require real-image supervision, does **not** need curated image-mask-text triplets, and avoids complex multi-stage training. Instead, it is described as a plug-in enhancement for existing few-step text-to-image models, with the principal change occurring in the initialization of the latent trajectory rather than in the denoiser architecture itself [2603.23463].

The reported inference overhead is small: about **0.06 s** extra on **SANA-Sprint** and about **0.04 s** extra on **SDXL-Turbo**. The paper therefore presents the method as suitable for low-latency applications such as photo restoration, object removal, and fast creative editing [2603.23463].

The main limitation explicitly identified is **color inconsistency** between the inpainted region and the background in some cases. This suggests that semantically aligned initialization does not fully solve global photometric matching. A plausible implication is that InverFill primarily improves the early latent trajectory and boundary harmonization, while some scene-level appearance mismatches remain dependent on the underlying few-step generator and prompt-conditioned sampling dynamics.

## 6. Terminological scope and related names

The term **InverFill** most directly denotes the 2026 diffusion-inpainting method described above [2603.23463]. However, adjacent literature uses similar strings for substantially different problems, and the distinction is important.

| Name | Domain | Paper |
|---|---|---|
| InverFill | Few-step diffusion inpainting | [2603.23463] |
| InvertFill | GAN inversion for image inpainting | [2208.11850] |
| infilling | Visual narrative generation | [2010.13944] |

“**InvertFill**” [2208.11850] is an encoder-based **GAN inversion** approach built around **StyleGAN2**, a **pre-modulation** module, the **$\mathcal{F}\mathcal{W}^+$ latent space**, and a **Soft-update Mean Latent** module. It addresses image inpainting, but its technical substrate is GAN inversion rather than few-step diffusion. By contrast, InverFill [2603.23463] is explicitly designed for few-step diffusion sampling and focuses on semantically aligned initial noise.

The broader term **infilling** in “Reading Between the Lines: Exploring Infilling in Visual Narratives” [2010.13944] refers to predicting a missing step in a visual procedure or story from surrounding left and right context. That work concerns image–text narrative interpolation rather than image-space masked-region synthesis. Other similarly named works in the provided literature refer to tabular missing-data imputation, astronomical image mask filling, and structural or additive-manufacturing infill; these are terminologically adjacent but methodologically unrelated to diffusion inpainting [2202.09484; 2312.03064; 2307.09518; 2007.00413; 2108.09675].

Within this landscape, InverFill is best understood as a specialized diffusion-inversion technique for low-NFE inpainting: a method that replaces random initialization with semantically aligned noise so that few-step generators can preserve background context and produce more coherent masked-region synthesis [2603.23463].

Source: https://www.emergentmind.com/topics/inverfill