Structure-Guided Inpainting Diffusion Model

• The paper's main contribution is a two-stage latent diffusion model that first predicts plausible edge maps before synthesizing detailed textures.
• The methodology leverages explicit structural guidance to address underconstrained geometry in large missing regions by separating structure and texture generation.
• The system enables editable intermediate edge maps for user-guided refinement and demonstrates superior performance on benchmarks compared to existing inpainting methods.

Structure-Guided Inpainting Diffusion Model denotes an image completion paradigm in which an explicit structural representation is inferred before texture synthesis, so that large missing regions are not reconstructed from appearance cues alone. In the formulation introduced by “A Structure-Guided Diffusion Model for Large-Hole Image Completion” (Horita et al., 2022), this idea is realized as SGDM, a two-stage latent diffusion system for large-hole image completion: a structure generator first predicts a plausible edge map inside the hole, and a texture generator then completes the RGB image conditioned on that edge map. The method is motivated by the observation that large-hole inpainting is underconstrained when the visible context does not sufficiently determine contours, object boundaries, or scene layout.

1. Problem formulation and structural prior

SGDM addresses the case in which the missing region is large enough that a single conditional image generator may fail to infer both geometry and appearance directly. The central claim is that diffusion-based inpainting benefits substantially from an explicit intermediate representation of scene structure. In SGDM, that representation is an edge map, not a segmentation map, depth map, or general structural token.

The image completion problem is defined over a target image $I \in \mathbb{R}^{H \times W \times 3}$, a binary mask $M \in \{0,1\}^{H \times W \times 1}$, and a masked image

$I_M = I \odot M,$

where masked regions are zero-filled. An edge image $E$ is extracted from the ground-truth image using DexiNed, and the masked edge image is

$E_M = E \odot M.$

The objective is to generate a completed image $\hat I$ by first generating a hole-filled edge image $\hat E$, then using $\hat E$ to guide texture completion.

This decomposition is meant to counter a specific failure mode of large-hole inpainting. Purely texture-oriented completion can produce semantically weak, structurally inconsistent, or contextually unrelated content when the visible boundary of the hole does not determine what should pass through it. SGDM therefore treats structure inference as a separate generative problem. The paper’s conceptual position is that large-hole completion is more manageable if plausible structure is inferred first, and only then translated into detailed appearance.

2. Cascaded latent diffusion architecture

At the architectural level, SGDM consists of two cascaded diffusion probabilistic models implemented in the latent diffusion framework (Horita et al., 2022). Both operate in latent space through a pretrained autoencoder with encoder $\mathcal E$ and decoder $\mathcal D$: $M \in \{0,1\}^{H \times W \times 1}$0 This choice is tied to the $M \in \{0,1\}^{H \times W \times 1}$1 experimental setting, where pixel-space diffusion would be expensive.

Module	Conditioning	Output
Structure generator $M \in \{0,1\}^{H \times W \times 1}$2	$M \in \{0,1\}^{H \times W \times 1}$3, $M \in \{0,1\}^{H \times W \times 1}$4, $M \in \{0,1\}^{H \times W \times 1}$5, noisy edge latent	Completed edge map $M \in \{0,1\}^{H \times W \times 1}$6
Texture generator $M \in \{0,1\}^{H \times W \times 1}$7	$M \in \{0,1\}^{H \times W \times 1}$8, $M \in \{0,1\}^{H \times W \times 1}$9, $I_M = I \odot M,$0, noisy image latent	Completed image $I_M = I \odot M,$1
Condition encoder $I_M = I \odot M,$2	Learned encoding of condition channels	Injected condition for diffusion backbone

The generators are initialized from stable-diffusion-2-1-base, with no prompt input used. Conditioning is encoded by a learned condition encoder $I_M = I \odot M,$3, trained following the conditioning mechanism of ControlNet. The paper states that each network uses a condition encoder over five channels comprising the mask, masked image, masked edge image, and hole-filled edge image; operationally, the structure and texture generators receive different subsets corresponding to their tasks. The encoded condition is injected into the diffusion backbone. The paper does not emphasize classifier-free guidance or custom attention blocks as a novel component.

The diffusion preliminaries follow standard DDPM notation: $I_M = I \odot M,$4

$I_M = I \odot M,$5

$I_M = I \odot M,$6

Training uses the usual denoising score-matching objective,

$I_M = I \odot M,$7

so both generators predict additive noise rather than the clean signal directly.

For the structure generator, the training loss is

$I_M = I \odot M,$8

For the texture generator, the corresponding objective is

$I_M = I \odot M,$9

During individual texture training, the condition edge image is the complete ground-truth edge image $E$0; at test time it is replaced with the generated $E$1.

3. Joint training and optimal Bayesian denoising

The technically distinctive element of SGDM is not only the edge-then-texture factorization, but the method used to jointly train the two diffusion models (Horita et al., 2022). A direct cascade is difficult because the structure generator naturally outputs a noisy latent at some diffusion timestep, whereas the texture generator expects a clean edge condition. Running a full reverse diffusion chain inside training would be computationally impractical and would obstruct gradient flow through the cascade.

To avoid that, the paper introduces optimal Bayesian denoising, specifically Tweedie’s formula, as a differentiable one-step denoising operator. Starting from

$E$2

the paper derives

$E$3

Applied to the noisy structure latent, this yields

$E$4

The role of $E$5 is operational rather than merely interpretive. It serves as a one-step denoised estimate of the clean edge sample that can be passed to the texture generator during training without unrolling the entire structure-sampling trajectory. Because the operator is differentiable, gradients from the texture loss can propagate back into the structure generator. This is the mechanism that enables end-to-end fine-tuning of the cascade.

The total joint objective is

$E$6

Here, $E$7 conditions the texture generator on the original ground-truth edge image $E$8, while $E$9 conditions it on the edge produced by the structure generator and denoised by Tweedie’s formula. The paper assigns distinct functions to these two terms. $E_M = E \odot M.$0 encourages the structure generator to produce edges that are useful for texture synthesis, effectively learning texture-aware edge prediction. $E_M = E \odot M.$1 regularizes the texture generator so that it does not overfit to imperfect generated edges.

The training protocol is explicitly stage-wise: both generators are first trained individually, then jointly fine-tuned. The paper explicitly states that joint training is done after individual training, not from scratch. It also states no alternating optimization and no explicit scalar weighting coefficients beyond the additive combination shown above.

4. Sampling, diversity, and editable control

At inference time, SGDM proceeds in two stochastic stages (Horita et al., 2022). Given $E_M = E \odot M.$2, $E_M = E \odot M.$3, and $E_M = E \odot M.$4, the structure generator starts from Gaussian noise in latent edge space and iteratively reverse-denoises under conditioning on $E_M = E \odot M.$5 to produce a completed edge map $E_M = E \odot M.$6. The texture generator then starts from Gaussian noise in latent image space and reverse-denoises under conditioning on $E_M = E \odot M.$7 to produce $E_M = E \odot M.$8. The paper uses the RePaint sampler for image generation.

Because both stages are stochastic diffusion samplers, different random seeds yield multiple plausible completions for the same masked input. Diversity is therefore an explicit property of the method rather than a side effect. This distinguishes SGDM from deterministic inpainting pipelines whose output is effectively single-valued once the mask and visible context are fixed.

A second important property is controllability through the intermediate edge representation. The paper emphasizes that the generated edge map is editable before texture synthesis. A user can manually modify edges inside the hole and then pass the edited structure to the texture generator. The paper further proposes refining user-edited edges with SDEdit under the structure generator’s prior: the edited edge map is perturbed with Gaussian noise at a chosen timestep, then reverse-diffused to obtain a refined edge compatible with the model prior. The examples use timestep 500 and 200 out of 1000. Raw user-edited edges can also be used directly, but the SDEdit refinement is reported to improve robustness.

This editable intermediate structure yields a sketch-guided editing interface. The paper also shows some simple language-guided editing, but it states that this is not a trained capability of the model. That distinction matters: SGDM is not a prompt-conditioned inpainting system, and its structural guidance is edge-based rather than text-based.

5. Experimental evaluation

SGDM is evaluated on Places and CelebA-HQ, both at $E_M = E \odot M.$9 resolution (Horita et al., 2022). Places uses an 8M-image training set and a 5,000-image test set; CelebA-HQ uses 24,183 training images and 2,993 test images. Following MAT, evaluation uses small and large masks. Masks are generated from random rectangles and random strokes; the training protocol creates zero to five full-size or half-size rectangles and zero to nine random strokes with width 12–48 and 4–18 vertices. The evaluation metrics are FID, P-IDS, and U-IDS; the ablation study also reports LPIPS.

The implementation details are also specific. Each network is individually trained for 25M images on both datasets, followed by 1M images of joint training. Optimization uses AdamW with $\hat I$0, $\hat I$1, learning rate $\hat I$2, batch size 1, and four NVIDIA A100 GPUs.

The baseline set spans diffusion, GAN, and autoregressive systems: Stable Diffusion inpainting, LDM, MAT, MISF, CoordFill, ZITS, MAE-FAR, LaMa, CoModGAN, and PUT. Among these, only ZITS also uses edge guidance, but unlike SGDM it is not diffusion-based and is described as struggling on large holes.

Quantitatively, the paper reports especially strong results on CelebA-HQ. SGDM achieves the best result on all reported metrics for both small and large masks. Under large masks on CelebA-HQ, SGDM obtains FID 4.72, P-IDS 13.99, and U-IDS 24.97, while MAT reports 5.04 / 11.42 / 24.13. On Places, SGDM achieves the best FID in both small and large mask settings and competitive or second-best P-IDS/U-IDS relative to MAT. For large masks on Places, SGDM reports FID 6.96, compared with MAT’s 7.11, while P-IDS/U-IDS are 18.12 / 31.78 for SGDM and 18.40 / 32.46 for MAT.

The ablation study isolates the roles of joint training and one-step denoising. On Places with large masks, individual training alone yields FID 32.28 and LPIPS 0.188. Adding joint training without optimal Bayesian denoising improves this to 28.68 / 0.175. Joint training with the denoising operator further improves performance to 27.47 / 0.170 with 0.1M joint-training samples, and 27.81 / 0.168 with 1M samples. The paper interprets this as evidence that Tweedie-based one-step denoising is important not only for computational convenience but also for making generated edge conditions semantically usable during end-to-end training.

Qualitatively, SGDM is reported to produce more plausible object boundaries and scene geometry inside large missing regions. The paper contrasts this with standard diffusion inpainting, which can generate content unrelated to context, and with GAN-based methods that may look sharp but distort structure. The reported visual strengths include coherent humans, furniture, facial structure, and scene layout, together with consistent boundary continuation between visible and completed regions.

6. Significance, related formulations, and limitations

The paper identifies three contributions: a diffusion-based structure-guided inpainting framework that uses a generated edge image to guide image completion, a joint-training method for cascaded diffusion models using optimal Bayesian denoising via Tweedie’s formula, and empirical performance on scene and face completion together with editable edge control (Horita et al., 2022). The authors claim that the framework is the first explicit combination of structural generation and diffusion for image completion.

Within the broader literature, SGDM exemplifies one specific interpretation of “structure-guided” diffusion. In StrDiffusion, structure guidance is formulated as a separate structure diffusion process whose denoised structure state guides texture denoising through a time-dependent sparse representation rather than through an explicit predicted edge map (Liu et al., 2024). In GSDM for text image restoration, the guiding structure is a predicted full foreground segmentation mask, and the model is designed for blind text inpainting where the corruption mask is not assumed at test time (Zhu et al., 2024). A later inpainting method conditions diffusion on a dense multi-scale anisotropic Gaussian splat map derived analytically from gradients and edge information around the hole, rather than on a predicted edge completion network (Fein-Ashley et al., 2024). This suggests that “structure-guided inpainting diffusion model” denotes a family of conditioning strategies rather than a single canonical architecture.

SGDM’s strengths are stated directly. It addresses structural ambiguity in large holes by introducing an explicit edge prior; it retains the diversity and high-fidelity synthesis capacity of diffusion models; and it exposes a natural editing interface at the intermediate edge level. A plausible implication is that the method is especially suited to settings where contour continuity is a more stable prior than direct texture extrapolation.

Its limitations are equally explicit. The structure generator can still fail to produce rational edges in difficult cases, although users may manually correct them. The system is computationally expensive because it performs iterative denoising in two stages: the paper reports about 133 seconds for a center-masked image with the RePaint sampler, compared with 0.098 s for MAT and 4.06 s for PUT. The approach also depends on edge extraction and on the assumption that edges are an effective structural representation for the target domain. Finally, while the paper shows some simple text-guided editing, the model is not trained for prompt conditioning and may fail with complex prompts.

In that sense, SGDM is best understood not as a generic multimodal inpainting framework, but as a precise two-stage latent diffusion design in which explicit edge generation mediates the transition from uncertain large-hole geometry to final image completion. Its enduring significance lies in making structure a first-class generative variable inside diffusion-based inpainting rather than leaving it implicit in texture synthesis alone.