AGInpaint: Automated Generative Inpainting

• AGInpaint is an automated, guidance-driven generative inpainting technique that replaces masked regions while preserving the structural consistency of the visible content.
• It combines automated mask generation, monocular depth estimation, and pretrained diffusion models to harmonize background, attire, and overall scene geometry.
• The approach is applied in portrait editing for identity preservation and in 3D scene synthesis via appearance-guided inpainting for enhanced cross-view consistency.

Searching arXiv for recent and directly relevant papers on AGInpaint and adjacent inpainting systems. AGInpaint denotes a class of automated or guidance-driven generative inpainting procedures whose common objective is to replace masked image content while preserving structural consistency with the visible context. In one usage, it refers to a fully automated inference-time pipeline for attire and background replacement in photographs of people, combining monocular depth estimation, automatic mask generation, Stable Diffusion XL, Latent Consistency Models, and SDXL inpainting without manual mask drawing (Perche-Mahlow et al., 2024). In another usage, it denotes Appearance-Guided Inpainting, a training-free modification of latent diffusion sampling inside the MeSS scene-generation system, where known RGB values in mesh-rendered views explicitly guide hole filling and Gaussian field densification (Chen et al., 21 Aug 2025). Taken together, these usages associate AGInpaint with automated perception, structured masking, and diffusion-based image completion under either prompt-driven or appearance-guided control.

1. Terminology and conceptual scope

In the portrait-editing literature, AGInpaint is presented as an automated, generative inpainting pipeline that takes a photo of a person and, with minimal user effort, changes both clothing and background in a coherent, photorealistic way. The defining characteristics are automatic foreground extraction, face preservation, prompt-conditioned synthesis, and a two-stage edit order in which the background is replaced before attire is inpainted (Perche-Mahlow et al., 2024).

In MeSS, AGInpaint has a narrower and explicitly named meaning: Appearance-Guided Inpainting. There it is not a separate neural network but a guided sampling strategy layered on top of an LCM-based latent diffusion inpainter. Its role is to inpaint missing or low-quality regions in intermediate, mesh-rendered views while forcing the generated content to match the appearance of surrounding known pixels (Chen et al., 21 Aug 2025).

These two usages place the term at two different levels of abstraction. In the first, AGInpaint denotes an end-to-end editing pipeline for people-centric image manipulation. In the second, it denotes a sampling-time guidance mechanism embedded in a larger 3D scene-synthesis system. This suggests that AGInpaint is best understood not as a single canonical architecture but as a label for systems that combine automatic scene understanding with structured generative inpainting.

2. Automated attire and background replacement

The portrait-editing pipeline begins from a single input photograph and is designed to produce an edited image in which the background is replaced, clothing is changed, the face is preserved, and no manual masks are required. The user provides a background prompt $B_P$, a clothing prompt $C_P$, and a depth threshold $T_h$, while the system performs depth estimation, mask creation, background generation, compositing, and clothing inpainting entirely at inference time (Perche-Mahlow et al., 2024).

Its principal stages are: MiDaS monocular depth estimation; depth-threshold mask construction; face exclusion via OpenCV Haar Cascade; morphological cleaning; SDXL-based background generation accelerated by Latent Consistency Models; foreground-background compositing; and attire replacement with the specialized SDXL inpainting model "stable-diffusion-xl-1.0-inpainting-0.1". Background generation uses an LCM-distilled UNet with LCMScheduler, allowing sampling in 2–4 steps. Clothing replacement is then driven by $C_P$ together with guidance_scale, num_inference_steps, strength, and generator for seed control. The pipeline performs no training or fine-tuning; it is an orchestration of existing pretrained modules (Perche-Mahlow et al., 2024).

A central design choice is the ordering of operations. Background replacement occurs before clothing inpainting, because the inpainting model can then harmonize attire with the new environment. The reported effect is background–clothing style coherence: painterly backgrounds tend to induce painterly attire, while photorealistic backgrounds tend to induce more realistic clothing. The same mechanism also produces broader aesthetic changes, including changes in palette and reinterpretation of nearby objects, without any explicit style-transfer loss.

The control surface is deliberately small. Users upload a photograph, specify $B_P$, $C_P$, and optionally adjust $T_h$ and advanced inpainting settings. This places AGInpaint, in this sense, at the intersection of prompt-based editing and automatic mask generation.

3. Depth-based mask formation and identity preservation

The mask-generation mechanism is the core automation step in the attire/background pipeline. MiDaS produces a dense monocular depth map $D$, which is thresholded to form a binary foreground mask $M$:

$$M_i = \begin{cases} 1, & \text{if } D_i > \text{Threshold} \ 0, & \text{otherwise} \end{cases}$$

Pixels with greater depth values are treated as subject or foreground, while lower values are treated as background. The threshold $C_P$0 is tuned per image and can be used to include nearby objects, such as an umbrella, in the preserved foreground region (Perche-Mahlow et al., 2024).

Face preservation is enforced after thresholding. The image is converted to grayscale, faces are detected with the Viola–Jones Haar Cascade classifier, and an elliptical preserve mask is drawn slightly larger than each detected face. Morphological operations, including opening and dilation/erosion with a kernel, clean noisy regions and smooth edges. The resulting binary mask preserves body and clothing regions while explicitly excluding the face from inpainting. In the attire stage, white areas designate regions to modify and black areas designate regions to preserve.

Background compositing is then expressed as a direct mask-controlled replacement:

$C_P$1

This mechanism is structurally simple, but it replaces manual mask authoring with automatic subject extraction and identity preservation. No ControlNet, segmentation map, or additional structural guide is used; depth and the derived mask are the sole structural control in this pipeline (Perche-Mahlow et al., 2024).

4. Appearance-Guided Inpainting in MeSS

Within MeSS, AGInpaint is the Stage II component of a three-stage pipeline consisting of sparse Gaussian field construction from key views, Gaussian field densification, and global consistency alignment. Sparse key views are first generated via Cascaded Outpainting ControlNets and lifted onto a city mesh as Gaussian surfels. Intermediate novel views rendered from this coarse Gaussian scene exhibit holes, silhouettes, blurry textures, and failures on slim structures such as poles and signs. AGInpaint fills these regions and projects the newly synthesized pixels back onto the mesh as additional Gaussians, thereby densifying the 3D Gaussian field (Chen et al., 21 Aug 2025).

The method is built on Latent Consistency Models. For latent $C_P$2, conditioning $C_P$3, and timestep $C_P$4, the consistency function satisfies

$C_P$5

In $C_P$6-prediction form, the paper gives

$C_P$7

AGInpaint modifies the resulting noise-free estimate by forcing agreement with known pixels in the unmasked region $C_P$8. If $C_P$9 is the current estimate and $T_h$0 is the target RGB image in known regions, the guidance loss is

$T_h$1

with gradient

$T_h$2

The guidance is repeated $T_h$3 times at each diffusion step with adaptive step size

$T_h$4

and updated estimate

$T_h$5

The final latent update uses the guided estimate in a DDPM/DDIM-style step:

$T_h$6

The role of this guidance is not merely local hole filling. Because known pixels come from a mesh-aligned Gaussian scene, enforcing their RGB values stabilizes exposure, texture, and color across intermediate views. MeSS therefore uses AGInpaint as the bridge between sparse key-view generation and dense, cross-view consistent scene reconstruction (Chen et al., 21 Aug 2025).

5. Relation to adjacent inpainting paradigms

AGInpaint occupies a broader research landscape defined by three recurrent problems: how masks are obtained, how geometry is preserved, and how generation is controlled. Several adjacent systems illuminate these axes.

Mask acquisition and interaction. "Inpaint Anything" introduces a “clicking and filling” paradigm in which Segment Anything produces masks from point prompts, after which LaMa or Stable Diffusion performs removal, filling, or background replacement. It explicitly characterizes this as a mask-free user experience in the sense that masks are used internally but not manually drawn by the user (Yu et al., 2023). The attire/background form of AGInpaint differs in that it automates mask generation from depth and face detection rather than click-based segmentation.

Geometry-aware inpainting. "Layout Aware Inpainting for Automated Furniture Removal in Indoor Scenes" addresses large indoor masks by using instance segmentation, PlaneRCNN, per-plane inpainting, automatic rectification, and texture refinement. Its central claim is that generic inpainting fails on wide-angle indoor scenes because it lacks explicit layout and plane reasoning (Kulshreshtha et al., 2022). MeSS-era AGInpaint advances a related principle in a different setting: geometry is imposed by a city mesh and Gaussian field rather than room planes, and appearance guidance is used to keep the inpainted content aligned with that geometry.

Hybrid speed–quality trade-offs. "DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs" proposes a two-stage hybrid in which a diffusion model denoises the image and a GAN generator performs the inpainting in a single forward pass. The paper emphasizes mask-agnostic generalization to arbitrary masks and reduced sampling cost relative to full DDPM sampling (Heidari et al., 2023). This differs from both principal AGInpaint usages: the portrait pipeline relies on pretrained diffusion modules only, while MeSS modifies LCM sampling directly rather than appending a GAN refinement stage.

Controllable semantics. "Interactive Image Inpainting Using Semantic Guidance" separates prior reconstruction from semantic control through a Stage 1 ESPA autoencoder, automatic semantic-mask derivation, and a SPADE-based semantic decoder guided by a user-edited mask (Yu et al., 2022). A plausible implication is that AGInpaint-like systems can be organized around a similar separation: perception and prior recovery in one stage, explicit semantic or appearance guidance in another.

Across these neighboring approaches, AGInpaint aligns most strongly with automated masking, structured conditioning, and modular composition of pretrained components.

6. Evaluation, limitations, and defensive implications

The attire/background pipeline is evaluated qualitatively on Pixabay images covering varied ages, genders, skin tones, indoor and outdoor backgrounds, single- and multiple-person scenarios, and some portrait-mode images. The reported observations are that clothing style matches theme prompts, background and attire are stylistically harmonized, and portrait-mode blurred backgrounds work particularly well for depth estimation. The paper explicitly reports no quantitative metrics such as FID or CLIP score and no ablation studies. Its stated limitations include failures when the subject is fused with the background, reduced robustness for multiple persons, difficulty with hands, feet, complex poses, and difficulty generating accurate afro-textured hair and loose hair; it also notes that the inpainting model often produces a slimming effect that may inadvertently reinforce body stereotypes (Perche-Mahlow et al., 2024).

MeSS provides a more explicit ablation for AGInpaint. Replacing AGInpaint with a Resample strategy yields LPIPS 0.422, FID 51.12, and KID 0.0459. The paper states that enabling AGInpaint significantly improves LPIPS and FID/KID, and qualitatively reports that Resample produces visible streak patterns and struggles to fill slim regions because mask downsampling in latent space loses detail (Chen et al., 21 Aug 2025). The limitation profile is therefore different from the portrait pipeline: the central concern is not facial identity or clothing realism but geometric fidelity and cross-view consistency under mesh-guided rendering.

The maturation of AGInpaint-like capabilities has also produced a defensive counter-literature. "Anti-Inpainting: A Proactive Defense against Malicious Diffusion-based Inpainters under Unknown Conditions" treats diffusion-based inpainting as a security threat and proposes a defense based on a multi-level deep feature extractor, multi-scale semantic-preserving data augmentation, and a selection-based distribution deviation optimization strategy. It is explicitly aimed at conditions where the attacker may choose unknown masks, prompts, and seeds (Guo et al., 19 May 2025). This indicates that advanced inpainting is now understood not only as a creative or reconstructive technology but also as a manipulation primitive requiring proactive safeguards.

Taken as a whole, the literature presents AGInpaint as a technically heterogeneous but coherent research theme: automatic perception defines editable structure; diffusion-based priors synthesize missing content; and additional guidance—prompt-based, semantic, geometric, or appearance-based—constrains the synthesis toward coherence with either user intent or known scene evidence.