Mask-Guided Generative Completion

• The paper surveys mask-guided generative completion, detailing how explicit mask injection strategies direct synthesis to reconstruct occluded or missing content.
• It outlines innovative network architectures, including multi-branch encoders and gated convolutions, that integrate mask cues with RGB, depth, and semantic inputs.
• Empirical results demonstrate improved perceptual quality and semantic consistency when employing mask-conditioned losses, while noting challenges like segmentation dependency and inference speed.

Mask-guided generative completion encompasses a class of methods in which pixel-wise or region-wise mask information directly modulates the generative process to complete or reconstruct missing, occluded, or degraded content. Techniques span modalities—RGB image, depth, and video—and exploit mask-conditioned spatial priors to focus synthesis and enhance controllability or fidelity. Recent literature demonstrates the efficacy of mask-guided strategies for inpainting, amodal completion, depth map restoration, object editing, and more. This article surveys the methodological frameworks, network architectures, encoding strategies, loss functions, and empirical findings associated with state-of-the-art mask-guided generative completion systems.

1. Mask-Guided Conditioning Paradigms

Mask-guidance refers to any strategy that uses an explicit mask as an input or conditioning signal to isolate target regions for generation or restoration. The mask may represent occlusions, missing pixels, instance masks (e.g., transparent regions (Cheng et al., 4 Aug 2025)), weighted attention (e.g., visible/invisible/background (Saleh et al., 2024)), or object part segmentation.

Mechanisms differ in how mask information is injected:

• Direct Concatenation: Most generative networks concatenate the binary or weighted mask channel-wise with other input modalities (e.g., RGB, raw depth, attributes) before the first convolutional or transformer layer, thus rendering the mask spatially explicit throughout the feature hierarchy (Cheng et al., 4 Aug 2025, Saleh et al., 2024, Saxena et al., 2023, Chen et al., 2018).
• Conditioned Gated Convolutions: Weighted masks enter as gating signals—multiplicatively modulating convolution outputs at every layer (not only the input) to enable selective spatial propagation of features, as in mask-guided gated convolution (Saleh et al., 2024).
• Cross-Attention and Diffusion Constraints: In diffusion or transformer-based models, mask channels steer denoising or cross-attention so that generation is strictly localized to masked (unknown or occluded) regions, while preserving the original content in known regions (Fan et al., 22 Sep 2025, Xu et al., 2023, Li et al., 2023).
• Mask-Attention Mechanisms: Early injection of instance or part masks into transformer encoders induces the backbone to focus on ambiguous or uncertain regions, particularly where measurements are missing (e.g., transparent object completion (Cheng et al., 4 Aug 2025)).

In all cases, the mask acts as a spatial prior—informing the generator "where to hallucinate" versus "where to preserve"—and can enable fine control for localized editing, inpainting, or attribute transfer.

2. Network Architectures and Mask Integration

Mixed approaches combine mask guidance with auxiliary cues (semantic segmentation, depth priors, geometry), using architectural motifs such as:

• Multi-Branch Encoders: Separate encoders for different modalities (RGB, depth, monocular estimate, mask) whose features are fused downstream, as in mask-attention + depth + monocular branches for depth completion (Cheng et al., 4 Aug 2025).
• Gated/Weighted Mask Pipelines: Layer-wise concatenation or gating of feature maps with weighted masks enables the network to prioritize visible (object) regions and minimize reliance on ambiguous contexts (Saleh et al., 2024).
• Cascade and Feedback Loops: Iterative methods alternate generation and mask refinement, employing segmentation feedback to progressively refine the inpainting mask (iterative mask denoising) and thus improve object completion (Li et al., 2023).
• Conditional Diffusion with ControlNet/Transformer Priors: Textual and mask conditioning via cross-attention or classifier-free guidance dominates recent large inpainting and amodal synthesis systems. Hierarchical strategies use boundary expansion masks and explicit multi-agent reasoning for compositional object synthesis (Fan et al., 22 Sep 2025).
• Self-Supervised and Weakly-Supervised Protocols: Self-supervised overlays generate synthetic occlusions for training, enabling mask-guided generative models to generalize from paired or unpaired real data (Saleh et al., 2024).

A representative high-level pipeline for mask-guided depth completion (Cheng et al., 4 Aug 2025):

Input Modalities	Encoder	Mask Injection	Feature Fusion/Decoding
RGB + instance mask	Swin-Transformer	Mask at input	Residual-MLP decoder
Monocular depth	Swin-Transformer	None
Raw depth	MLP	None

Such hybrid architectures allow explicit spatial supervision and context preservation, realizing higher fidelity and generalizability.

3. Loss Functions and Training Supervision

Mask-guided generative systems employ loss functions that are spatially modulated by the mask, often complemented by adversarial or perceptual objectives:

• Region-Restricted Losses: Reconstruction loss restricted to masked/unmasked regions—either mask-only losses or global losses, depending on whether the supervision should focus exclusively on unknown areas or trade-off with global consistency (Cheng et al., 4 Aug 2025, Saxena et al., 2023, Xu et al., 2023).

$$\mathcal{L}_{mask} = \frac{1}{|\mathcal{M}|} \sum_{(u, v) \in \mathcal{M}} |D_o(u,v) - D_{gt}(u,v)|$$

where $\mathcal{M}$ is the set of masked pixels.

• Adversarial Losses: Patch-level, global, or local GAN discriminators drive the realism of both local completions and whole-image compositions. Hinge loss, WGAN-GP, or PatchGAN variants are common (Saleh et al., 2024, Saxena et al., 2023, Li et al., 2017).
• Perceptual/Feature Losses: Deep feature matching (e.g., via VGG activations or CLIP embeddings) augments pixel-wise losses, driving semantic consistency, and mitigating blurry reconstructions (Saleh et al., 2024).
• Style and Patch Losses: Style loss via Gram matrices enforces textural statistics; hole-region $L_1$ losses focus the optimization on the inpainted area.
• Auxiliary Mask/Boundary Losses: Auxiliary mask prediction losses (dice, BCE) and blurred-boundary losses improve edge blending and segmentation consistency, especially in cascaded generator-segmenter frameworks (Li et al., 2023, Chen et al., 2018).
• Low-Rank Regularization: Nuclear norm penalization encourages low-rank structure in predicted masks for disentanglement and denoising (Song et al., 2018).
• No Additional Consistency or Smoothness: In some settings, a single global $L_1$ or $L_2$ suffices to induce faithful and smooth completions, with no explicit smoothness regularization required (Cheng et al., 4 Aug 2025).

4. Applications Across Modalities

Mask-guided generative completion underpins several tasks across vision domains:

A. Depth Completion for Transparent Objects

ReMake integrates an instance mask (denoting transparent regions) and monocular depth estimation to guide generative depth completion, improving robotic grasping by localizing completion to physically ambiguous areas (Cheng et al., 4 Aug 2025).

B. Amodal and Object Completion

Weighted mask-guided gated convolution enables accurate amodal (invisible content) synthesis by adjusting the attention paid to visible, background, and occluded pixels (Saleh et al., 2024). Iterative mask denoising and segmentation-generation feedback loops yield high-fidelity object shape and mask recovery (Li et al., 2023).

C. Portrait and Semantic Image Completion

Conditional GANs with mask and part-level embedding (eyes, mouth, hair) enable attribute-controllable portrait completion and editing, preserving consistency in swapped or reconstructed parts (Gu et al., 2019). High-resolution completion is achieved by progressive GAN training and mask-aware residual blocks (Chen et al., 2018).

D. Diffusion and Flow-based Inpainting

Unconditional flow-matching priors (Restora-Flow) and diffusion inpainting systems (PD-MC, OWAAC, Flux-ControlNet) employ mask fusion and trajectory correction to reconstruct missing or degraded content while enforcing boundary sharpness and high perceptual similarity. These approaches generalize to video, denoising, and attribute editing with minimal architectural modification (Hadzic et al., 25 Nov 2025, Fan et al., 22 Sep 2025, Xu et al., 2023).

E. Video and Multimodal Completion

Multimodal masked video generation fuses tokenized visual and textual cues with frame-wise masking, addressing prediction, rewind, and infilling scenarios in one unified transformer framework (Fu et al., 2022).

5. Empirical Results and Comparative Evaluation

Mask-guided systems consistently outperform non-mask or implicitly conditioned baselines on both quantitative and qualitative measures:

• Perceptual Quality: Mask-guided models yield superior LPIPS, SSIM, PSNR, and FID metrics versus unconditioned or non-guided generative inpainting, particularly in tasks with high structural ambiguity or sparse visible context (Saleh et al., 2024, Cheng et al., 4 Aug 2025, Li et al., 2023, Hadzic et al., 25 Nov 2025).
• Semantic Consistency: CLIP similarity, DreamSim, VGG-16 feature distances, and user studies indicate significantly higher semantic realism and user preference for mask-guided completions, notably in amodal or out-of-distribution context (Xu et al., 2023, Fan et al., 22 Sep 2025).
• Task-Specific Metrics: In robotics, ReMake achieves grasping success rates of 85.7% in top-down and 73.5% in bird-eye configurations, far exceeding previous depth completion methods (Cheng et al., 4 Aug 2025).
• Ablation Analyses: Mask ablation increases RMSE; removing contextual cues degrades out-of-distribution robustness. Iterative segmentation/generation cycles demonstrate monotonic improvement in mask accuracy and realism (Li et al., 2023).
• Inference Speed and Modularity: Flow-matching and training-free pipelines accelerate sampling (Restora-Flow is 10× faster than diffusion baseline RePaint) with negligible sacrifice in perceptual metrics (Hadzic et al., 25 Nov 2025). Iterative methods incur slower inference due to multiple generation/segmentation passes but yield the highest object alignment and realism.
• Application Generality: Mask-guided pipelines generalize to complex, heavily occluded scenarios (robust up to 60–80% missing), various occlusion patterns, video, and multi-object setups.

6. Limitations, Open Problems, and Future Directions

Mask-guided completion faces several open challenges:

• Segmentation Quality Dependency: Completion quality depends critically on the accuracy of the input mask. Segmentation failures, especially for complex occlusion or fine structures, can propagate errors through mask-gated pipelines (Cheng et al., 4 Aug 2025, Li et al., 2023). Voting and repeated IMD steps offer partial mitigation.
• Solid Surface Bias: Depth completion often yields solid (watertight) surfaces, limiting reconstruction of hollow structures; explicit volumetric priors or multi-view cues could address such cases (Cheng et al., 4 Aug 2025).
• Diversity and Attribute Control: While mask guidance improves consistency and controllability, generating diverse plausible completions for highly occluded or ambiguous regions remains challenging—especially when visible context is limited or object categories have highly heterogeneous parts (Saleh et al., 2024, Fu et al., 2022).
• Inference Speed: Multi-pass diffusion and iterative denoising-refinement cycles increase inference latency, motivating research into more efficient inpainting loops and mask updating (Li et al., 2023).
• Generalization to Real-World and Open-Set Scenarios: Mask-guided systems trained on synthetic overlays or specific depth ranges may not generalize without further adaptation or expanded datasets (Cheng et al., 4 Aug 2025).
• Fusion of Semantic, Textual, and Visual Cues: Integration of fine-grained text prompts (multimodal semantic guidance) and boundary expansion priors (multi-agent reasoning) shows promise in guiding RGBA layer synthesis and direct AR asset generation (Fan et al., 22 Sep 2025).
• Unified Multimodal Frameworks: Extending mask-conditioning to video, multi-modal fusion (text, geometry, segmentation), and non-Euclidean domains (3D, MRI, etc.) is ongoing.

7. Representative Implementations and Outcomes

Approach	Mask Type	Key Modality	Mask Integration	Highlights	Reference
ReMake	Instance (binary)	Depth completion	Mask-attention + fusion	Generalizes to real transparent object grasping	(Cheng et al., 4 Aug 2025)
Mask Guided Gated Conv	Weighted	RGB object completion	Gated conv, layerwise	Texture-rich amodal fill, better PSNR/SSIM	(Saleh et al., 2024)
MaskComp	Partial/iterative	Amodal object	Iterative gen–seg loop	Large FID gains, strong shape guidance	(Li et al., 2023)
Restora-Flow	Binary	Restoration/inpainting	Mask-fused flow ODE	Fast, high-quality, training-free	(Hadzic et al., 25 Nov 2025)
Multi-Agent Amodal	Vis./occluder+text	Diffusion RGBA	Multi-agent composite	SOTA metrics, direct AR-ready RGBA composition	(Fan et al., 22 Sep 2025)
MMVG	Temporal frame	Video+text	Tokenized mask & text	Unified prediction, rewind, infilling	(Fu et al., 2022)

The literature establishes mask-guided generative completion as the dominant paradigm for localized, robust, and controllable object, image, and depth reconstruction. Current research explores increasingly modular, multimodal, and efficient pipelines, aiming to fuse segmentation, synthesis, and semantic reasoning in a unified generation framework.