Spatially-Conditioned Part Synthesis

• Spatially-conditioned part synthesis is a generative modeling approach that uses spatial cues (e.g., masks, coordinates, continuous fields) to selectively synthesize, manipulate, or restore object parts.
• It employs advanced architectures like SPADE, GANs, diffusion and implicit function-based models to encode spatial information and achieve precise localization and controllable diversity.
• The framework has significant applications in semantic image synthesis, medical imaging, and 3D asset creation, delivering improved metrics such as FID, SSIM, and IoU through tailored loss functions and conditioning strategies.

Spatially-conditioned part synthesis encompasses generative modeling techniques in which the synthesis, manipulation, or restoration of specific object parts is explicitly controlled via spatial signals. These signals frequently include coordinates, semantic/part segmentation, bounding boxes, spatial masks, or continuous fields that encode geometric or anatomical priors. Developed across both image and 3D domains for applications such as semantic image synthesis, medical imaging, content creation, and industrial inspection, spatially-conditioned part synthesis frameworks achieve precise localization, controllable diversity, and structural plausibility in the generated outputs.

1. Conditioning Mechanisms: Spatial Inputs and Their Encoding

Conditioning on spatial information is fundamental to part-level synthesis. Conditioning modalities include:

• Semantic and part masks: One-hot or multi-channel binary masks localize parts at pixel or voxel level. Semantic class, instance, and hierarchical part information can be encoded (Park et al., 2019, Jiang et al., 2019, Wei et al., 2023, Hanifi et al., 12 Nov 2025).
• Continuous fields: Scalar or vector-valued fields, such as continuous tumor concentration maps in MRI synthesis, enable granular control beyond binary boundaries (Biller et al., 10 Oct 2025).
• Bounding boxes and spatial coordinates: Axis-aligned boxes or normalized (x, y, z) coordinates define localities for patch-based generation or part synthesis (Yang et al., 8 Jul 2025, Lin et al., 2019).
• Reference features and appearance tokens: Local or part-wise appearance features paired with spatial indices (e.g., concatenated part tokens extracted from IP-Adapter+ in (Richardson et al., 13 Mar 2025)), preserve both identity and placement.
• Pose and structural cues: Skeletons, contour maps, or pose vectors provide geometric constraints relevant in articulated figures (Pandey et al., 2019, Huang et al., 23 Apr 2024).

Common spatial conditioning strategies include channel concatenation at each network layer, modulation via spatially-adaptive normalization (SPADE), and injection of part-specific features through cross-attention or shared attention in transformer/U-Net blocks (Park et al., 2019, Huang et al., 23 Apr 2024).

2. Model Architectures for Spatially-Conditioned Part Synthesis

Architectural paradigms for spatially-conditioned part synthesis include:

• SPADE Generators and Variants: SPADE modulates feature activations in normalization layers using spatial maps, enabling fine-grained mask- or part-level control (Park et al., 2019). Hierarchical or multi-branch SPADE extensions support sub-part and compositional synthesis (Wei et al., 2023).
• Conditional and Part-Modulated GANs: Generators with input streams for spatial maps and latent codes (usually z for style, c for attributes, s for masks) are observed to best preserve spatial detail when mask features are injected at multiple encoder/decoder levels (Jiang et al., 2019, Pandey et al., 2019).
• Patch-based and Coordinate-aware GANs: Architectures such as COCO-GAN condition part generators on coordinate embeddings (micro/macro), enabling localized generation and seamless assembly via grid-based or even topology-aware layouts (Lin et al., 2019).
• Diffusion and Flow-based Models: Denoising diffusion probabilistic models integrate spatial masks or continuous fields at every U-Net resolution (via concatenation or SPADE), while rectified-flow approaches condition noise prediction on part-aware layouts or part tokens (Hanifi et al., 12 Nov 2025, Biller et al., 10 Oct 2025, Richardson et al., 13 Mar 2025, Yang et al., 8 Jul 2025).
• Implicit Function-based Architectures: For 3D shape composition, implicit decoders model part geometry as fields conditioned on spatial parameters, with spatial transformers mapping normalized parts to their locations within the composite shape (Guan et al., 17 Jan 2024).
• Self- and Cross-attention with Mask Guidance: Parts2Whole uses self-attention across reference and target feature maps, with binary masks restricting attention to appropriate spatial regions (Huang et al., 23 Apr 2024).

3. Loss Functions and Training Objectives

Models employ loss functions to enforce both realism and spatial/part consistency:

• Adversarial loss: Critical for realism in both global and patch-level discriminators; spatial patch discriminators may focus on boundary and regional coherence (Jiang et al., 2019, Lin et al., 2019, Wei et al., 2023).
• Denoising/objective loss: DDPMs and rectified-flow models minimize noise prediction or velocity matching losses, conditioned on spatial controls (Hanifi et al., 12 Nov 2025, Biller et al., 10 Oct 2025, Yang et al., 8 Jul 2025, Richardson et al., 13 Mar 2025).
• Segmentation consistency: Synthetic outputs are penalized according to their agreement with spatial masks or segmentation ground truth, either via cross-entropy or mask regularization (Jiang et al., 2019, Wei et al., 2023).
• Spatial coverage/IoU losses: In 3D, coverage losses ensure part bounding boxes encompass the true part voxels (Yang et al., 8 Jul 2025).
• Modality- and context-aware losses: Incorporation of feature-matching, perceptual, and style losses (e.g., CLIP-based patch correspondence) aligns the synthesized content to the prescribed spatial template (Wei et al., 2023).

Dropout on branch selection (to increase robustness to partial conditioning), region-weighted denoising loss, and data augmentation via geometric mask transformations support generalization and spatial flexibility (Huang et al., 23 Apr 2024, Hanifi et al., 12 Nov 2025).

4. Applications, Performance, and Evaluation

Spatially-conditioned part synthesis enables critical applications and supports quantitative/qualitative assessment:

• Medical imaging: Control over spatial lesion synthesis/inpainting in volumetric MRI with tumor concentration fields, evaluated via PSNR, SSIM, and region-wise error (Biller et al., 10 Oct 2025).
• Photovoltaic defect generation: Mask-based DDPM synthesis of localized anomalies for data augmentation, measured by FID/KID and cluster analysis in feature-space (Hanifi et al., 12 Nov 2025).
• Semantic and part-aware image synthesis: Rich control over parts (hair, clothing, limbs) for face/fashion/human image generation, with user studies, mIoU, FID, and attribute accuracy as benchmarks (Park et al., 2019, Jiang et al., 2019, Wei et al., 2023, Huang et al., 23 Apr 2024).
• 3D object and asset creation: Omnipart attains interpretable, manipulable 3D assemblies with explicit user-chosen part arrangements, evaluated by Chamfer Distance, F1-score, IoU, and generation time (Yang et al., 8 Jul 2025, Guan et al., 17 Jan 2024).
• Coherent multi-part concept synthesis: Piece-it-Together and Parts2Whole enable artists to specify arbitrary part fragments and receive full, plausible completions, assessed via CLIP/DINO alignment, user preference scores, and visual plausibility (Richardson et al., 13 Mar 2025, Huang et al., 23 Apr 2024).

5. Algorithmic and Practical Innovations

Recent research introduces several innovations unique to spatially-conditioned part synthesis:

• Continuous Field Conditioning: Encoding tumor concentration as a continuous scalar-field (not binary), realized via PDE simulation, enables smooth control of infiltration and supports morphability at clinical resolution (Biller et al., 10 Oct 2025).
• Mask-guided attention: Reference and generated features are fused with explicit in-mask attention gating, enabling precise transfer of part structure/content (Huang et al., 23 Apr 2024).
• Autoregressive structure planning: OmniPart decouples part layout (autoregressive bounding box generation) from geometry synthesis (conditional rectified-flow), supporting multi-granularity manipulation (Yang et al., 8 Jul 2025).
• Known-region injection and boundary harmonization: In inpainting, explicit injection of known voxels at each diffusion step, combined with repeated back/forward steps at interfaces, preserves context and smooths seams (Biller et al., 10 Oct 2025).
• Implicit spatial transformer for positioning: 3D part-based frameworks use a transformer module to align generated part geometry into holistic global context, enabling interactive assembly and restructuring (Guan et al., 17 Jan 2024).
• In-situ encoding: Embedding user-provided parts at designated canvas positions (without separate coordinate channels), with transformers attending to these localized tokens, achieves high-fidelity compositionality (Richardson et al., 13 Mar 2025).

6. Limitations, Open Challenges, and Future Directions

Current spatially-conditioned part synthesis frameworks exhibit several limitations:

• Computational cost: 3D models with large spatial domains and multi-stage architectures (e.g., latent VAE + 3D U-Net diffusion) entail high training/inference latency and GPU requirements (Biller et al., 10 Oct 2025, Yang et al., 8 Jul 2025).
• Boundary artifacts: Naive known-region or mask concatenation can create seams; thus, harmonization steps (e.g., RePaint, Poisson blending) are essential but not always fully effective (Biller et al., 10 Oct 2025).
• Fidelity at fine scales: Subtle details in very small or thin parts may blur, particularly at mid-range resolutions (e.g., accessories in 512×512 images) (Huang et al., 23 Apr 2024).
• Mode collapse and diversity: GAN-based models may exhibit limited diversity or mode dropping in part suggestions, motivating diffusion-based or multimodal flows for faithful coverage (Guan et al., 17 Jan 2024).
• Manual mask/part annotation: Reliance on accurate segmentation or bounding box labeling poses a data bottleneck; semi-supervised and few-shot part inference are active research fronts (Wei et al., 2023, Yang et al., 8 Jul 2025).
• Conditional prompt adherence: Adapters for prompt control can impair visual/structural faithfulness if not properly designed; LoRA-based adapters mitigate this to some extent (Richardson et al., 13 Mar 2025).

Future work is focusing on integrated multimodal conditioning, real-time or interactive synthesis, compositional transfer across domains, fine-scale detail preservation, and more automated or weakly-supervised spatial annotation procedures (Biller et al., 10 Oct 2025, Huang et al., 23 Apr 2024, Guan et al., 17 Jan 2024).

7. Summary, Impact, and Theoretical Significance

Spatially-conditioned part synthesis integrates explicit spatial constraints with deep generative modeling, producing outputs with localized, interpretable, and controllable part structure. The approach spans both image and volumetric 3D domains, providing state-of-the-art capabilities for content creation, clinical imaging, anomaly simulation, and interactive asset generation. By leveraging spatial signals at each stage of the generation process—through normalization, coordinate embedding, conditional attention, or explicit field modeling—these frameworks reconcile global coherence with localized controllability. Quantitative results demonstrate significant gains in metrics such as FID, SSIM, mIoU, and Chamfer-F1 compared to unconditioned or purely text/latent-driven generative models. As part synthesis matures, it is poised to become a crucial axis of controllability in generative modeling, supporting both high-fidelity realism and precise user-driven customization (Park et al., 2019, Wei et al., 2023, Biller et al., 10 Oct 2025, Hanifi et al., 12 Nov 2025, Yang et al., 8 Jul 2025, Richardson et al., 13 Mar 2025).