Depth-to-Image Diffusion Model

• Depth-to-image diffusion models are generative frameworks that use neural diffusion conditioned on depth maps to enable 3D-aware image synthesis with explicit geometric guidance.
• They integrate techniques such as channel-wise concatenation, parallel depth encoder streams, and cross-attention to merge depth and appearance cues for photorealistic outputs.
• Practical applications include multi-view image generation, precise facial editing, and compositional object synthesis, with evaluations showing robust geometrical consistency and semantic reliability.

A depth-to-image diffusion model is a generative framework in which a neural diffusion process synthesizes realistic RGB images conditioned directly on input depth maps or depth-derived cues. By leveraging paired or inferred depth information, these models enable explicit geometric and 3D structural guidance in image synthesis and facilitate 3D-aware generation, compositional control, semantically consistent cross-modality registration, and geometry-respecting inpainting. Contemporary depth-to-image diffusion models operate in both pixel-space and latent-space, and often employ auxiliary architectures (depth encoders, channel-wise inpainting heads, cross-attention injectors) to achieve joint or conditional modeling of appearance and geometry.

1. Mathematical Formulation and Conditioning

Depth-to-image diffusion models build upon denoising diffusion probabilistic models (DDPMs), in which the generative process is framed as a sequence of noise reduction steps over an initial stochastic sample. The forward process is defined by: $$q(x_t|x_{t-1}) = \mathcal N(x_t;\,\sqrt{1-\beta_t}\,x_{t-1},\,\beta_t I)$$ for a prescribed noise schedule $\{\beta_t\}$, with $$x_t = \sqrt{\alpha_t}\,x_0 + \sqrt{1-\alpha_t}\,\epsilon$$, where $\alpha_t = \prod_{i=1}^t(1-\beta_i)$, $\epsilon\sim \mathcal N(0,I)$. The reverse process is parameterized via a neural network, usually a U-Net $\epsilon_\theta(x_t, t)$, trained with the $\epsilon$-prediction loss: $$\mathcal L(\theta) = \mathbb E_{x_0, \epsilon, t}\bigl[\|\epsilon - \epsilon_\theta(x_t,t)\|^2\bigr]$$ For depth conditioning, various mechanisms are employed:

• Channel-wise concatenation: Depth data is concatenated with RGB or latent channels at model input (e.g., 4-channel RGBD for pixel models (Xiang et al., 2023); 8-channel RGB+depth latent for latent models (Ji et al., 15 Jan 2025)).
• Parallel depth encoder streams: Depth maps are processed via additional convolutional streams, whose activations are injected via adapters or normalization layers at each U-Net stage (e.g., ControlNet zero-conv (Wang et al., 2023), local fuser SPADE-like norms (Lee et al., 2024)).
• Cross-attention with reference features: Extraction of multi-scale features from a reference image via a mirror encoder (ReferenceNet), injected by cross-attention at each block to preserve semantic consistency (Ji et al., 15 Jan 2025).

Conditioning on depth permits the model to respect geometric structure, enforce 3D-awareness, and support spatial disentanglement of appearances.

2. Construction and Utilization of Depth Data

High-quality paired RGB-depth training data is a prerequisite for effective depth-to-image modeling. Multiple approaches have been adopted:

• Monocular depth estimation: Depth maps are inferred from single-view RGB images with pretrained monocular estimators (e.g., MiDaS dpt_beit_large_512 (Xiang et al., 2023), ZoeDepth (Wang et al., 2023)) to bootstrap depth data from large-scale image collections.
• Synthetic triplet generation: “Depth disentanglement training” leverages image triplets $(I_f, I_b, M)$, where $I_f$ (foreground) and $I_b$ (background) masks are extracted from in-the-wild images, depth computed via estimation, and used to train explicit occlusion ordering between regions (Lee et al., 2024).
• Multi-view and studio capture: Controlled datasets with ground-truth multi-view RGBD pairs facilitate joint training for specialized domains (portrait generation (Ji et al., 15 Jan 2025)).

During training or inference, paired depth and color data are fused into consistent, mutually-constraining representations. For multi-view synthesis (Xiang et al., 2023), depth is used for visibility-aware mesh warping and aggregation of conditioning signals across views.

3. Network Architectures and Conditioning Mechanisms

Depth-to-image diffusion networks typically extend canonical U-Net backbones with depth-specific modules or expanded channel dimensionality:

Model	Conditioning on Depth	Backbone	Additional Modules
(Xiang et al., 2023)	RGBD concat, 10-ch	ADM U-Net	No cross-attention; mesh warp
(Ji et al., 15 Jan 2025)	8-ch latent concat	Latent U-Net	ReferenceNet + inpainting mask
(Wang et al., 2023)	ControlNet	Stable Diffusion	Parallel depth encoder; zero-conv
(Lee et al., 2024)	Local/Global fuser	SD U-Net (clone)	SPADE-like norms, cross-attn

• Channel expansion: Input/output convolutions are expanded to accommodate latent depth and appearance (e.g., RGB+depth latents in (Ji et al., 15 Jan 2025)).
• ReferenceNet and soft guidance: Use of clone encoders for multi-scale semantic injection (ReferenceNet (Ji et al., 15 Jan 2025)) and masking of cross-attention affinities for region-specific style conditioning (soft guidance (Lee et al., 2024)).
• Mesh-based warping: For multi-view synthesis, depth is used to construct visibility masks and perform forward-backward warping, supporting conditional generation conditioned on physically plausible geometry (Xiang et al., 2023).

4. Training Procedures and Objectives

Training is structured around standard diffusion objectives, adapted for depth and appearance:

• Joint RGB+depth denoising: Models are trained to jointly denoise both appearance and depth latent channels, typically via MSE loss on predicted noise for both modalities (Ji et al., 15 Jan 2025).
• Asymmetric masked inpainting: During fine-tuning, binary masks specify which latents (depth or appearance) are held fixed versus synthesized, enabling flexible channel-wise inpainting (Ji et al., 15 Jan 2025).
• Multi-condition dropouts: For robustness, modalities (depth, text, style) are randomly dropped during training so the model is agnostic to incomplete conditioning (Lee et al., 2024).
• Relative depth ordering: Training objectives enforce depth-aware occlusion and layer assignment by always conditioning on foreground/background triplets (Lee et al., 2024).
• Augmentations: Blur and texture erosion are applied to conditional inputs to mitigate artifacts and improve geometric generalization in large-angle synthesis (Xiang et al., 2023).

5. Inference Pipelines and Practical Applications

Depth-to-image diffusion architectures enable a broad array of practical tasks:

• 3D-aware image generation: Sequential unconditional-conditional sampling generates multi-view image sets for a single 3D asset, conditioned on depth-warped prior views (Xiang et al., 2023).
• Facial depth→image editing: Depth maps are edited (shape, pose, expression); RGB channels are inpainted to match, yielding high-fidelity, identity-preserving portraits (Ji et al., 15 Jan 2025).
• Composable multi-object synthesis: Depth maps and exemplar semantics condition the placement and appearance of multiple objects at distinct depths, with region-specific style injection (Lee et al., 2024).
• Image-to-point-cloud registration: Intermediate U-Net features (“diffusion features”) extracted from both RGB and depth inputs provide cross-modality anchors, unified with geometric FCGF descriptors for robust correspondence and alignment (Wang et al., 2023).

6. Quantitative and Qualitative Evaluation

Empirical evaluation employs both standard generative and geometric metrics:

Model	Domain	FID ↓	IS ↑	SSIM ↑	Depth MAE ↓
(Xiang et al., 2023) Baseline	ImageNet	9.45	68.7	—	—
(Lee et al., 2024) CnC-FT	COCO-Stuff	18.19	29.30	0.2248	0.0990
(Wang et al., 2023) FreeReg	2D–3D Reg.	RR ↑ 48.6%	IR ↑ 20.6%	—	—
(Ji et al., 15 Jan 2025)	Portrait	—	—	—	—

• Large-angle view synthesis demonstrates slow FID degradation and robust geometric parallax (Xiang et al., 2023).
• Qualitative editing examples exhibit photorealistic adaptation to depth edits and identity preservation (Ji et al., 15 Jan 2025).
• COCO-Stuff benchmarks show CnC-Finetuned outperforming Uni-ControlNet and T2I-Adapter in SSIM, LPIPS, and depth MAE (Lee et al., 2024).
• In registration tasks, fusion of diffusion and geometric features yields 3–4× higher inlier numbers and registration recall over metric learning baselines (Wang et al., 2023).

7. Limitations and Future Directions

Current depth-to-image diffusion methods face several domain and scalability constraints:

• Domain limitation: Portrait-specific models are restricted to regions with depth supervision (e.g., facial skin), and generalization outside these domains is untested (Ji et al., 15 Jan 2025).
• Full-body and non-human objects: Generalization to diverse objects would require retraining or architectural adaptation.
• Multi-modal extension: Channel-expansion and masked inpainting could be extended to normals, semantic segmentation, or surface reflectance (Ji et al., 15 Jan 2025).
• Separate vs shared encoders: Extending the VAE to separately encode RGB/depth, or employing dual-encoder architectures, may improve non-face fidelity.
• Curricular training: Progressive schedules generating coarse geometry before fine detail could enhance high-frequency quality.
• Scaling: Large-scale joint training using text-to-3D paired data may further reinforce geometric priors and enable broader generalization.

A plausible implication is that future research may focus on unified frameworks allowing disentangled multi-modal conditioning, seamless cross-domain adaptation, and robust compositional control for 3D-aware generative models.

---

References:

• "3D-aware Image Generation using 2D Diffusion Models" (Xiang et al., 2023)
• "Joint Learning of Depth and Appearance for Portrait Image Animation" (Ji et al., 15 Jan 2025)
• "FreeReg: Image-to-Point Cloud Registration Leveraging Pretrained Diffusion Models and Monocular Depth Estimators" (Wang et al., 2023)
• "Compose and Conquer: Diffusion-Based 3D Depth Aware Composable Image Synthesis" (Lee et al., 2024)