Joint Image-Depth Generation

• Joint image-depth generation is a technique that simultaneously synthesizes RGB images and aligned depth maps by modeling their joint distribution.
• Diffusion models and latent variable architectures leverage cross-modal attention and adaptive noise schedules to ensure coherent and consistent synthesis.
• This approach enhances 3D reconstruction, image inpainting, and bidirectional manipulation while addressing challenges in computational efficiency and data requirements.

Joint image-depth generation refers to the class of algorithms and models that simultaneously synthesize (or infer) both photorealistic RGB images and their aligned geometric depth maps, typically by modeling the joint distribution over appearance and 3D structure. Unlike cascaded or post-hoc systems—which estimate depth from images or hallucinate images from geometry—joint models explicitly learn or optimize for cross-modal coherence, permitting strongly coupled generation, bidirectional conditional inference, and improved consistency for downstream 3D applications.

1. Probabilistic Formulations and Modeling Paradigms

Joint image-depth models are grounded in explicit representations of the joint distribution $p(\mathbf{x}, \mathbf{d})$, where $\mathbf{x}$ denotes the image (typically RGB) and $\mathbf{d}$ denotes the depth map. Early approaches formulated this coupling via structured sparse generative models: paired overcomplete dictionaries for intensity and depth, subject to joint sparsity constraints, as in the Joint Basis Pursuit (JBP) algorithm (Tosic et al., 2012). In deep learning, most joint generation methods employ generative latent variable models—such as VAEs and diffusion models—where both modalities are integrated at the latent, decoder, or denoising levels.

Recent generative models such as JointDiT (Byung-Ki et al., 1 May 2025), Modality Forcing (Duisterhof et al., 11 Jun 2026), Orchid (Krishnan et al., 22 Jan 2025), and dual-modal adaptations of latent diffusion architectures share the design principle of parameterizing noise or generation processes over both modalities, sometimes with independent or per-modality noise schedules, allowing joint or conditional sampling via noise-level selection.

2. Diffusion Models: Architectures for Joint Generation

Diffusion-based approaches dominate the contemporary state of the art for joint image-depth synthesis. The canonical structure involves paired noising and denoising processes for each modality:

• Forward Process: For modalities $m \in \{\mathrm{RGB}, \mathrm{depth}\}$, independent noise schedules are applied:

$$q(x_t^m \mid x_{t-1}^m) = \mathcal{N}(x_t^m; \sqrt{1-\beta_t^m}\,x_{t-1}^m, \beta_t^m I)$$

with closed-form propagation over noise levels $t$.

• Reverse Process: Rather than learning modality-separate denoisers, models such as JointDiT (Byung-Ki et al., 1 May 2025) or IDOL (Zhai et al., 2024) employ a single transformer-based backbone, or a U-Net with weight sharing, to predict the joint noise components for both modalities, integrating them via cross-attention or fused representation blocks.
• Adaptive Scheduling and Sampling: JointDiT introduces adaptive weighting of cross-modal feature exchange depending on the relative noise schedules (timesteps) of each modality, ensuring information flows from the "cleaner" to the "noisier" modality. Training employs an unbalanced sampling strategy to guarantee adequate coverage of all combinations of noise levels, which is essential for both joint and conditional generation scenarios.
• Latent Diffusion and Joint VAEs: Orchid (Krishnan et al., 22 Jan 2025) extends the latent diffusion paradigm by encoding RGB, depth, and surface normals into a joint latent space via a VAE, with the subsequent latent diffusion model operating on the combined embedding.
• Cross-modal Attention: Shared or unified backbones (e.g., in IDOL and Modality Forcing (Duisterhof et al., 11 Jun 2026)) are augmented with cross-modal self-attention or cross-modal communication layers, encouraging spatial and semantic coupling.

3. Conditional and Joint Generation Modes

These architectures support multiple usage regimes by manipulating modality-specific noise schedules:

• Joint Sampling: Both modalities are initialized with maximal noise and jointly denoised, producing image-depth pairs with learned statistical coherence.
• Conditional Inference: Fixing one modality as noise-free (e.g., t_rgb = T for RGB) and the other as high-noise (t_depth = 0) implements image-to-depth (monocular depth estimation) or depth-to-image translation (depth-conditioned synthesis). Cross-modal conditioning is then mediated by the network's fused architecture and intermodal attention.
• Zero-shot Inpainting and Regularization: Models like Orchid leverage their learned joint priors as regularizers, for example, filling missing regions in both color and depth simultaneously via guided inpainting in latent space.

The flexibility to switch between these regimes with no retraining is a direct outcome of the unified noise schedule and network design.

4. Alternative Generative and Physics-Based Formulations

Prior to the diffusion revolution, generative formulations employed:

• Sparse Joint Dictionary Learning: JBP (Tosic et al., 2012) learns paired dictionaries for intensity and depth, enforcing shared support (feature activation) between modalities, but allowing coefficients to differ. This structure ensures that the same "3D causes" can explain both apparent intensity and measured depth, and supports joint synthesis, inpainting, and theoretical guarantees on recovery error.
• Joint Optimization in Computational Imaging: In jointly optimized mask-based lensless cameras (Zheng et al., 2019), the forward model directly links observed sensor measurements to both scene intensity and a per-pixel continuous depth map. Alternating nonconvex optimization recovers both intensity and depth jointly, subject to smoothness priors, achieving improved depth fidelity over plane-restricted or sparsity-enforcing approaches.
• End-to-End Differentiable Imaging Pipelines: In end-to-end learned diffracted rotation-based imagers (Mel et al., 2022), the physical design of the aperture and the reconstruction networks are optimized together, supporting robust joint reconstruction of sharp images and depth from coded observations.

5. Evaluation Metrics and Empirical Results

Benchmarking of joint image-depth models leverages standard image and depth fidelity metrics:

Task	Key Metric(s)	SOTA Results/Models
Joint RGB–Depth Generation	FID, IS, ImageReward (human preference)	JointDiT FID ≈ 24, IS ≈ 37
Monocular Depth Estimation	AbsRel (mean abs rel error), δ₁ (inlier %)	Modality Forcing AbsRel 2.52% on NYUv2, δ₁ 98.9% (Duisterhof et al., 11 Jun 2026)
Depth-Conditioned Generation	FID for RGB, AbsRel for depth adherence	JointDiT FID ≈ 12.6, AbsRel 6.99% (Byung-Ki et al., 1 May 2025)
Zero-shot Geometry Prediction	AbsRel, Mean Angular Error (normals)	Orchid AbsRel 5.7% (NYUv2), normal error 15.2° (Krishnan et al., 22 Jan 2025)

These models consistently outperform vanilla cascaded or independent approaches, improving both intra- and inter-modal consistency, and, in several cases, approach or surpass discriminative single-modality baselines.

6. Scalability, Pretraining, and Generalization

Recent findings demonstrate that the accuracy and robustness of joint image-depth generation scale strongly with diffusion model capacity and the amount of image-only pretraining. Modality Forcing (Duisterhof et al., 11 Jun 2026) shows that a single DiT backbone, pretrained on web-scale text-to-image data and extended post hoc to dual-modality via lightweight adapters, delivers depth estimation quality rivaling state-of-the-art monocular estimators, while requiring orders of magnitude less dense depth supervision. The model's depth accuracy (AbsRel, δ₁) improves monotonically with model size and pretraining set size, highlighting image generation as a scalable "proxy task" for depth learning.

Hybrid-training approaches, such as those in IDOL (Zhai et al., 2024) and Orchid (Krishnan et al., 22 Jan 2025), benefit similarly from large, diverse image datasets (COCO, LAION, virtual scans), augmented with real or pseudo-labeled depth, to reach generalizability across scenes and domains.

7. Applications, Limitations, and Extensions

Joint image-depth generators enable applications beyond unconditional synthesis, such as:

• Image Editing and Inpainting: Models such as Orchid perform semantically coupled editing and completion across color, depth, and normals.
• 3D Scene Reconstruction: Generated RGB-D outputs can be directly fused into partial 3D meshes or splats for AR/VR.
• Bidirectional Manipulation: Portrait animation frameworks (Ji et al., 15 Jan 2025) support conditional depth-to-image editing, relighting, and even audio-driven 3D video animation with alignment maintained via the joint latent backbone.
• Physics-Integrated Computation: Approaches integrating mask design and end-to-end learning (Mel et al., 2022, Zheng et al., 2019) highlight avenues for hardware-algorithm co-design.

Limitations remain in computational efficiency, especially for large diffusion or transformer models. Certain architectures are more data-hungry, relying on extensive aligned RGB-D datasets. Precision on sparse, reflective, or transparent surfaces is still an open problem. A plausible implication is that adaptive weighting and intermodal fusion methods, as introduced in JointDiT (Byung-Ki et al., 1 May 2025), are critical to maintaining cross-modal coherence, especially under noise or when only partial input is available.

• JointDiT: Enhancing RGB-Depth Joint Modeling with Diffusion Transformers (Byung-Ki et al., 1 May 2025)
• Learning joint intensity-depth sparse representations (Tosic et al., 2012)
• IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation (Zhai et al., 2024)
• Modality Forcing for Scalable Spatial Generation (Duisterhof et al., 11 Jun 2026)
• Joint Learning of Depth and Appearance for Portrait Image Animation (Ji et al., 15 Jan 2025)
• Orchid: Image Latent Diffusion for Joint Appearance and Geometry Generation (Krishnan et al., 22 Jan 2025)
• End-to-end Learning for Joint Depth and Image Reconstruction from Diffracted Rotation (Mel et al., 2022)
• Joint Image and Depth Estimation with Mask-Based Lensless Cameras (Zheng et al., 2019)