Multi-View Depth Diffusion Models

• Multi-View Depth Diffusion Models are generative methods that integrate denoising diffusion with multi-view depth maps to achieve efficient and 3D-consistent synthesis.
• They leverage epipolar cross-view attention and depth fusion to enforce geometric consistency and mitigate typical 3D reconstruction artifacts.
• MVDD delivers robust performance in dense 3D point cloud and mesh generation, depth completion, and shape regularization for downstream applications.

Multi-View Depth Diffusion Models (MVDD) are a class of generative methods that integrate denoising diffusion models with multi-view depth map representations to enable 3D-consistent synthesis, dense point cloud and mesh generation, and downstream tasks such as depth completion and shape regularization. By leveraging image-gridded depth as a 2.5D proxy, MVDD achieves scalability and detail beyond explicit 3D voxel, point cloud, or implicit field methods, while maintaining the powerful inductive structure of diffusion-based generative modeling (Wang et al., 2023).

1. Multi-View Depth as a Generative Representation

MVDD encodes a 3D object $\mathcal{X}$ as a tuple of metric depth maps: $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$ where each $x^v$ corresponds to a calibrated camera with known intrinsics $A$ and relative poses $\pi_{v \rightarrow r}$. This 2.5D formulation enables:

• Alignment with diffusion backbones: UNet-based diffusion models natively operate on 2D grids, allowing efficient and effective modeling of depth map distributions.
• Dimensionality reduction: Producing a set of depth images over $N$ views at $H\times W$ resolution is exponentially more efficient than directly generating 3D voxels ($H \times W \times D$).
• Facilitated 3D recovery: Dense, view-aligned depth maps can be back-projected to generate point clouds with high spatial resolution (e.g., $N=8$, $H=W=128$ yields $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$0K points).
• Seamless integration as 3D priors: The same depth predictions serve as structural regularizers in tasks such as GAN inversion and depth completion (Wang et al., 2023).

2. Diffusion Process and Epipolar Cross-View Attention

MVDD leverages the denoising diffusion probabilistic modeling (DDPM) formalism. The forward process applies time-indexed Gaussian noise independently to each view’s depth map: $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$1 The reverse process reconstructs clean samples from noise, crucially conditioning each view’s denoising on its neighbors ($$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$2): $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$3 The objective is a simple $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$4-prediction loss, under the DDPM framework: $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$5 where $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$6.

To enforce cross-view geometric consistency, MVDD introduces epipolar “line-segment” attention. For each pixel $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$7 in view $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$8, attention is restricted to $$x = \{ x^v \in \mathbb{R}^{H \times W} \mid v=1,...,N \}$$9 samples along the corresponding epipolar line in each neighbor $x^v$0, centered on the current depth-based 3D back-projection. Cross-attention is formulated as: $x^v$1 with a visibility mask $x^v$2 that sets logits to $x^v$3 if depth disagreement exceeds a threshold $x^v$4. Value vectors are depth-augmented to bias toward geometric consistency in the denoised outputs.

3. Denoising Depth Fusion and Cross-View Consistency

Despite epipolar attention enforcing local consistency, multi-view denoising can still produce “double layers” or geometric artifacts. Depth fusion mitigates this by further aligning predicted depths during sampling:

• For pixel $x^v$5 in view $x^v$6 and neighbor $x^v$7, the depth at the projection is reprojected back to $x^v$8 using $x^v$9. If reprojected and predicted depths agree within a threshold, the final value is pooled across views before adding noise.
• Depth fusion is applied in the final stages of sampling, with an additional filtering step at $A$0 (Wang et al., 2023).

This mechanism leads to robust cross-view geometric coherence, further highlighted in ablation studies where disabling depth fusion leads to significant increases in nearest-neighbor errors (1-NNA rise from 54.5% to nearly random 92%).

4. Network Architecture and Training Protocol

The generative backbone is a vanilla 2D UNet augmented with inter-view cross-attention. At each layer, features from the view $A$1 are supplemented with those from $A$2 nearest neighbors via epipolar attention, supporting efficient message-passing while retaining per-view feature locality.

Key hyperparameters:

• $A$3 views at $A$4
• $A$5 epipolar samples per neighbor
• $A$6 diffusion steps (cosine noise schedule)
• Adam optimizer, learning rate $A$7
• Training for $A$8 epochs on ShapeNet categories (car, chair, airplane)

5. Generation of Dense 3D Point Clouds and Meshes

After ancestral sampling through the diffusion process, $A$9 depth maps are back-projected to form a dense 3D point cloud (>20K points for $\pi_{v \rightarrow r}$0, $\pi_{v \rightarrow r}$1). For mesh reconstruction, surface reconstruction algorithms such as Poisson or SAP can be employed for watertight outputs.

Quantitative benchmarks (ShapeNet chairs) demonstrate high fidelity relative to state-of-the-art: | Method | MMD-EMD | COV-EMD | 1-NNA-EMD | |----------------|---------|---------|-----------| | DPM | 3.32 | 42.60 | 80.10 | | PVD | 0.98 | 52.07 | 64.04 | | 3D-LDM | 0.92 | 48.27 | 63.49 | | MVDD (Ours) | 0.92 | 53.00 | 62.50 |

MVDD is able to faithfully model fine structures (e.g., thin legs, wing tips) that are missed by other point cloud or volumetric methods (Wang et al., 2023).

6. Downstream Applications: Depth Completion and 3D Prior Regularization

Depth Completion

MVDD performs two-pass reverse diffusion for multi-view completion given a single input view $\pi_{v \rightarrow r}$2:

• First pass: All views attend to each other, generating intermediate completions.
• Second pass: Synthesis is conditioned on the input view to enforce global coherence.

MVDD surpasses PVD, PointFlow, DPF-Net, and SoftFlow on the GenRe benchmark, with 10–30% improvements in Chamfer and EMD metrics.

Shape Prior for GAN Inversion

Inversion of a pretrained 3D GAN (such as EG3D) typically suffers from degenerate geometry under extreme poses. Incorporating an MVDD-based SDS-style prior on multi-view depths guides the optimization toward plausible 3D structures, mitigating failure modes such as “collapse” and producing accurate novel-view renderings. The additional MVDD loss steers the GAN's output depths toward the learned multi-view shape manifold (Wang et al., 2023).

7. Extensions, Limitations, and Position Within the Broader Field

MVDD represents a departure from direct 3D generation (voxels, point clouds, implicit fields) by reducing generative complexity and leveraging superior network architectures for 2D grids. The integration of epipolar-attention cross-view coupling and depth fusion establishes a new state of the art in unconditional 3D shape generation, multi-view-consistent synthesis, and shape prior embedding for diverse downstream tasks.

Critical ablations reveal the necessity of each cross-view mechanism for geometric fidelity. Limitations include reliance on accurate camera calibrations and the potential for artifacts from misestimated cross-view visibility or extreme occlusions.

MVDD’s formulation has inspired further research on diffusion-based multi-view or multi-modal 3D generative models across RGB-D, GAN inversion, and architectural synthesis domains (Du et al., 5 Mar 2025, Hu et al., 2024, Cao et al., 2024), consolidating its role as a foundational model for 3D-aware generative perception and content creation (Wang et al., 2023).