Depth Diffusion Features

• Depth diffusion features are intermediate representations that condition diffusion models to recover scene geometry by integrating RGB images with sparse depth inputs.
• They manifest in various forms—such as multi-scale pyramids, latent tokens, and texture-enriched maps—that improve geometric consistency in tasks like depth completion and segmentation.
• Their use enhances model robustness, efficiency, and generalization across diverse applications including monocular depth estimation, cross-modal registration, and 3D reconstruction.

Searching arXiv for papers on depth diffusion features, diffusion-based depth estimation, and depth completion. Depth diffusion features are the intermediate, conditioning, or transformed representations through which diffusion models encode, guide, or refine scene geometry in depth-related tasks. Across recent literature, the term does not denote a single canonical tensor or module; instead, it refers to a family of depth-centered representations that appear in conditional denoisers, latent depth spaces, semantically prompted diffusion transformers, geometry-aware multi-view mechanisms, and feature-steering pipelines for depth completion, monocular depth estimation, semantic segmentation, registration, and navigation (Yuan et al., 31 Mar 2025, Viola et al., 2024, Sun et al., 2024, Wang et al., 2023, Xu et al., 8 Oct 2025).

1. Conceptual scope and definitions

In diffusion-based depth completion, depth diffusion features often denote the condition features that drive denoising toward a dense depth map. DenseFormer, for example, introduces a feature extraction module that leverages a feature pyramid structure and multi-layer deformable attention to extract and integrate features from sparse depth maps and RGB images, and these features serve as the guiding condition for the diffusion process; the method further includes a depth refinement module and reports that it outperforms classical depth completion methods on KITTI (Yuan et al., 31 Mar 2025).

Other works use the term more broadly for the internal depth prior carried by a pretrained diffusion backbone. Marigold-DC treats monocular depth completion as image-conditioned depth generation guided by sparse measurements, and explicitly attributes its behavior to the pretrained U-Net’s internal multi-scale features and conditioning pathways, which act as a strong visual-depth prior during test-time guidance (Viola et al., 2024). In "Depth-guided Texture Diffusion for Image Semantic Segmentation," depth diffusion features are defined as enriched structural features obtained by selectively diffusing texture cues extracted from RGB into the depth map before RGB–depth fusion (Sun et al., 2024).

A further usage appears in cross-modal geometry. FreeReg calls the intermediate U-Net activations extracted from a pretrained depth-to-image diffusion model “diffusion features” and uses them as semantically consistent descriptors shared between RGB images and point-cloud-derived depth maps (Wang et al., 2023). Pixel-space diffusion work extends the concept again: "Pixel-Perfect Depth" treats depth diffusion features as the representational and architectural elements inside a diffusion generator that preserve global semantic consistency and fine-grained geometric detail while avoiding VAE-induced flying pixels (Xu et al., 8 Oct 2025).

This suggests that “depth diffusion features” is best understood as an umbrella term for representations that make denoising depth-aware: they may be explicit conditioning features, latent depth states, semantically aligned intermediate activations, or engineered feature transforms that reshape one modality so that diffusion can recover geometry more faithfully.

2. Representational forms and conditioning pathways

Recent papers instantiate depth diffusion features in several distinct forms. Some are explicit multi-scale RGB–depth condition pyramids; some are latent depth tokens; some are semantic prompts injected into a DiT; some are joint RGBD channels; and some are feature maps extracted from a pretrained diffusion backbone and reused for another geometric task.

Form of depth diffusion feature	Representative role	Example papers
Multi-scale RGB–depth condition features	Guide reverse denoising toward dense depth	(Yuan et al., 31 Mar 2025, Duan et al., 2023, Saxena et al., 2023)
Pretrained latent or U-Net depth priors	Supply geometry and semantics without retraining or with minimal adaptation	(Viola et al., 2024, Song et al., 5 Jan 2025)
Texture-enriched depth features	Bridge the modality gap between RGB and depth before fusion	(Sun et al., 2024)
Multi-view geometry-aware depth features	Enforce cross-view consistency in depth diffusion	(Wang et al., 2023)
Joint RGBD diffusion features	Model RGB–depth coherence directly in a shared denoising process	(Papa et al., 2024)
Extracted diffusion descriptors for cross-modal matching	Build correspondences between images and point clouds	(Wang et al., 2023)

Conditioning mechanisms vary accordingly. DiffusionDepth builds a condition tensor from backbone features enhanced by a HAHI neck and FPN, then fuses that condition with the latent depth state through the Monocular Conditioned Denoising Block, combining local projection, CNN processing, self-attention, and channel-wise attention (Duan et al., 2023). DMD conditions an Efficient U-Net on the scalar field-of-view variable $c_{\mathrm{FOV}}=\tan(\theta/2)$, embeds it sinusoidally, and injects it via FiLM modulation throughout the backbone so that metric scale is tied explicitly to camera intrinsics (Saxena et al., 2023).

Single-step models shift the representational emphasis. DepthMaster retains Stable Diffusion v2 as backbone but adapts its generative features for discriminative depth by aligning the U-Net middle-block representation to external semantic features such as DINOv2 and by applying a Fourier Enhancement module to $F_{\mathrm{mid}}$ so that low-frequency structure and high-frequency detail are rebalanced in one pass (Song et al., 5 Jan 2025). Pixel-Perfect Depth moves to pixel-space diffusion and prompts DiT tokens with normalized vision-foundation-model semantics through the fusion $z' = h_\phi(z \oplus B(\hat e))$, where $\hat e = e/\|e\|_2$ and $e=f(c)$ (Xu et al., 8 Oct 2025).

A notable contrast is Marigold-DC, which does not add explicit feature adapters at all. Instead, it exploits the pretrained latent diffusion model’s own hierarchical features by concatenating the image latent $z_I$ with the depth latent $x_t$ at the U-Net input and by backpropagating sparse-depth guidance through the decoder, Tweedie preview, and denoiser at each denoising step (Viola et al., 2024).

3. Diffusion dynamics, supervision, and steering

Most depth diffusion formulations retain the standard Gaussian forward process,

$$q(x_t \mid x_0)=\mathcal{N}\!\left(x_t;\sqrt{\bar{\alpha}_t}\,x_0,\,(1-\bar{\alpha}_t)I\right),$$

and learn a reverse map conditioned on image, depth, or auxiliary geometry. Marigold-DC follows this latent diffusion pattern, uses DDIM sampling, previews the clean latent with

$$\hat x_0=\frac{x_t-\sqrt{1-\bar{\alpha}_t}\,\hat\epsilon_\theta(x_t,t,z_I)}{\sqrt{\bar{\alpha}_t}},$$

decodes it to affine-invariant depth, and optimizes a sparse measurement-consistency objective jointly over the latent and global affine parameters at test time (Viola et al., 2024).

Other methods alter the supervisory target or denoising schedule to suit depth data. DiffusionDepth performs latent-space self-diffusion rather than diffusing sparse ground-truth depth directly, because directly diffusing sparse GT leads to failure on KITTI; it instead diffuses the model’s own refined latent depth and trains with a DDIM-style $x_{t-1}$ prediction objective, together with scale-invariant pixel loss and latent alignment loss on valid pixels (Duan et al., 2023). DepthGen addresses noisy and incomplete depth supervision with masked $F_{\mathrm{mid}}$0 noise prediction, training-time depth infilling, and step-unrolled denoising diffusion, thereby reducing the train–test mismatch in noisy latent marginals (Saxena et al., 2023).

Several papers explicitly depart from multi-step ancestral denoising. DidSee identifies signal leakage bias and exposure bias in vanilla latent diffusion for depth completion on non-Lambertian objects, introduces a rescaled scheduler with zero terminal SNR, and trains a noise-agnostic single-step model at $F_{\mathrm{mid}}$1 with a pixel-space objective

$F_{\mathrm{mid}}$2

where the second term supervises palette-based semantic regression (Lyu et al., 26 Jun 2025). DMD uses v-parameterization,

$F_{\mathrm{mid}}$3

with reconstructions

$F_{\mathrm{mid}}$4

to reach competitive metric depth with only a small number of denoising steps (Saxena et al., 2023). Pixel-Perfect Depth replaces DDPM-style training with flow matching,

$F_{\mathrm{mid}}$5

and minimizes

$F_{\mathrm{mid}}$6

which is coupled with a gradient-matching loss to sharpen edges (Xu et al., 8 Oct 2025).

Inference-time steering provides another route. SteeredMarigold leaves the pretrained Marigold backbone unchanged, decodes the clean-sample estimate at every reverse step, replaces its low-frequency depth component by a sparse-depth-consistent interpolation, re-encodes the image-space delta, and updates the latent through

$F_{\mathrm{mid}}$7

thereby steering multi-scale latent features toward sparse-depth-consistent geometry without retraining (Gregorek et al., 2024).

4. Major application domains

Depth diffusion features now support a broad set of geometric and perception tasks rather than a single monocular depth pipeline. In depth completion, DenseFormer combines sparse depth and RGB guidance in a conditional diffusion model and supplements denoising with multi-step refinement (Yuan et al., 31 Mar 2025). Marigold-DC reframes monocular depth completion as image-conditioned depth generation with sparse test-time guidance and reports strong zero-shot performance across NYU-Depth V2, ScanNet, iBims-1, VOID, KITTI DC, and DDAD (Viola et al., 2024). DidSee adapts diffusion priors to non-Lambertian robotic depth completion and jointly predicts semantics to separate objects from background, improving downstream category-level pose estimation and robotic grasping (Lyu et al., 26 Jun 2025). SteeredMarigold targets largely incomplete depth maps and reports state-of-the-art performance on NYUv2 when large contiguous regions contain no valid depth (Gregorek et al., 2024).

In monocular depth estimation, diffusion-based representations appear in iterative latent denoisers, few-step metric-depth models, and single-step deterministic systems. DiffusionDepth formulates monocular depth as self-diffusion in latent depth space and reports state-of-the-art performance on KITTI and NYU-Depth-V2 with acceptable inference time (Duan et al., 2023). DepthGen shows that a generic image-to-image diffusion architecture, combined with pretraining, infilling, and step-unrolled training, achieves state-of-the-art REL on NYU and near state-of-the-art results on KITTI while naturally representing multimodal uncertainty (Saxena et al., 2023). DMD conditions depth diffusion on field of view to obtain zero-shot metric depth across indoor and outdoor domains (Saxena et al., 2023). DepthMaster uses adapted Stable Diffusion features in a single deterministic step to improve generalization and detail preservation (Song et al., 5 Jan 2025). Pixel-Perfect Depth moves the entire process to pixel space and reports the best performance among published generative models across five benchmarks, together with substantially improved edge-aware point cloud quality (Xu et al., 8 Oct 2025).

The same feature family also appears in multi-view and cross-modal geometry. MVDD uses multi-view depth as a 2D diffusion representation of 3D shape, introduces epipolar line segment attention and a depth fusion module, and reports state-of-the-art results in 3D shape generation, depth completion, and downstream 3D prior applications (Wang et al., 2023). FreeReg extracts diffusion features from Stable Diffusion and ControlNet depth conditioning, fuses them with local geometric descriptors, and reports average gains of $F_{\mathrm{mid}}$8 percent in Inlier Ratio, roughly a three-fold higher Inlier Number, and $F_{\mathrm{mid}}$9 percent improvement in Registration Recall for image-to-point-cloud registration (Wang et al., 2023).

Depth diffusion features also act as transferable structure priors in non-depth outputs. Depth-guided Texture Diffusion diffuses RGB texture into depth before semantic segmentation and reports state-of-the-art results in camouflaged object detection, salient object detection, and indoor semantic segmentation (Sun et al., 2024). DUViN conditions a diffusion navigation policy on knowledge-transferred depth features and uses them to generate 4-DoF action sequences for underwater obstacle avoidance and altitude-aware navigation (Yang et al., 3 Sep 2025).

5. Robustness, efficiency, and generalization

A central motivation for depth diffusion features is robustness under sparse labels, corrupted imagery, incomplete depth, or domain shift. DepthGen addresses missing and noisy supervision through nearest-neighbor depth infilling, masked $z' = h_\phi(z \oplus B(\hat e))$0 noise prediction, and step-unrolled denoising diffusion; the resulting model improves from $z' = h_\phi(z \oplus B(\hat e))$1 to $z' = h_\phi(z \oplus B(\hat e))$2 REL on NYU and from $z' = h_\phi(z \oplus B(\hat e))$3 to $z' = h_\phi(z \oplus B(\hat e))$4 on KITTI when infilling and SUD are combined (Saxena et al., 2023). D4RD targets adverse weather and corruption by introducing a multi-level “trinity” contrastive scheme over noise, feature, and image levels, together with pseudo-depth distillation enhancement, outlier depth removal, and feature-image joint conditioning; it reports robustness gains on WeatherKITTI, KITTI-C, DrivingStereo, and Dense (Wang et al., 2024). A self-supervised hierarchical feature-guided diffusion model conditions denoising on a pyramid of image features and combines photometric loss, smoothness, DDIM consistency, and implicit depth consistency, reporting improved robustness under motion blur, rain, and sensor noise (Liu et al., 2024).

Generalization across domains is equally prominent. Marigold-DC builds on a synthetic-only pretrained monocular depth diffusion prior and still reports strong zero-shot depth completion on real indoor and outdoor datasets (Viola et al., 2024). DMD jointly models indoor and outdoor metric depth through log-depth parameterization, FOV conditioning, synthetic FOV augmentation, and a diverse training mixture, yielding about a $z' = h_\phi(z \oplus B(\hat e))$5 reduction in indoor REL and $z' = h_\phi(z \oplus B(\hat e))$6 reduction in outdoor REL over the prior zero-shot state of the art (Saxena et al., 2023). DepthMaster trains on approximately $z' = h_\phi(z \oplus B(\hat e))$7k synthetic samples and nonetheless reports state-of-the-art zero-shot performance among diffusion-based methods across KITTI, NYUv2, ETH3D, ScanNet, and DIODE (Song et al., 5 Jan 2025). DUViN explicitly transfers depth features from in-air to underwater environments by adapting the encoder on underwater depth estimation while retaining the trained diffusion navigation policy, and reports improved success and collision-free rates in simulated and real underwater environments (Yang et al., 3 Sep 2025).

Efficiency has become a defining fault line within the field. Multi-step latent diffusion remains computationally heavy, but several lines of work decouple depth diffusion features from long ancestral chains. DepthMaster reduces inference time from $z' = h_\phi(z \oplus B(\hat e))$8 s for iterative denoising to $z' = h_\phi(z \oplus B(\hat e))$9 s in a single deterministic pass, while an iterative deterministic four-pass variant runs at $\hat e = e/\|e\|_2$0 s (Song et al., 5 Jan 2025). DidSee removes multi-step exposure bias entirely through one-step prediction at $\hat e = e/\|e\|_2$1 (Lyu et al., 26 Jun 2025). DMD reaches strong zero-shot metric depth with only $\hat e = e/\|e\|_2$2 denoising steps indoors and $\hat e = e/\|e\|_2$3 outdoors (Saxena et al., 2023). Pixel-Perfect Depth reports $\hat e = e/\|e\|_2$4 s runtime for its large cascaded model and $\hat e = e/\|e\|_2$5 ms for a small variant (Xu et al., 8 Oct 2025). This suggests that depth diffusion features are increasingly being treated as reusable geometry-aware representations rather than as evidence that long stochastic sampling is intrinsically necessary.

6. Limitations, misconceptions, and open directions

One common misconception is that depth diffusion features refer to one standard module analogous to a ResNet feature pyramid. The literature shows the opposite: they may be sparse-depth-and-RGB conditions in a deformable-attention pyramid, pretrained latent U-Net activations, texture-enriched depth maps, semantically prompted transformer tokens, or extracted descriptors from a depth-conditioned diffusion model (Yuan et al., 31 Mar 2025, Viola et al., 2024, Sun et al., 2024, Wang et al., 2023, Xu et al., 8 Oct 2025). A second misconception is that diffusion-based depth must always operate in latent VAE space. Pixel-Perfect Depth performs diffusion directly in pixel space to avoid VAE-induced flying pixels, while DidSee and DepthMaster show that single-step paradigms can still exploit diffusion-trained representations effectively (Xu et al., 8 Oct 2025, Lyu et al., 26 Jun 2025, Song et al., 5 Jan 2025).

Despite progress, the limitations are substantial. Runtime and memory remain dominant concerns in multi-step systems and in pipelines that repeatedly decode and re-encode during steering or ControlNet conditioning (Gregorek et al., 2024, Wang et al., 2023). Strong pretrained backbones improve generalization, but they may encode a generative–discriminative gap, texture bias, or domain-specific appearance statistics that require explicit adaptation through feature alignment, physics-informed transfer, or robust contrastive constraints (Song et al., 5 Jan 2025, Yang et al., 3 Sep 2025, Wang et al., 2024). Sparse-depth guidance can also conflict with the learned visual prior when measurements are noisy, inconsistent, or extremely narrow in range; Marigold-DC explicitly identifies this prior–measurement conflict, and DidSee targets analogous failures on non-Lambertian surfaces (Viola et al., 2024, Lyu et al., 26 Jun 2025).

A plausible implication is that future work will continue to split along three axes. The first is representational: whether depth diffusion features should remain latent, move to pixel space, or become hybrid semantic–geometric tokens. The second is algorithmic: whether geometry should be enforced by iterative stochastic denoising, deterministic few-step solvers, or one-step discriminative adaptation of diffusion backbones. The third is systems-oriented: whether pretrained depth diffusion features can be exported as transferable geometry priors for tasks such as registration, segmentation, 3D reconstruction, or control rather than being used only to predict a final depth image. The recent literature already contains all three tendencies, indicating that “depth diffusion features” has evolved from a model-internal detail into a general design language for geometry-aware generative perception.