---
title: 'Marigold-SSD: Zero-Shot Depth Completion'
url: https://www.emergentmind.com/topics/marigold-ssd
type: topic
---

# Marigold-SSD: Zero-Shot Depth Completion

Marigold-SSD denotes, in the provided literature, two distinct uses built on the Marigold diffusion framework. Most commonly, it refers to a zero-shot depth completion method that converts an RGB image $M$ and a sparse depth map $C$ into a dense depth map $D$ by combining a Marigold-derived diffusion prior with single-step inference and late fusion of sparse depth inside a conditional decoder [2603.10584]. In a separate usage, the label is described as an apt informal shorthand for Marigold with Single-pair Same-view Defocus guidance: a training-free procedure that injects defocus blur cues at inference time to turn Marigold from a scale-invariant monocular depth estimator into a metric depth predictor, although the original paper does not name the method “SSD” [2505.17358]. Both usages are rooted in the broader Marigold family, which adapts pretrained latent diffusion models such as Stable Diffusion v2 to dense image analysis tasks with minimal architectural changes [2505.09358].

## 1. Terminology and lineage

Marigold is a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models like Stable Diffusion and adapts them for dense image analysis tasks, including monocular depth estimation, surface normals prediction, and intrinsic decomposition [2505.09358]. Its core design reuses the Stable Diffusion v2 VAE to encode both the input RGB and the target modality, conditions the UNet by latent concatenation, disables text conditioning, and fine-tunes only the UNet. For depth, the resulting predictor is affine-invariant rather than metric: the original Marigold formulation estimates depth only up to global scale and shift, and absolute metric depth is not estimated [2505.09358].

Within that lineage, the acronym “SSD” is not part of the original Marigold family nomenclature. The Marigold paper states that there is no “Marigold-SSD” variant in the paper, and the acronym “SSD” is not used anywhere in the Marigold family [2505.09358]. It further notes that, if “SSD” is intended to mean single-step diffusion, the closest Marigold components are Marigold v1.1 with DDIM “trailing” timesteps and Marigold-LCM, which explicitly distills one-step or few-step generation; if “SSD” is intended to mean self-supervised depth, that interpretation does not apply, because Marigold uses supervised synthetic data only [2505.09358].

This terminological point matters because later work uses “Marigold-SSD” in a precise and explicit sense for single-step depth completion, while another paper discusses the label only as an apt descriptor for a defocus-guided metric-depth method. A plausible implication is that the same string identifies related but non-identical descendants of the Marigold prior.

## 2. Marigold-SSD as zero-shot depth completion

In its explicit, named form, Marigold-SSD is a zero-shot depth completion framework that turns sparse sensor measurements into a dense depth map, guided by an input RGB image [2603.10584]. The problem setting is defined by an RGB image $M \in \mathbb{R}^{H \times W \times 3}$, a sparse depth condition $C \in \mathbb{R}^{H \times W}$, the implicit set of valid measurements $\Omega$, and an output dense depth map $D \in \mathbb{R}^{H \times W}$ [2603.10584]. It differs from monocular depth estimation in that completion must respect the provided sparse measurements while hallucinating missing regions under varying sparsity patterns and domain shifts.

The method starts from Marigold trained for monocular depth in the latent space of a frozen VAE encoder and decoder. A UNet denoiser is conditioned on the RGB latent $m = \mathcal{E}(M)$ and predicts a $v$-parameterized velocity for the latent depth [2603.10584]. The defining change is the removal of iterative diffusion sampling at test time. Single-step, in this setting, means that the model predicts the clean latent in one denoising step at test time, with no iterative sampling. Following recent scheduler insights, the timestep is fixed to $t = T$ and the noise is set to zero, so inference becomes deterministic:
$$
\hat{x}_0 = \sqrt{\bar{\alpha}_T} x_T - \sqrt{1 - \bar{\alpha}_T}\,\hat{v}_T, \qquad x_T = 0.
$$
The sparse depth is not injected into the latent denoiser. Instead, it is fused late, inside a conditional decoder $\mathcal{D}_{C,\phi}$, so that the UNet is trained to produce a good clean latent and the decoder learns to respect $C$ [2603.10584].

Late fusion is central to the method’s design. The sparse depth condition is injected downstream in the decoder, not into the UNet encoder or the latent denoiser; conditioning features computed from $C$ are fused with decoder features at multiple scales through zero-initialized $1 \times 1$ convolutional gates [2603.10584]. This yields a single forward pass at inference, with no classifier-free guidance, no guidance scales, and no ensembling [2603.10584].

## 3. Architecture, objective, and training regime

The depth completion architecture consists of a frozen VAE encoder $\mathcal{E}$, a UNet denoiser $v_\theta$, a conditional decoder $\mathcal{D}_{C,\phi}$ that replaces the original VAE decoder, and a condition feature extractor $\mathcal{F}$ initialized from $\mathcal{E}$ with adapted downsampling positions [2603.10584]. The conditional decoder mirrors the VAE’s multi-scale hierarchy with $L = 5$ scales. At each scale $l$, decoder features from $\hat{x}_0$ and condition features from $C$ are concatenated and fused via a $1 \times 1$ convolution:
$$
f_l = \mathrm{CONV}_{1 \times 1}\!\left(f_l^{\mathcal{D}_C} \oplus f_l^{\mathcal{F}}\right).
$$
These fusion gates are zero-initialized in a ControlNet-style manner so that the original decoder behavior is preserved at the start of fine-tuning [2603.10584]. UP, DOWN, and MID blocks are ResNet-based, and the MID block includes attention [2603.10584].

The training objective for the original Marigold depth pretraining uses the standard diffusion $v$-parameterization. With clean latent $x_0 = \mathcal{E}(D)$, forward process
$$
x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1 - \bar{\alpha}_t}\,\epsilon,
$$
and target
$$
v_t^* = \sqrt{\bar{\alpha}_t}\,\epsilon - \sqrt{1 - \bar{\alpha}_t}\,x_0,
$$
the denoising objective is
$$
\mathbb{E}_{t,\epsilon}\!\left[\left\|v_t^* - v_\theta(x_t \oplus m, t)\right\|_2^2\right].
$$
For Marigold-SSD fine-tuning, the task loss is an $L_1$ loss on dense depth:
$$
\mathcal{L}_{\text{depth}} = \mathbb{E}\!\left[\left\|\hat{D} - D\right\|_1\right],
$$
where $\hat{D} = \mathcal{D}_{C,\phi}(\hat{x}_0, C)$ [2603.10584].

Metric depth is recovered after decoding through least-squares alignment over valid sparse pixels:
$$
D^* = a\hat{D} + b, \qquad
(a,b) = \arg\min_{a,b} \sum_{i \in \Omega} (a\hat{D}_i + b - C_i^*)^2.
$$
This preserves the relative-depth output of the latent model while forcing agreement with metric sparse measurements [2603.10584].

Training is initialized from Marigold-E2E. The encoder remains frozen, while the UNet and conditional decoder are fine-tuned with higher learning rate for the decoder to emphasize adaptation for completion. The optimizer is AdamW, with learning rates $3 \times 10^{-5}$ for $\mathcal{D}_{C,\phi}$ and $3 \times 10^{-6}$ for the UNet, warmup of 100 steps, exponential decay, and gradient accumulation over 32 steps with micro-batch size 1. Training runs for 20K iterations on a single NVIDIA H100, costs 4.5 GPU-days per model, and mixes Hypersim and Virtual KITTI at a 9:1 ratio [2603.10584]. During fine-tuning, sparsity is uniformly sampled from either $[0.16\%, 5\%]$ or $[0.16\%, 0.5\%]$ for an indoor-only model [2603.10584].

## 4. Benchmarks, efficiency, and comparative performance

Marigold-SSD is evaluated zero-shot on four indoor datasets—NYUv2, ScanNet, VOID, and IBims-1—and two outdoor datasets—KITTI and DDAD [2603.10584]. The training sets are Hypersim, with 365 scenes and approximately 54K samples at $640 \times 480$, and Virtual KITTI, with 5 scenes times weather variants and approximately 21K samples cropped to $1216 \times 352$ [2603.10584]. Performance is reported with Mean Absolute Error and Root Mean Squared Error in meters.

Across the six benchmarks, the method achieves average RMSE 1.500 and MAE 0.474, outperforming Marigold-DC, which records RMSE 1.758 and MAE 0.640, while running at approximately 66 times lower latency [2603.10584]. With ensembling, Marigold-DC improves to RMSE 1.469 and MAE 0.510, but at approximately 10 times higher runtime than its non-ensemble version and thus approximately 660 times slower than Marigold-SSD overall [2603.10584].

| Model | Average latency across 6 datasets | Average RMSE / MAE |
|---|---:|---:|
| Marigold-SSD | 0.42 s per frame ($\approx 2.4$ FPS) | 1.500 / 0.474 |
| Marigold-DC | 27.49 s ($\approx 0.04$ FPS) | 1.758 / 0.640 |
| Marigold-DC with ensembling | $\approx 10\times$ higher runtime than non-ensemble | 1.469 / 0.510 |

Dataset-specific results show strong indoor and mixed-domain behavior. For the non-star model trained on $[0.16\%, 5\%]$, Marigold-SSD reports ScanNet MAE 0.027 and RMSE 0.068, IBims-1 MAE 0.060 and RMSE 0.185, VOID MAE 0.182 and RMSE 0.590, NYUv2 MAE 0.052 and RMSE 0.134, KITTI MAE 0.454 and RMSE 1.496, and DDAD MAE 2.066 and RMSE 6.524 [2603.10584]. On KITTI, the speed–performance comparison is especially notable: Marigold-SSD runs at 0.527 s with RMSE 1.496, compared with Marigold-DC at 35.103 s and RMSE 1.676; discriminative baselines at original resolution include VPP4DC at 0.164 s and RMSE 1.609, CompletionFormer at 0.151 s and RMSE 1.935, and NLSPN at 0.039 s and RMSE 2.076 [2603.10584].

These figures support two technical conclusions stated in the paper. First, Marigold-SSD substantially narrows the efficiency gap between diffusion-based and discriminative depth completion. Second, it preserves strong zero-shot behavior under domain shift while removing the costly iterative inference characteristic of guided diffusion completion [2603.10584].

## 5. Robustness, ablations, and limitations

A central theme of the Marigold-SSD evaluation is robustness under varying sparsity levels. The paper explicitly varies the number of sparse points—for example 500, 1500, 15360, or approximately 5000 on DDAD—and compares Marigold-SSD, Marigold-DC, and barycentric interpolation over a Delaunay triangulation [2603.10584]. Accuracy improves as input density increases for all methods. At high densities, simple interpolation becomes competitive on IBims-1 and NYUv2, and on DDAD as few as approximately 5000 points allow interpolation to surpass more sophisticated models; interpolation on DDAD achieves MAE 1.598 and RMSE 6.831 [2603.10584]. At lower densities, such as 1500 or 500 points, Marigold-SSD outperforms both Marigold-DC and interpolation, highlighting the value of strong priors when measurements are scarce [2603.10584].

The ablation on fusion strategy shows that early-fusion variants consistently underperform the proposed late-fusion decoder. On KITTI, a Frozen-VAE early-fusion variant with interpolated condition yields RMSE approximately 1.791 versus Marigold-SSD’s 1.496; on ScanNet, Frozen-VAE with interpolated condition yields RMSE approximately 0.070 versus 0.068 for Marigold-SSD, but the paper reports that late fusion has a clear overall advantage across datasets [2603.10584]. The stated takeaway is that off-the-shelf VAE encoders are ill-suited for sparse maps, and that pre-completing sparse depth helps but remains weaker than the proposed multi-scale late-fusion decoder [2603.10584].

The indoor “star” model trained only on lower densities, $[0.16\%, 0.5\%]$, is stronger on indoor datasets but degrades markedly outdoors, with KITTI MAE 2.443 and RMSE 4.070, and DDAD MAE 3.870 and RMSE 7.841 [2603.10584]. This confirms the importance of training across a broad sparsity spectrum when generalization to outdoor LiDAR patterns is required.

The reported limitations are correspondingly specific. The model is sensitive to the sparsity distribution seen during training; both Marigold-SSD and Marigold-DC can underestimate sky depth on outdoor scenes; and when input depth becomes very dense, classical interpolation may suffice or even outperform learned models [2603.10584]. The paper suggests adaptive normalization of the condition to the VAE’s operational range and improved modeling of outdoor depth distributions as possible mitigation directions [2603.10584].

## 6. Informal usage for defocus-guided metric depth

A different paper uses “Marigold-SSD” only descriptively, not as an official model name. It states that the original paper does not name the method “SSD,” but that, if referred to as Marigold-SSD, the term aptly describes Marigold with Single-pair Same-view Defocus guidance: a training-free, zero-shot, metric depth estimator that optimizes the Marigold noise latents and global scale parameters against a defocus physics loss [2505.17358]. This usage concerns monocular metric depth estimation rather than depth completion.

The method begins from Marigold as a diffusion-based monocular depth estimator fine-tuned from Stable Diffusion v2 to generate scale-invariant depth maps $D_{\text{si}}$ from a single input image. Scale-invariant means the predicted depth is determined only up to an affine transform, whereas metric depth $D$ in meters is required by downstream applications [2505.17358]. To break that global scale ambiguity, the method captures two radiometrically linear images of the same static scene and viewpoint with the same focus distance $z_f = F$ and focal length $f$ but different apertures: a small-aperture image $I_{\text{small}}$ at approximately $N_{\text{aif}} \approx 22$ that approximates an all-in-focus image, and a large-aperture image $I_{\text{large}}$ at smaller $N_b$, for example 8, that exhibits depth-dependent defocus blur [2505.17358]. Exposure matching is done in raw linear space by
$$
I_{\text{large}} \leftarrow I_{\text{large}} \times (t_{\text{aif}}/t_b) \times (N_b^2 / N_{\text{aif}}^2),
$$
with fixed camera gain [2505.17358].

The defocus image formation model adopts thin-lens and circle-of-confusion physics. The circle-of-confusion radius in pixels is
$$
c(z) = \frac{f^2}{N} \cdot \frac{|z - F|}{z(F - f)s},
$$
where $f$ is focal length, $F$ the focus distance, $N$ the F-number, and $s$ the pixel pitch [2505.17358]. The corresponding point spread function is modeled as a normalized disc kernel with a small linear fall-off at the edge, and the defocused image is formed by a spatially varying convolution
$$
I_{\text{large}}(i,j) = \iint I_{\text{small}}(u,v)\, h(i-u, j-v \mid D[u,v])\,du\,dv.
$$
Compared with a Gaussian PSF, the disc PSF more closely matches real camera bokeh and improves metric recovery [2505.17358].

Metric depth is parameterized as
$$
D = \alpha D_{\text{si}} + \beta,
$$
with $\alpha = s_{\max}\sigma(a)$ and $\beta = s_{\min}\sigma(b)$, where $a$ and $b$ are learned scalars and $s_{\min}, s_{\max}$ are conservative scene bounds [2505.17358]. Crucially, the method also optimizes the diffusion noise latent $z$ controlling Marigold’s relative depth output. With Marigold-LCM in single-step mode, $D_{\text{si}}$ becomes differentiable with respect to $z$, and the optimization minimizes
$$
\min_{a,b,z}\; \mathcal{L}(a,b,z) = \left\| I_{\text{large}} - g(I_{\text{small}}, y(a,b,z); \theta)\right\|_2^2,
$$
subject to $\|z\|_2 = \sqrt{M}$ [2505.17358]. Adam is used with learning rates $1.5 \times 10^{-3}$ for $z$ and $5 \times 10^{-3}$ for $a,b$, for 200 iterations, with rescaling of $z$ to $\sqrt{M}$ after each step [2505.17358].

On a self-collected real dataset of 7 indoor scenes—thordog, books, stairs, plane, toys, shoerack, and kitchen—captured with a Canon 5D Mark II and RealSense D435 ground truth, the defocus-guided method reports RMSE 0.273, REL 0.125, $\log_{10}$ 0.052, $\delta_1$ 0.879, $\delta_2$ 0.975, and $\delta_3$ 0.991 [2505.17358]. Baselines on the same dataset are MLPro with RMSE 0.468, REL 0.246, $\log_{10}$ 0.105, $\delta_1$ 0.597, $\delta_2$ 0.821, $\delta_3$ 0.990; UniDepth with RMSE 0.644, REL 0.376, $\log_{10}$ 0.157, $\delta_1$ 0.259, $\delta_2$ 0.684, $\delta_3$ 0.954; and Metric3D with RMSE 0.459, REL 0.295, $\log_{10}$ 0.106, $\delta_1$ 0.650, $\delta_2$ 0.825, $\delta_3$ 0.895 [2505.17358]. Ablations show that replacing the disc PSF with a Gaussian increases RMSE from 0.273 to 0.528 and lowers $\delta_1$ from 0.879 to 0.422, and that optimizing only scale parameters while keeping $z$ fixed degrades performance to RMSE 0.297, REL 0.156, and $\delta_1$ 0.743 [2505.17358].

This alternative usage should not be conflated with the explicit single-step depth completion framework. One is a training-free, inference-time optimization method for metric monocular depth using an all-in-focus / blurred image pair; the other is an end-to-end fine-tuned, single-step depth completion method using RGB plus sparse depth. The shared nomenclature arises from their common dependence on Marigold and on single-step or physically guided adaptations of its diffusion prior.

Source: https://www.emergentmind.com/topics/marigold-ssd