Marigold-SSD: Zero-Shot Depth Completion

• The paper introduces a zero-shot depth completion method that combines Marigold diffusion priors with single-step inference and a conditional decoder for late fusion of sparse depth.
• The approach bypasses iterative sampling by fixing the inference timestep, significantly reducing computational latency while maintaining robust performance.
• Benchmark evaluations on diverse indoor and outdoor datasets show improved MAE and RMSE over baseline methods, even under varying sparsity levels.

Marigold-SSD denotes, in the provided literature, two distinct uses built on the Marigold diffusion framework. Most commonly, it refers to a zero-shot depth completion method that converts an RGB image $M$ and a sparse depth map $C$ into a dense depth map $D$ by combining a Marigold-derived diffusion prior with single-step inference and late fusion of sparse depth inside a conditional decoder (Gregorek et al., 11 Mar 2026). In a separate usage, the label is described as an apt informal shorthand for Marigold with Single-pair Same-view Defocus guidance: a training-free procedure that injects defocus blur cues at inference time to turn Marigold from a scale-invariant monocular depth estimator into a metric depth predictor, although the original paper does not name the method “SSD” (Talegaonkar et al., 23 May 2025). Both usages are rooted in the broader Marigold family, which adapts pretrained latent diffusion models such as Stable Diffusion v2 to dense image analysis tasks with minimal architectural changes (Ke et al., 14 May 2025).

1. Terminology and lineage

Marigold is a family of conditional generative models and a fine-tuning protocol that extracts the knowledge from pretrained latent diffusion models like Stable Diffusion and adapts them for dense image analysis tasks, including monocular depth estimation, surface normals prediction, and intrinsic decomposition (Ke et al., 14 May 2025). Its core design reuses the Stable Diffusion v2 VAE to encode both the input RGB and the target modality, conditions the UNet by latent concatenation, disables text conditioning, and fine-tunes only the UNet. For depth, the resulting predictor is affine-invariant rather than metric: the original Marigold formulation estimates depth only up to global scale and shift, and absolute metric depth is not estimated (Ke et al., 14 May 2025).

Within that lineage, the acronym “SSD” is not part of the original Marigold family nomenclature. The Marigold paper states that there is no “Marigold-SSD” variant in the paper, and the acronym “SSD” is not used anywhere in the Marigold family (Ke et al., 14 May 2025). It further notes that, if “SSD” is intended to mean single-step diffusion, the closest Marigold components are Marigold v1.1 with DDIM “trailing” timesteps and Marigold-LCM, which explicitly distills one-step or few-step generation; if “SSD” is intended to mean self-supervised depth, that interpretation does not apply, because Marigold uses supervised synthetic data only (Ke et al., 14 May 2025).

This terminological point matters because later work uses “Marigold-SSD” in a precise and explicit sense for single-step depth completion, while another paper discusses the label only as an apt descriptor for a defocus-guided metric-depth method. A plausible implication is that the same string identifies related but non-identical descendants of the Marigold prior.

2. Marigold-SSD as zero-shot depth completion

In its explicit, named form, Marigold-SSD is a zero-shot depth completion framework that turns sparse sensor measurements into a dense depth map, guided by an input RGB image (Gregorek et al., 11 Mar 2026). The problem setting is defined by an RGB image $M \in \mathbb{R}^{H \times W \times 3}$, a sparse depth condition $C \in \mathbb{R}^{H \times W}$, the implicit set of valid measurements $\Omega$, and an output dense depth map $D \in \mathbb{R}^{H \times W}$ (Gregorek et al., 11 Mar 2026). It differs from monocular depth estimation in that completion must respect the provided sparse measurements while hallucinating missing regions under varying sparsity patterns and domain shifts.

The method starts from Marigold trained for monocular depth in the latent space of a frozen VAE encoder and decoder. A UNet denoiser is conditioned on the RGB latent $m = \mathcal{E}(M)$ and predicts a $v$-parameterized velocity for the latent depth (Gregorek et al., 11 Mar 2026). The defining change is the removal of iterative diffusion sampling at test time. Single-step, in this setting, means that the model predicts the clean latent in one denoising step at test time, with no iterative sampling. Following recent scheduler insights, the timestep is fixed to $t = T$ and the noise is set to zero, so inference becomes deterministic:

$C$0

The sparse depth is not injected into the latent denoiser. Instead, it is fused late, inside a conditional decoder $C$1, so that the UNet is trained to produce a good clean latent and the decoder learns to respect $C$2 (Gregorek et al., 11 Mar 2026).

Late fusion is central to the method’s design. The sparse depth condition is injected downstream in the decoder, not into the UNet encoder or the latent denoiser; conditioning features computed from $C$3 are fused with decoder features at multiple scales through zero-initialized $C$4 convolutional gates (Gregorek et al., 11 Mar 2026). This yields a single forward pass at inference, with no classifier-free guidance, no guidance scales, and no ensembling (Gregorek et al., 11 Mar 2026).

3. Architecture, objective, and training regime

The depth completion architecture consists of a frozen VAE encoder $C$5, a UNet denoiser $C$6, a conditional decoder $C$7 that replaces the original VAE decoder, and a condition feature extractor $C$8 initialized from $C$9 with adapted downsampling positions (Gregorek et al., 11 Mar 2026). The conditional decoder mirrors the VAE’s multi-scale hierarchy with $D$0 scales. At each scale $D$1, decoder features from $D$2 and condition features from $D$3 are concatenated and fused via a $D$4 convolution:

$D$5

These fusion gates are zero-initialized in a ControlNet-style manner so that the original decoder behavior is preserved at the start of fine-tuning (Gregorek et al., 11 Mar 2026). UP, DOWN, and MID blocks are ResNet-based, and the MID block includes attention (Gregorek et al., 11 Mar 2026).

The training objective for the original Marigold depth pretraining uses the standard diffusion $D$6-parameterization. With clean latent $D$7, forward process

$D$8

and target

$D$9

the denoising objective is

$M \in \mathbb{R}^{H \times W \times 3}$0

For Marigold-SSD fine-tuning, the task loss is an $M \in \mathbb{R}^{H \times W \times 3}$1 loss on dense depth:

$M \in \mathbb{R}^{H \times W \times 3}$2

where $M \in \mathbb{R}^{H \times W \times 3}$3 (Gregorek et al., 11 Mar 2026).

Metric depth is recovered after decoding through least-squares alignment over valid sparse pixels:

$M \in \mathbb{R}^{H \times W \times 3}$4

This preserves the relative-depth output of the latent model while forcing agreement with metric sparse measurements (Gregorek et al., 11 Mar 2026).

Training is initialized from Marigold-E2E. The encoder remains frozen, while the UNet and conditional decoder are fine-tuned with higher learning rate for the decoder to emphasize adaptation for completion. The optimizer is AdamW, with learning rates $M \in \mathbb{R}^{H \times W \times 3}$5 for $M \in \mathbb{R}^{H \times W \times 3}$6 and $M \in \mathbb{R}^{H \times W \times 3}$7 for the UNet, warmup of 100 steps, exponential decay, and gradient accumulation over 32 steps with micro-batch size 1. Training runs for 20K iterations on a single NVIDIA H100, costs 4.5 GPU-days per model, and mixes Hypersim and Virtual KITTI at a 9:1 ratio (Gregorek et al., 11 Mar 2026). During fine-tuning, sparsity is uniformly sampled from either $M \in \mathbb{R}^{H \times W \times 3}$8 or $M \in \mathbb{R}^{H \times W \times 3}$9 for an indoor-only model (Gregorek et al., 11 Mar 2026).

4. Benchmarks, efficiency, and comparative performance

Marigold-SSD is evaluated zero-shot on four indoor datasets—NYUv2, ScanNet, VOID, and IBims-1—and two outdoor datasets—KITTI and DDAD (Gregorek et al., 11 Mar 2026). The training sets are Hypersim, with 365 scenes and approximately 54K samples at $C \in \mathbb{R}^{H \times W}$0, and Virtual KITTI, with 5 scenes times weather variants and approximately 21K samples cropped to $C \in \mathbb{R}^{H \times W}$1 (Gregorek et al., 11 Mar 2026). Performance is reported with Mean Absolute Error and Root Mean Squared Error in meters.

Across the six benchmarks, the method achieves average RMSE 1.500 and MAE 0.474, outperforming Marigold-DC, which records RMSE 1.758 and MAE 0.640, while running at approximately 66 times lower latency (Gregorek et al., 11 Mar 2026). With ensembling, Marigold-DC improves to RMSE 1.469 and MAE 0.510, but at approximately 10 times higher runtime than its non-ensemble version and thus approximately 660 times slower than Marigold-SSD overall (Gregorek et al., 11 Mar 2026).

Model	Average latency across 6 datasets	Average RMSE / MAE
Marigold-SSD	0.42 s per frame ($C \in \mathbb{R}^{H \times W}$2 FPS)	1.500 / 0.474
Marigold-DC	27.49 s ($C \in \mathbb{R}^{H \times W}$3 FPS)	1.758 / 0.640
Marigold-DC with ensembling	$C \in \mathbb{R}^{H \times W}$4 higher runtime than non-ensemble	1.469 / 0.510

Dataset-specific results show strong indoor and mixed-domain behavior. For the non-star model trained on $C \in \mathbb{R}^{H \times W}$5, Marigold-SSD reports ScanNet MAE 0.027 and RMSE 0.068, IBims-1 MAE 0.060 and RMSE 0.185, VOID MAE 0.182 and RMSE 0.590, NYUv2 MAE 0.052 and RMSE 0.134, KITTI MAE 0.454 and RMSE 1.496, and DDAD MAE 2.066 and RMSE 6.524 (Gregorek et al., 11 Mar 2026). On KITTI, the speed–performance comparison is especially notable: Marigold-SSD runs at 0.527 s with RMSE 1.496, compared with Marigold-DC at 35.103 s and RMSE 1.676; discriminative baselines at original resolution include VPP4DC at 0.164 s and RMSE 1.609, CompletionFormer at 0.151 s and RMSE 1.935, and NLSPN at 0.039 s and RMSE 2.076 (Gregorek et al., 11 Mar 2026).

These figures support two technical conclusions stated in the paper. First, Marigold-SSD substantially narrows the efficiency gap between diffusion-based and discriminative depth completion. Second, it preserves strong zero-shot behavior under domain shift while removing the costly iterative inference characteristic of guided diffusion completion (Gregorek et al., 11 Mar 2026).

5. Robustness, ablations, and limitations

A central theme of the Marigold-SSD evaluation is robustness under varying sparsity levels. The paper explicitly varies the number of sparse points—for example 500, 1500, 15360, or approximately 5000 on DDAD—and compares Marigold-SSD, Marigold-DC, and barycentric interpolation over a Delaunay triangulation (Gregorek et al., 11 Mar 2026). Accuracy improves as input density increases for all methods. At high densities, simple interpolation becomes competitive on IBims-1 and NYUv2, and on DDAD as few as approximately 5000 points allow interpolation to surpass more sophisticated models; interpolation on DDAD achieves MAE 1.598 and RMSE 6.831 (Gregorek et al., 11 Mar 2026). At lower densities, such as 1500 or 500 points, Marigold-SSD outperforms both Marigold-DC and interpolation, highlighting the value of strong priors when measurements are scarce (Gregorek et al., 11 Mar 2026).

The ablation on fusion strategy shows that early-fusion variants consistently underperform the proposed late-fusion decoder. On KITTI, a Frozen-VAE early-fusion variant with interpolated condition yields RMSE approximately 1.791 versus Marigold-SSD’s 1.496; on ScanNet, Frozen-VAE with interpolated condition yields RMSE approximately 0.070 versus 0.068 for Marigold-SSD, but the paper reports that late fusion has a clear overall advantage across datasets (Gregorek et al., 11 Mar 2026). The stated takeaway is that off-the-shelf VAE encoders are ill-suited for sparse maps, and that pre-completing sparse depth helps but remains weaker than the proposed multi-scale late-fusion decoder (Gregorek et al., 11 Mar 2026).

The indoor “star” model trained only on lower densities, $C \in \mathbb{R}^{H \times W}$6, is stronger on indoor datasets but degrades markedly outdoors, with KITTI MAE 2.443 and RMSE 4.070, and DDAD MAE 3.870 and RMSE 7.841 (Gregorek et al., 11 Mar 2026). This confirms the importance of training across a broad sparsity spectrum when generalization to outdoor LiDAR patterns is required.

The reported limitations are correspondingly specific. The model is sensitive to the sparsity distribution seen during training; both Marigold-SSD and Marigold-DC can underestimate sky depth on outdoor scenes; and when input depth becomes very dense, classical interpolation may suffice or even outperform learned models (Gregorek et al., 11 Mar 2026). The paper suggests adaptive normalization of the condition to the VAE’s operational range and improved modeling of outdoor depth distributions as possible mitigation directions (Gregorek et al., 11 Mar 2026).

6. Informal usage for defocus-guided metric depth

A different paper uses “Marigold-SSD” only descriptively, not as an official model name. It states that the original paper does not name the method “SSD,” but that, if referred to as Marigold-SSD, the term aptly describes Marigold with Single-pair Same-view Defocus guidance: a training-free, zero-shot, metric depth estimator that optimizes the Marigold noise latents and global scale parameters against a defocus physics loss (Talegaonkar et al., 23 May 2025). This usage concerns monocular metric depth estimation rather than depth completion.

The method begins from Marigold as a diffusion-based monocular depth estimator fine-tuned from Stable Diffusion v2 to generate scale-invariant depth maps $C \in \mathbb{R}^{H \times W}$7 from a single input image. Scale-invariant means the predicted depth is determined only up to an affine transform, whereas metric depth $C \in \mathbb{R}^{H \times W}$8 in meters is required by downstream applications (Talegaonkar et al., 23 May 2025). To break that global scale ambiguity, the method captures two radiometrically linear images of the same static scene and viewpoint with the same focus distance $C \in \mathbb{R}^{H \times W}$9 and focal length $\Omega$0 but different apertures: a small-aperture image $\Omega$1 at approximately $\Omega$2 that approximates an all-in-focus image, and a large-aperture image $\Omega$3 at smaller $\Omega$4, for example 8, that exhibits depth-dependent defocus blur (Talegaonkar et al., 23 May 2025). Exposure matching is done in raw linear space by

$\Omega$5

with fixed camera gain (Talegaonkar et al., 23 May 2025).

The defocus image formation model adopts thin-lens and circle-of-confusion physics. The circle-of-confusion radius in pixels is

$\Omega$6

where $\Omega$7 is focal length, $\Omega$8 the focus distance, $\Omega$9 the F-number, and $D \in \mathbb{R}^{H \times W}$0 the pixel pitch (Talegaonkar et al., 23 May 2025). The corresponding point spread function is modeled as a normalized disc kernel with a small linear fall-off at the edge, and the defocused image is formed by a spatially varying convolution

$D \in \mathbb{R}^{H \times W}$1

Compared with a Gaussian PSF, the disc PSF more closely matches real camera bokeh and improves metric recovery (Talegaonkar et al., 23 May 2025).

Metric depth is parameterized as

$D \in \mathbb{R}^{H \times W}$2

with $D \in \mathbb{R}^{H \times W}$3 and $D \in \mathbb{R}^{H \times W}$4, where $D \in \mathbb{R}^{H \times W}$5 and $D \in \mathbb{R}^{H \times W}$6 are learned scalars and $D \in \mathbb{R}^{H \times W}$7 are conservative scene bounds (Talegaonkar et al., 23 May 2025). Crucially, the method also optimizes the diffusion noise latent $D \in \mathbb{R}^{H \times W}$8 controlling Marigold’s relative depth output. With Marigold-LCM in single-step mode, $D \in \mathbb{R}^{H \times W}$9 becomes differentiable with respect to $m = \mathcal{E}(M)$0, and the optimization minimizes

$m = \mathcal{E}(M)$1

subject to $m = \mathcal{E}(M)$2 (Talegaonkar et al., 23 May 2025). Adam is used with learning rates $m = \mathcal{E}(M)$3 for $m = \mathcal{E}(M)$4 and $m = \mathcal{E}(M)$5 for $m = \mathcal{E}(M)$6, for 200 iterations, with rescaling of $m = \mathcal{E}(M)$7 to $m = \mathcal{E}(M)$8 after each step (Talegaonkar et al., 23 May 2025).

On a self-collected real dataset of 7 indoor scenes—thordog, books, stairs, plane, toys, shoerack, and kitchen—captured with a Canon 5D Mark II and RealSense D435 ground truth, the defocus-guided method reports RMSE 0.273, REL 0.125, $m = \mathcal{E}(M)$9 0.052, $v$0 0.879, $v$1 0.975, and $v$2 0.991 (Talegaonkar et al., 23 May 2025). Baselines on the same dataset are MLPro with RMSE 0.468, REL 0.246, $v$3 0.105, $v$4 0.597, $v$5 0.821, $v$6 0.990; UniDepth with RMSE 0.644, REL 0.376, $v$7 0.157, $v$8 0.259, $v$9 0.684, $t = T$0 0.954; and Metric3D with RMSE 0.459, REL 0.295, $t = T$1 0.106, $t = T$2 0.650, $t = T$3 0.825, $t = T$4 0.895 (Talegaonkar et al., 23 May 2025). Ablations show that replacing the disc PSF with a Gaussian increases RMSE from 0.273 to 0.528 and lowers $t = T$5 from 0.879 to 0.422, and that optimizing only scale parameters while keeping $t = T$6 fixed degrades performance to RMSE 0.297, REL 0.156, and $t = T$7 0.743 (Talegaonkar et al., 23 May 2025).

This alternative usage should not be conflated with the explicit single-step depth completion framework. One is a training-free, inference-time optimization method for metric monocular depth using an all-in-focus / blurred image pair; the other is an end-to-end fine-tuned, single-step depth completion method using RGB plus sparse depth. The shared nomenclature arises from their common dependence on Marigold and on single-step or physically guided adaptations of its diffusion prior.