Marigold-Based Defocus Blur Depth Estimation

• The Marigold-based Defocus Blur Approach is a zero-shot method for metric depth estimation that combines a pre-trained diffusion model with defocus blur cues at inference.
• It employs dual-aperture image acquisition and a differentiable blur forward model, leveraging thin-lens physics to accurately estimate absolute depth.
• Experimental results show improved RMSE and depth precision over state-of-the-art methods, particularly in out-of-distribution and low-texture scenarios.

The Marigold-based Defocus Blur Approach is a zero-shot, training-free method for metric monocular depth estimation that augments a pre-trained diffusion model, Marigold, with defocus blur cues at inference time. By leveraging the optical physics of depth-dependent defocus and a differentiable blur forward model, this approach enables inference-time optimization for absolute depth, outperforming state-of-the-art monocular metric depth estimation (MMDE) systems in generalization to out-of-distribution scenes (Talegaonkar et al., 23 May 2025).

1. Defocus Blur Image-Formation Physics

The method operationalizes the thin-lens model, quantifying defocus using the Circle-of-Confusion (CoC), which models the diameter of the blur disk for a point at distance $d$:

$c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$

where $f$ is focal length, $F$ is focus distance, $N$ is F-stop, and $s$ is the pixel size. The point-spread function (PSF) for defocus is parameterized as a spatially varying disc with linear rim fall-off, allowing differentiable rendering of the blurred image:

$$x_{\mathrm{blur}}(i,j) = \iint x_{\text{AIF}}(u,v)\, h(i-u,j-v \mid \mathbf d^{\mathrm m}[u,v])\, du\, dv$$

where $x_\text{AIF}$ is the all-in-focus radiance, $\mathbf d^{\mathrm m}$ is the metric depth map, and $h$ is the normalized disc-PSF at each location.

2. Dual-Aperture Image Acquisition and Preprocessing

Inference requires acquisition of two raw images from a fixed viewpoint with known optical parameters:

• A high-aperture ($c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$0) all-in-focus image ($c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$1)
• A lower-aperture ($c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$2) defocused blurred image ($c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$3)

An exposure compensation factor ensures energy constancy between images:

$c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$4

where $c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$5 denotes exposure times. This preprocessing aligns the linear radiance scales of both images, necessary for accurate forward modeling.

3. Inference-Time Optimization Framework

The pipeline formulates metric depth estimation as a constrained optimization task over latent variables:

• Marigold-LCM denoiser $c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$6 operates on a noise latent $c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$7 and AIF-encoded latent $c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$8, producing a depth latent $c(d) = \frac{f^2}{N} \frac{|d-F|}{d(F-f)s}$9.
• The decoded depth $f$0 is affinely mapped to metric depth:

$f$1

with affine scale $f$2 and offset $f$3 determined through unconstrained optimization variables $f$4, $f$5, and scene bounds $f$6, $f$7.

The loss function penalizes the $f$8 norm between the captured blurred image and the model-predicted blur given the current metric depth, all-in-focus image, and camera parameters:

$f$9

Optimization proceeds over $F$0 subject to the norm constraint $F$1, consistent with the sampling from standard normal for $F$2-dimensional latents.

4. Integration into the Marigold Pipeline

The inference-time algorithm is initialized by sampling $F$3 with $F$4 (implying $F$5). For $F$6 iterations:

1. Compute $F$7
2. Decode $F$8
3. Affinely convert to metric depth
4. Forward-blur image synthesis $F$9
5. Evaluate loss and gradients
6. Update $N$0 using Adam or SGD
7. Renormalize $N$1 to maintain $N$2

Convergence is by fixed iteration count or saturation of $N$3. All modules (Marigold-LCM inference, depth decoding, metric mapping, forward blur) are differentiable, enabling unified backpropagation.

5. Approximations and Regularization

The blur forward model utilizes a disc-PSF with linear rim fall-off, eschewing full diffraction simulation for computational efficiency. Single-step latent-consistency sampling is used for Marigold-LCM in place of typical diffusion with 20–50 steps. Sigmoid parameterization for mapping ensures well-behaved depth scales. Latent renormalization enforces a "Gaussian-annulus" prior, with known scene bounds $N$4 constraining metric depth range.

6. Experimental Evaluation and Results

Experiments are conducted on a custom real dataset acquired with a Canon 5D Mark II (50mm lens) and Intel RealSense D435 for depth ground truth, capturing seven indoor scenes across various F-stops (AIF at $N$5, blur at $N$6). Focus is fixed at 0.8m; RealSense provides depth in [0.3, 3.8] m. Standard metrics—RMSE, AbsRel, log10 error, and accuracy thresholds $N$7—are employed [bhat2023zoedepth].

Comparison with zero-shot MMDE baselines (MLPro, UniDepth, Metric3D) demonstrates strong performance:

Method	RMSE (m)	Rel	log10	$N$8
Marigold+Disc-PSF	0.273	0.125	0.052	0.879
MLPro	0.468	0.246	0.105	0.597
UniDepth	0.644	0.376	0.157	0.259
Metric3D	0.459	0.295	0.106	0.650
Gaussian-PSF ablation	0.528	—	—	—

Ablation studies indicate:

• Fixing the latent $N$9 and optimizing only affine parameters yields degraded RMSE (0.297)
• The $s$0 blur achieves best results; too little or too much blur is suboptimal
• Method displays low sensitivity to initialization (std. dev. $s$1 0.02)
• Single Marigold-LCM step is sufficient; additional steps yield marginal gains

Synthetic plane tests ("Texture-Plane" toy) verify that the approach robustly recovers flat depth in the presence of ambiguous texture seen to challenge conventional methods (RMSE $s$2 0.01).

7. Qualitative Outcomes and Broader Implications

Qualitative results on both synthetic and real scenes reveal accurate recovery of both relative and absolute metric scale, with sharper boundaries and correct geometric ordering where prior methods frequently fail on scale or relative depth. The integration of defocus cues at inference reliably disambiguates depth in repetitive or low-texture regions. A plausible implication is that depth-from-blur provides significant complementary signals to deep-learning-based priors when upgraded with physical-camera information.

By posing metric depth estimation as a differentiable inverse problem utilizing both the strong diffusion prior of Marigold and explicit depth-varying blur cues, the framework achieves state-of-the-art results in zero-shot depth estimation under out-of-distribution conditions (Talegaonkar et al., 23 May 2025).