Iris: Bringing Real-World Priors into Diffusion Model for Monocular Depth Estimation

Abstract: In this paper, we propose \textbf{Iris}, a deterministic framework for Monocular Depth Estimation (MDE) that integrates real-world priors into the diffusion model. Conventional feed-forward methods rely on massive training data, yet still miss details. Previous diffusion-based methods leverage rich generative priors yet struggle with synthetic-to-real domain transfer. Iris, in contrast, preserves fine details, generalizes strongly from synthetic to real scenes, and remains efficient with limited training data. To this end, we introduce a two-stage Priors-to-Geometry Deterministic (PGD) schedule: the prior stage uses Spectral-Gated Distillation (SGD) to transfer low-frequency real priors while leaving high-frequency details unconstrained, and the geometry stage applies Spectral-Gated Consistency (SGC) to enforce high-frequency fidelity while refining with synthetic ground truth. The two stages share weights and are executed with a high-to-low timestep schedule. Extensive experimental results confirm that Iris achieves significant improvements in MDE performance with strong in-the-wild generalization.

• The paper introduces Iris, a deterministic two-stage diffusion framework that separates low-frequency real-world prior alignment from high-frequency synthetic geometry refinement using spectral-gated distillation.
• Iris achieves leading diffusion-based zero-shot depth results, including 7.2% AbsRel on KITTI, 5.5% on ETH3D, and 5.0% on ScanNet, using only 159K training images.
• The method surpasses or approaches much larger models while running in 1.3 seconds at 1536² resolution, though indoor performance and dependence on teacher pseudo-label quality remain limitations.

Iris addresses a persistent trade-off in monocular depth estimation (MDE): conventional feed-forward estimators such as the Depth Anything family achieve strong cross-domain generalization but require tens of millions of training images and still produce over-smoothed details, while diffusion-based perception models inherit rich generative priors from Stable Diffusion and preserve fine boundaries, but generalize poorly beyond their synthetic training domain. Iris proposes a deterministic, single-step diffusion framework that injects real-world priors into the diffusion backbone through a two-stage spectral-gated distillation scheme, achieving competitive accuracy with only 59K synthetic images plus 100K pseudo-labeled real images (2603.16340).

Motivation and problem formulation

The paper identifies two supervision bottlenecks: real-world datasets provide imperfect depth annotations that suppress fine detail, while synthetic datasets offer precise annotations but suffer from a pronounced domain gap. The Depth Anything V2 (DAv2) pipeline mitigates this via massive pseudo-labeling and distillation at a scale of 62.6M images, which is difficult to replicate and still underperforms on boundary fidelity. Diffusion-based methods such as Marigold, GenPercept, and Lotus fine-tune on small synthetic sets and retain sharp detail, yet degrade under synthetic-to-real shift.

A key diagnostic observation motivates the design: teacher pseudo labels from DAv2 are reliable in low-frequency bands (global layout, object extents) but underspecify high-frequency content, whereas synthetic ground truth supplies accurate high-frequency geometry. Training a student to regress both signals simultaneously induces gradient interference. Iris therefore decouples these objectives across two diffusion states.

Priors-to-Geometry Deterministic framework

Following Lotus and GenPercept, Iris discards iterative sampling and uses the U-Net denoiser of Stable Diffusion V2 as a deterministic feed-forward predictor $\hat{z}^y = f_\theta(z^x, t)$, with the timestep $t$ serving purely as a conditioning index. The Priors-to-Geometry Deterministic (PGD) schedule runs a shared-weight predictor through two stages:

• Stage 1 (prior alignment) operates at $t=1000$, the low-SNR regime, conditioning the network toward global layout. Spectral-Gated Distillation (SGD) supervises it with pseudo labels from a frozen DAv2-Large teacher on real images.
• Stage 2 (geometry refinement) takes the stage-1 output as input and operates at $t=500$, the high-SNR regime, trained on synthetic ground truth for metric calibration and high-frequency geometry.

Spectral-Gated Distillation and Consistency

SGD learns a three-parameter Fourier-domain low-pass gate with learnable cutoff, slope, and residual strength, and constrains the student to match only the gated low-band spectrum of the teacher's latent prediction. High-frequency components are intentionally left unconstrained in this stage, preventing imprinting of teacher-specific artifacts — an ablation shows vanilla full-band distillation degrades KITTI AbsRel from 7.4% to 7.5% and ScanNet from 5.2% to 5.3% relative to gated distillation.

A counterintuitive finding drives the second component: despite low-pass alignment, the stage-1 output often exhibits sharper boundaries than the refined stage-2 output, because concentrating supervision on stable global structure implicitly favors steeper edge transitions. SGC exploits this by aligning stage-2 to stage-1 via a complementary high-pass gate with stop-gradient, plus a weighted ($\beta=0.1$) over-activation constraint; omitting this constraint lets stage-1 over-amplify high frequencies and degrades final accuracy.

An auxiliary image reconstruction loss (following Lotus) preserves the text-to-image backbone's detail capacity against catastrophic forgetting, applied at stage 2 on both real and synthetic samples.

Results

On zero-shot affine-invariant depth estimation, Iris ranks first among 16 methods in both overall and group-average ranking:

Benchmark	Iris AbsRel	Best prior diffusion baseline
KITTI	7.2	7.8 (GenPercept)
ETH3D	5.5	5.9 (Lotus-G)
ScanNet	5.0	5.1 (Lotus-D)
NYUv2	4.9	4.3 (DepthAnything)
DIODE	24.3	22.8 (Lotus-D/GenPercept)

Notably, Iris surpasses DAv2 on KITTI, ETH3D, and ScanNet despite using roughly 0.25% of its training data, and achieves the best DA-2K accuracy (94.5%) among all diffusion-based methods, substantially narrowing the gap to DAv2 (97.1%). Inference takes 1.3 s at $1536^2$ resolution — faster than DAv2 (2.2 s) and orders of magnitude faster than multi-step Marigold (377.7 s). The appendix additionally demonstrates joint depth and surface-normal estimation within a single shared-weight model via task switchers.

Limitations and open questions

The gains on indoor benchmarks are more modest than outdoor ones; NYUv2 AbsRel remains behind Depth Anything (4.9 vs. 4.3). The authors attribute this to the SA-1B distillation subset being dominated by outdoor scenes, and note that incorporating indoor-heavy real data is left for future work. More broadly, the method depends on the quality of the frozen teacher's pseudo labels — SGD filters but does not eliminate systematic teacher bias in the low-frequency band — and the fixed timesteps ($t=1000$, $t=500$) were chosen empirically without a study of schedule sensitivity.

Conclusion

Iris shows that frequency-resolved, stage-decoupled distillation can transfer real-world priors into a deterministic diffusion backbone efficiently, achieving state-of-the-art zero-shot MDE among diffusion methods and competitive performance against models trained on orders-of-magnitude more data, at modest compute cost.