Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Abstract: Monocular depth estimation is a ubiquitous yet highly ill-posed computer vision task, with downstream applications in scene reconstruction, computational photography, and robotics, among others. Despite the field's maturity, recent models still struggle to generalize to out-of-distribution inputs and to produce sharp and detailed depth maps. In this paper, we revisit Marigold, a set of techniques for repurposing modern image generation and editing models, powered by the diffusion transformer (DiT) architecture, into state-of-the-art monocular depth estimators. Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run. We analyze the artifacts of naive training and identify two effective remedies: aligning the model's internal representations with semantic features extracted from ground-truth, and adopting a 2-stage fine-tuning protocol built around a novel Sinkhorn-based loss. The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D. Qualitatively, our model resolves fur, foliage, and hair-thin edges that have eluded prior models. Furthermore, Marigold V2 achieves state-of-the-art results when applied to other dense regression tasks, such as surface normals estimation and intrinsic image decomposition. Project website: https://hf.co/spaces/huawei-bayerlab/marigold-v2-web

• The paper revisits diffusion transformers, introducing Marigold V2, which repurposes an image-editing DiT using 4-bit quantization and QLoRA adapters and trains with novel representations and losses.
• The methodology combines three key properties—strong zero-shot accuracy, improved boundary fidelity, and practical adaptation cost—achieving reductions in global depth error down to 3.6 and improvements in local accurancy up to 98.0%
• The system outperforms previous best models by up to ~15% in absolute relative error (AbsRel) on benchmarks like KITTI and ETH3D, effectively addressing common issues in depth estimation with deep learning.

Problem setting and central contribution

"Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation" (2609.08084) studies whether a pretrained image-editing diffusion transformer can be converted into a high-quality monocular depth estimator without the compute typically associated with adapting large DiTs. The paper addresses two persistent weaknesses of diffusion-based dense prediction: oversmoothed depth discontinuities and the loss of thin, high-frequency structures such as fur, foliage, and hair. Its central claim is that these weaknesses arise not from an intrinsic incompatibility between diffusion priors and geometric prediction, but from the interaction between VAE representations, pixelwise supervision, and noisy or spatially ambiguous depth annotations.

The proposed system repurposes Qwen-Image-Edit-2509 using 4-bit quantization and rank-128 QLoRA adapters. Training is conducted on a mixture of HyperSim and vKITTI using a two-stage protocol. Stage 1 performs single-step latent rectified-flow regression, augmented with pixel-space reconstruction, spatial-gradient, and semantic representation losses. Stage 2 unfreezes the VAE decoder and introduces SinkLoss, an entropy-regularized optimal-transport objective that relaxes exact pixel-to-pixel correspondence within local image blocks.

The resulting model is notable for combining three properties that are usually difficult to obtain simultaneously: strong zero-shot depth accuracy, improved boundary fidelity, and practical adaptation cost. The final Stage-1 run uses 160,000 optimization steps and takes slightly more than five days on a single 32 GB GPU; Stage 2 requires a further approximately 30,000 steps. Inference remains single-pass through the DiT and VAE rather than iterative diffusion sampling.

Model adaptation and training protocol

The target depth representation is affine-invariant log-depth. Metric depth is transformed using robust 2nd and 98th percentile clipping, followed by normalization to $[-1,1]$. This choice removes global scale and shift ambiguity while making the target better aligned with relative-depth evaluation. The authors also report that log-depth outperforms linear depth and disparity under a common training configuration, obtaining an average AbsRel of 4.72 compared with 5.04 for linear depth and 5.28 for disparity.

The Qwen image-editing backbone receives an RGB latent and is trained at a fixed flow-matching timestep, $t=0.5$, to predict the latent displacement between the RGB and depth representations. Because the timestep is fixed, the formulation reduces to direct latent regression: the predicted depth latent is obtained by subtracting the predicted velocity from the RGB latent. This design eliminates the integration cost of multi-step diffusion while retaining the pretrained DiT's representational capacity.

The loss hierarchy is important. Latent-space regression supplies the principal generative supervision, while pixel-space $L_1$ reconstruction constrains decoded outputs and a spatial-gradient loss emphasizes local depth transitions. The paper reports that pixel-space supervision has only a limited effect on AbsRel but consistently improves $\delta_1$, indicating that it primarily benefits local accuracy rather than global affine-invariant structure.

The complete training protocol is summarized below.

Figure 1: Marigold's two-stage training protocol: Stage 1 adapts the DiT with QLoRA, latent and pixel-space supervision, and iREPA-depth; Stage 2 unfreezes the VAE decoder and adds SinkLoss.

iREPA-depth and semantic-geometric alignment

The first major methodological contribution is iREPA-depth, a representation-alignment loss applied to intermediate DiT features. Earlier uses of representation alignment for depth estimation generally extract target features from the RGB input. Marigold V2 instead extracts DINOv3 features from the ground-truth depth map and aligns the model's internal representations with these geometric features. The distinction is substantive: RGB features encode semantic appearance, whereas depth-derived features expose structural organization more directly.

The ablation results support this design. In the 30,000-step experiments, the depth-targeted iREPA variant provides the strongest overall performance across NYUv2, KITTI, ETH3D, ScanNet, and DIODE. For example, it reduces KITTI AbsRel from 7.84 in the pixel-and-gradient baseline to 6.72, while improving $\delta_1$ from 95.62 to 96.38. On DIODE, it reduces AbsRel from 6.22 to 5.55 and increases $\delta_1$ from 96.01 to 96.82.

The advantage of iREPA-depth is less pronounced after 160,000 steps in standard metrics, but the paper reports that qualitative improvements remain visible. This distinction matters: iREPA-depth functions partly as a convergence accelerator and partly as a perceptual regularizer. The authors do not claim that its benefit is uniformly measurable through aggregate depth metrics after sufficiently long training. Instead, its principal residual effect is preservation of fine structure in visually dense regions.

Figure 2: Semantic feature supervision reduces artifacts and improves structural detail, with iREPA-depth producing a sharper prediction than the baseline and LPIPS alternatives.

The choice to use ground-truth depth features also introduces a training-only dependency rather than an inference-time module. At deployment, the model requires only the input image and the adapted DiT-VAE pipeline. This preserves the operational simplicity of the original single-pass formulation.

SinkLoss and the treatment of ambiguous supervision

The second major contribution is SinkLoss, designed for a specific failure mode of dense depth supervision. Thin and transparent structures can be visually ambiguous in RGB images, and their synthetic depth annotations can be unstable. In HyperSim, for example, stochastic rendering can assign neighboring edge pixels to foreground or background depth inconsistently. Under strict pixelwise losses, the model is encouraged either to reproduce annotation noise or to average incompatible alternatives, producing flying pixels and incoherent boundaries.

Figure 3: HyperSim contains thin structures whose RGB appearance and stochastic rendering make exact pixelwise depth correspondence unreliable.

SinkLoss addresses this problem by partitioning the prediction and target into non-overlapping $5 \times 5$ blocks and matching the predicted depth values to target values through entropy-regularized optimal transport. Within each block, the objective preserves the local distribution of depths while relaxing exact spatial correspondence. The implementation uses temperature $\tau=0.1$, invalid-pixel penalty $B=10^6$, and five Sinkhorn iterations.

This is not merely a perceptual sharpness term. It changes the supervision geometry: the model is penalized for producing the wrong collection of local depth values, but it is not forced to associate every potentially ambiguous target pixel with one exact predicted location. Invalid pixels are excluded through large matching costs, preventing unusable annotations from supervising valid predictions.

Stage 2 combines SinkLoss with VAE-decoder unfreezing. The qualitative comparison shows that unfreezing the decoder alone does not reliably eliminate flying pixels, whereas adding SinkLoss substantially cleans object boundaries while retaining fine detail.

Figure 4: SinkLoss combined with VAE-decoder fine-tuning reduces flying pixels more effectively than decoder unfreezing alone.

The quantitative ablation reveals a deliberate tradeoff between conventional depth metrics and boundary-sensitive quality. SinkLoss produces a large improvement in Soft Edge Error while leaving AbsRel and $\delta_1$ essentially unchanged. On HyperSim, the final model achieves SEE3, SEE5, and SEE7 values of 0.352, 0.333, and 0.320, respectively, compared with 0.404, 0.385, and 0.371 for Pixel-Perfect Depth. The implication is that standard global metrics do not adequately capture the benefit of local matching: SinkLoss improves the geometry most relevant to discontinuities without necessarily changing average pixelwise error.

The transfer experiments further support the claim that SinkLoss is a recipe-level contribution rather than a Qwen-specific artifact. Applying it to Stable Diffusion V1.5 and FLUX.2 improves SEE metrics in both cases. For FLUX.2, SEE3 decreases from 0.491 to 0.377; for Stable Diffusion V1.5, it decreases from 0.553 to 0.485.

Zero-shot depth estimation

Marigold V2 is evaluated on NYUv2, KITTI, ETH3D, ScanNet, and DIODE under a robust affine alignment protocol. Among methods trained on comparable amounts of data, it obtains the best reported AbsRel and $t=0.5$0 results on all five datasets.

Dataset	AbsRel	$t=0.5$1
NYUv2	3.6	98.0
KITTI	5.4	97.4
ETH3D	2.8	99.2
ScanNet	3.7	97.9
DIODE	5.2	97.1

The strongest numerical gains occur on KITTI and ETH3D. On KITTI, Marigold V2 improves AbsRel from the strongest comparable baseline value of 6.5 to 5.4, an approximately 17% reduction. On ETH3D, it improves from 3.8 to 2.8, approximately 26%. These results substantiate the abstract's reported 16–26% improvement over the previous best on those benchmarks.

Figure 5: Across indoor, outdoor, and in-the-wild scenes, Marigold V2 preserves thin structures and local detail while limiting flying-pixel artifacts.

The comparison with Pixel-Perfect Depth is particularly informative. Pixel-Perfect Depth can produce fewer flying pixels in some qualitative examples, but it loses substantial detail. Marigold V2 instead achieves the best edge-sensitive scores while retaining details that are visibly suppressed by PPD. This supports the paper's stronger claim that detail preservation and artifact suppression need not be treated as mutually exclusive objectives when local supervision is relaxed appropriately.

The model also has a favorable, though not dominant, computational profile. At $t=0.5$2, it requires 1.9 seconds and 16.9 GB of memory on the reported GPU. At $t=0.5$3, it remains feasible at 9.6 seconds and 29.3 GB, whereas Pixel-Perfect Depth, Lotus-2, and FE2E exceed memory limits. InfiniDepth is substantially faster and more memory-efficient, so Marigold V2's advantage is resolution scalability and quality rather than absolute latency.

Figure 6: SinkLoss improves boundary-sensitive SEE metrics substantially while leaving standard AbsRel and $t=0.5$4 nearly unchanged.

Extension to other dense modalities

The paper tests whether the proposed adaptation recipe is specific to relative depth. The results indicate that the same architecture and loss design can be transferred to several dense regression tasks, although each task requires task-specific supervision.

For metric depth completion, the authors freeze the affine-invariant depth prior and fit a rank-16 test-time LoRA on sparse depth measurements, together with learned scale and shift parameters. High-resolution inference and tiled local adaptation improve performance further. The resulting model achieves competitive or best results across iBims-1, NYUv2, KITTI-DC, and DDAD, including the lowest RMSE on all four benchmarks according to the paper. This result implies that an affine-invariant generative prior can be converted into a metric estimator through sparse test-time adaptation without retraining the full model.

For see-through depth, the authors identify a systematic annotation bias: HyperSim often labels transparent surfaces at the glass rather than at the geometry visible behind it. Fine-tuning on the final depth layer of LayeredDepth-Syn reduces AbsRel from 13.66 to 8.17 and increases $t=0.5$5 from 83.96 to 92.65. The improvement demonstrates that the base model's transparent-surface behavior is not fixed by architecture alone; it is strongly determined by the target-depth convention.

Figure 7: Fine-tuning on layered depth enables predictions of geometry behind transparent surfaces rather than depth at the glass interface.

The surface-normal experiments replace the pixel-space depth loss with an angular loss while retaining iREPA and SinkLoss. Marigold obtains the best or near-best results across the evaluated datasets, including a mean angular error of 14.1 on ScanNet and 15.9 on iBims-1. SinkLoss improves boundary-sensitive SAEE metrics even when the conventional mean angular error changes little, reproducing the same separation between global accuracy and local discontinuity quality observed for depth.

For albedo estimation, a 30,000-step adaptation with $t=0.5$6 and ground-truth-albedo iREPA obtains 20.78 PSNR and 0.195 LPIPS on HyperSim, improving over the compared methods on both metrics while remaining competitive in SSIM. These results support the broader technical premise that a pretrained image-editing DiT can serve as a general dense-modality regressor when its latent and decoded representations are appropriately supervised.

Figure 8: iREPA and SinkLoss improve surface-normal detail, while the same adaptation framework produces high-quality albedo estimates.

Limitations and open questions

The paper explicitly concedes that the large Qwen backbone precludes real-time deployment. Although the single-step formulation is substantially cheaper than iterative diffusion, 1.9 seconds at $t=0.5$7 is still unsuitable for many interactive or robotic settings. The method is also slower and more memory-intensive than InfiniDepth, which achieves 0.2 seconds and 1.9 GB at the same resolution.

The evaluation is primarily affine-invariant. Consequently, the strongest depth results establish relative geometric quality rather than universal metric-depth accuracy. Metric depth completion is addressed through test-time adaptation, learned scale and shift, and sparse measurements; it is therefore not evidence that the base model directly recovers metric scale.

The training data are compact but synthetic-heavy, consisting principally of HyperSim and vKITTI. This makes the reported out-of-distribution performance strong relative to the baselines, but it leaves open how the method behaves under systematic real-world corruptions such as severe motion blur, defocus, reflections, weather, and transparent or non-Lambertian surfaces absent from the training distribution. The authors note that such regions remain ambiguous.

SinkLoss also relies on design choices whose generality is not fully established: fixed $t=0.5$8 blocks, a fixed entropy temperature, a large invalid-pixel penalty, and a small number of Sinkhorn iterations. Local permutation invariance is beneficial when annotations are spatially ambiguous, but it can in principle weaken precise localization if applied to structures whose position is unambiguous. The paper demonstrates strong empirical results, yet does not fully characterize when relaxed correspondence helps or harms geometry.

Finally, the reported evaluation uses robust RANSAC alignment and standard metrics that can understate improvements in boundary structure. The paper partly addresses this with SEE and SAEE, but these metrics are computed on synthetic boundary annotations and do not fully resolve the relationship between perceptual sharpness, 3D reconstruction accuracy, and downstream task utility.

Conclusion

Marigold V2 presents a technically coherent procedure for converting a pretrained image-editing DiT into a single-pass monocular depth estimator using limited hardware and data. Its principal contributions are the use of ground-truth-depth representation alignment through iREPA-depth and local optimal-transport supervision through SinkLoss. Together with pixel-space losses and VAE-decoder adaptation, these components improve both benchmark accuracy and boundary fidelity.

The strongest evidence is the combination of best comparable zero-shot results across five datasets, 17% and 26% AbsRel reductions on KITTI and ETH3D relative to the previous best comparable methods, and substantially improved edge-sensitive metrics without sacrificing standard accuracy. The same recipe also transfers to depth completion, see-through depth, surface normals, and albedo estimation. Its remaining constraints are computational cost, dependence on affine evaluation for the base task, synthetic-heavy training, and unresolved ambiguity in reflections, blur, motion, and transparency.