---
title: 'ZoeDepth: Unified Monocular Depth Estimation'
url: https://www.emergentmind.com/topics/zoedepth
type: topic
---

# ZoeDepth: Unified Monocular Depth Estimation

ZoeDepth is a monocular depth estimation framework designed to bridge the gap between relative depth generalization and metric depth precision. It introduces a unified approach that combines scale-invariant pre-training on diverse datasets with domain-specific metric fine-tuning, delivering both robust zero-shot transfer capabilities and high-fidelity metric depth predictions. Key technical innovations include a modular architecture featuring a shared encoder–decoder (based on transformer backbones), novel multi-scale bin-based metric heads, attractor layer refinement, and an automatic domain routing mechanism. ZoeDepth achieves competitive or state-of-the-art results on varied benchmarks, including NYU Depth v2 and KITTI, and demonstrates unprecedented zero-shot generalization to unseen indoor and outdoor datasets.

## 1. Design Principles and Architecture

ZoeDepth employs a two-stage encoder–decoder structure. The initial pre-training stage utilizes a MiDaS-style network with a transformer backbone (e.g., BEiT-L, Swin Transformer) trained on scale-invariant losses across twelve heterogeneous datasets. The decoder aggregates multi-scale image features, enabling the extraction of robust geometric representations agnostic to scene type.

Domain-specific metric heads—each realized via a metric bins module—are attached during fine-tuning. Metric heads are lightweight, facilitating efficient training and inference. Each head predicts a discrete set of depth bin centers $c_i(k)$, which are subsequently refined using several attractor points $a_k$ drawn from decoder feature pyramids. The bin refinement process operates as follows (inverse attractor variant):
$$
\Delta c_i = \sum_{k=1}^{n_a} \frac{a_k - c_i}{1 + \alpha|a_k - c_i|^\gamma}
$$
where $\alpha$ and $\gamma$ control attraction strength. This multi-scale adjustment contracts bin centers towards higher accuracy without explicit hierarchical splitting.

The network further predicts a unimodal, ordering-aware bin probability distribution using a binomial model:
$$
p(k; N, q) = \binom{N}{k}q^{k}(1-q)^{N-k}
$$
where $q$, and optionally a temperature $t$, are learned for each pixel. The metric depth at pixel $i$ is computed as a weighted sum:
$$
d(i) = \sum_{k=1}^{N_{\mathrm{total}}} p_i(k) \cdot c_i(k)
$$
Multi-domain deployments (e.g., NYU for indoor, KITTI for outdoor) are handled by associating separate metric heads with a latent classifier for automatic routing at inference.

## 2. Training Methodology

Training is organized in two stages:

- **Stage 1:** Relative depth pre-training on twelve datasets (indoor, outdoor, synthetic, real) using scale-invariant log losses. This enables the encoder–decoder to learn geometric abstractions invariant to global scale or dataset-specific biases:
  $$
  L_{\mathrm{rel}} = \frac{1}{n}\sum_i (g_i)^2 - \frac{1}{n^2}(\sum_i g_i)^2, \quad g_i = \log d_i - \log d^*_i
  $$
- **Stage 2:** Metric fine-tuning on datasets with available ground-truth metric depth (typically NYU Depth v2 and KITTI). Each domain-specific head is fine-tuned separately or jointly, with a router determining selection during inference.

The combination of diverse pre-training and minimal metric fine-tuning enables accurate scale reintroduction on target domains, while preserving the generalization benefits of training on heterogeneous data.

## 3. The Metric Bins Module: Attractor Layers and Probability Formulation

The metric bins module refines a set of candidate bin centers $c_i(k)$ by integrating multi-scale attractor information. Rather than hierarchical splitting (such as AdaBins), ZoeDepth refines all bins at the decoder bottleneck and contracts them through attractor layers.

The final depth prediction uses a binomial probability model instead of softmax, explicitly exploiting the ordinal structure of depth and generating sharply unimodal, order-preserving distributions. For numerical stability, log-probabilities are computed via Stirling’s approximation, and temperature scaling is used as necessary.

This module is central to ZoeDepth’s ability to reintroduce metric scale in a way that is efficient, robust against varying depth distributions across domains, and resistant to common binning artifacts.

## 4. Quantitative and Qualitative Performance

On the NYU Depth v2 dataset, ZoeDepth in configuration ZoeD-X-N (no relative pre-training) achieves REL = 0.082, improving by 13.7% over NeWCRFs (REL = 0.095). With full relative pre-training (ZoeD-M12-N), REL drops to 0.075 (a total improvement of nearly 21%). Threshold accuracies $\delta_1$, $\delta_2$, and $\delta_3$ are consistently higher (e.g., $\delta_1$ = 0.955) than prior baselines.

In multi-domain joint training (ZoeD-M12-NK), performance drop on NYU is modest, while zero-shot accuracy on eight unseen datasets is unprecedented, including up to 976% improvement over NeWCRFs on the DIML Outdoor dataset. ZoeDepth demonstrates robust cross-domain transfer, delivering reliable metric depth predictions without further fine-tuning.

## 5. Zero-Shot Generalization and Routing

The combination of relative depth pre-training and domain-specific metric heads allows ZoeDepth to generalize to entirely unseen data distributions, scene geometries, and camera parameters. Automatic routing via a latent classifier or simple multi-layer perceptron (MLP) determines the optimal metric head, avoiding the pitfalls of applying a generic model to both indoor and outdoor scenes.

Adaptive bin refinement and attractor-based contraction allow the model to accommodate dataset and domain shifts while maintaining reliable metric predictions and boundary delineation.

## 6. Technical Implementation and Mathematical Details

Key technical elements include:

- Transformer-based encoder
- Multi-scale feature aggregation in the decoder
- Attractor-based bin refinement (inverse and exponential variants)
- Binomial probability modeling for depth prediction
- Modular domain routing for zero-shot adaptation

The main equations governing predictions are:
$$
d(i) = \sum_{k=1}^{N_{\mathrm{total}}} p_i(k) \cdot c_i(k)
$$
and attractor bin refinement
$$
\Delta c_i = \sum_{k=1}^{n_a} \frac{a_k - c_i}{1 + \alpha|a_k - c_i|^\gamma}
$$
with bin probabilities as
$$
p(k; N, q) = \binom{N}{k}q^{k}(1-q)^{N-k}
$$

## 7. Applications, Limitations, and Comparisons

ZoeDepth’s dense metric depth estimation supports diverse applications: autonomous navigation, augmented reality, virtual reality, scene reconstruction, and photo editing. Its robust zero-shot generalization makes it especially suitable for environments where domain-specific fine-tuning is infeasible.

Subsequent research identified competitive and superior approaches in specialized settings (e.g., ECoDepth’s ViT-based diffusion conditioning [2403.18807], PatchFusion’s high-resolution fusion [2312.02284], and wildlife monitoring benchmarks [2510.04723]). While ZoeDepth offers rapid inference (0.17 s per image), its spatial accuracy can degrade in challenging outdoor environments—recording a mean absolute error of 3.087 m on wildlife camera trap images, compared to 0.454 m from Depth Anything V2 [2510.04723]. Median-based depth aggregation is recommended for improved robustness in noisy conditions.

The metric bins module and latent routing are often adapted for downstream tasks such as 3D occupancy prediction [2407.12294], NeRF-based scene editing [2405.00630], and scene enhancement in degraded domains [2404.17883].

## Summary Table: ZoeDepth Configuration Performance

| Configuration           | Relative Pre-training | Metric Fine-tuning | REL (NYU) | Zero-shot generalization |
|-------------------------|----------------------|--------------------|-----------|-------------------------|
| ZoeD-X-N                | No                   | NYU                | 0.082     | Moderate                |
| ZoeD-M12-N              | M12 (12 datasets)    | NYU                | 0.075     | Excellent               |
| ZoeD-M12-NK             | M12                  | NYU + KITTI        | ~0.078    | Unprecedented           |

ZoeDepth’s modular approach and technical innovations have established new standards for cross-domain metric depth estimation, with the caveat that accuracy may be limited in highly unstructured outdoor environments unless further domain adaptation is performed.

Source: https://www.emergentmind.com/topics/zoedepth