Foundational Monocular Depth Estimators

Updated 8 August 2025

Foundational Monocular Depth Estimators (FMDEs) are neural models that generate per-pixel depth maps from single RGB images by integrating supervised, weakly, and self-supervised learning regimes.
They leverage advanced architectures such as Transformer encoders and diffusion models along with innovative loss functions to enhance edge preservation, feature fusion, and domain adaptation.
FMDEs underpin breakthroughs in autonomous perception, AR/VR, and geometric vision by robustly combining multi-scale cues and semantic features for versatile real-world applications.

Foundational Monocular Depth Estimators (FMDEs) are a class of large-scale, high-capacity neural models trained on vast datasets for the task of predicting dense per-pixel depth maps from single RGB images. These estimators, designed to robustly generalize across a spectrum of real-world and synthetic domains, represent a paradigm shift from incrementally specialized monocular depth networks to flexible, "foundation-like" vision architectures. FMDEs integrate diverse architectural, training, and data-centric advances, supporting modalities from strictly supervised to weakly and self-supervised regimes, and form the backbone of current breakthroughs in autonomous perception, spatial understanding, and robust geometric reasoning.

1. Historical Evolution and Core Architectures

The development of FMDEs reflects a progression from conventional convolutional approaches to Transformer-based and generative diffusion architectures. Early deep monocular depth networks (e.g., Eigen et al.) utilized multi-scale CNNs with global and local branches to directly regress per-pixel depth, introducing scale-invariant losses in log space to mitigate scale ambiguity. Subsequent advances layered multi-scale fusion using Conditional Random Fields (CRFs) and, later, structured attention guided CRFs to selectively merge hierarchical features and preserve scene structure and object boundaries (Bhoi, 2019).

Recent FMDEs are dominated by large-scale Transformer encoders (such as BEiT, ViT in Depth Anything (Spencer et al., 25 Apr 2024), DPT), which model global image dependencies via self-attention. In parallel, diffusion-based monocular depth models (e.g., Marigold, DepthGen (Saxena et al., 2023, Ke et al., 2023)) adapt denoising probabilistic frameworks (originally for image synthesis) to predict depth via iterative refinement in a VAE-latent or spatial domain, capitalizing on the extensive visual priors internalized by generative models trained on internet-scale data (e.g., Stable Diffusion/LAION-5B (Ke et al., 2023)). In certain hybrid settings, FMDEs are augmented with efficient context modules, multiscale pyramids, and cross-task (semantic, motion) heads (Xu et al., 15 Jul 2025).

FMDEs now serve as universal backbones interfacing with auxiliary modules for application-specific contexts, e.g., federated settings (Soares et al., 2023), adaptation to camera types (Gangopadhyay et al., 6 Aug 2025), or low-light scenarios (Zeng et al., 24 Jul 2025).

2. Methodologies and Loss Design

FMDEs leverage a blend of supervised, ordinal, and self-supervised learning schemes:

Supervised Regression and Ordinal Reformulation: Early models used L2/L1 regression and scale-invariant log loss for absolute metric depth, while ordinal regression recasts depth as discrete intervals with space-increasing discretization and cross-entropy or specialized ordinal losses to capture ordering and uncertainty (Bhoi, 2019).
Feature and Scale Fusion: Deep feature fusion occurs via multi-scale CRFs, attention-guided CRFs, dilated convolutions, or explicit pyramidal modules that balance global and local information (Bhoi, 2019, Sagar, 2020).
Self-supervised and Weakly-supervised Paradigms: Unsupervised approaches, particularly popularized by Godard et al., exploit image reconstruction from predicted depth/disparity and adjacent views, using photometric (appearance), smoothness, and left-right consistency losses (Bhoi, 2019). Self-supervised learning with additional pose (egomotion) heads leverages monocular video sequences (Gurram et al., 2021, Soares et al., 2023). GradNorm, domain adaptation losses (gradient reversal), and multi-task semantic or normal alignment regularize representations (Gurram et al., 2021).

Novel losses target model weaknesses:

Boundary-aware depth loss increases penalty on edge pixels, improving sharpness of depth discontinuities (Yang et al., 2021).
Ambiguity-masking and frequency-adaptive loss filter out loss contributions at ambiguous boundaries or high-frequency regions in the photometric loss (Chen et al., 2022).
Deep feature annihilation loss targets the internal feature space for adversarial robustness (Mathew et al., 2020).

Scale and shift-invariant loss functions and adaptive binning (e.g., AdaptiveBins, ZoeDepth) reinforce generalization across camera intrinsics and varied scenes (Zhang, 21 Jan 2025, Xu et al., 15 Jul 2025, Spencer et al., 25 Apr 2024).

3. Generalization, Robustness, and Adaptation Strategies

A defining property of FMDEs is strong zero-shot generalization:

Large-scale Pre-training: Models like Depth Anything are pretrained on millions of diverse, labeled and unlabeled images, providing rich depth and scene priors that support transfer across domains (Spencer et al., 25 Apr 2024, Xu et al., 15 Jul 2025).
Synthetic and Domain Bridging: Synthetic datasets (e.g., Virtual KITTI, Hypersim) are critical for filling annotation gaps and enable fine-tuning with clean dense labels (Ke et al., 2023). Approaches such as MonoDEVSNet blend real-world self-supervision (SfM) and virtual-world ground truth, using adversarial domain alignment to unify representations (Gurram et al., 2021).
Biologically Inspired and Semantic Cues: FMDEs augmented with semantic segmentation, instance segmentation, size priors, and language embeddings emulate biological depth cues (relative, familiar, and absolute size) to better infer geometry from monocular images (Auty et al., 2022).

Adaptation to new sensor domains: The calibration token mechanism extends FMDEs to fisheye or severely distorted cameras by modulating transformer-layer latent embeddings with learned tokens. This sidesteps information loss from reprojection and enables plug-and-play correction for covariate shift introduced by camera intrinsic/distortion change. Self-supervised consistency losses with synthetic fisheye transformations underpin this adaptation (Gangopadhyay et al., 6 Aug 2025).

Robustness concerns: FMDEs, particularly those based on deep CNNs or Transformers, remain vulnerable to adversarial perturbations and patch attacks, which can significantly degrade predicted depth even under imperceptible modifications. Attacks targeting internal feature spaces (via deep feature annihilation losses) demonstrate the need for robust architectures and regularization to ensure reliability in safety-critical deployments such as autonomous driving or medical robotics (Mathew et al., 2020).

Low-Light and Adverse Conditions: Models like DepthDark address domain-specific degradations, introducing simulation-based data augmentation (flare, noise synthesis), illumination-aware fine-tuning, and feature fusion to sustain performance in nighttime or adverse illumination (Zeng et al., 24 Jul 2025).

4. Empirical Results and Application Domains

FMDEs have achieved state-of-the-art results in prominent benchmarks, both indoor (NYU-Depth V2, ScanNet++) and outdoor (KITTI, RobotCar-Night, Cityscapes, KITTI-360, nuScenes-Night) (Spencer et al., 25 Apr 2024, Ke et al., 2023, Zeng et al., 24 Jul 2025, Gangopadhyay et al., 6 Aug 2025). Metrics typically reported include RMSE, SILog, AbsRel, accuracy thresholds (δ₁, δ₂, δ₃), and application-specific 3D F-Score (Spencer et al., 25 Apr 2024).

Challenge results demonstrate that Depth Anything, when used as a backbone and appropriately fine-tuned, can dramatically boost 3D F-Score (up to ~23–24% on the SYNS-Patches test set) (Spencer et al., 25 Apr 2024).
Robust generalization to fisheye cameras, domain adaptation to nighttime, and efficient federated/self-supervised learning further substantiate FMDEs as practical foundation models for real-world, diverse scenarios (Soares et al., 2023, Gangopadhyay et al., 6 Aug 2025, Zeng et al., 24 Jul 2025).

Application domains for FMDEs include:

Robotics and Autonomous Vehicles: Real-time scene understanding, navigation, SLAM, and mapping, requiring robust, accurate, and metrically consistent depth estimates in the wild (Xu et al., 15 Jul 2025, Soares et al., 2023).
Augmented/Virtual Reality (AR/VR): Scene geometry inference, view synthesis, and point cloud completion from monocular input (Ke et al., 2023, Viola et al., 18 Dec 2024).
Geometric Vision and Registration: As in FreeReg, FMDEs paired with cross-modal features enable image-to-point cloud registration via modality unification and robust correspondence matching (Wang et al., 2023).
Depth Completion: Guided diffusion-based FMDEs (e.g., Marigold-DC) now serve as strong priors for sparse-to-dense depth completion tasks, demonstrating zero-shot generalization and robust test-time adaptation (Viola et al., 18 Dec 2024).

5. Remaining Challenges and Future Perspectives

Despite substantial progress, several technical challenges remain:

Generalization Across Domains: Although FMDEs exhibit improved cross-dataset performance, domain gaps (synthetic ↔ real, sensor change) and tasks requiring precise metric depth still pose difficulties, especially under severe environmental variation or novel artifacts (Zhang, 21 Jan 2025, Xu et al., 15 Jul 2025).
Efficiency vs. Fidelity Trade-offs: Generative diffusion-based models, while excelling at fine structure recovery and ambiguity modeling, are slower to sample than feedforward networks. The community continues to explore trade-offs for real-time deployment (Ke et al., 2023, Viola et al., 18 Dec 2024).
High-Frequency and Boundary Preservation: Edge and detail smoothing remains an issue, especially under single-pass inference. Patch-based and multiscale designs improve fidelity at the cost of computational burden (Zhang, 21 Jan 2025).
Robustness: Adversarial vulnerability, particularly to patch and feature-space attacks, necessitates further development in regularization, robust loss formulations, and possibly built-in anomaly detection (Mathew et al., 2020).
Scalability of Self-Supervision and Federated Learning: Distributed self-supervised finetuning (e.g., via federated learning) demonstrates promise for privacy-aware and scalable training on autonomous fleets, though efficiency, communication costs, and non-IID data present ongoing challenges (Soares et al., 2023).

Ongoing research directions identified include: expanding large-scale, realistically simulated and synthetic datasets; more effective self-supervised and multi-task learning strategies; architectural advances (efficient ViTs, hybrid models); plug-and-play adaptation techniques (e.g., calibration tokens); and tightly integrating metric recovery and geometric constraints for robust monocular metric depth (Xu et al., 15 Jul 2025, Zhang, 21 Jan 2025).

6. Tables — Core FMDE Methodologies and Their Distinctives

Method	Key Features	Learning Paradigm
Multi-scale CNN (Bhoi, 2019)	Global-to-local, scale-invariant loss	Supervised
Multi-scale CRF (Bhoi, 2019)	Hierarchical fusion, energy models	Supervised
Ordinal Regression (Bhoi, 2019)	SID discretization, ordering loss	Supervised (ordinal)
Unsupervised via Stereo (Bhoi, 2019)	Left-right consistency, photometric	Unsupervised
Depth Anything (Spencer et al., 25 Apr 2024)	Large ViT backbone, massive pretrain	(Supervised/)Self-supervised
Marigold (diffusion) (Ke et al., 2023)	Latent denoising, strong prior	Diffusion, synthetic pretrain
DEFOM-Stereo (Jiang et al., 16 Jan 2025)	Foundation depth cues in stereo	Hybrid/Zero-shot
FreeReg (Wang et al., 2023)	Fusion: monocular geometry + diffusion semantic	Registration

This table illustrates methodological diversity and foundational advances present in modern FMDEs.

7. Impact and Outlook

FMDEs have redefined monocular depth estimation from a narrowly-scoped learned task to a robust visual abstraction layer, underpinning complex perception systems. By systematizing architectural, training, and adaptation advances—incorporating global priors, strong generalization, and plug-and-play adaptation—FMDEs now make possible real-time, privacy-preserving, and cross-device scalable geometric reasoning. Their continued development is central to advances in autonomous robotics, spatial AI, immersive reality, and beyond. Persistent challenges regarding detail preservation, adversarial robustness, and cross-domain adaptation motivate ongoing research into even more adaptable and resilient monocular depth estimation foundations.