Depth Foundation Models

• Depth foundation models are large-scale, reusable geometric encoders trained on diverse datasets to enable robust zero-shot transfer across various 3D perception tasks.
• They employ advanced architectures like ViTs with encoder-decoder systems, cost-volume modules, and diffusion backbones to achieve state-of-the-art performance on tasks such as SLAM and monocular 3D reconstruction.
• The models are adaptable via methods like low-rank adaptation and teacher-student training pipelines, which tailor performance for specialized domains including surgical and panoramic imaging.

Depth foundation models are large, reusable depth predictors or geometric encoders trained on heterogeneous data with the aim of broad transfer across domains, camera models, and downstream tasks. In the recent literature, the term is used in both a broad and a strict sense. BenchDepth treats depth foundation models as general-purpose monocular or geometric models trained on large, diverse datasets and intended as reusable components for many downstream 3D tasks (Li et al., 21 Jul 2025). A 2025 survey proposes a stricter criterion—training on more than $10$ million images with architectures exceeding $1$ billion parameters—while emphasizing strong zero-shot generalization across monocular, stereo, multi-view, and video depth settings (Xu et al., 15 Jul 2025). Across these formulations, the common theme is a pretrained depth prior that can be adapted, fused, or distilled into specialized systems rather than retrained from scratch.

1. Definition, scope, and representations

The recent literature frames depth foundation models as the depth analogue of foundation models in language and vision: they are trained on large, diverse corpora, aim for cross-domain transfer, and are intended to support many downstream tasks with limited adaptation (Li et al., 21 Jul 2025). One survey formalizes this by associating depth foundation models with data scale, parameter scale, and emergent zero-shot behavior across monocular, stereo, multi-view, and monocular-video depth estimation (Xu et al., 15 Jul 2025). This formulation places depth foundation models within a broader transition from dataset-specific depth regressors toward reusable geometric backbones.

A defining feature of this literature is representational pluralism. BenchDepth explicitly benchmarks models that predict affine-invariant disparity, affine-invariant depth, metric depth, and affine-invariant point maps, and argues that these representations should not be conflated during evaluation (Li et al., 21 Jul 2025). This matters because the depth foundation model ecosystem is not limited to calibrated metric estimators: MiDaS and Depth Anything V2 relative models operate in affine-invariant regimes, Metric3D V2 and UniDepth target metric depth, and MoGe predicts point maps rather than depth directly (Li et al., 21 Jul 2025).

The scope of the field is correspondingly broad. The 2025 survey organizes the space around monocular depth, stereo disparity, multi-view image depth estimation, and monocular video depth estimation, and treats depth foundation models as potentially task-unified systems whose shared representations can be adapted across these settings (Xu et al., 15 Jul 2025). A plausible implication is that “depth foundation model” denotes less a single architecture class than a training-and-transfer regime for geometric representation learning.

2. Architectural patterns and training paradigms

Architecturally, the field is centered on large encoder-decoder systems built on ViTs, DPT-style dense prediction heads, cost-volume modules, recurrent refinement, and, increasingly, diffusion backbones. The survey identifies DPT with ViT encoders as a canonical large-scale monocular backbone, transformer-based stereo and MVS systems as the dominant cross-view reasoning pattern, and diffusion-based depth models such as Marigold and GenPercept as a separate line that repurposes generative backbones for geometry prediction (Xu et al., 15 Jul 2025). These systems are usually paired with scale- or affine-invariant objectives, multi-dataset training, and synthetic-to-real transfer.

Depth Anything and Depth Anything V2 exemplify the dominant training strategy. A 2025 uncertainty study describes Depth Anything V2 as a DINOv2-based depth foundation model that builds on a student-teacher pipeline: a teacher produces pseudo-depth for about $62$ million unlabeled real images, the student is trained on those pseudo-labels plus about $1.5$ million labeled images, and V2 further replaces labeled real images with synthetic scenes, uses DINOv2-G as a stronger teacher, and adds large-scale pseudo-labeled real images to mitigate the synthetic-real gap (Landgraf et al., 14 Jan 2025). This pipeline yields a model that is simultaneously semantically rich and geometrically detailed.

Other work generalizes the same pattern to new camera models. Depth Any Panoramas is described as a panoramic metric depth foundation model trained on about $2$ million panoramas using a data-in-the-loop pipeline that combines public datasets, UE5-generated synthetic data, text-to-image panoramas, and real panoramic web imagery; it uses DINOv3-Large, a range mask head, sharpness-centric optimization, and geometry-centric optimization to achieve stable metric predictions across diverse scene distances (Lin et al., 18 Dec 2025). This extends the foundation-model recipe from perspective imagery to equirectangular geometry.

3. Adaptation and specialization

A recurrent result in the literature is that zero-shot transfer from generalist depth models is often insufficient in specialized domains. Surgical-DINO provides a direct demonstration in endoscopic surgery: zero-shot DINOv2 with a depth head trained on natural images performs worst among compared methods on SCARED, while a low-rank adaptation of a frozen DINOv2 ViT-Base encoder, applied to the $q$ and $v$ projections and training only about $0.14$M parameters ($0.17\%$ of the full model), achieves the best SCARED results and better cross-dataset generalization to Hamlyn (Cui et al., 2024). The paper’s core conclusion is that naive transfer or decoder-only fine-tuning is inadequate for the surgical domain.

The same pattern appears in more general modular adaptation. BriGeS keeps both a depth foundation model, Depth Anything, and a segmentation foundation model, SAM, frozen, and trains only a multi-scale Bridging Gate to fuse semantic and geometric features. Using roughly $1\%$ of Depth Anything’s original training data and only one or two epochs, BriGeS reports an average $1$0 reduction in AbsRel relative to the underlying Depth Anything baselines and improved zero-shot performance on KITTI, NYUv2, ETH3D, DIODE, and DA-2K (Ma et al., 29 May 2025). The result is a geometric-semantic depth system rather than a depth-only model.

Specialized medical adaptation has also moved beyond semantic transfer. SpecDepth argues that the main failure mode of natural-image foundation models in colonoscopy is a spectral mismatch rather than a semantic mismatch: colonoscopy images lack the structured high-frequency content on which the pretrained models rely. Its adaptive spectral rectification module, implemented through learnable wavelet decomposition and low-dimensional subband reweighting, reaches state-of-the-art performance on C3VD and SimCol3D with AbsRel $1$1 and $1$2, respectively (Zhang et al., 16 Mar 2026). This reframes domain adaptation as signal-statistics alignment.

Weakly supervised and source-free adaptation pushes the same idea into OOD robustness. WeSTAR adapts Depth Anything v2 or MiDaS through teacher-student self-training, semantically aware hierarchical normalization using SAM2 masks, pairwise ordinal depth annotations, and LoRA regularized by an explicit penalty on weight updates, yielding state-of-the-art results on clean and corrupted OOD benchmarks such as NYU-C, KITTI-C, Sintel-C, DIODE-C, NuScenes night, and DrivingStereo weather conditions (Huang et al., 18 Nov 2025). A related line, Surgical Depth Anything, shows that straightforward supervised fine-tuning of Depth Anything on surgical datasets can reduce AbsRel by $1$3, $1$4, and $1$5 on EndoSLAM stomach, small intestine, and colon cases relative to the generic small model (Lou et al., 2024).

4. Depth foundation models as reusable geometric priors

The field increasingly treats depth foundation models as modular geometric priors that can be injected into systems whose primary output is not depth itself. BenchDepth makes this explicit by evaluating depth foundation models through five downstream proxy tasks—depth completion, stereo matching, monocular feed-forward 3D scene reconstruction, SLAM, and vision-language spatial understanding—rather than only through aligned depth metrics (Li et al., 21 Jul 2025). This framing matches how the strongest recent systems are actually built.

DEFOM-Stereo is a clear example. It treats Depth Anything V2 as a depth foundation model, uses its features in combined context and matching encoders, uses its monocular relative depth to initialize recurrent disparity, and introduces a scale update module that resolves monocular scale ambiguity inside a stereo matcher. The resulting system reports top performance on KITTI 2012, KITTI 2015, Middlebury, and ETH3D, with first-place rankings on many metrics and strong robust-vision-challenge results (Jiang et al., 16 Jan 2025). Here the foundation model supplies shape priors and transferable features, while stereo resolves metric scale.

FlowSeek applies the same principle to optical flow. It freezes a single-image depth foundation model such as Depth Anything v2, injects both its decoder features and inverse-depth maps into a RAFT-style flow system, and derives motion bases from the predicted inverse depth to encode the low-dimensional flow subspace induced by rigid motion. The paper reports training on a single RTX 3090 and relative improvements of $1$6 and $1$7 over SEA-RAFT on Sintel Final and KITTI, respectively, together with gains on Spring and LayeredFlow (Poggi et al., 5 Sep 2025). The depth model is not optimized for flow, but its geometric prior reduces the burden on the flow network.

FoundationSLAM extends the reuse pattern further. It imports frozen FeatureNet and ContextNet encoders from FoundationStereo, which itself builds on Depth Anything v2, and uses them as geometry-aware feature extractors inside a monocular dense SLAM loop with a Hybrid Flow Network, a Bi-Consistent Bundle Adjustment Layer, and a Reliability-Aware Refinement mechanism. The system reports real-time operation at $1$8 FPS together with superior trajectory accuracy and dense reconstruction quality across multiple datasets (Wu et al., 31 Dec 2025). In this formulation, the depth foundation model acts as a feature prior embedded in an optimization loop rather than as a direct source of final depth.

The same reuse logic appears outside classical geometry tasks. Depth Any Canopy fine-tunes Depth Anything v2 into a canopy-height estimator and reports superior or comparable performance to a large in-domain baseline with much lower compute, requiring less than $1$9 in compute and an estimated carbon footprint of $62$0 kgCO$62$1 for the small model (Cambrin et al., 2024). In laparoscopic liver landmark segmentation, a dual-encoder system couples SAM2 and Depth Anything V2 through cross-attention fusion and SRFT-GaLore adaptation, improving Dice by $62$2 and reducing ASSD by $62$3 points relative to D2GPLand on L3D (Lin et al., 5 Nov 2025). These systems treat depth foundation models as general geometric encoders rather than narrowly as monocular depth regressors.

5. Evaluation, uncertainty, and reliability

Evaluation has become a central controversy in the field. BenchDepth argues that conventional depth benchmarks rely too heavily on alignment-based metrics such as AbsRel, RMSE, and $62$4-accuracy after scale-and-shift alignment, and that these protocols introduce representation-dependent bias, sensitivity to outliers, and a smoothing bias that can favor some depth parameterizations over others (Li et al., 21 Jul 2025). The paper shows that alignment in depth space and disparity space behaves differently under noise, that least-squares alignment is sensitive to localized outliers, and that cross-representation comparison becomes especially opaque for outputs such as point maps.

Its alternative is to evaluate depth models by practical utility. In that benchmark, Depth Anything V2 relative emerges as the best all-rounder across depth completion, stereo, and SLAM; Depth Anything V2 metric is especially strong for monocular feed-forward 3D reconstruction; diffusion-based models vary substantially, with GenPercept substantially stronger than Marigold; and depth inputs only weakly improve vision-language spatial understanding because the bottleneck appears to be the VLM rather than the depth model (Li et al., 21 Jul 2025). This suggests that a depth foundation model should be judged by how it improves systems, not only by how well it aligns to a single ground-truth map.

Reliability is the second major evaluation axis. A critical synthesis of uncertainty quantification and foundation models attaches five UQ methods—Learned Confidence, GNLL, Monte Carlo Dropout, Sub-Ensembles, and Test-Time Augmentation—to Depth Anything V2 and evaluates them on NYUv2, Cityscapes, UseGeo, and HOPE. The main finding is that fine-tuning with Gaussian Negative Log-Likelihood is a particularly promising compromise: it provides strong uncertainty metrics on NYUv2, Cityscapes, and HOPE while keeping computational cost essentially unchanged relative to the baseline, though its performance is weaker on UseGeo, where large depth ranges complicate variance estimation (Landgraf et al., 14 Jan 2025). This makes probabilistic depth outputs a concrete next step rather than an external add-on.

OOD depth completion produces a complementary reliability lesson. PSD freezes a depth foundation model, aligns its output to sparse metric depth, performs dual-space propagation in 3D and 2D without learnable parameters, and then applies a lightweight correction module; trained on NYUv2 and KITTI, it is evaluated on $62$5 additional datasets and reports strong OOD robustness relative to prior completion systems (Chen et al., 7 Aug 2025). The important point is not only the result but the mechanism: foundation-model structure can be used as a prior while metric supervision remains sparse.

6. Open problems and research directions

The surveys and case studies converge on several unresolved issues. Data scale remains central: one survey explicitly associates depth foundation models with more than $62$6 million training images and architectures beyond $62$7 billion parameters, while also noting that the field is still moving toward such scale in practice (Xu et al., 15 Jul 2025). At the same time, domain-specific results in surgery, colonoscopy, panoramas, and remote sensing show that large generic pretraining does not eliminate the need for specialization (Cui et al., 2024). A plausible implication is that future systems will combine ever larger base models with increasingly modular adaptation layers.

A second open problem is the tension between relative and metric depth. Much of the strongest generalization still comes from affine-invariant or relative depth models, yet downstream robotics, AR/VR, SLAM, and scientific mapping often require metric outputs. DMD$62$8C addresses this by distilling a monocular foundation model into a depth completion network through synthetic LiDAR pretraining and then resolving scale ambiguity with a scale- and shift-invariant loss during real-world fine-tuning, achieving first place on the KITTI depth completion benchmark (Liang et al., 21 Mar 2025). This suggests that metric depth may increasingly be obtained by coupling general relative priors with sparse metric cues rather than by purely metric pretraining.

A third direction is broader multimodal and temporal integration. BriGeS shows that segmentation foundation models can improve depth when fused through a small adapter (Ma et al., 29 May 2025), while FoundationSLAM and FlowSeek show that depth priors can regularize motion, mapping, and correspondence estimation (Wu et al., 31 Dec 2025). Depth Any Panoramas indicates that camera-specific geometry can be handled by task-specific heads and losses without abandoning the foundation-model recipe (Lin et al., 18 Dec 2025). The surveys further emphasize multi-task geometry, cross-view consistency, and video depth as key steps toward unified 3D perception backbones (Xu et al., 15 Jul 2025).

Finally, evaluation itself is likely to change. BenchDepth’s proxy-task perspective argues against treating aligned depth scores as the sole criterion for progress (Li et al., 21 Jul 2025). In parallel, the uncertainty literature argues that safe deployment requires calibrated confidence maps, not just lower RMSE (Landgraf et al., 14 Jan 2025). Together these trends suggest that the mature depth foundation model will be defined not only by scale and zero-shot accuracy, but by calibrated uncertainty, modular transfer, camera generality, and demonstrable utility in downstream geometric systems.