Unsupervised Monocular Depth Estimation

• Unsupervised monocular depth estimation is a technique that infers dense scene depth from single images by leveraging geometric and photometric consistency without using ground-truth labels.
• It employs encoder–decoder architectures with skip connections, attention modules, and transformer backbones, using photometric reprojection and edge-aware smoothing to enhance accuracy.
• Ongoing research tackles challenges like non-rigid motion, illumination variability, and domain gaps while striving for improved scale-consistency and real-time performance.

Unsupervised monocular depth estimation is the task of inferring dense per-pixel scene depth from a single image (or sequence of images) using training protocols that do not require direct supervision by ground-truth depth labels. Instead, these methods exploit geometric, photometric, or temporal consistency across sequences, stereo pairs, or domains to provide learning signals. Since its introduction, unsupervised monocular depth estimation has become a cornerstone in 3D scene understanding, autonomous navigation, and robotic perception, with extensive research focused on improving robustness, accuracy, scale-consistency, and generalizability across domains and highly dynamic or complex environments.

1. Fundamental Principles and Core Losses

The core idea in unsupervised monocular depth estimation is to replace direct ground-truth supervision with a geometric proxy: given adjacent frames in a monocular sequence (or stereo pairs), a predicted depth map and relative pose are used to warp one frame into the viewpoint of another. The network is then penalized if the warped view disagrees with the observed one.

The generic photometric reprojection loss takes the form

$$L_{\text{photo}}=\sum_{p} \min_{s}\Psi\left(I_t(p),I_{s\rightarrow t}(p)\right)$$

with

$$I_{s\rightarrow t}(p) = I_s\Bigl(\pi\bigl(T_{t\to s} D_t(p) K^{-1} p\bigr)\Bigr)$$

where $I_t$ is the target frame, $I_s$ is a source frame (typically $t-1$ or $t+1$), $D_t(p)$ is the predicted depth at pixel $p$, $K$ is the intrinsics matrix, and $T_{t\to s}$ is the predicted pose. The similarity function $\Psi$ typically mixes SSIM and L1 norms: $$\Psi(a, b) = \alpha \cdot \tfrac{1 - \mathrm{SSIM}(a, b)}{2} + (1-\alpha)\|a-b\|_1$$ where $\alpha=0.85$ is standard (Godard et al., 2016).

This is typically complemented by an edge-aware depth smoothness loss,

$$L_{\text{smooth}} = \sum_{p} \left| \partial_x D_t(p) \right| e^{-|\partial_x I_t(p)|} + \left| \partial_y D_t(p) \right| e^{-|\partial_y I_t(p)|}$$

to regularize in textureless regions while preserving sharp depth at image edges (Godard et al., 2016, Zhu et al., 2023).

Occlusions and dynamic objects are handled through pixel-wise masking or more advanced modeling (see Section 4). The total loss often takes the form of a weighted sum: $$L_{\text{total}} = \alpha L_{\text{photo}} + \beta L_{\text{SSIM}} + \gamma L_{\text{smooth}} + \ldots$$ e.g., with typical weights $$(\alpha, \beta, \gamma, \delta) = (1.0, 0.1, 0.1, 1.0)$$ (Almalioglu et al., 2020).

2. Network Architectures and Geometric Integration

Encoder–decoder architectures define the backbone for most unsupervised monocular depth methods. Early systems employ fully-convolutional CNN encoders (e.g., VGG, ResNet-18/50), with UNet-style decoders featuring skip-connections and multi-scale outputs (Godard et al., 2016, Zhu et al., 2023). Modern methods add attention modules (e.g., channel and spatial attention as in Depth-Enhancement modules (Almalioglu et al., 2020)), plug in transformer encoders (e.g., Swin Transformer (Shim et al., 2023)) or hybrid CNN/Transformer backbones (Sun et al., 2023, Li et al., 2024).

Architectural enhancements include:

• Hierarchical feature fusion: Multi-level features are densely cascaded to aggregation points in the decoder for finer details (DCMNet in SwinDepth (Shim et al., 2023)):
• Residual blocks via depthwise separable convolutions to recover local textures when using transformer encoders (e.g., Res-DSC (Li et al., 2024))
• Recurrent modulation units (RMU) to iteratively fuse features and allow compact but effective depth models (RM-Depth (Hui, 2023))
• Super-resolution sub-networks for high-resolution depth upsampling without interpolation artifacts (Zhou et al., 2018)
• Lightweight depth heads for foundation models, e.g., trainable fusion on top of frozen transformer encoder outputs (Li et al., 2024)

Pose networks generally employ shallow CNN or U-Net backbones that consume stacked RGB frames and predict SE(3) transforms. In dynamic-scene settings, residual per-pixel motion and motion masks further extend the pose module (Almalioglu et al., 2020, Sun et al., 2023).

3. Advances in Loss Design and Robustness

Photometric loss is subject to failure modes under illumination change, non-Lambertian surfaces, and dynamic objects. To address this, key innovations include:

• Occlusion-aware photometric losses: Geometric reasoning detects occluded points by depth comparison in reprojected clouds, masking them in both photometric and geometry-consistency terms (Almalioglu et al., 2020).
• Scale-aware geometry consistency: Enforcing cross-frame depth agreement up to a single coherent scale is crucial for persistent monocular visual odometry (Almalioglu et al., 2020).
• Edge-aware, adaptive, or robust smoothness: Loss weights may be adaptively tuned per-pixel via reconstruction residuals for improved support near depth discontinuities (Wong et al., 2019).
• Robust and improved SSIM: Additive forms of SSIM (as opposed to classic multiplicative) yield smoother gradients and empirically better accuracy (Cao et al., 5 Jun 2025). SSIM-a (additive) penalizes errors in luminance, contrast, and structure with individual weights, improving both convergence and final metrics.
• Outlier clipping: Residuals exceeding a high quantile (e.g., 95th) are clipped to suppress influence from non-modeled dynamic regions or occlusion artifacts (Zhou et al., 2018).
• Intrinsic-decomposition: In endoscopic and adverse-illumination scenarios, photometric consistency is replaced or augmented with losses on decomposed reflectance and shading, mitigating specularity and non-Lambertian artifacts (Li et al., 2024).
• Flow distillation: Instead of photometric matching, rigid flow predictions from depth+pose can be directly supervised by an external pretrained flow estimator with masking for unreliable pixels (Zhu et al., 2023).

Regularization for dynamic or adverse environments often involves additional constraints:

• Sparsity and piecewise-constancy priors on dense 3D motion fields (Li et al., 2020).
• Joint cycle or geometric consistency across multiple views or artificial domains (Zhou et al., 2018, Zhao et al., 2021).
• Consistency regularization under augmentations or domain shifts (for domain adaptation) (El-Ghoussani et al., 2024).

4. Dynamic Scenes, Motion Segmentation, and Robustness

A key theoretical and empirical challenge is the epipolar ambiguity in monocular depth learning in dynamic environments. The observed image-plane flow at a pixel can be equally explained by hypothesizing a different depth or by postulating independent object motion. If left unresolved, moving objects are often mis-inferred as arbitrarily far or flat (Li et al., 2020, Sun et al., 2023).

Solutions include:

• Joint learning of depth, pose, and dense 3D motion: Networks predict per-pixel translation fields and a binary motion mask, fusing rigid and nonrigid flows in a learned gating mechanism (Almalioglu et al., 2020, Sun et al., 2023, Hui, 2023).
• Explicit pre-training for motion segmentation: Early-stage depth networks are trained under static assumptions, then motion masks and flow fields are initialized to separate static and dynamic pixels before joint fine-tuning (critical for stability) (Sun et al., 2023).
• Sparse+piecewise-constant regularisation: Encourages object-motion fields to be zero almost everywhere (background) and constant on rigid objects, resolving underdeterminedness (Li et al., 2020).
• Cycle-consistency and cross-view geometry: Enforce that motion forward, backward, and round-trip reconstructs original geometry and pose, penalizing implausible solutions (Li et al., 2020).

Performance gains on dynamic scenes (e.g., Waymo, nuScenes) can be dramatic: Dynamo-Depth reduces AbsRel on moving objects by up to 68% compared to static-scene baselines, with large improvements in δ<1.25 accuracy for moving regions (e.g., +48–62%) (Sun et al., 2023). Similar trends hold for RM-Depth, which achieves state-of-the-art on challenging urban benchmarks for both static and dynamic scenes with only 2.97 M parameters (Hui, 2023).

5. Domain Generalization, Adaptation, and Robustness

Unsupervised monocular depth methods are increasingly deployed across diverse domains: outdoor driving, indoor navigation, underground robotics, endoscopy, and simulated-to-real settings.

• Domain adaptation and transfer: Approaches such as image-transfer and domain-feature adaptation align latent features and outputs across source and target domains (e.g., day-to-night, synthetic-to-real). Sample streams are weighted by cycle-reconstruction quality to discount poor style-transferred images (Zhao et al., 2021). Consistency regularisation, e.g., enforcing perturbation-invariant depth (El-Ghoussani et al., 2024), provides robust unsupervised adaptation without auxiliary networks or adversarial objectives.
• Intrinsic-based robustness: Medical endoscopy requires depth methods tolerant to specularities and non-Lambertian effects, addressed via intrinsic decomposition and transformer adaptation with local-feature augmentation (Li et al., 2024).
• Implicit depth consistency: Implicit depth, computed analytically under known camera transformations, is matched with network predictions to enforce scale-consistency across long sequences (Liu et al., 2024).

Performance across domains is summarized in the following table for selected methods (AbsRel = lower is better):

Method	Environment	AbsRel
Monodepth2	KITTI	0.115
FG-Depth	KITTI	0.099
Dynamo-Depth	Waymo (all)	0.116
SwinDepth	KITTI	0.106
RM-Depth	KITTI	0.107
Adv. DepthAny	Endoscopy SCARED	0.048
ITDFA (night)	Oxford RobotCar	0.1469
Consist.Reg.	vKITTI→KITTI	0.161

This suggests that recent advances have significantly closed the domain gap, particularly when explicit domain adaptation or robust photometric surrogates are used.

6. Evaluation Protocols and Metrics

Standardized evaluation on depth benchmarks employs the Eigen split for KITTI (697 test images), Make3D, NYUv2 for indoors, and SCARED or Hamlyn for endoscopy. Reported metrics include:

• AbsRel = mean$|D_{pred} - D_{gt}| / D_{gt}|$
• SqRel = mean$(D_{pred} - D_{gt})^2 / D_{gt}$
• RMSE, RMSElog
• δ thresholds: fraction of pixels where $\max(D_{pred}/D_{gt}, D_{gt}/D_{pred}) < 1.25^i$ for $i=1,2,3$
• Application-specific: ATE drift in pose (trajectory error), motion-mask F1, or cycle error (for teacher–student distillation)

Qualitative evaluation focuses on edge sharpness, recovery of thin and distant structures, removal of "texture copy" artifacts on moving objects, and depth-map fidelity near occlusion boundaries and under adverse conditions (e.g., blur, noise, illumination shifts).

Ablation studies consistently demonstrate:

• The necessity of cross-frame or domain-consistency regularizers
• The substantial impact of occlusion and motion modeling in dynamic scenes
• The benefit of additive SSIM formulations for stable training and improved metrics (Cao et al., 5 Jun 2025)
• The effectiveness of lightweight architectural innovations (e.g., RVLoRA, Res-DSC, RMUs) in parameter-efficient regimes (Li et al., 2024, Hui, 2023)

7. Open Challenges and Future Directions

Key remaining challenges in unsupervised monocular depth estimation include:

• Handling non-rigid motion: Current protocols regularize rigidity or employ primitive object-motion fields, but scenes with significant deformation or articulated objects remain difficult (Li et al., 2020, Hui, 2023).
• Illumination and appearance variability: Even intrinsic-based models may fail under extreme lighting or specularity not covered by training domains (Li et al., 2024).
• Robust scale-consistency over long horizons: While scale-aware loss terms and bundle-adjustment modules improve temporal stability (Almalioglu et al., 2020, Zhou et al., 2018), persistence over hours and in unseen conditions is still open.
• Computational efficiency: Achieving real-time performance with few parameters without sacrificing fine spatial details remains an active area, especially on embedded hardware (Hui, 2023, Li et al., 2024).

Potential directions include integrating uncertainty quantification, self-adaptive loss scheduling, higher-order similarity measures, and truly domain-agnostic adaptation modules. A plausible implication is that plug-and-play photometric surrogates that generalize beyond SSIM and encode task-aware invariances (e.g., via learned augmentation or feature distillation) may offer the next leap in generality and robustness.

---

Unsupervised monocular depth estimation now encompasses a diverse family of techniques—geometric, photometric, motion-aware, and domain-adaptive. Its evolution is marked by increasingly sophisticated representations, loss functions, and integration with large-scale foundation models. Advances documented in (Almalioglu et al., 2020, Li et al., 2020, Sun et al., 2023, Li et al., 2024, Shim et al., 2023, Cao et al., 5 Jun 2025, Zhu et al., 2023, Zhao et al., 2021, Hui, 2023), and others have established competitive or state-of-the-art accuracy across outdoor, indoor, medical, and adverse domains without reliance on ground-truth depth, setting the stage for robust monocular 3D perception in challenging real-world deployments.