Scale-Aware Visual Depth Estimation

• Visual depth estimation refers to methods that compute per-pixel metric depth by decoupling a global scene scale from a normalized relative depth map.
• It utilizes specialized modules like SASP for semantic-based scale prediction and ARDE for adaptive relative depth estimation via transformer attention.
• This integrated approach enables unified, robust depth inference across indoor and outdoor scenes, benefiting applications in robotics, navigation, and AR.

A visual depth estimation approach refers to methods and algorithms that computationally infer scene geometry—specifically per-pixel depth, often in metric scale—from one or several visual inputs such as monocular images, stereo pairs, or video sequences. These approaches are foundational for perception in robotics, autonomous navigation, augmented reality, and 3D scene understanding. As of 2025, the field encompasses a spectrum of architectures and methodologies, ranging from non-parametric example-based inference to deep learning, transformer-based models, and fusion with supplementary cues such as semantics, geometry, inertial data, or vision-language knowledge.

1. Decomposition and Key Principles in Deep Metric Depth Estimation

A central breakthrough in robust, generalizable monocular depth estimation is the explicit decomposition of metric depth ($M$) into two factors: an image-level scene scale ($S$) and a per-pixel, normalized relative depth map ($R$). Mathematically, the relation is: $M = S \cdot R$ where $M$ is the target metric depth, $S$ is a scalar (or low-dimensional vector) predicting the scale of the scene, and $R \in [0,1]^{H \times W}$ encodes intra-image depth order.

This factorization, as operationalized in the ScaleDepth approach, is motivated by the observation that scene scale generalizes poorly across domains (e.g., from indoor to outdoor), while relative depth (ordinal structure) is widely transferable. By decoupling global scale inference from local depth structure, depth estimation systems can be trained and deployed across diverse environments, handling variations in camera intrinsics, scale, and context without per-domain fine-tuning or explicit range normalization.

The method further structures its prediction pipeline around two specialized modules:

• The Semantic-Aware Scale Prediction (SASP) module aggregates both structural visual cues and scene semantics using external text–image embeddings (e.g., CLIP), ensuring the predicted scale $S$ is informed by both geometric configuration and categorical context.
• The Adaptive Relative Depth Estimation (ARDE) module discretizes the normalized depth range using adaptive bin queries, then regresses per-pixel relative depth using transformer-based attention and bin-wise mask generation.

2. Integration of Semantic and Structural Scene Cues

Semantic context is critical for predicting the physical scene scale in an input image. The SASP module integrates multi-level features from a CLIP-based image encoder with text embeddings derived from scene categories (e.g., “a photo of a conference room”).

Semantic and structural features are fused via learned query aggregation, and supervised through a contrastive similarity loss: $$T_i = \frac{\exp(\cos \langle F_t^i, F_c \rangle / \tau ) }{ \sum_{j=1}^C \exp(\cos \langle F_t^j, F_c \rangle / \tau ) }$$ where $F_t^i$ are text embeddings (per scene category) and $F_c$ is a projected image feature. The scene scale output $S$ is then supervised with classification or regression losses to match the correct (dataset-provided) metric scale.

This semantic infusion ensures that, for instance, images recognized as “outdoor street” yield globally larger $S$ than “indoor kitchen,” rectifying depth range predictions across highly variable environmental contexts without hand-crafted rules.

3. Adaptive Relative Depth Estimation Module

The ARDE module employs a bin-based discretization of the [0,1] relative depth range, where each bin’s location, length, and feature are adaptively regressed for each input.

• Bin queries represent candidate segments within the depth range.
• Transformer layers propagate attention between image features and bin representations, enabling each bin to specialize in particular scene regions or depth structures.
• Mask generation leverages learned similarity between bin features and spatial locations:

$P = E \times F^T$

allowing soft assignment of image pixels to bins for explicit structure localization.

Per-pixel relative depth is then predicted as a weighted sum of bin centers: $$R = \sum_{i=1}^N \text{softmax}(\ldots) \cdot \theta_i$$ where $\theta_i$ is the center of bin $i$.

Training uses the scale-invariant (SI) loss: $$L_\text{SI} = \alpha \sqrt{\mathbb{V}[\delta] + \lambda \mathbb{E}^2[\epsilon]}$$ with $\delta=\log \overline{M} - \log R$, $\overline{M}$ ground-truth metric depth, so the model is robust to global scale errors.

4. Unified Indoor–Outdoor and Unconstrained Scene Generalization

Traditional depth networks often require separate models for indoor and outdoor scenes or explicit range settings during inference. In contrast, decomposing metric depth into $(S, R)$ enables a single, unified model to perform metric estimation across diverse settings.

Empirical evaluation demonstrates strong zero-shot generalization across eight unseen datasets (SUN RGB-D, iBims-1, DIODE indoor/outdoor, HyperSim, Virtual KITTI 2, DDAD, DIML Outdoor), with consistent state-of-the-art accuracy in both error (ARel, RMSE, SILog) and threshold metrics ($\delta_1$).

For example, on KITTI (outdoor road scenes), ScaleDepth-K achieves:

• $ARel = 0.048$ (best among SOTA)
• $RMSE = 1.987$ and on NYU-Depth V2 (indoor), ScaleDepth-N achieves:
• $ARel = 0.074$

Generalization to mixed and unseen environments is accomplished without explicitly setting depth ranges or fine-tuning, directly estimating metric depth under arbitrary real-world conditions.

5. Practical Applications and Performance Implications

The decoupled architecture of ScaleDepth enables direct, physically interpretable depth output—$M = S \cdot R$—crucial for downstream robotics, autonomous navigation, AR/VR, and 3D modeling applications that require depth in metric units or cross-domain robustness.

Notable features include:

• Unified deployment: No need to train or fine-tune per-domain.
• Scale-flexible inference: Seamless transition between scenes of vastly different depths, including high-rise exteriors and small interiors, within one model.
• Semantic awareness: Improves disambiguation of scale in ambiguous scenarios, such as distinguishing between a photo of a model and a real environment.

This approach addresses a principal challenge in monocular metric depth estimation: catastrophic performance drops under domain shift due to mis-calibrated scene scale. By isolating scale inference and leveraging category priors and transformer-based attention, errors are dramatically reduced in cross-dataset, cross-domain, and open-environment deployment settings.

6. Limitations and Future Outlook

Although the explicit scale–relative decomposition enables superior generalization, failures may occur when semantic priors are misapplied or when scene categories are outside the range of training data, impeding scale prediction accuracy. Additionally, the current approach assumes the availability of strong pretrained visual and text encoders (e.g., CLIP), which may be less robust in highly novel visual domains.

Research directions include:

• Scaling training to broader, open-vocabulary or weakly supervised data for increased semantic coverage.
• Incorporating uncertainty estimation or open-set recognition in the SASP module for improved robustness.
• Optimizing computational complexity for edge deployment and real-time applications, preserving metric accuracy in resource-constrained scenarios.

7. Summary Table: Main Components of the ScaleDepth Framework

Module	Role	Mechanism/Details
SASP	Scene scale prediction	Aggregates CLIP-based visual+semantic features, supervised via scene category
ARDE	Relative depth regression	Adaptive bin queries, transformer attention, soft mask-based prediction
Output	Metric depth synthesis	$M = S \cdot R$

This approach advances the state-of-the-art in domain-general monocular metric depth estimation, enabling direct, interpretable, and robust deployment in a broad range of real-world scenarios.