Focusable Monocular Depth Estimation

Abstract: Monocular depth foundation models generalize well across scenes, yet they are typically optimized with uniform pixel-wise objectives that do not distinguish user-specified or task-relevant target regions from the surrounding context. We therefore introduce Focusable Monocular Depth Estimation (FDE), a region-aware depth estimation task in which, given a specified target region, the model is required to prioritize foreground depth accuracy, preserve sharp boundary transitions, and maintain coherent global scene geometry. To prioritize task-critical region modeling, we propose FocusDepth, a prompt-conditioned monocular relative depth estimation framework that guides depth modeling to focus on target regions via box/text prompts. The core Multi-Scale Spatial-Aligned Fusion (MSSA) in FocusDepth spatially aligns multi-scale features from Segment Anything Model 3 to the Depth Anything family and injects them through scale-specific, gated conditional fusion. This enables dense prompt cue injection without disrupting geometric representations, thereby endowing the depth estimation model with focused perception capability. To study FDE, we establish FDE-Bench, a target-centric monocular relative depth benchmark built from image-target-depth triplets across five datasets, containing 252.9K/72.5K train/val triplets and 972 categories spanning real-world and embodied simulation environments. On FDE-Bench, FocusDepth consistently improves over globally fine-tuned DA2/DA3 baselines under both box and text prompts, with the largest gains appearing in target boundary and foreground regions while preserving global scene geometry. Ablations show that MSSA's spatial alignment is the key design factor, as disrupting prompt-geometry correspondence increases AbsRel by up to 13.8%.

• The paper redefines monocular depth estimation as a prompt-conditioned task, emphasizing improved accuracy in targeted regions.
• It introduces FocusDepth, a dual-branch model that fuses global depth priors with local prompt features using MSSA, MoE fusion, and gated residuals.
• Empirical results on the FDE-Bench benchmark show significant enhancements in boundary and foreground depth accuracy across diverse datasets.

Focusable Monocular Depth Estimation: Target-Aware Depth via Prompt-Conditioned Fusion

Introduction and Motivation

Monocular depth estimation is critical in robotics, embodied AI, AR, and 3D perception, with recent years witnessing the rise of foundation models providing strong global geometric priors from large-scale training. However, prevailing approaches treat depth estimation as a uniform, image-wide task, neglecting the heterogeneous requirements of spatially-focused downstream usage. Task-driven scenarios—such as robot manipulation—demand heightened foreground accuracy and sharp boundary transitions on user-specified regions, not well-supported by uniform objectives.

The "Focusable Monocular Depth Estimation" paper (2605.11756) redefines monocular depth estimation as a region-aware, prompt-conditioned task. Instead of optimizing indiscriminately over all pixels, the goal is to yield highly accurate depth within a prompted region (via bounding box or text description), while maintaining global scene coherence. This is operationalized through FocusDepth—a hybrid framework leveraging the architectural strengths of both state-of-the-art segmentation (SAM3) and depth (Depth Anything family) foundation models, guided by Multi-Scale Spatial-Aligned Fusion (MSSA) for controlled, spatially-aware feature injection.

FocusDepth Architecture and Methodology

FocusDepth reframes monocular depth estimation as a prompt-guided, dual-branch model:

• The geometry branch leverages pretrained depth foundation models (DA2/DA3) for global dense depth priors.
• The prompt-conditioned branch utilizes SAM3, encoding user-specified targets as either text or bounding box prompts, and extracting region-focused features.

The central innovation, MSSA, fuses prompt-conditioned and geometric features at multiple scales, enforcing spatial alignment via patch-wise correspondence (enabled by ViT-based tokenization). This design incorporates several mechanisms:

• Spatial-aligned token concatenation: Ensures local target cues modulate depth at precisely corresponding spatial locations.
• Mixture-of-experts (MoE) fusion: Allows conditional, input-dependent blending of prompt and geometric cues, essential for region-specific corrections.
• Gated residual fusion: Uses learnable sigmoid gates to prevent the overwriting of the pretrained geometric prior, ensuring global coherence is not compromised by local injections.

The overall supervision objective is a weighted sum of region-aware losses—foreground, boundary, and global—plus auxiliary segmentation, reflecting the explicit prioritization of boundary and foreground accuracy. The authors adopt a two-stage training regime, first optimizing only MSSA (frozen backbone), then adapting downstream heads, which is shown empirically to stabilize the local-global tradeoff.

FDE-Bench: Target-Centric Benchmark for Depth Estimation

To facilitate systematic evaluation under this new paradigm, the paper introduces FDE-Bench, a large-scale benchmark constructed from five RGB-D datasets (NYU v2, TUM RGB-D, RLBench, RoboTwin, YCB-Video). Each benchmark entry is an image-target-depth triplet, with region definitions and prompt specification (box/text) standardized across datasets. Evaluation metrics are region-aware: δ1 and AbsRel are reported on the specified foreground, target boundary (10-pixel ring), and full image.

Dataset construction ensures coverage of diverse spatial contexts, target sizes, and prompt types, allowing for robust comparison of prompt-conditioned and standard depth models under task-centric conditions.

Empirical Results

FocusDepth demonstrates consistent and substantial improvements in region-focused depth accuracy, supported by strong quantitative and qualitative results:

• On RLBench (box-prompt), FocusDepth(DA3) improves boundary AbsRel from 0.073 (DA3-ft) to 0.049, and foreground AbsRel from 0.095 to 0.056, while improving or at least matching global region performance.
• On RoboTwin, FocusDepth(DA3) outperforms DA3-ft both in target boundaries and foreground, reducing AbsRel (e.g., boundary: 0.082 vs. 0.058).
• On NYU v2 and YCB-Video, global metrics are nearly saturated for all methods, yet boundary and foreground accuracy see additional gains with FocusDepth, demonstrating the efficacy of prompt-conditioned injection particularly in object-centric and manipulation-focused scenarios.

Ablation studies underline the primacy of spatial alignment in MSSA: disrupting prompt-geometry correspondence degrades global AbsRel by up to 13.8%. Scale-specific MoE fusion and gated blending are also necessary for optimized region adaptation. Region-aware supervision and staged training are key for stabilizing the local-global tradeoff.

The prompt-correctness analysis demonstrates FocusDepth's robustness: correct prompts yield optimal local improvements, but even absent or wrong prompts do not collapse local/global accuracy, indicating useful depth cues are injected even without explicit target specification.

Qualitative analysis highlights sharper boundaries and accurate gradient transitions in prompted regions, indicating that FocusDepth succeeds not only in reducing average error but also in localizing depth discontinuities vital for manipulation and segmentation applications.

Implications and Future Directions

The explicit framing of region-prioritized monocular depth estimation re-aligns model objectives with practical task requirements in robotics, AR, and spatial understanding—they often require accurate, local geometry (e.g., at manipulation points) rather than uniform scene understanding. FocusDepth's design provides a blueprint for controlling the spatial distribution of prediction quality by harmonizing dense geometric priors with prompt-driven specificity, a model which may generalize beyond depth to other dense prediction tasks.

Practically, these methods pave the way for more robust, context-aware scene understanding pipelines in embodied agents, particularly where user attention or task relevance changes dynamically. The design of MSSA as a prompt-conditioned, gated, and spatially-resolved fusion mechanism is especially promising for the next wave of interactive, multimodal perception models.

Potential future work includes:

• Large-scale joint training between segmentation and depth branches to further harmonize representations.
• Extension to metric depth estimation and non-relative, zero-shot generalization.
• Downstream task integration, such as closed-loop robot control, where local geometry accuracy can be directly linked to manipulation or navigation performance.

Conclusion

This paper establishes Focusable Depth Estimation as a task wherein monocular depth models are expected to focus depth accuracy and boundary fidelity to user- or task-specified targets. The FocusDepth framework, enabled by Multi-Scale Spatial-Aligned Fusion, delivers consistent region-aware improvements across heterogeneous datasets without degrading global performance. The comprehensive FDE-Bench further facilitates reproducible, task-centric evaluation. As downstream applications increasingly demand spatially-controllable perception, the architectural and evaluation methodology introduced here represents a significant step toward targeted, prompt-driven dense understanding in computer vision (2605.11756).