Prompt Depth Anything in Estimation

• Prompt Depth Anything is a framework that uses prompt-based techniques to adapt depth estimation models through sensor inputs and contextual cues.
• It employs strategies such as background prompting, sensor-based metric cues, and modular feature-level prompts to enhance model adaptability and precision.
• The approach integrates architectural prompt injections and self-supervised training to achieve robust performance in both zero-shot and sim2real scenarios.

Prompt Depth Anything encompasses a set of techniques that use prompt-based strategies—either via learnable embeddings, input-dependent cues, or symbolic instance alignment—to enhance or control depth estimation models, especially foundation models such as Depth Anything, Segment Anything, and their derivatives. The methods operationalize a broad prompt concept, spanning background/context manipulation, sensor-provided measurements (e.g., LiDAR), explicit depth-aware prompt representations, architectural prompt injection, and compositional reasoning protocols in vision-language pipelines. The field is characterized by modularity, the use of depth as either a conditioning input or symbolic constraint, and a unification of metric priors with relative monocular depth inference.

1. Foundations of Prompt-Based Depth Estimation

Prompting in monocular depth estimation originated through the recognition that large, generic depth models lack adaptability to distribution shifts, sensor idiosyncrasies, or complex scene compositions. Foundational models such as Depth Anything (Yang et al., 19 Jan 2024) provide dense depth maps trained on vast, unlabeled collections augmented by strong teacher-student pipelines and semantic anchor losses inherited from large-scale ViT feature alignment. However, these models are subject to limitations when absolute metric accuracy, robustness to sensor bias, or object-level reasoning is required.

Prompting circumvents these issues by providing auxiliary information—either as explicit numeric priors (sparse LiDAR, ToF), visual context (background prompts), or learned prompt embeddings injected into model architectures. These techniques enable rapid adaptation, enhanced metric fidelity, and increased generalization beyond the vanilla capacity of pre-trained foundation depth models (Lin et al., 18 Dec 2024, Park et al., 20 May 2024).

2. Prompt Strategies: Representation and Mechanisms

Prompt Depth Anything is instantiated through several distinct but sometimes overlapping mechanisms:

• Background/Context Prompting: Synthetic context or a learnable background (B) is composed with the object of interest, stabilizing sim2real transfer and minimizing context bias (Baradad et al., 2023). The prompt is learned on synthetic object crops and later composited with real imagery, making pretrained depth networks invariant to irrelevant backgrounds by normalizing input statistics:

$$C_i = M_i \odot \phi(x_i) + (1-M_i)\odot \phi(\sigma(B))$$

This permission of both unconditional and conditional (mask-dependent) prompting allows for foreground-focused estimation with enhanced sharpness and realism.

• Sensor-Based Metric Prompting: Sparse, metrically accurate cues (e.g., LiDAR, ToF) are injected as prompts into depth foundation models (Lin et al., 18 Dec 2024, Wang et al., 15 May 2025). In the Prompt Fusion Block paradigm, LiDAR maps are resized and convolved, then projected and fused at multiple decoder layers as:

$$F_i' = F_i + P_i,\quad P_i = \phi_i(\text{Conv}_2(\text{ReLU}(\text{Conv}_1(L_i))))$$

This design endows the model with early, spatially aligned metric priors, enforcing scale consistency and landmark anchoring across resolutions up to 4K.

• Feature-Level Prompting and Modular Prompt Banks: Depth prompt modules constructed from explicit ResNet encoders or ViT layers produce multi-scale prompt features concatenated or cross-attended with image features (Park et al., 20 May 2024, Chen et al., 24 Sep 2024). These modules may be task- or sensor-adaptive, supporting generalization across density, pattern, and range biases.
• Symbolic/Instance-Level Prompting for Vision-Language: Depth maps are used post-hoc as instance-level symbolic features, enabling per-object reasoning in composition pipelines for multi-modal understanding (Huo et al., 7 Jun 2024). Here, the model returns a depth map, and mean depth is computed per segment, providing structured inputs for compositional VQA or spatial grounding.
• Learnable Visual Prompt Embeddings: Architecture-level prompt tensors (learnable, domain-specific, multi-token) are fused with deep feature maps via cross-prompting attention blocks (Wang et al., 23 Jan 2025). These learnable prompts, often realized as $P \in \mathbb{R}^{C \times N \times H \times W}$, adapt model behavior across domains (e.g., night, rain), improving self-supervised generalization and robustness.

3. Prompt Injection and Training Methodologies

Prompt Depth Anything systems employ specific architectural or optimization strategies to inject and leverage prompts:

• Cross-Attention and Prompt Fusion: Prompt features are injected via cross-attention within UNet bottlenecks or decoder upsampling stages (Chen et al., 24 Sep 2024, Wang et al., 23 Jan 2025). For example, the Depth-Anything-constrained prompt is L2-normalized and injected with cross-attention weights, ensuring that the contribution of the prompt is adaptive and robust to input corruption.
• Zero-Initialized Convolutions and Fine-Tuning: When prompts (e.g., predicted depth or metric prior) are concatenated with RGB at model input, zero-initialized 1×1 convolutions are used so that fusion does not degrade initial performance (Wang et al., 15 May 2025).
• Self-Supervised and Multi-Objective Training: Losses typically include scale-invariant error, photometric reconstruction, edge-aware smoothness, and contrastive or feature-alignment terms. In multiple-task pipelines, additional losses enforce consistency between depth outputs and metric priors or semantic segmentation features (Lin et al., 18 Dec 2024, Wang et al., 23 Jan 2025).
• Adapter and Query-Based Prompting in Segmentation: Prompt-free adaptation of segmentation models (SAM) employs lightweight, depth-guided adapters and parallel memory/query modules, enabling effective integration of dense depth cues for salient video segmentation without hand-crafted prompts (Lin et al., 13 Nov 2025).

4. Multimodal Applications: Vision-Language Reasoning and Segmentation

Prompt Depth Anything methods have demonstrable impact in multi-modal contexts:

• Compositional Reasoning: By fusing segment masks (from SAM) and per-instance mean depths (from DAM), models provide structured, symbolic spatial information for vision-LLMs such as GPT-4V. The input pipeline is:

$$\text{masks}, \text{classes} = \mathrm{SAM}(I, \text{prompt}=\psi) \ \text{depth\_map} = \mathrm{DAM}(I) \ \bar d_i = \frac{1}{|P_i|} \sum_{p \in P_i} d(p)$$

enabling the reasoning module to utilize relational depth at the segment level (Huo et al., 7 Jun 2024).

• Layered-Depth-Based Prompting for MLLMs: LDP divides depth maps into percentile-based layers (e.g., closest/mid/farthest), and region captions are obtained per mask using a grounded VLM (e.g., KOSMOS-2). The resulting structured, depth-aware textual prompt seeds downstream reasoning in MLLMs, significantly reducing hallucinations and improving spatial grounding in VQA (Roy et al., 11 Jul 2025).
• Depth Prompted Segmentation: For camouflaged object or RGB-D video salient-object detection, depth-driven prompt modules or adapters—informed by knowledge distillation or explicit multi-query memory—allow fine-grained segmentation beyond RGB-only or prompt-requiring baselines (Yu et al., 17 Jul 2024, Lin et al., 13 Nov 2025).

5. Zero-Shot Generalization and Empirical Performance

Prompt Depth Anything methods consistently deliver strong empirical results across zero-shot, cross-modal, and sim2real tasks:

• Metric Depth Generalization: Fusing sparse LiDAR prompts with foundation models achieves state-of-the-art results on ARKitScenes/ScanNet++ at resolutions up to 4K, along with robust downstream 3D reconstruction and generalized robotic grasping (Lin et al., 18 Dec 2024).
• Sensor-Agnostic Robustness: Prompt-based depth modules mitigate density/pattern/range biases, achieving low RMSE even with minimal (e.g., 4-point or 4-scanline) sensor input and robust performance in cross-domain evaluations (Park et al., 20 May 2024).
• Sim2Real Object Depth: Background prompting lowers domain gap, yielding dramatic improvements on out-of-distribution test sets, with quantitative gains such as si-RMSE dropping from 0.93→0.43 on Google Scans using DPT+prompt (Baradad et al., 2023).
• Multimodal Reasoning: Layered depth prompting increases binary VQA accuracy by 0.01–0.08 and spatial reasoning accuracy by 0.1–0.15 across BLIP, ViLT, Qwen2.5-VL, and GPT-4o without retraining or parameter modification (Roy et al., 11 Jul 2025).

6. Future Directions and Open Challenges

Despite substantial advances, prompt-based depth modeling poses open questions:

• Hierarchical and Universal Prompting: Prospects include learning scene-category or patch-level depth prompts, universal prompt banks spanning multiple foundation models, or adaptive resizing and attention-based prompt designs to further expand generalization (Baradad et al., 2023, Park et al., 20 May 2024).
• Multi-Modal Fusion and Uncertainty: Extending prompt design to handle uncertainty, multi-modal inputs (thermal, radar), and dynamic scene evolution may broaden applicability (Park et al., 20 May 2024, Wang et al., 15 May 2025).
• Symbolic Integration and Scalability: Neural-symbolic fusion at larger scale—using per-instance, per-layer, or per-object prompt representations—remains fertile ground, particularly in the context of foundation models for vision-language and compositional reasoning (Huo et al., 7 Jun 2024).
• Parameter Efficiency and Speed: Methods such as low-rank adaptation (RVLoRA) and prompt-injection with adapters demonstrate that state-of-the-art performance can be attained with parameter-efficient modifications suitable for both real-time and resource-constrained settings (Li et al., 12 Sep 2024, Lin et al., 13 Nov 2025).
• Interpretability and Transferability: The interpretability of explicit prompt cues (background, LiDAR, learned tokens) provides a pathway to controlled behavior and explanatory outputs, supporting human-in-the-loop systems and safety-critical deployments.

In summary, Prompt Depth Anything encapsulates a unifying principle for augmenting, specializing, and operationalizing depth foundation models—by means of direct, sensor-aware, or symbolic prompt strategies—across simulation, robotics, segmentation, and vision-language reasoning domains. It provides a template for broader prompt engineering across perception modalities.

References: (Yang et al., 19 Jan 2024, Baradad et al., 2023, Park et al., 20 May 2024, Lin et al., 18 Dec 2024, Wang et al., 15 May 2025, Huo et al., 7 Jun 2024, Roy et al., 11 Jul 2025, Chen et al., 24 Sep 2024, Yu et al., 17 Jul 2024, Wang et al., 23 Jan 2025, Li et al., 12 Sep 2024, Lin et al., 13 Nov 2025, Tian et al., 2023).