Depth Prompting: Sensor-Agnostic Estimation

• Depth prompting is a prompt-based formulation that uses a dedicated depth prompt from sparse measurements to condition a frozen monocular model for producing absolute-scale depth maps.
• The approach decouples image features from depth cues, employing adaptive spatial propagation to overcome sensor biases in density, pattern, and range.
• Experimental results show that the method maintains low RMSE and high DELTA1 across varying sensor setups and sparse measurement conditions.

Searching arXiv for the target paper and closely related depth-estimation work to ground the article in current literature. "Depth Prompting" denotes a prompt-based formulation of image-guided depth estimation in which sparse metric depth is not fused into a joint RGB-depth representation, but instead encoded as a dedicated conditioning signal for a monocular depth foundation model. In "Depth Prompting for Sensor-Agnostic Depth Estimation" (Park et al., 2024), the central claim is that systematic sensor biases—density, sensing pattern, and scan range—degrade the generalization of conventional joint-fusion depth completion methods, because the learned feature space becomes tied to the training sensor. The proposed alternative disentangles image and depth modalities, uses a depth prompt to condition adaptive spatial propagation, and thereby produces absolute-scale dense depth maps that are sensor-agnostic and scan-range–agnostic (Park et al., 2024).

1. Depth estimation and the role of sensor bias

Depth estimation predicts the distance from the camera to scene surfaces, typically in the form of a dense depth map that assigns a depth value to every pixel. Such maps are a fundamental representation for 3D reconstruction and mapping, robotics and autonomous driving, and AR/VR scene understanding (Park et al., 2024). In monocular depth estimation, a model receives a single RGB image $I \in \mathbb{R}^{3\times H\times W}$ and predicts a dense depth field, often via a foundation model $f_{\mathcal{F}}$ that also exposes intermediate multi-scale image features: $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$ Monocular models are effective at recovering relative depth, but absolute metric scale remains strongly dependent on camera intrinsics and training distribution, which limits out-of-domain generalization (Park et al., 2024).

Active depth sensors provide sparse but metric depth, yet introduce systematic biases. The paper distinguishes three such biases. Density bias arises when a model trained with one sparsity level fails under sparser test measurements. Pattern bias arises when the spatial sampling geometry changes, as between LiDAR line scans, Kinect-like quasi-dense measurements with holes, or random point sampling. Range bias arises when training and testing differ in scan range, for example indoor $0\text{–}10$ m versus outdoor $0\text{–}80$ m conditions (Park et al., 2024). The core observation is that recent RGB-depth completion systems generally learn joint representations of the two modalities, and those fused representations inherit the statistics of the training sensor, making them brittle under sensor shift (Park et al., 2024).

This diagnosis is consistent with a broader trend in depth completion research: methods that are highly adapted to a particular discretization or sensing regime can lose flexibility when the scene depth distribution changes. "Progressive Depth Decoupling and Modulating for Flexible Depth Completion" (Yang et al., 2024) addresses scene-dependent variation through adaptive depth binning, while "Depth Prompting for Sensor-Agnostic Depth Estimation" (Park et al., 2024) addresses sensor-dependent variation through modality disentanglement and prompt-conditioned propagation. This suggests that generalization in depth estimation depends not only on predictor capacity, but also on how depth measurements are represented and injected.

2. Prompt-based formulation and architecture

The defining object in the method is the depth prompt: a compact learned representation derived from sparse depth measurements that conditions a monocular depth foundation model for dense metric estimation (Park et al., 2024). The overall system contains three components: a monocular depth foundation model $f_{\mathcal{F}}$, a depth prompt encoder-decoder $f_{\mathcal{E}}, f_{\mathcal{D}}$, and a spatial propagation module (Park et al., 2024).

Sparse depth $D_S$ is processed by a ResNet34-based encoder-decoder: $F^d, F^d_k = f_{\mathcal{E}}(D_S).$ Here $F^d$ is a global prompt embedding and $f_{\mathcal{F}}$0 are multi-scale depth features at strides $f_{\mathcal{F}}$1 of input resolution (Park et al., 2024). The encoder is described as sensor-agnostic: it maps sparse depth of any pattern or density into a unified embedding space that captures both where measurements exist and how densely they are distributed (Park et al., 2024).

In parallel, the foundation model processes the RGB image: $f_{\mathcal{F}}$2 The model is largely frozen; only bias terms are fine-tuned, corresponding to about $f_{\mathcal{F}}$3 of backbone parameters (Park et al., 2024). This preserves the monocular prior learned from large-scale data while allowing modest adaptation to prompt-conditioned refinement.

The decoder fuses image features and depth-prompt features to predict an adaptive affinity map: $f_{\mathcal{F}}$4 with $f_{\mathcal{F}}$5 and $f_{\mathcal{F}}$6, so each pixel is associated with a $f_{\mathcal{F}}$7 propagation neighborhood (Park et al., 2024). The prompt therefore does not act as a token appended to a transformer sequence; rather, it modulates spatial propagation by controlling how metric information diffuses through the image-conditioned feature field (Park et al., 2024).

The paper emphasizes that this differs fundamentally from direct RGB-depth fusion. The backbone remains predominantly image-centric, while sparse depth influences the estimator through a prompt-conditioned affinity structure. A plausible implication is that the approach separates generic visual understanding from sensor-specific measurement structure more cleanly than fused encoders.

3. Spatial propagation and absolute-scale reconstruction

Depth completion is formulated as iterative spatial propagation: $f_{\mathcal{F}}$8 In classical spatial propagation networks, the affinity $f_{\mathcal{F}}$9 is learned as a fixed function of the training data distribution. In Depth Prompting, this is replaced by the prompt-conditioned affinity $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$0, so propagation depends jointly on image content and the sparse depth distribution present at test time (Park et al., 2024). This is the operational mechanism by which density, pattern, and range shifts are handled.

Because monocular foundation models generally predict relative depth, the method aligns $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$1 to the sparse metric measurements via least squares: $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$2 Here $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$3 denotes the monocular prediction restricted to pixels where $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$4 is observed (Park et al., 2024). This scalar alignment anchors the monocular prediction to metric scale wherever sparse depth exists, while the propagation process extends that metric consistency into unmeasured regions (Park et al., 2024).

The resulting interpretation is precise. Where the sensor measures depth, sparse points provide metric anchors. Where the sensor has holes or is out of range, the monocular model supplies dense relative structure, and prompt-conditioned propagation converts that structure into an absolute-scale estimate (Park et al., 2024). This is how the method claims scan-range agnosticism: sparse measurements fix scale locally, while monocular priors and adaptive propagation extrapolate beyond the native sensing envelope (Park et al., 2024).

This architecture places the work in a broader class of hybrid metric-relative methods. "Depth Anything with Any Prior" (Wang et al., 15 May 2025) also combines incomplete metric priors with dense monocular predictions through a coarse-to-fine fusion pipeline, but it does so by pre-filling priors and conditioning a monocular estimator. Depth Prompting instead injects sparse depth through a prompt that modulates propagation (Park et al., 2024). The two approaches share the view that dense monocular geometry and sparse metric anchors are complementary, but differ in how that complementarity is operationalized.

4. Integration with foundation models and optimization

The approach is designed as a plug-in module for monocular depth foundations including DepthFormer, MiDaS v3.1, and KBR (Park et al., 2024). Integration follows three principles. First, almost all backbone parameters are frozen. Second, only bias terms are tuned. Third, the depth prompt encoder-decoder is trained from scratch, with gradients flowing through the differentiable spatial propagation module (Park et al., 2024).

The parameterization is correspondingly economical. Total learnable parameters are about $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$5M, with only $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$6M assigned to foundation-model tuning, approximately $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$7 of the backbone (Park et al., 2024). The paper presents this as a memory-efficient adaptation scheme analogous to prompt tuning in LLMs, where the core model remains general and prompts steer behavior toward the target sensing regime (Park et al., 2024).

Training is supervised with metric ground-truth depth $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$8. The initial monocular prediction is constrained with a scale-invariant loss: $$\hat{D}_I, F^i_k = f_{\mathcal{F}}(I, \Theta_{f_{\mathcal{F}}}).$$9

$0\text{–}10$0

The final dense output is supervised with a combined L1/L2 regression loss: $0\text{–}10$1 The total objective is

$0\text{–}10$2

The role of this decomposition is explicit: $0\text{–}10$3 preserves the relative-depth competence of the monocular foundation, while $0\text{–}10$4 drives the prompt module and propagation mechanism toward accurate metric completion (Park et al., 2024).

In comparison with conventional domain adaptation, which often fine-tunes most or all of the backbone for a new sensor or environment, the prompting approach pushes adaptation into bias tuning and an explicit depth-conditioned module (Park et al., 2024). This suggests a more modular route to sensor adaptation, one that preserves broad monocular generalization while specializing only the interaction between sparse metric cues and dense visual context.

5. Experimental evidence across sensors, patterns, and ranges

The empirical study spans indoor, outdoor, and zero-shot cross-sensor settings. NYUv2 uses $0\text{–}10$5 indoor RGB-D imagery, with sparse depth formed by randomly sampling 500 points and then testing densities from 200 down to 1 point, random versus grid sampling patterns, and range splits $0\text{–}10$6 m versus $0\text{–}10$7 m (Park et al., 2024). KITTI Depth Completion uses Velodyne HDL-64E LiDAR with approximately $0\text{–}10$8 lines and $0\text{–}10$9 density, with tests varying from 32 down to 1 LiDAR line, switching between line and random patterns, and shifting range from $0\text{–}80$0 m to $0\text{–}80$1 m (Park et al., 2024). Additional zero-shot sensors include Apple iPhone/iPad LiDAR, Intel RealSense, and 32-line LiDAR from nuScenes and VOID, with no further training (Park et al., 2024).

Evaluation uses RMSE, MAE, and DELTA1, the inlier ratio at $0\text{–}80$2 (Park et al., 2024). The reported findings are organized around three bias types and two transfer settings.

Density robustness is demonstrated on both NYUv2 and KITTI. The paper states that baseline methods including CSPN, S2D, NLSPN, DySPN, CompletionFormer, and SAN degrade sharply as the number of points or LiDAR lines decreases, whereas the proposed method maintains significantly lower RMSE and MAE and higher DELTA1 even at extreme sparsity such as a single NYUv2 point or a single KITTI LiDAR line (Park et al., 2024).

Pattern robustness is tested by switching random-to-grid and grid-to-random on NYUv2, and line-to-random or random-to-line on KITTI. Conventional spatial propagation models are described as strongly biased to their training pattern, while depth prompting reduces the increase in RMSE and MAE under pattern shift and preserves DELTA1 more effectively (Park et al., 2024).

Range robustness is evaluated by training on one depth interval and testing on another. Examples include training on $0\text{–}80$3 m and testing on $0\text{–}80$4 m in NYUv2, or training on near-range and testing on far-range in KITTI. Standard methods are reported to struggle outside the training scan range, while the proposed method maintains markedly better accuracy through the combination of foundation-model relative depth, least-squares alignment, and prompt-conditioned propagation (Park et al., 2024).

Cross-domain transfer is examined through few-shot adaptation between NYU and KITTI, using 10 or 100 images with sparse depth seeds. The prompt-based model outperforms CSPN, NLSPN, and CompletionFormer in both indoor-to-outdoor and outdoor-to-indoor transfer directions (Park et al., 2024).

Backbone generality is tested by plugging the prompt module into MiDaS and KBR. The same prompting scheme remains effective across these backbones, and the self-supervised KBR backbone is reported to show particularly strong generalization when combined with depth prompting (Park et al., 2024).

Qualitatively, traditional methods are described as producing fragmented depth maps, over-smoothed backgrounds, and mis-scaled near or far objects under extreme sparsity or pattern mismatch, whereas the prompt-based system yields coherent geometry, correct global metric scale, and plausible extrapolation into unmeasured regions (Park et al., 2024). In zero-shot tests on commercial sensors such as Apple LiDAR and RealSense, the model reportedly completes depth maps with high fidelity and consistent metric scale, outperforming built-in frameworks such as ARKit and dataset-specific learning-based baselines (Park et al., 2024).

6. Conceptual significance, limitations, and relation to adjacent work

The paper advances a particular interpretation of depth estimation: depth is not merely a scalar field to regress, but a modality whose measurement distribution depends on sensor density, sampling geometry, and range. The authors argue that a model that ignores these distributional aspects will overfit to training sensors, and that separating image understanding from depth conditioning improves robustness (Park et al., 2024). In that sense, "Depth Prompting for Sensor-Agnostic Depth Estimation" (Park et al., 2024) reframes prompt engineering as a mechanism for low-level geometric adaptation rather than only high-level task specification.

This perspective relates to several adjacent lines of work. "Depth Coefficients for Depth Completion" (Imran et al., 2019) argues that scalar depth representations promote inter-object depth mixing and proposes a multi-channel probabilistic depth representation to better handle discontinuities. "Progressive Depth Decoupling and Modulating for Flexible Depth Completion" (Yang et al., 2024) similarly emphasizes scene-adaptive depth distributions, using progressive bin refinement and bi-directional interactions between depth priors and per-pixel probabilities. A plausible synthesis is that depth generalization depends critically on respecting structure in the depth modality itself—whether that structure is distributional, categorical, or sensor-specific.

The method also belongs to a broader movement toward leveraging strong monocular backbones as geometric priors. "Prior Depth Anything" (Wang et al., 15 May 2025) combines incomplete metric priors with dense relative predictions in a coarse-to-fine pipeline and reports zero-shot generalization across completion, super-resolution, and inpainting. "DEPTHOR" (Xiang et al., 2 Apr 2025) addresses practical dToF enhancement by integrating monocular depth estimation with noisy sensor completion, explicitly treating real sensor artifacts as a robustness problem. These works differ in architecture and domain, but all treat monocular priors as globally informative and sparse metric cues as scale anchors.

The limitations stated in the paper are concrete. Performance depends on the quality and domain coverage of the foundation model; if $0\text{–}80$5 provides poor relative depth, the entire system suffers (Park et al., 2024). Extreme sensor differences or highly corrupted measurements may still be problematic (Park et al., 2024). The model contains roughly $0\text{–}80$6M parameters and uses spatial propagation, with inference times of $0\text{–}80$7 s on NYU and $0\text{–}80$8 s on KITTI on high-end GPUs, so edge deployment may require further optimization (Park et al., 2024). Finally, propagation remains constrained by the availability of sparse seeds and by the ambiguity of monocular cues in unmeasured regions (Park et al., 2024).

The future directions suggested in the paper include multi-sensor prompting, dynamic per-scene prompts, temporal integration, and closer integration with multimodal foundation models (Park et al., 2024). These directions are consistent with the article’s central insight: robust metric depth estimation in the wild may require keeping general visual semantics in a largely frozen backbone while encoding sensor behavior, scene configuration, and temporal context through explicit conditioning pathways rather than entangled fusion.