Prompt-based Monocular Depth Estimation

• Prompt-based monocular depth estimation is a paradigm that conditions depth prediction on explicit prompts such as boxes, text, or sparse measurements.
• This approach enhances regional accuracy and semantic alignment by integrating domain-specific cues into established geometric backbones.
• It employs diverse prompt injection techniques like gated attention, affinity modulation, and adapter-based adaptation to improve performance under challenging conditions.

Searching arXiv for papers on prompt-based monocular depth estimation and related methods. Prompt-based monocular depth estimation designates a family of monocular depth methods in which depth prediction is conditioned by an explicit prompt rather than being treated as a purely image-centric mapping. In this formulation, the prompt can specify a target region, a semantic concept, sparse metric measurements, global scene context, or latent domain information, and the estimator uses that conditioning to bias depth inference toward a particular geometric objective. Across recent work, the term encompasses region-aware prompting for target-centric refinement, language prompting for semantic and layout priors, depth-map prompting for sensor-agnostic absolute-scale prediction, visual prompt learning for adverse-domain robustness, and lightweight prompt-guided adaptation of vision-language backbones (Du et al., 12 May 2026, Zhang et al., 18 Mar 2025, Park et al., 2024, Xia et al., 2024, Wang et al., 23 Jan 2025, Manghotay et al., 1 Apr 2026).

1. Conceptual scope and task formulations

Prompt-based monocular depth estimation arises from the observation that standard monocular depth models are commonly optimized with uniform pixel-wise objectives over the full image, even though many downstream tasks are spatially or semantically asymmetric. One line of work therefore reframes depth prediction as an interactive, target-aware process. In Focusable Monocular Depth Estimation (FDE), given an RGB image $I \in R^{H\times W\times 3}$ and an interactive prompt $p$ in the form of a box or text prompt, the model outputs both a dense relative depth map $\hat{D} \in R^{H\times W}$ and an auxiliary target-foreground mask $\hat{M} \in [0,1]^{H\times W}$, with the forward formulation

$\hat{D}, \hat{M} = \mathcal{F}(I, p).$

The objective is explicitly spatially heterogeneous: improve depth accuracy inside the specified foreground, preserve sharp depth transitions around the target boundary, and maintain coherent global scene geometry (Du et al., 12 May 2026).

A second formulation uses language prompts as semantic priors over plausible 3D layouts. In the embodied language-and-depth prior model for road environments, textual descriptions are generated online from the current image and combined with embodied scene depth derived from camera intrinsics and known mounting height. These prompts encode scene captions, object categories, relative distance ordering, approximate distances, and environmental layout, and are mapped into a latent distribution over plausible scene geometry by a text VAE (Zhang et al., 18 Mar 2025). In LLM-MDE, prompt-based depth additionally refers to two mechanisms: automatically generated language prompts and learned-soft or continuous prompt tokens obtained by reprogramming visual features into a pretrained LLM’s embedding space (Xia et al., 2024).

A third formulation broadens the notion of prompt beyond language. “Depth Prompting for Sensor-Agnostic Depth Estimation” treats a sparse metric depth map as a continuous prompt. Rather than mixing image and sparse depth early into a joint representation, it encodes sparse sensor depth into a dedicated prompt embedding and uses that conditioning to predict adaptive spatial propagation affinities. This prompt is not a token sequence but a structured feature map encoding density, pattern, and positional characteristics of the measurements (Park et al., 2024).

A fourth formulation employs latent visual prompts that are learnable tensors injected into the decoder. PromptMono introduces visual prompt learning into a self-supervised monocular depth pipeline so that a single model can estimate depth across disparate visual domains such as day, night, rain, night-rain, and motion blur. These prompts are purely visual parameters rather than textual cues, and they are used to encode domain-specific priors through gated cross prompting attention (Wang et al., 23 Jan 2025).

A fifth formulation uses prompt-guided adaptation of vision-LLMs. MoA-DepthCLIP encodes a small set of indoor scene phrases using the frozen CLIP text encoder, averages them into a single global semantic context vector, and combines that fixed textual prior with lightweight Mixture-of-Adapters inside the visual backbone. In this setting, “prompt-guided” does not mean per-pixel text matching, but rather a global semantic anchor plus parameter-efficient visual adaptation (Manghotay et al., 1 Apr 2026).

These formulations suggest that prompt-based monocular depth estimation is best understood as a conditioning paradigm rather than a single architecture class. The prompt can be spatial, semantic, metric, or latent; the common feature is that it changes the estimator’s objective or feature routing relative to conventional uniform prediction.

2. Prompt modalities and conditioning semantics

The most explicit prompt taxonomy in this area distinguishes box prompts and text prompts. In FocusDepth, box prompts specify a tight bounding box of the target region, such as an object instance, while text prompts specify a semantic concept or class name such as “mug” or “drawer.” Segment Anything Model 3 provides a prompt encoder that transforms these prompts into conditioning signals for the image encoder and a prompt-guided transformer, producing prompt-conditioned token features aligned to the image. Because both the prompt branch and the geometry branch operate on the same ViT patch grid at input resolution 1008 with patch size 14, dense prompt cues can be spatially aligned and injected without disrupting global geometry (Du et al., 12 May 2026).

Language-conditioned systems use prompts with broader semantic content. In the embodied language-and-depth prior model, prompts are generated online every frame. ExpansionNet-v2 produces scene captions, and object-specific depth descriptions are instantiated as

$$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$

These descriptions are merged with scene captions into a combined prompt encoding category, spatial context, and depth-related cues. A CLIP text encoder with ResNet-50 backbone extracts 1024-dimensional text features, and an MLP maps them to the mean and standard deviation of a latent distribution over probable scene layouts (Zhang et al., 18 Mar 2025).

LLM-MDE uses prompts at two levels. The first is a natural-language prompt produced by adaptive prompt estimation from four perspectives: Dataset, Task, Pixel statistics, and Class category. An example string is “Dataset: NYU Depth v2. Task: estimate per-pixel depth from a single image. Pixel stats: min=14, median=102, max=240. Global depth class: extremely close.” The second is a set of prompt tokens obtained by cross-modal reprogramming, where visual patch embeddings attend to a small set of text prototypes such as “giant,” “extremely close,” “close,” “not in distance,” “a little remote,” “far,” and “unseen” (Xia et al., 2024).

Depth Prompting replaces language with sparse sensor depth as the prompt modality. The prompt encoder

$F^d, F^d_k = f_E(D_S)$

maps the sparse depth map $D_S$ to a prompt embedding and multi-scale prompt features. This design is expressly sensor-agnostic because it does not require explicit sensor metadata; it treats the sparse depth map itself as the prompt carrying the sensor distribution (Park et al., 2024).

PromptMono’s prompt modality is a learnable 4D tensor

$P \in \mathbb{R}^{C\times N\times H\times W},$

that is, a set of $N$ spatial prompt maps with $p$0 channels aligned to the deepest decoder feature resolution. These prompts are randomly initialized and optimized end-to-end. They are not condition labels and do not use text or language supervision; instead, they are content-dependent modifiers that encode domain-specific knowledge through interaction with image features (Wang et al., 23 Jan 2025).

MoA-DepthCLIP uses a small set of indoor scene phrases such as “a photo of a kitchen” and “a photo of a classroom.” Their CLIP text embeddings are L2-normalized and averaged into a global semantic context vector $p$1, which is then broadcast spatially and concatenated channel-wise to the adapted visual feature map. The textual prompt is therefore global and fixed throughout training rather than dynamically generated per image (Manghotay et al., 1 Apr 2026).

These designs show that “prompt” in monocular depth estimation is semantically overloaded. It may refer to user interaction, scene descriptions, sparse metric support, latent decoder parameters, or fixed text-derived context vectors. A plausible implication is that the field’s unifying concern is controllable conditioning, while the concrete prompt semantics depend on which ambiguity—regional focus, metric scale, domain shift, or semantic layout—is being targeted.

3. Architectural mechanisms for prompt injection

The architectural diversity of prompt-based monocular depth estimation is substantial, but several recurrent patterns appear: modality disentanglement, feature-space alignment, parameter-efficient adaptation, and controlled fusion.

FocusDepth is built around a geometry branch and a prompt branch. The geometry branch uses the Depth Anything family, specifically DA2 and DA3, as the dense geometry backbone. The prompt branch uses Segment Anything Model 3 for prompt-grounded spatial selectivity. The core Multi-Scale Spatial-Aligned Fusion (MSSA) module aligns prompt-conditioned tokens and depth tokens on the shared ViT patch grid. For geometry scale $p$2, the DA encoder produces $p$3, and the SAM3 branch produces $p$4. MSSA projects prompt features into the geometry feature space,

$p$5

concatenates them token-wise,

$p$6

then applies a per-scale routed MoE,

$p$7

with $p$8 experts, followed by learnable gated injection:

$p$9

The gating explicitly preserves pretrained geometry by selectively injecting prompt-conditioned corrections (Du et al., 12 May 2026).

Depth Prompting adopts a fundamentally different injection mechanism. A pretrained monocular depth foundation model $\hat{D} \in R^{H\times W}$0 produces a dense relative-scale depth prediction $\hat{D} \in R^{H\times W}$1 and multi-scale image features $\hat{D} \in R^{H\times W}$2. Sparse depth is encoded by $\hat{D} \in R^{H\times W}$3 into prompt features $\hat{D} \in R^{H\times W}$4. A least-squares global scaling first aligns the relative prediction to metric sparse measurements,

$\hat{D} \in R^{H\times W}$5

The adaptive affinity decoder then fuses prompt and image features by concatenation,

$\hat{D} \in R^{H\times W}$6

with $\hat{D} \in R^{H\times W}$7, and uses the predicted affinity in spatial propagation:

$\hat{D} \in R^{H\times W}$8

Prompt injection therefore affects the final depth through affinity prediction rather than token-level cross-attention (Park et al., 2024).

The embodied language-and-depth prior model uses three encoders—RGB, embodied scene depth, and text—and fuses the RGB and depth streams by cross-attention. Queries from one modality attend to keys and values from the other:

$\hat{D} \in R^{H\times W}$9

A conditional sampler then predicts a spatially varying noise tensor $\hat{M} \in [0,1]^{H\times W}$0 and uses it in reparameterization with the text-derived latent distribution,

$\hat{M} \in [0,1]^{H\times W}$1

where the depth decoder is shared between text-only and image-conditioned routes (Zhang et al., 18 Mar 2025).

LLM-MDE places the prompt mechanism inside a pretrained BERT. A ViT-base encoder produces patch embeddings $\hat{M} \in [0,1]^{H\times W}$2. Cross-modal reprogramming aligns these with a reduced prototype bank derived from the LLM embedding table:

$\hat{M} \in [0,1]^{H\times W}$3

The resulting reprogrammed visual tokens $\hat{M} \in [0,1]^{H\times W}$4 are concatenated with prompt tokens $\hat{M} \in [0,1]^{H\times W}$5 from adaptive prompt estimation,

$\hat{M} \in [0,1]^{H\times W}$6

and passed through the 12-layer BERT before a lightweight Adaptation Head with three UpsampleBN blocks outputs the depth map (Xia et al., 2024).

PromptMono uses Gated Cross Prompting Attention (GCPA) to fuse image features with learnable prompts at the deepest upsampling stage. Prompts are first gated by current image content:

$\hat{M} \in [0,1]^{H\times W}$7

Queries come from image features, keys and values from gated prompts, and multi-head cross attention produces

$\hat{M} \in [0,1]^{H\times W}$8

followed by a GDFN block (Wang et al., 23 Jan 2025).

MoA-DepthCLIP inserts Mixture-of-Adapters into CLIP’s ViT-B/32 encoder at layers $\hat{M} \in [0,1]^{H\times W}$9. Each expert is a bottleneck MLP,

$\hat{D}, \hat{M} = \mathcal{F}(I, p).$0

with $\hat{D}, \hat{M} = \mathcal{F}(I, p).$1 and $\hat{D}, \hat{M} = \mathcal{F}(I, p).$2, and token-specific gating is given by

$\hat{D}, \hat{M} = \mathcal{F}(I, p).$3

where $\hat{D}, \hat{M} = \mathcal{F}(I, p).$4 is a 2-layer MLP $\hat{D}, \hat{M} = \mathcal{F}(I, p).$5 with $\hat{D}, \hat{M} = \mathcal{F}(I, p).$6 experts and $\hat{D}, \hat{M} = \mathcal{F}(I, p).$7. The adapted token is

$\hat{D}, \hat{M} = \mathcal{F}(I, p).$8

This token-level routing is combined with a hybrid classification–regression head and the fixed global semantic context vector (Manghotay et al., 1 Apr 2026).

Taken together, these systems illustrate several distinct prompt-injection paradigms: token-space alignment and gated correction, affinity modulation for propagation, latent sampling conditioned by text priors, decoder-side cross-attention with learnable prompt maps, and adapter-based backbone modulation. The field therefore does not converge on a single prompting primitive; rather, it explores where in the pipeline conditioning is most effective.

4. Objectives, supervision, and optimization

Prompt-based depth methods also differ in what the prompt is meant to optimize. Some pursue regional prioritization, some metric grounding, some semantic plausibility, and some robustness across domains.

FocusDepth defines an explicitly region-aware objective. Predicted relative depth is first aligned to ground truth per image by scale $\hat{D}, \hat{M} = \mathcal{F}(I, p).$9 and shift $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$0,

$$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$1

Let $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$2 be ground-truth depth, $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$3 the target-foreground mask, $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$4 the valid-depth mask, and $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$5 the valid foreground, boundary, and global regions. Region-wise supervision is defined through $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$6 combining masked MSE and masked gradient losses within each region, and the auxiliary segmentation loss is

$$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$7

The final objective uses equal weights with per-region normalization:

$$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$8

An optimization ablation shows that using only a global loss improves global AbsRel but substantially worsens foreground AbsRel by $$T_i = \text{“This object seems to be } d_i \text{ and ranks as the } r_i\text{-th farthest in distance.”}$$9 for box prompts and $F^d, F^d_k = f_E(D_S)$0 for text prompts, quantifying the local–global trade-off (Du et al., 12 May 2026).

The embodied language-and-depth prior model uses supervised depth learning with a text VAE and conditional sampler. The text latent follows reparameterization

$F^d, F^d_k = f_E(D_S)$1

and KL regularization

$F^d, F^d_k = f_E(D_S)$2

Depth regression uses the scale-invariant log loss

$F^d, F^d_k = f_E(D_S)$3

Training alternates between text-VAE optimization and conditional-sampler optimization following Wordepth, with no photometric self-supervision or explicit cross-modal contrastive losses (Zhang et al., 18 Mar 2025).

Depth Prompting combines a scale-invariant loss on the backbone output with a final dense depth loss after propagation:

$F^d, F^d_k = f_E(D_S)$4

with $F^d, F^d_k = f_E(D_S)$5, and

$F^d, F^d_k = f_E(D_S)$6

The total loss is

$F^d, F^d_k = f_E(D_S)$7

with $F^d, F^d_k = f_E(D_S)$8 (Park et al., 2024).

LLM-MDE employs the scale-invariant squared loss

$F^d, F^d_k = f_E(D_S)$9

and does not introduce an explicit cross-modal contrastive or prototype-matching loss beyond this depth regression objective (Xia et al., 2024).

PromptMono remains within the self-supervised monocular depth tradition. It uses differentiable view synthesis and photometric loss,

$D_S$0

with

$D_S$1

edge-aware smoothness

$D_S$2

and self-distillation between easy and hard branches,

$D_S$3

The full multi-scale objective is

$D_S$4

with $D_S$5 (Wang et al., 23 Jan 2025).

MoA-DepthCLIP uses a composite supervised objective:

$D_S$6

with $D_S$7, $D_S$8, and $D_S$9. The terms are per-pixel cross-entropy for fixed depth-bin classification, L1 regression loss on metric depth, and SILog with $P \in \mathbb{R}^{C\times N\times H\times W},$0 and $P \in \mathbb{R}^{C\times N\times H\times W},$1 (Manghotay et al., 1 Apr 2026).

Across these methods, prompt conditioning is never merely auxiliary metadata. It directly changes either the prediction target, the latent geometry prior, the propagation operator, or the optimization landscape itself.

5. Benchmarks, datasets, and reported empirical behavior

The empirical profile of prompt-based monocular depth estimation is heterogeneous because the methods target different regimes. Some are evaluated for target-centric refinement, some for road-scene metric depth, some for sensor-shift robustness, some for few-shot capability, and some for indoor CLIP adaptation.

FocusDepth introduces FDE-Bench, a target-centric monocular relative depth benchmark constructed from image–target–depth triplets across NYU v2, TUM RGB-D, YCB-Video, RLBench, and RoboTwin. The benchmark contains $P \in \mathbb{R}^{C\times N\times H\times W},$2K/$P \in \mathbb{R}^{C\times N\times H\times W},$3K train/val images and $P \in \mathbb{R}^{C\times N\times H\times W},$4K/$P \in \mathbb{R}^{C\times N\times H\times W},$5K train/val triplets spanning $P \in \mathbb{R}^{C\times N\times H\times W},$6 target categories, with box prompts for all five datasets and text prompts for NYU v2, RLBench, and YCB-Video. Evaluation is region-aware: valid foreground, boundary, and global masks are defined using target masks and a boundary ring of radius $P \in \mathbb{R}^{C\times N\times H\times W},$7 pixels, and $P \in \mathbb{R}^{C\times N\times H\times W},$8 and AbsRel are reported per region per target, aggregated by the median over targets (Du et al., 12 May 2026).

On RLBench with box prompts, FocusDepth(DA3) improves over DA3-ft from boundary AbsRel $P \in \mathbb{R}^{C\times N\times H\times W},$9 to $N$0, foreground AbsRel $N$1 to $N$2, and global AbsRel $N$3 to $N$4. On RoboTwin, FocusDepth(DA3) improves boundary AbsRel from $N$5 to $N$6, foreground from $N$7 to $N$8, and global from $N$9 to $p$00. On YCB-Video with text prompts, FocusDepth(DA3) achieves boundary AbsRel $p$01, foreground $p$02, and global $p$03, improving over DA3-ft values $p$04. Ablations show that disrupting spatial alignment by shuffling token correspondences increases text-prompt global AbsRel by $p$05 (Du et al., 12 May 2026).

The embodied language-and-depth prior model is evaluated on KITTI (Eigen split) and DDAD. On KITTI, it reports AbsRel $p$06, SqRel $p$07, RMSE $p$08, RMSE log $p$09, and $p$10 of $p$11, outperforming selected reported methods including Depth Anything, Metric3D v2, UniDepth, Wordepth, and AFNet on that split. On DDAD, it reports AbsRel $p$12, RMSE $p$13, and $p$14 of $p$15 (Zhang et al., 18 Mar 2025). The paper also studies the quality of the embodied prior itself on KITTI, reporting that Embodied Road Depth achieves $p$16 error for $p$17 of pixels and $p$18 for $p$19.

Depth Prompting evaluates sensor-agnostic robustness on NYUv2 and KITTI Depth Completion. On NYUv2 random-sample sparsity tests, when trained on 500 samples and tested at 100 samples, it reports RMSE $p$20, MAE $p$21, and $p$22 $p$23, compared with NLSPN RMSE $p$24. At 1 sample, it reports RMSE $p$25, compared with $p$26 for NLSPN and $p$27 for CompletionFormer. On KITTI Depth Completion, when trained at 64 LiDAR lines and tested at 1 line, it reports RMSE $p$28, compared with $p$29 for NLSPN and $p$30 for CompletionFormer (Park et al., 2024). The method also reports improved performance under pattern bias, range bias, and few-shot cross-domain transfer.

LLM-MDE is evaluated on the NYU raw dataset. Its few-shot results span 1-shot through 4-shot and a 28-image few-shot group. For 1-shot it reports RMSE $p$31, AbsRel $p$32, and $p$33 $p$34; for 4-shot RMSE $p$35, AbsRel $p$36, and $p$37 $p$38. An ablation comparing adaptive prompt estimation, fixed prompts, and no prompts shows RMSE $p$39 for APG, $p$40 for fixed prompts, and $p$41 for no prompts, indicating that adaptive prompt construction materially improves the model’s performance (Xia et al., 2024).

PromptMono targets challenging outdoor conditions. On Oxford RobotCar with depth cap 40 m and ResNet-18, it reports day AbsRel $p$42 and night AbsRel $p$43, surpassing ADDS values of $p$44 and $p$45 respectively. With depth cap 60 m, day AbsRel is $p$46 and night AbsRel $p$47. In the higher-resolution 576×320, ResNet-50 setting, it reports night AbsRel $p$48, SqRel $p$49, and RMSE $p$50. On nuScenes under fully self-supervised monocular training, PromptMono reports night AbsRel $p$51 versus Monodepth2’s $p$52, and day-rain AbsRel $p$53 versus $p$54 (Wang et al., 23 Jan 2025).

MoA-DepthCLIP evaluates on NYU Depth V2. Relative to the reproduced DepthCLIP baseline with $p$55 and RMSE $p$56, the final system with ViT-B/32, MoA, composite loss, and $p$57 bins reports $p$58 and RMSE $p$59. The progression through ablations shows the effect of the composite loss, MoA insertion, and bin count selection. For example, the ViT-B/32 baseline without composite loss or MoA reports $p$60 and RMSE $p$61, while the addition of composite loss reduces RMSE to $p$62 (Manghotay et al., 1 Apr 2026).

The following table summarizes the empirical targets of the principal methods.

Method	Prompt form	Evaluation emphasis
FocusDepth (Du et al., 12 May 2026)	Box/text prompts	Target foreground, boundary, and global relative depth
Vision-Language Embodiment (Zhang et al., 18 Mar 2025)	Text descriptions + embodied scene depth	Metric road-scene depth with camera priors
Depth Prompting (Park et al., 2024)	Sparse depth prompt	Sensor-agnostic absolute-scale depth under density, pattern, and range shifts
LLM-MDE (Xia et al., 2024)	APG language prompts + prompt tokens	Few-shot and zero-shot language-informed depth
PromptMono (Wang et al., 23 Jan 2025)	Learnable visual prompts	Self-supervised robustness in challenging environments
MoA-DepthCLIP (Manghotay et al., 1 Apr 2026)	Global text-derived context + visual adapters	Parameter-efficient CLIP adaptation for indoor depth

A common misconception is that prompt-based depth estimation is primarily a few-shot or language-only phenomenon. The literature does not support that narrow interpretation. Prompting is used for target-centric supervision, metric alignment, sensor-agnostic propagation, adverse-condition robustness, and parameter-efficient VLM transfer, often without any generative LLM in the loop.

6. Relations to foundation models, vision-language systems, and parameter-efficient adaptation

Prompt-based monocular depth estimation is closely tied to recent foundation-model practice. Several systems rely on pretrained backbones with limited adaptation rather than end-to-end retraining from scratch.

FocusDepth explicitly builds on the Depth Anything family and Segment Anything Model 3. Its design premise is that depth backbones such as DA2 and DA3 already encode robust geometry priors, while SAM3 provides prompt-grounded spatial selectivity. MSSA is constructed to preserve the geometry prior rather than overwrite it, and the reported ablations show that removing gated injection harms both foreground and global AbsRel (Du et al., 12 May 2026). This positioning treats prompting as controlled refinement of a foundation geometry model.

Depth Prompting similarly assumes that large monocular depth foundation models trained on diverse web-scale imagery capture useful geometry priors in relative scale. It uses a frozen backbone with bias tuning, updating only convolutional bias terms, approximately $p$63M of $p$64M parameters, or $p$65 of parameters. The sparse depth prompt then steers the backbone’s relative prediction toward sensor-agnostic, absolute-scale output without retraining the backbone (Park et al., 2024).

LLM-MDE is anchored in pretrained ViT and BERT components and emphasizes efficient adaptation via LoRA. Large parts of both backbones remain frozen, and only LoRA adapters plus the adaptation head are primarily trained. The best reported scheme in the ablation uses ViT Alpha $p$66, Rank $p$67, and LLM Rank $p$68 (Xia et al., 2024). Here the prompt is inseparable from the hypothesis that linguistic priors can support depth reasoning with minimal supervision.

MoA-DepthCLIP is explicitly a parameter-efficient transfer method for CLIP. Most of the CLIP backbone and the entire text encoder remain frozen; trainable components are sparse MoA modules, the final four transformer blocks, and the prediction heads. The method emphasizes that exact parameter counts are not reported, but the trainable parameters are a small fraction of the full backbone because of sparse insertion, bottleneck experts, and selective fine-tuning (Manghotay et al., 1 Apr 2026).

PromptMono differs from the frozen-backbone paradigm. It does not treat prompting as a PEFT mechanism in the usual VLM sense. Instead, it uses a unified self-supervised depth model with learned visual prompts and a weight-shared siamese architecture during training. Its prompt design is closer to latent feature conditioning in restoration or domain-robust vision than to text prompting (Wang et al., 23 Jan 2025).

The embodied language-and-depth prior model draws on CLIP for text encoding but is conceptually distinct from VLM adaptation methods. Its primary novelty lies in coupling language priors with a physically embodied camera model. A dense metric prior derived from camera intrinsics and mounting height constrains global scale, while textual priors encode scene semantics and object ordering (Zhang et al., 18 Mar 2025).

These works collectively indicate that prompt-based monocular depth estimation is one of the sites where PEFT, VLM transfer, and depth-specific geometry priors intersect. This suggests that prompting in depth estimation is often less about natural-language interaction per se than about how to attach controllable priors to pretrained geometric representations.

7. Limitations, controversies, and emerging directions

The limitations reported across these papers reveal that prompt-based conditioning does not eliminate the classical ambiguities of monocular depth; it redistributes them.

In FocusDepth, prompt ambiguity is a direct source of error. Vague or incorrect text prompts reduce local gains, and poorly aligned box prompts may underperform when the box does not match the true object mask. Extremely fine structures, strong occlusions at boundaries, and transparency remain challenging. The authors also note that aggressive local optimization can risk global consistency, even though MSSA gating mitigates the trade-off (Du et al., 12 May 2026).

The embodied language-and-depth prior model relies on a ground-plane assumption and accurate camera calibration. Sidewalks, rail tracks, and uneven terrain degrade the embodied prior, and miscalibration of intrinsics or mounting height biases the metric prior. Noisy prompt generation from captioning can also misguide the text prior, while domain shift in night or weather conditions may affect captioning and segmentation (Zhang et al., 18 Mar 2025). This makes clear that prompt quality is only one component; physically grounded priors can fail if their assumptions are violated.

Depth Prompting remains dependent on the foundation model’s relative prediction when sparse measurements are extremely scarce. Since the least-squares scale alignment is global, localized systematic depth errors in the sparse input may not be corrected. The paper also notes that no explicit use is made of sensor or domain metadata, even when such metadata might further improve conditioning (Park et al., 2024).

LLM-MDE identifies coarse grounding, ambiguity in mapping continuous geometry to discrete language descriptors, dependence on prompt quality, and indoor bias as limitations. The work explicitly states that it is not competitive with top supervised MDE systems on absolute accuracy and that its emphasis is flexibility and low supervision rather than state-of-the-art performance (Xia et al., 2024). This directly challenges any assumption that prompt-based language methods currently dominate conventional supervised depth estimation on standard accuracy metrics.

PromptMono relies on CycleGAN-generated easy-hard pairs, so translation artifacts can affect performance. Only one GCPA placement, at the deepest decoder stage, is studied. Explicit handling of dynamic objects or heavy adverse weather such as dense fog is not incorporated beyond the Monodepth2 per-pixel minimum strategy (Wang et al., 23 Jan 2025).

MoA-DepthCLIP is limited by its fixed “indoor scene” global context, unspecified bin spacing, and the coarse detail implied by ViT-B/32 patching and limited adapter capacity. The authors point to extension to outdoor datasets and dynamic prompt selection as future directions (Manghotay et al., 1 Apr 2026).

Several future directions recur across the literature. FocusDepth proposes richer prompt types such as points and polygons, larger-scale joint training, and integration into downstream robotic manipulation pipelines (Du et al., 12 May 2026). The embodied language-and-depth prior model suggests continued use of real-time prompt regeneration and relaxing the plane assumption for non-road domains (Zhang et al., 18 Mar 2025). LLM-MDE points toward richer prototype banks, better prompt synthesis, hybrid supervision, and multi-view or video depth with temporal prompts (Xia et al., 2024). PromptMono suggests multi-modal prompts, language-conditioned prompts via CLIP or LLMs, and test-time prompt adaptation (Wang et al., 23 Jan 2025). MoA-DepthCLIP identifies image-aware semantic contexts rather than a fixed average vector as a next step (Manghotay et al., 1 Apr 2026).

A broader interpretive point follows from these limitations. Prompt-based monocular depth estimation does not constitute a single settled methodology. It is a research direction organized around conditional priors: user-specified regions, textual scene models, sparse metric anchors, latent domain codes, and fixed semantic context vectors. The central technical question is how to introduce such priors without degrading geometric fidelity. The most mature recent answer is not universal prompting, but controlled prompt injection—via alignment, gating, propagation, or selective adaptation—so that conditioning improves the relevant ambiguity without destabilizing the remainder of the scene representation.