PureCLIP-Depth: Prompt-Free Monocular Depth Estimation

• PureCLIP-Depth is a monocular depth estimation model that maps RGB images to depth using CLIP's final-layer embeddings without prompts or decoders.
• It employs dual MLPs and a learnable depth table with contrastive, alignment, and RMSE losses to achieve superior performance on NYU and KITTI datasets.
• The approach demonstrates that rich, pretrained image–text representations can directly support metric depth prediction through embedding-space alignment.

PureCLIP-Depth is a monocular depth estimation model that is explicitly formulated as prompt-free and decoder-free, with the stated objective of performing monocular depth estimation “entirely within the Contrastive Language-Image Pre-training (CLIP) embedding space” (Miya et al., 17 Mar 2026). Introduced in “PureCLIP-Depth: Prompt-Free and Decoder-Free Monocular Depth Estimation within CLIP Embedding Space” (Miya et al., 17 Mar 2026), it learns a direct mapping from RGB imagery to depth representations using final-layer CLIP visual embeddings, two small multilayer perceptrons, and a learnable depth table, rather than textual prompting or a conventional convolutional or transformer decoder. The method is positioned as a test of whether CLIP’s conceptual embedding space can support metric depth prediction without the architectural machinery that dominates geometry-focused depth networks.

1. Problem formulation and conceptual scope

Monocular depth estimation takes a single RGB image $I \in \mathbb{R}^{H\times W\times 3}$ and predicts a per-pixel depth map $D \in \mathbb{R}^{H\times W}$ (Miya et al., 17 Mar 2026). In the framing adopted by PureCLIP-Depth, the task is challenging not only because monocular depth is ill-posed, but also because most CLIP-based depth methods either query the text encoder with prompts such as “near” and “far” or attach a dense decoder to CLIP visual features. PureCLIP-Depth removes both mechanisms and instead treats depth as a structure that can be read out from the conceptual geometry of CLIP’s joint embedding space (Miya et al., 17 Mar 2026).

The paper motivates this design through the claim that CLIP’s large-scale image–text pretraining embeds rich world knowledge and object semantics, and that “there exists a learnable mapping from RGB to depth entirely within CLIP’s conceptual embedding space” (Miya et al., 17 Mar 2026). In this formulation, depth is not queried through language at inference time and is not reconstructed through a UNet-like or DPT-like decoding hierarchy. Rather, patch embeddings are rotated into a depth-oriented subspace and compared against learnable depth embeddings.

A recurrent source of confusion is the meaning of “prompt-free.” In PureCLIP-Depth, prompt-free does not mean that text-derived information never appears. The depth table is initialized from CLIP text embeddings of phrases such as “1 meter”, “2 meters”, and so forth, but after initialization no text is fed into CLIP during training or inference (Miya et al., 17 Mar 2026). Similarly, “decoder-free” does not mean parameter-free. The model still learns two MLPs and a depth table, but it does not employ a conventional convolutional or transformer decoder (Miya et al., 17 Mar 2026).

2. Embedding-space architecture

PureCLIP-Depth uses CLIP ViT-L/14@336px as its visual backbone (Miya et al., 17 Mar 2026). With a $14\times14$ patch size and $336\times336$ input resolution, the image is decomposed into $24\times24$ patches, yielding $N_{\text{patch}}=576$ patch tokens plus a CLS token (Miya et al., 17 Mar 2026). The method relies on the final CLIP ViT visual embeddings, not intermediate multiscale features.

Let

$$\mathbf{z} = f(I) \in \mathbb{R}^{B \times (N_{\text{patch}}+1) \times D},$$

where $f$ is the frozen CLIP image encoder (Miya et al., 17 Mar 2026). For each patch token $\mathbf{z}_{\text{patch},i}$, the model applies a two-layer RGB adapter MLP $\phi$: $D \in \mathbb{R}^{H\times W}$0 The CLS token $D \in \mathbb{R}^{H\times W}$1 is then concatenated with $D \in \mathbb{R}^{H\times W}$2 and fused by a second two-layer MLP $D \in \mathbb{R}^{H\times W}$3: $D \in \mathbb{R}^{H\times W}$4 Both $D \in \mathbb{R}^{H\times W}$5 and $D \in \mathbb{R}^{H\times W}$6 use LayerNorm, Linear, GELU, Dropout, and Linear layers (Miya et al., 17 Mar 2026).

The depth representation is a learnable table

$D \in \mathbb{R}^{H\times W}$7

with $D \in \mathbb{R}^{H\times W}$8 on both NYU and KITTI (Miya et al., 17 Mar 2026). On NYU the bins cover 0–10 m, and on KITTI they cover 0–30 m (Miya et al., 17 Mar 2026). These vectors are initialized from CLIP text embeddings of phrases such as “1 meter”, “2 meters”, and so on, with the KITTI range extended to 30 m (Miya et al., 17 Mar 2026).

Prediction is performed by similarity in embedding space. For patch $D \in \mathbb{R}^{H\times W}$9 and depth embedding $14\times14$0,

$14\times14$1

with temperature $14\times14$2 (Miya et al., 17 Mar 2026). After softmax,

$14\times14$3

the patch depth is obtained as a soft expectation over bin centers $14\times14$4: $14\times14$5 The resulting $14\times14$6 is assigned to all pixels in the corresponding patch region (Miya et al., 17 Mar 2026).

3. Training objectives and optimization scheme

PureCLIP-Depth combines three losses: an InfoNCE contrastive loss, an alignment loss, and an RMSE loss in metric depth space (Miya et al., 17 Mar 2026). Ground-truth patch depth is defined as the average of valid pixels in each patch region $14\times14$7: $14\times14$8 where $14\times14$9 excludes pixels outside the valid depth range and NaN pixels (Miya et al., 17 Mar 2026).

The InfoNCE term is

$336\times336$0

where $336\times336$1 is the target depth-bin index and $336\times336$2 is the valid-patch mask (Miya et al., 17 Mar 2026). The alignment term is

$336\times336$3

The metric supervision is

$336\times336$4

A distinctive aspect of the method is its alternating optimization. In the embedding phase, the model updates $336\times336$5, $336\times336$6, and $336\times336$7 using

$336\times336$8

with $336\times336$9 (Miya et al., 17 Mar 2026). In the depth phase, the depth table $24\times24$0 is frozen and only $24\times24$1 and $24\times24$2 are updated using $24\times24$3 (Miya et al., 17 Mar 2026). The two phases alternate every 100 training steps (Miya et al., 17 Mar 2026).

Training uses AdamW. In the embedding phase, the learning rate is $24\times24$4 with weight decay $24\times24$5; in the depth phase, the learning rate is $24\times24$6 with no weight decay (Miya et al., 17 Mar 2026). The batch size is 8 on both NYU and KITTI (Miya et al., 17 Mar 2026). The paper reports a constant learning rate until 20 epochs without validation improvement, followed by linear decay to one-tenth by 50 epochs without improvement, with early stopping patience 50 (Miya et al., 17 Mar 2026).

4. Datasets, evaluation protocol, and empirical results

On NYU Depth V2, PureCLIP-Depth resizes images to $24\times24$7, uses the Eigen crop, and evaluates over the valid depth range 0–10 m with 15 depth bins (Miya et al., 17 Mar 2026). On KITTI, images and depth maps are resized to $24\times24$8 and tiled horizontally into four segments; the protocol uses the Garg mask, the valid range 0–30 m, and 15 bins (Miya et al., 17 Mar 2026). KITTI inference additionally uses horizontal-flip test-time augmentation (Miya et al., 17 Mar 2026).

The paper evaluates using AbsRel, RMSE, log10, and threshold accuracies $24\times24$9, $N_{\text{patch}}=576$0, and $N_{\text{patch}}=576$1 (Miya et al., 17 Mar 2026).

Dataset	Method	Selected metrics
NYU Depth V2	DepthCLIP	AbsRel 0.388, RMSE 1.167, $N_{\text{patch}}=576$2 0.394
NYU Depth V2	Hu et al.	AbsRel 0.347, RMSE 1.049, $N_{\text{patch}}=576$3 0.428
NYU Depth V2	Auty & Mikolajczyk	AbsRel 0.319, RMSE 0.970, $N_{\text{patch}}=576$4 0.465
NYU Depth V2	PureCLIP-Depth	AbsRel 0.201, RMSE 0.670, $N_{\text{patch}}=576$5 0.671
KITTI	DepthCLIP	AbsRel 0.473, RMSE 12.958, $N_{\text{patch}}=576$6 0.281
KITTI	Hu et al.	AbsRel 0.384, RMSE 12.290, $N_{\text{patch}}=576$7 0.312
KITTI	Auty & Mikolajczyk	AbsRel 0.238, RMSE 5.756, $N_{\text{patch}}=576$8 0.652
KITTI	PureCLIP-Depth	AbsRel 0.172, RMSE 1.062, $N_{\text{patch}}=576$9 0.739

These results are the basis for the paper’s claim of state-of-the-art performance among CLIP embedding-based models on both indoor and outdoor datasets (Miya et al., 17 Mar 2026). The gains are especially large relative to prompt-based CLIP baselines.

The ablation study isolates three factors. First, the two MLPs are essential: without $$\mathbf{z} = f(I) \in \mathbb{R}^{B \times (N_{\text{patch}}+1) \times D},$$0 and $$\mathbf{z} = f(I) \in \mathbb{R}^{B \times (N_{\text{patch}}+1) \times D},$$1, NYU AbsRel rises from 0.201 to 0.320, and KITTI AbsRel rises from 0.172 to 0.371 (Miya et al., 17 Mar 2026). Second, the CLS token provides a smaller but consistent gain: removing it changes NYU AbsRel from 0.201 to 0.215, and KITTI AbsRel from 0.172 to 0.182 (Miya et al., 17 Mar 2026). Third, the full loss combination is strongest: InfoNCE alone is worse than InfoNCE + alignment, which is worse than InfoNCE + alignment + RMSE (Miya et al., 17 Mar 2026).

The paper also reports qualitative improvements in object boundaries, fine structure, and small objects, and notes that uncertainty increases at far depths, producing a tendency toward underestimation in the far range (Miya et al., 17 Mar 2026).

5. Relation to earlier CLIP-based depth paradigms

PureCLIP-Depth belongs to a rapidly developing family of CLIP-based depth methods, but it occupies a specific architectural position. DepthCLIP reframed depth as semantic distance classification using handcrafted prompts such as “This object is [distance class]”, a CLIP ResNet-50 visual encoder, and fixed depth bins, all in a training-free pipeline (Zhang et al., 2022). Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation introduced learnable prompts and a scene-adaptive depth codebook, while keeping CLIP frozen and training only a small number of parameters in a one-shot-per-scene regime (Hu et al., 2023). CLIP2Depth kept CLIP’s image and text encoders frozen but added a mirror—a static non-human prompt for the text encoder—and a CLIPSeg-style decoder with FiLM conditioning (Kim et al., 2024). SPACE-CLIP removed the text encoder at inference time and used a dual-pathway decoder over a frozen CLIP ViT-B/16 vision encoder (Cho et al., 25 Jan 2026). MoA-DepthCLIP inserted Mixture-of-Adapters into ViT-B/32, fine-tuned the last four transformer blocks, and combined classification, regression, and composite geometric loss terms (Manghotay et al., 1 Apr 2026).

Against that background, PureCLIP-Depth differs in three ways. First, it uses no prompts at training or inference in the operational sense; second, it uses no decoder; third, it works with the final CLIP visual embeddings rather than intermediate spatial feature hierarchies (Miya et al., 17 Mar 2026). The model therefore strips CLIP-based depth estimation down to the question of whether metric depth can be recovered by embedding-space alignment alone.

A plausible implication is that PureCLIP-Depth tests a narrower hypothesis than methods such as SPACE-CLIP or MoA-DepthCLIP. Those methods examine whether CLIP can become a strong depth backbone when supplemented with decoders, adapters, or geometric losses, whereas PureCLIP-Depth examines whether the conceptual structure of CLIP space is already sufficient once an appropriate mapping and depth table are learned. The reported benchmark gains over earlier CLIP embedding-based methods suggest that this hypothesis is nontrivial rather than merely minimalist (Miya et al., 17 Mar 2026).

6. Limitations, interpretation, and significance

The paper identifies three main limitations. The first is far-range underestimation: empirical uncertainty increases for far depths, and the prediction distribution becomes biased toward shallower values in sparsely represented depth regions (Miya et al., 17 Mar 2026). The second is sky-region instability, which is linked to missing or noisy LiDAR supervision and the fact that sky corresponds to effectively infinite depth (Miya et al., 17 Mar 2026). The third is the intrinsic constraint of patch resolution: depth is predicted per $$\mathbf{z} = f(I) \in \mathbb{R}^{B \times (N_{\text{patch}}+1) \times D},$$2 patch region, which limits spatial detail relative to pixel-level decoders (Miya et al., 17 Mar 2026).

These limitations clarify the meaning of the model’s “decoder-free” claim. PureCLIP-Depth does not attempt to recover high-resolution local geometry by multiscale reconstruction; its strength lies instead in aligning conceptual image representations with depth anchors. This suggests that its strongest use case is not necessarily maximal geometric precision, but rather the demonstration that a foundation model trained for image–text contrastive learning contains a depth-relevant structure that can be exposed through lightweight embedding-space learning.

The paper explicitly points to low-contrast and thermal infrared imagery as potential future applications (Miya et al., 17 Mar 2026). That suggestion is consistent with the method’s emphasis on semantic priors rather than classical texture-heavy depth cues. More broadly, PureCLIP-Depth supports a specific claim about multimodal foundation models: conceptual embedding spaces can support geometric prediction without prompts and without a decoder, provided that the mapping from RGB patch embeddings to depth embeddings is learned with appropriate contrastive and metric supervision (Miya et al., 17 Mar 2026).

In the literature of CLIP-based monocular depth estimation, PureCLIP-Depth therefore marks a distinct line of inquiry. It is less a replacement for geometry-aware depth architectures than a controlled demonstration that prompt engineering and dense decoders are not the only route by which CLIP can be made depth-sensitive.