---
title: 'PureCLIP-Depth: Prompt-Free Monocular Depth Estimation'
url: https://www.emergentmind.com/topics/pureclip-depth
type: topic
---

# PureCLIP-Depth: Prompt-Free Monocular Depth Estimation

PureCLIP-Depth is a monocular depth estimation model that is explicitly formulated as **prompt-free** and **decoder-free**, with the stated objective of performing monocular depth estimation “entirely within the Contrastive Language-Image Pre-training (CLIP) embedding space” [2603.16238]. Introduced in “PureCLIP-Depth: Prompt-Free and Decoder-Free Monocular Depth Estimation within CLIP Embedding Space” [2603.16238], it learns a direct mapping from RGB imagery to depth representations using final-layer CLIP visual embeddings, two small multilayer perceptrons, and a learnable depth table, rather than textual prompting or a conventional convolutional or transformer decoder. The method is positioned as a test of whether CLIP’s conceptual embedding space can support metric depth prediction without the architectural machinery that dominates geometry-focused depth networks.

## 1. Problem formulation and conceptual scope

Monocular depth estimation takes a single RGB image \(I \in \mathbb{R}^{H\times W\times 3}\) and predicts a per-pixel depth map \(D \in \mathbb{R}^{H\times W}\) [2603.16238]. In the framing adopted by PureCLIP-Depth, the task is challenging not only because monocular depth is ill-posed, but also because most CLIP-based depth methods either query the text encoder with prompts such as “near” and “far” or attach a dense decoder to CLIP visual features. PureCLIP-Depth removes both mechanisms and instead treats depth as a structure that can be read out from the conceptual geometry of CLIP’s joint embedding space [2603.16238].

The paper motivates this design through the claim that CLIP’s large-scale image–text pretraining embeds rich world knowledge and object semantics, and that “there exists a learnable mapping from RGB to depth entirely within CLIP’s conceptual embedding space” [2603.16238]. In this formulation, depth is not queried through language at inference time and is not reconstructed through a UNet-like or DPT-like decoding hierarchy. Rather, patch embeddings are rotated into a depth-oriented subspace and compared against learnable depth embeddings.

A recurrent source of confusion is the meaning of “prompt-free.” In PureCLIP-Depth, prompt-free does **not** mean that text-derived information never appears. The depth table is initialized from CLIP text embeddings of phrases such as “1 meter”, “2 meters”, and so forth, but after initialization no text is fed into CLIP during training or inference [2603.16238]. Similarly, “decoder-free” does **not** mean parameter-free. The model still learns two MLPs and a depth table, but it does not employ a conventional convolutional or transformer decoder [2603.16238].

## 2. Embedding-space architecture

PureCLIP-Depth uses **CLIP ViT-L/14@336px** as its visual backbone [2603.16238]. With a \(14\times14\) patch size and \(336\times336\) input resolution, the image is decomposed into \(24\times24\) patches, yielding \(N_{\text{patch}}=576\) patch tokens plus a CLS token [2603.16238]. The method relies on the **final CLIP ViT visual embeddings**, not intermediate multiscale features.

Let
\[
\mathbf{z} = f(I) \in \mathbb{R}^{B \times (N_{\text{patch}}+1) \times D},
\]
where \(f\) is the frozen CLIP image encoder [2603.16238]. For each patch token \(\mathbf{z}_{\text{patch},i}\), the model applies a two-layer RGB adapter MLP \(\phi\):
\[
\mathbf{z}'_i = \phi(\mathbf{z}_{\text{patch},i}) \in \mathbb{R}^D.
\]
The CLS token \(\mathbf{z}_{CLS}\) is then concatenated with \(\mathbf{z}'_i\) and fused by a second two-layer MLP \(\psi\):
\[
\tilde{\mathbf{z}}_i = \psi([\mathbf{z}'_i ; \mathbf{z}_{CLS}]) \in \mathbb{R}^D.
\]
Both \(\phi\) and \(\psi\) use LayerNorm, Linear, GELU, Dropout, and Linear layers [2603.16238].

The depth representation is a learnable table
\[
\mathcal{W} = \{\mathbf{w}_1,\dots,\mathbf{w}_K\}, \quad \mathbf{w}_j \in \mathbb{R}^D,
\]
with **\(K=15\)** on both NYU and KITTI [2603.16238]. On NYU the bins cover **0–10 m**, and on KITTI they cover **0–30 m** [2603.16238]. These vectors are initialized from CLIP text embeddings of phrases such as “1 meter”, “2 meters”, and so on, with the KITTI range extended to 30 m [2603.16238].

Prediction is performed by similarity in embedding space. For patch \(i\) and depth embedding \(j\),
\[
s_{ij} = \frac{\tilde{\mathbf{z}}_i^\top \mathbf{w}_j}{\tau},
\]
with temperature \(\tau = 0.07\) [2603.16238]. After softmax,
\[
p_{ij} = \frac{\exp(s_{ij})}{\sum_k \exp(s_{ik})},
\]
the patch depth is obtained as a soft expectation over bin centers \(c_j\):
\[
\hat{d}_i = \sum_{j=1}^K p_{ij} c_j.
\]
The resulting \(\hat{d}_i\) is assigned to all pixels in the corresponding patch region [2603.16238].

## 3. Training objectives and optimization scheme

PureCLIP-Depth combines three losses: an **InfoNCE contrastive loss**, an **alignment loss**, and an **RMSE loss** in metric depth space [2603.16238]. Ground-truth patch depth is defined as the average of valid pixels in each patch region \(\Omega_i\):
\[
d_i^{GT} = \frac{1}{|\mathcal{V}_i|}\sum_{p\in \mathcal{V}_i} D_{GT}(p),
\]
where \(\mathcal{V}_i\) excludes pixels outside the valid depth range and NaN pixels [2603.16238].

The InfoNCE term is
\[
\mathcal{L}_{\mathrm{InfoNCE}} = - \frac{1}{\sum_i m_i} \sum_{i=1}^{N_{\text{patch}}} m_i
\log \frac{\exp((\tilde{\mathbf{z}}_i \cdot \mathbf{w}_{y_i}) / \tau)}{\sum_{j=1}^K \exp((\tilde{\mathbf{z}}_i \cdot \mathbf{w}_j) / \tau)},
\]
where \(y_i\) is the target depth-bin index and \(m_i\) is the valid-patch mask [2603.16238]. The alignment term is
\[
\mathcal{L}_{align} = \frac{1}{\sum_i m_i} \sum_{i=1}^{N_{\text{patch}}} m_i \left[ 1 - \tilde{\mathbf{z}}_i^\top \mathbf{w}_{y_i} \right].
\]
The metric supervision is
\[
\mathcal{L}_{RMSE} = \sqrt{ \frac{1}{|\mathcal{V}|}\sum_{p\in \mathcal{V}} (\hat{D}_p - D_{GT,p})^2 }.
\]

A distinctive aspect of the method is its **alternating optimization**. In the **embedding phase**, the model updates \(\phi\), \(\psi\), and \(\mathcal{W}\) using
\[
\mathcal{L}_{emb} = \mathcal{L}_{align} + \lambda_{\text{InfoNCE}}\mathcal{L}_{\text{InfoNCE}},
\]
with \(\lambda_{\text{InfoNCE}}=1.0\) [2603.16238]. In the **depth phase**, the depth table \(\mathcal{W}\) is frozen and only \(\phi\) and \(\psi\) are updated using \(\mathcal{L}_{RMSE}\) [2603.16238]. The two phases alternate every **100 training steps** [2603.16238].

Training uses **AdamW**. In the embedding phase, the learning rate is **\(3\times 10^{-4}\)** with weight decay **\(10^{-2}\)**; in the depth phase, the learning rate is **\(1\times 10^{-3}\)** with no weight decay [2603.16238]. The batch size is **8** on both NYU and KITTI [2603.16238]. The paper reports a constant learning rate until **20 epochs without validation improvement**, followed by linear decay to one-tenth by **50 epochs without improvement**, with early stopping patience **50** [2603.16238].

## 4. Datasets, evaluation protocol, and empirical results

On **NYU Depth V2**, PureCLIP-Depth resizes images to **\(336\times336\)**, uses the **Eigen crop**, and evaluates over the valid depth range **0–10 m** with **15 depth bins** [2603.16238]. On **KITTI**, images and depth maps are resized to **\(336 \times (336\times4)\)** and tiled horizontally into four segments; the protocol uses the **Garg mask**, the valid range **0–30 m**, and **15 bins** [2603.16238]. KITTI inference additionally uses horizontal-flip test-time augmentation [2603.16238].

The paper evaluates using **AbsRel**, **RMSE**, **log10**, and threshold accuracies \(\delta<1.25\), \(\delta<1.25^2\), and \(\delta<1.25^3\) [2603.16238].

| Dataset | Method | Selected metrics |
|---|---|---|
| NYU Depth V2 | DepthCLIP | AbsRel 0.388, RMSE 1.167, \(\delta<1.25\) 0.394 |
| NYU Depth V2 | Hu et al. | AbsRel 0.347, RMSE 1.049, \(\delta<1.25\) 0.428 |
| NYU Depth V2 | Auty & Mikolajczyk | AbsRel 0.319, RMSE 0.970, \(\delta<1.25\) 0.465 |
| NYU Depth V2 | PureCLIP-Depth | AbsRel 0.201, RMSE 0.670, \(\delta<1.25\) 0.671 |
| KITTI | DepthCLIP | AbsRel 0.473, RMSE 12.958, \(\delta<1.25\) 0.281 |
| KITTI | Hu et al. | AbsRel 0.384, RMSE 12.290, \(\delta<1.25\) 0.312 |
| KITTI | Auty & Mikolajczyk | AbsRel 0.238, RMSE 5.756, \(\delta<1.25\) 0.652 |
| KITTI | PureCLIP-Depth | AbsRel 0.172, RMSE 1.062, \(\delta<1.25\) 0.739 |

These results are the basis for the paper’s claim of **state-of-the-art performance among CLIP embedding-based models on both indoor and outdoor datasets** [2603.16238]. The gains are especially large relative to prompt-based CLIP baselines.

The ablation study isolates three factors. First, the two MLPs are essential: without \(\phi\) and \(\psi\), NYU **AbsRel** rises from **0.201** to **0.320**, and KITTI **AbsRel** rises from **0.172** to **0.371** [2603.16238]. Second, the CLS token provides a smaller but consistent gain: removing it changes NYU **AbsRel** from **0.201** to **0.215**, and KITTI **AbsRel** from **0.172** to **0.182** [2603.16238]. Third, the full loss combination is strongest: **InfoNCE alone** is worse than **InfoNCE + alignment**, which is worse than **InfoNCE + alignment + RMSE** [2603.16238].

The paper also reports qualitative improvements in **object boundaries**, **fine structure**, and **small objects**, and notes that uncertainty increases at far depths, producing a tendency toward **underestimation** in the far range [2603.16238].

## 5. Relation to earlier CLIP-based depth paradigms

PureCLIP-Depth belongs to a rapidly developing family of CLIP-based depth methods, but it occupies a specific architectural position. **DepthCLIP** reframed depth as semantic distance classification using handcrafted prompts such as “This object is [distance class]”, a CLIP **ResNet-50** visual encoder, and fixed depth bins, all in a **training-free** pipeline [2207.01077]. **Learning to Adapt CLIP for Few-Shot Monocular Depth Estimation** introduced **learnable prompts** and a **scene-adaptive depth codebook**, while keeping CLIP frozen and training only a small number of parameters in a one-shot-per-scene regime [2311.01034]. **CLIP2Depth** kept CLIP’s image and text encoders frozen but added a **mirror**—a static non-human prompt for the text encoder—and a **CLIPSeg**-style decoder with FiLM conditioning [2402.03251]. **SPACE-CLIP** removed the text encoder at inference time and used a **dual-pathway decoder** over a frozen CLIP **ViT-B/16** vision encoder [2601.17657]. **MoA-DepthCLIP** inserted **Mixture-of-Adapters** into **ViT-B/32**, fine-tuned the last four transformer blocks, and combined classification, regression, and composite geometric loss terms [2604.01118].

Against that background, PureCLIP-Depth differs in three ways. First, it uses **no prompts at training or inference** in the operational sense; second, it uses **no decoder**; third, it works with the **final CLIP visual embeddings** rather than intermediate spatial feature hierarchies [2603.16238]. The model therefore strips CLIP-based depth estimation down to the question of whether metric depth can be recovered by embedding-space alignment alone.

A plausible implication is that PureCLIP-Depth tests a narrower hypothesis than methods such as SPACE-CLIP or MoA-DepthCLIP. Those methods examine whether CLIP can become a strong depth backbone when supplemented with decoders, adapters, or geometric losses, whereas PureCLIP-Depth examines whether the **conceptual structure** of CLIP space is already sufficient once an appropriate mapping and depth table are learned. The reported benchmark gains over earlier CLIP embedding-based methods suggest that this hypothesis is nontrivial rather than merely minimalist [2603.16238].

## 6. Limitations, interpretation, and significance

The paper identifies three main limitations. The first is **far-range underestimation**: empirical uncertainty increases for far depths, and the prediction distribution becomes biased toward shallower values in sparsely represented depth regions [2603.16238]. The second is **sky-region instability**, which is linked to missing or noisy LiDAR supervision and the fact that sky corresponds to effectively infinite depth [2603.16238]. The third is the intrinsic constraint of **patch resolution**: depth is predicted per \(14\times14\) patch region, which limits spatial detail relative to pixel-level decoders [2603.16238].

These limitations clarify the meaning of the model’s “decoder-free” claim. PureCLIP-Depth does not attempt to recover high-resolution local geometry by multiscale reconstruction; its strength lies instead in aligning conceptual image representations with depth anchors. This suggests that its strongest use case is not necessarily maximal geometric precision, but rather the demonstration that a foundation model trained for image–text contrastive learning contains a depth-relevant structure that can be exposed through lightweight embedding-space learning.

The paper explicitly points to **low-contrast** and **thermal infrared** imagery as potential future applications [2603.16238]. That suggestion is consistent with the method’s emphasis on semantic priors rather than classical texture-heavy depth cues. More broadly, PureCLIP-Depth supports a specific claim about multimodal foundation models: **conceptual embedding spaces can support geometric prediction without prompts and without a decoder**, provided that the mapping from RGB patch embeddings to depth embeddings is learned with appropriate contrastive and metric supervision [2603.16238].

In the literature of CLIP-based monocular depth estimation, PureCLIP-Depth therefore marks a distinct line of inquiry. It is less a replacement for geometry-aware depth architectures than a controlled demonstration that prompt engineering and dense decoders are not the only route by which CLIP can be made depth-sensitive.

Source: https://www.emergentmind.com/topics/pureclip-depth