CLIP-Guided Lexical Certainty Adapter

• The paper demonstrates that masking high-certainty tokens compels the model to rely on subtle spatial cues, significantly improving 3D referential accuracy.
• CLIP-LCA integrates CLIP-based feature extraction with k-means clustering for token certainty scoring, effectively distinguishing spatial descriptors from explicit lexical cues.
• Empirical results show notable performance gains in far-range localization, underscoring the utility of the adapter in scenarios with ambiguous object appearances.

The CLIP-Guided Lexical Certainty Adapter (CLIP-LCA) is a neural masking mechanism designed for spatial-aware text encoding in the Monocular 3D Visual Grounding (Mono3DVG) task. This adapter addresses the challenge of models over-relying on explicit lexical cues that directly identify objects—such as color or object class—while neglecting implicit spatial language vital for robust object localization. CLIP-LCA dynamically distinguishes and masks high-certainty tokens in captions during training, compelling the model to extract and reason about less explicit, spatial descriptors, thereby advancing spatial reasoning in 3D referential comprehension.

1. Motivation and Problem Setting

Mono3DVG aims to localize objects in 3D from RGB images using natural language descriptions that often mix high-certainty and low-certainty lexical elements. High-certainty keywords (e.g., “red car”) alone may suffice for object identification, leading models to disregard spatial descriptors (e.g., “on the right side,” “20 m away”) that demand genuine geometric reasoning. This frequently results in failures when captions become lexically ambiguous or omit direct object identifiers. CLIP-LCA is engineered specifically to counteract this tendency by occluding high-certainty tokens during training, forcing the model’s language encoder to utilize spatial descriptions for target localization.

2. Architectural Design and Workflow

CLIP-LCA is integrated at the text encoding stage of the Mono3DVG-EnSD framework:

1. Region Cropping: Given an image $I$ and ground-truth box $B$, crop the target region $R$ from $I$.
2. CLIP Feature Extraction: Encode $R$ into a visual embedding $v = \text{CLIP}_\text{Image}(R) \in \mathbb{R}^d$. Encode each caption token $w_i$ into $$e_i = \text{CLIP}_\text{Text}(w_i) \in \mathbb{R}^d$$.
3. Lexical Certainty Scoring: Compute cosine similarity $c_i = \frac{e_i^T v}{\|e_i\|\|v\|}$ for each token $w_i$.
4. Clustering & Masking: Apply k-means ($k=2$) clustering to $\{c_i\}$, separating tokens into high-certainty and low-certainty groups. Define binary mask $m_i$:

$$m_i = \begin{cases} 1 & \text{if } c_i < \tau \ (\text{low-certainty, keep}) \ 0 & \text{if } c_i \geq \tau \ (\text{high-certainty, mask}) \end{cases}$$

1. Masked Caption Encoding: Substitute $w_i$ with “[MASK]” if $m_i=0$, retaining original $w_i$ if $m_i=1$. Feed masked sequence into RoBERTa, yielding spatially enhanced features $T_t$.

At inference, masking is skipped and the unaltered caption provides both lexical and spatial cues.

3. Mathematical Formulation

Let $d$ denote CLIP’s embedding dimension. For each token $w_i$,

• Token Embedding: $$e_i = \text{CLIP}_\text{text}(w_i) \in \mathbb{R}^d$$
• Region Embedding: $v = \text{CLIP}_\text{image}(R) \in \mathbb{R}^d$
• Certainty Score: $c_i = \frac{e_i^T v}{\|e_i\|\|v\|}$
• Mask Assignment: Using threshold $\tau$ from 2-means clustering on $\{c_i\}$:

$$m_i = \begin{cases} 1 & c_i < \tau \ 0 & c_i \geq \tau \end{cases}$$

• Masked Embedding: $\hat{e}_i = m_i e_i + (1-m_i) e_\text{mask}$
• Sequence Output: $\{\hat{e}_1,\ldots, \hat{e}_n\} \rightarrow$ RoBERTa $\rightarrow T_t$

This step enforces the use of spatial reasoning by prioritizing low-certainty tokens during training.

4. Integration with CLIP and Downstream Modules

The following pseudocode abstracts integration with CLIP and the downstream encoder:

R = crop_image(I, B) v = CLIP_Image_Encoder(R) c = [] for w_i in w_1...w_n: e_i = CLIP_Text_Encoder(w_i) c_i = cosine_similarity(e_i, v) c.append(c_i) tau_low, tau_high = kmeans(c, k=2) tau = midpoint(tau_low, tau_high) masked_caption = [] for i, c_i in enumerate(c): if c_i >= tau: w_i_masked = "[MASK]" # high-certainty, mask else: w_i_masked = w_i # low-certainty, keep masked_caption.append(w_i_masked) T_t = RoBERTa(masked_caption)

During model inference, steps involving certainty scoring, clustering, and masking are omitted; thus, captions retain both lexical and spatial content.

5. Training Objectives and Optimization

CLIP-LCA is a masking-based adapter that does not introduce additional loss terms; it is coupled with the main loss function of Mono3DVG-EnSD:

$$L_\text{total} = L_\text{2D} + L_\text{3D} + L_\text{dmap}$$

where

• $$L_\text{2D} = \lambda_1 L_\text{class} + \lambda_2 L_\text{lrtb} + \lambda_3 L_\text{GIoU} + \lambda_4 L_\text{xy3D}$$
• $$L_\text{3D} = L_\text{size3D} + L_\text{orien} + L_\text{depth}$$
• $$L_\text{dmap} = \text{FocalLoss}(\text{depth\_map})$$

CLIP-LCA operates strictly as a plug-in masking strategy within the training loop and does not alter the downstream optimization schema.

6. Empirical Performance and Ablation Analysis

Quantitative evidence from ablation studies on the Mono3DRefer dataset demonstrates the benefits of CLIP-LCA on spatial reasoning and referential accuracy:

Model Variant	[email protected] (%)	[email protected] (%)	Far ([email protected], %)
Baseline (no modules)	64.36	44.25	15.35
CLIP-LCA only	66.57	49.29	—
D2M only	68.11	51.08	—
CLIP-LCA + D2M (full)	69.51	52.85	28.89

A major finding is the +13.54 percentage point gain in the challenging Far-range scenario (Hard/Far [email protected]: 15.35% → 28.89%). CLIP-LCA is especially effective when object appearance is ambiguous, due to its forced exploitation of spatial language.

Qualitative results further illustrate CLIP-LCA’s impact on captions containing multiple objects and spatial relations, such as “the car on the right of the red truck 40 m ahead,” where conventional keyword matching fails.

7. Implications and Context

CLIP-LCA advances the state-of-the-art by supporting spatial generalization and robust performance in referential tasks where lexical cues are insufficient. A plausible implication is that similar certainty-based masking mechanisms could be adapted for other multimodal grounding tasks where explicit keyword bias impedes deeper semantic parsing. The adapter remains agnostic to loss function modifications, situating itself as an effective, low-overhead enhancement to standard CLIP-based architectures within the spatial language domain. The demonstrated consistency of gains across distance and complexity regimes supports its utility as a mainstay component in future Mono3DVG research and applications.