---
title: 'LABCLIP: Enhanced Compositional Binding in CLIP'
url: https://www.emergentmind.com/topics/labclip
type: topic
---

# LABCLIP: Enhanced Compositional Binding in CLIP

LABCLIP (Linear Attribute Binding CLIP) is a lightweight architectural modification of CLIP designed to address compositional binding failures—specifically, the tendency of CLIP to behave as a bag-of-words (BoW) model in cross-modal retrieval tasks. While standard CLIP encoders preserve attribute–object binding information within uni-modal representations, the canonical cosine similarity-based cross-modal alignment does not exploit this, resulting in a failure to correctly bind attributes to objects in compositional settings. LABCLIP introduces a learned linear transformation on the text embedding space prior to cross-modal similarity computation, substantially increasing compositional understanding and attribute binding accuracy for image–text retrieval and related tasks [2502.03566].

## 1. Mathematical Formulation

Let $f_{\text{text}}: T \to \mathbb{R}^d$ and $f_{\text{image}}: I \to \mathbb{R}^d$ be the original, frozen CLIP encoders for text and image, respectively. For a text input $x^{\mathrm{txt}}$, its embedding is $t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$. LABCLIP incorporates a trainable linear mapping $T \in \mathbb{R}^{d \times d}$ applied to $t$:

$$ t' = T t $$

Cross-modal similarity between an image $x^{\mathrm{img}}$ and a transformed text $x^{\mathrm{txt}}$ is computed as

$$
s(x^{\mathrm{img}}, T t) = \frac{\langle f_{\text{image}}(x^{\mathrm{img}}),\, T t \rangle}{\| f_{\text{image}}(x^{\mathrm{img}})\| \cdot \| T t \|}
$$

where $\langle \cdot, \cdot \rangle$ denotes the Euclidean dot product. The key distinction from CLIP is the learned transformation $T$ on the text embeddings, while both image and text encoders remain fixed.

## 2. Training Objective and Optimization

LABCLIP adopts the standard CLIP-style symmetric contrastive objective, but exclusively trains the linear map $T$. For a minibatch of $B$ aligned image–text pairs $\{(i_i, t_i)\}$:

- Positives: $s_{i,i} = s(i_i,\,T t_i)$
- Negatives: Optionally, "hard negatives" via attribute–object–permuted captions, where negatives are $t_{j,{\rm neg}} = f_{\text{text}}({\rm permute}(x^{\mathrm{txt}}_j))$ before transformation.

For a combined batch of $2B$ candidates (positives and negatives), the symmetric cross-entropy losses are

$$
L_{\text{img} \to \text{txt}} = -\sum_{i} \log \frac{\exp(s_{i,i}/\tau)}{\sum_{j=1}^{2B} \exp(s_{i,j}/\tau)}, \qquad
L_{\text{txt} \to \text{img}} = -\sum_{j} \log \frac{\exp(s_{j,j}/\tau)}{\sum_{i=1}^{2B} \exp(s_{i,j}/\tau)}
$$
with total contrastive loss $L = L_{\text{img}\to\text{txt}} + L_{\text{txt}\to\text{img}}$, and $\tau$ (temperature parameter) initialized and trained as in CLIP.

## 3. Architectural Details and Training Protocol

The transformation $T$ is a single, fully connected $d \times d$ layer, initialized to the identity matrix $I_d$, ensuring LABCLIP initially behaves identically to CLIP. CLIP's encoders are used in frozen mode throughout. Training data for $T$ covers both synthetic (CLEVR, PUG:SPAR, PUG:SPARE) and real (COCO) datasets, with synthetic hard negative captions constructed by permuting attribute–object pairs.

Hyperparameter defaults (LABCLIP-HNB):

- Optimizer: Adam
- Learning rate: $1 \times 10^{-4}$–$1 \times 10^{-3}$, tuned per dataset
- Batch size: 64–2048
- Negative sampling: Hard Negative Batch (attribute–object permutations)
- Training epochs: 10–50, selected via validation

Inference involves computing similarity $s(x^{\text{img}}, T f_{\text{text}}(x^{\text{txt}}))$ for ranking.

## 4. Experimental Results and Quantitative Evaluation

Empirical evaluation demonstrates substantial improvements over standard CLIP and alternatives on synthetic and real-world compositional benchmarks. Observed metrics include cross-modal attribute–object binding accuracy and Recall@1 for retrieval.

| Dataset      | Model         | Accuracy (test) | Recall@1 (test) |
|--------------|--------------|-----------------|-----------------|
| CLEVR        | CLIP         | 0.58            | 0.36            |
|              | LABCLIP-SB   | 0.95            | 0.94            |
|              | LABCLIP-HNB  | 0.95            | 0.93            |
| PUG:SPAR     | CLIP         | 0.53            | 0.09            |
|              | LABCLIP-SB   | 0.97            | 0.90            |
|              | LABCLIP-HNB  | 0.97            | 0.91            |
| PUG:SPARE    | CLIP         | 0.50            | 0.06            |
|              | LABCLIP-SB   | 0.90            | 0.86            |
|              | LABCLIP-HNB  | 0.94            | 0.90            |

For real-world compositions (ViT-B/32):

| Benchmark          | Metric     | CLIP | NegCLIP | LABCLIP-SB | LABCLIP-HNB |
|--------------------|------------|------|---------|------------|-------------|
| ARO-VG-R           | Accuracy   | 0.63 | 0.71    | 0.64       | 0.69        |
| ARO-Flickr-PRC     | Accuracy   | 0.60 | 0.91    | 0.42       | 0.84        |
| SugarCrepe-Swap    | Accuracy   | 0.62 | 0.75    | 0.69       | 0.74        |
| COCO retrieval     | Recall@1   | 0.30 | 0.41    | 0.41       | 0.41        |

## 5. Ablation Analyses and Probing

Ablation studies indicate:

- Nearly identical gains whether using vanilla (SB) or explicit hard negatives (HNB), implying $T$ alone recovers binding information.
- Uni-modal linear probes demonstrate that both CLIP image and text encoders linearly encode attribute–object bindings (≈95–100% probe accuracy), ruling out absence of information in the modalities.
- After applying $T$, correct vs permuted text caption similarities diverge more; image–text similarities more strongly separate positives from negatives.
- The mean embedding gap $\| \mu_{\text{img}} - T\mu_{\text{txt}} \|_2$ decreases after training, signifying improved cross-modal alignment.
- Robustness trials show that text probes maintain accuracy $\geq 0.8$ as the number of objects increases; image probes degrade moderately from ≈0.9 to ≈0.6 with complexity.

## 6. Interpretations and Broader Implications

The core finding is that the canonical cross-modal alignment in CLIP (cosine dot product) treats the two embedding spaces too symmetrically, "flattening" structural relations and inducing BoW-like matching. LABCLIP, via a learned linear map $T$, warps the text embedding space to better match compositional geometry in the image space, thereby enabling precise attribute–object binding.

Implications include:

- Modality-specific alignment layers, such as $T$, can substantially improve cross-modal compositionality without retraining large encoders.
- Synthetic and hard-negative sampling should be considered during vision–language model pretraining to increase robustness to compositional structure.
- The widespread practice of using a shared, fixed embedding space (with raw cosine similarity) is insufficient for structure-sensitive tasks and merits reevaluation.

LABCLIP introduces only $\mathcal{O}(d^2)$ additional parameters and can be deployed post-hoc on any frozen CLIP model, providing a scalable path towards structured alignment in large multimodal systems [2502.03566].

Source: https://www.emergentmind.com/topics/labclip