LABCLIP: Enhanced Compositional Binding in CLIP

• The paper introduces a trainable linear map on text embeddings to overcome CLIP’s bag-of-words limitation in compositional image–text retrieval.
• It employs a symmetric contrastive objective and synthetic hard negatives to optimize attribute–object binding accuracy.
• Empirical results show significant improvements in accuracy and recall over standard CLIP across benchmarks like CLEVR and COCO.

LABCLIP (Linear Attribute Binding CLIP) is a lightweight architectural modification of CLIP designed to address compositional binding failures—specifically, the tendency of CLIP to behave as a bag-of-words (BoW) model in cross-modal retrieval tasks. While standard CLIP encoders preserve attribute–object binding information within uni-modal representations, the canonical cosine similarity-based cross-modal alignment does not exploit this, resulting in a failure to correctly bind attributes to objects in compositional settings. LABCLIP introduces a learned linear transformation on the text embedding space prior to cross-modal similarity computation, substantially increasing compositional understanding and attribute binding accuracy for image–text retrieval and related tasks (Koishigarina et al., 5 Feb 2025).

1. Mathematical Formulation

Let $f_{\text{text}}: T \to \mathbb{R}^d$ and $f_{\text{image}}: I \to \mathbb{R}^d$ be the original, frozen CLIP encoders for text and image, respectively. For a text input $x^{\mathrm{txt}}$, its embedding is $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$. LABCLIP incorporates a trainable linear mapping $T \in \mathbb{R}^{d \times d}$ applied to $t$:

$t' = T t$

Cross-modal similarity between an image $x^{\mathrm{img}}$ and a transformed text $x^{\mathrm{txt}}$ is computed as

$$s(x^{\mathrm{img}}, T t) = \frac{\langle f_{\text{image}}(x^{\mathrm{img}}),\, T t \rangle}{\| f_{\text{image}}(x^{\mathrm{img}})\| \cdot \| T t \|}$$

where $f_{\text{image}}: I \to \mathbb{R}^d$0 denotes the Euclidean dot product. The key distinction from CLIP is the learned transformation $f_{\text{image}}: I \to \mathbb{R}^d$1 on the text embeddings, while both image and text encoders remain fixed.

2. Training Objective and Optimization

LABCLIP adopts the standard CLIP-style symmetric contrastive objective, but exclusively trains the linear map $f_{\text{image}}: I \to \mathbb{R}^d$2. For a minibatch of $f_{\text{image}}: I \to \mathbb{R}^d$3 aligned image–text pairs $f_{\text{image}}: I \to \mathbb{R}^d$4:

• Positives: $f_{\text{image}}: I \to \mathbb{R}^d$5
• Negatives: Optionally, "hard negatives" via attribute–object–permuted captions, where negatives are $f_{\text{image}}: I \to \mathbb{R}^d$6 before transformation.

For a combined batch of $f_{\text{image}}: I \to \mathbb{R}^d$7 candidates (positives and negatives), the symmetric cross-entropy losses are

$f_{\text{image}}: I \to \mathbb{R}^d$8

with total contrastive loss $f_{\text{image}}: I \to \mathbb{R}^d$9, and $x^{\mathrm{txt}}$0 (temperature parameter) initialized and trained as in CLIP.

3. Architectural Details and Training Protocol

The transformation $x^{\mathrm{txt}}$1 is a single, fully connected $x^{\mathrm{txt}}$2 layer, initialized to the identity matrix $x^{\mathrm{txt}}$3, ensuring LABCLIP initially behaves identically to CLIP. CLIP's encoders are used in frozen mode throughout. Training data for $x^{\mathrm{txt}}$4 covers both synthetic (CLEVR, PUG:SPAR, PUG:SPARE) and real (COCO) datasets, with synthetic hard negative captions constructed by permuting attribute–object pairs.

Hyperparameter defaults (LABCLIP-HNB):

• Optimizer: Adam
• Learning rate: $x^{\mathrm{txt}}$5–$x^{\mathrm{txt}}$6, tuned per dataset
• Batch size: 64–2048
• Negative sampling: Hard Negative Batch (attribute–object permutations)
• Training epochs: 10–50, selected via validation

Inference involves computing similarity $x^{\mathrm{txt}}$7 for ranking.

4. Experimental Results and Quantitative Evaluation

Empirical evaluation demonstrates substantial improvements over standard CLIP and alternatives on synthetic and real-world compositional benchmarks. Observed metrics include cross-modal attribute–object binding accuracy and Recall@1 for retrieval.

Dataset	Model	Accuracy (test)	Recall@1 (test)
CLEVR	CLIP	0.58	0.36
	LABCLIP-SB	0.95	0.94
	LABCLIP-HNB	0.95	0.93
PUG:SPAR	CLIP	0.53	0.09
	LABCLIP-SB	0.97	0.90
	LABCLIP-HNB	0.97	0.91
PUG:SPARE	CLIP	0.50	0.06
	LABCLIP-SB	0.90	0.86
	LABCLIP-HNB	0.94	0.90

For real-world compositions (ViT-B/32):

Benchmark	Metric	CLIP	NegCLIP	LABCLIP-SB	LABCLIP-HNB
ARO-VG-R	Accuracy	0.63	0.71	0.64	0.69
ARO-Flickr-PRC	Accuracy	0.60	0.91	0.42	0.84
SugarCrepe-Swap	Accuracy	0.62	0.75	0.69	0.74
COCO retrieval	Recall@1	0.30	0.41	0.41	0.41

5. Ablation Analyses and Probing

Ablation studies indicate:

• Nearly identical gains whether using vanilla (SB) or explicit hard negatives (HNB), implying $x^{\mathrm{txt}}$8 alone recovers binding information.
• Uni-modal linear probes demonstrate that both CLIP image and text encoders linearly encode attribute–object bindings (≈95–100% probe accuracy), ruling out absence of information in the modalities.
• After applying $x^{\mathrm{txt}}$9, correct vs permuted text caption similarities diverge more; image–text similarities more strongly separate positives from negatives.
• The mean embedding gap $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$0 decreases after training, signifying improved cross-modal alignment.
• Robustness trials show that text probes maintain accuracy $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$1 as the number of objects increases; image probes degrade moderately from ≈0.9 to ≈0.6 with complexity.

6. Interpretations and Broader Implications

The core finding is that the canonical cross-modal alignment in CLIP (cosine dot product) treats the two embedding spaces too symmetrically, "flattening" structural relations and inducing BoW-like matching. LABCLIP, via a learned linear map $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$2, warps the text embedding space to better match compositional geometry in the image space, thereby enabling precise attribute–object binding.

Implications include:

• Modality-specific alignment layers, such as $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$3, can substantially improve cross-modal compositionality without retraining large encoders.
• Synthetic and hard-negative sampling should be considered during vision–LLM pretraining to increase robustness to compositional structure.
• The widespread practice of using a shared, fixed embedding space (with raw cosine similarity) is insufficient for structure-sensitive tasks and merits reevaluation.

LABCLIP introduces only $$t = f_{\text{text}}(x^{\mathrm{txt}}) \in \mathbb{R}^d$$4 additional parameters and can be deployed post-hoc on any frozen CLIP model, providing a scalable path towards structured alignment in large multimodal systems (Koishigarina et al., 5 Feb 2025).