CLIP: Contrastive Language-Image Pre-training

• CLIP is a pre-training paradigm that aligns image and text modalities using a contrastive loss, enabling zero-shot classification and diverse vision-language tasks.
• It employs a dual-encoder architecture with independent vision (ResNet/ViT) and text (transformer) branches, projecting data into a shared, ℓ₂-normalized embedding space.
• Studies show that pooling strategies critically affect explainability, with innovations like Masked Max Pooling (MMP) reducing semantic shift and enhancing spatial attribution.

Contrastive Language-Image Pre-training (CLIP) is a pre-training paradigm that leverages natural language supervision to align image and text modalities in a shared embedding space using a contrastive loss. It underpins a variety of vision and vision-LLMs, enabling robust zero-shot transfer, retrieval, segmentation, and captioning. CLIP’s architecture has served as the foundation for further advancements in interpretability, pooling methods, and model transparency. However, challenges remain in understanding and improving the localization and explainability of its predictions, particularly at the level of raw spatial features.

1. CLIP Model Structure and Contrastive Learning Objective

CLIP employs a dual-encoder architecture with independent vision and text branches:

• Vision encoder: Either a ResNet-50/101 or Vision Transformer (ViT) backbone. In the ResNet case, a learned attention pooling layer aggregates the final convolutional feature map into a single image embedding; ViT architecture produces a class token as image representation.
• Text encoder: A transformer-based module (similar to BERT) processes tokenized prompts (e.g., "a photo of the {class}") and projects them into a shared embedding space.
• Projection heads: Linear transformations φᵢ (image) and φₜ (text) produce D-dimensional, ℓ₂-normalized embeddings for both modalities.

The core learning objective is a symmetric InfoNCE-style contrastive loss. Given a minibatch of N image–text pairs $\{(x_i, y_i)\}_{i=1}^N$, let $f_v(x_i)$ and $f_t(y_j)$ be their respective embeddings. The similarity is measured as $\mathrm{sim}(u, v) = \frac{u \cdot v}{\|u\|\|v\|}$, and the loss is:

$$L(\theta) = -\frac{1}{N} \sum_{i=1}^N \left[ \log\frac{\exp(\mathrm{sim}(f_v(x_i),f_t(y_i))/\tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(f_v(x_i),f_t(y_j))/\tau)} + \log\frac{\exp(\mathrm{sim}(f_v(x_i),f_t(y_i))/\tau)}{\sum_{j=1}^N \exp(\mathrm{sim}(f_v(x_j),f_t(y_i))/\tau)} \right]$$

where $\tau > 0$ is a temperature hyperparameter.

During inference, images and candidate labels (tokenized as prompts) are embedded and scored by cosine similarity. This enables zero-shot classification and retrieval without explicit retraining.

2. Visual Interpretability and the Image-Text Similarity Map (ITSM)

A central tool for probing CLIP’s spatial reasoning is the Image-Text Similarity Map (ITSM):

• Given a spatial feature map $F \in \mathbb{R}^{H \times W \times D}$ from the vision encoder and a normalized text embedding $t \in \mathbb{R}^D$, the ITSM is given by $S_{u,v} = \mathrm{sim}(F_{u,v}, t)$ for all locations $(u,v)$.
• ITSM can be upsampled and min-max normalized for overlay as a heatmap visualizing which regions contribute most to the model’s image–text alignment.

Empirical findings indicate that the CLIP ITSM frequently highlights background regions more than true object regions. This is incongruous with conventional interpretability techniques (e.g., class activation mapping), suggesting a problematic misalignment with human intuition and ground-truth semantics.

This phenomenon, termed “semantic shift”, arises because the pooling performed in CLIP encoders (especially averaging or attention-based pooling) allows discriminative foreground features to “leak” into background regions in the global embedding. As a result, the final embedding can be dominated by spatially diffuse or background-associated features.

3. Pooling Mechanisms and the Semantic Shift Problem

Pooling is key to feature aggregation in both CNNs and transformers, but its impact on explainability differs by strategy:

• Attention pooling: Aggregates spatial features with softmax weights $$\alpha_{u,v} = \mathrm{softmax}(W \cdot \mathrm{flatten}(F))$$, resulting in $c = \sum_{u,v} \alpha_{u,v} F_{u,v}$. This can blur fine spatial distinctions and blend foreground/background.
• Global average pooling: Computes $c = \frac{1}{HW} \sum_{u,v} F_{u,v}$, similarly prone to blending spatially disparate features.
• Global max pooling: Selects $c = \max_{u,v} F_{u,v}$ for each channel, which empirically localizes activations to true object regions and preserves discriminative cues.

The analysis demonstrates that max pooling is less susceptible to semantic shift, as it avoids spatial mixing and tends to pick out the most semantically relevant activations. In contrast, attention and average pooling cause feature shift—foreground features are diluted and background features can dominate the pooled representation.

4. Explainable CLIP (ECLIP) and the Masked Max Pooling (MMP) Solution

ECLIP modifies CLIP's pooling layer to align spatial explanations with human intuition, relying on Masked Max Pooling (MMP):

• Guidance: For each image, a self-supervised attention map $A \in \mathbb{R}^{H \times W}$ is computed using a model such as DINO.
• Masking: The map is thresholded to give $M_{u,v} = \mathbb{1}[A_{u,v} \geq \tau_A]$, a binary mask selecting confident foreground regions.
• Pooling: The pooled token is $z = \mathrm{MaxPool}(F \odot M)$, restricting aggregation to confident foreground locations.

During training:

• All original CLIP parameters are frozen.
• New projection heads are introduced and trained on masked foreground pooled features with the standard contrastive loss.
• No manual masks or ground-truth segmentation is used; the unsupervised attention map supplies training guidance.

At inference, the mask is omitted, and global max pooling alone is used — introducing no test-time computational overhead.

5. Experimental Quantification of Explainability Gains

ECLIP’s explainability and recognition ability are evaluated across standard benchmarks (Pascal VOC 2012, MS COCO 2017, ImageNet-S50) with the following metrics:

• mIoU (mean intersection-over-union): Measures alignment of binarized ITSM maps with ground-truth object masks.
• mSC (mean score-contrast): The average difference in cosine similarity for foreground vs. background regions; negative values indicate background dominance, positive values favor correct object localization.
• Zero-shot mAP: Maintains CLIP’s downstream recognition accuracy.

Key results (ViT-B/32, Pascal VOC):

• Original CLIP: mIoU ≈ 18.5%, mSC ≈ −23.5%
• Grad-CAM: mIoU ≈ 19.0%, mSC ≈ −9.8%
• Bi-Model Transformer: mIoU ≈ 32.6%, mSC ≈ 16.8%
• Inverted CLIP ITSM: mIoU ≈ 39.8%, mSC ≈ 25.0%
• ECLIP (MMP): mIoU ≈ 48.4%, mSC ≈ 34.7%

ECLIP achieves an absolute gain of ≈30 points in mIoU and reverses mSC from strongly negative to strongly positive, surpassing existing explainability baselines by large margins. Zero-shot mAP is preserved, confirming that explainability gains do not compromise recognition performance. These improvements generalize across both ResNet and ViT backbones.

6. Interpretability Principles and Implications for Design

The investigation reveals that the pooling layer's design is a decisive factor for CLIP's spatial explainability:

• Average-style pooling (attention or mean) inherently degrades spatial attribution, making CLIP’s heatmaps unreliable for localization.
• Masked max pooling restores spatial specificity by preventing foreground signals from being washed out.
• The use of self-supervised attention maps as guides introduces object-level explainability without requiring ground-truth masks or increasing inference latency.

This result challenges conventional wisdom that pooling operations are architecturally neutral with respect to model interpretability. Instead, it establishes that for contrastively-trained multi-modal encoders, the pooling strategy governs whether attribution maps reflect true semantic organization.

7. Broader Significance and Future Directions

ECLIP demonstrates that explainability can be substantially improved in contrastive vision-LLMs without altering their core recognition ability or requiring manual annotation. This approach generalizes across both convolutional and transformer-based encoders, indicating the universality of the semantic shift issue and the effectiveness of masked max pooling as a remedy. These findings suggest that:

• Pooling operations should be explicitly reconsidered in future multi-modal model design when interpretability is critical.
• Free attention mechanisms (e.g., DINO) can be leveraged as supervisory signals for explainability, even in the absence of manually curated segmentation masks.
• There is a new architectural direction for models that are both performant on transfer tasks and inherently aligned with human spatial reasoning.

The paradigm established by ECLIP thus sets a new baseline for the integration of human-aligned interpretability in contrastively trained vision-language systems.