SuperCLIP: Fine-Grained Vision-Language Alignment

• SuperCLIP is an extension of CLIP that incorporates a lightweight token-level classification head to leverage fine-grained textual cues during vision-language alignment.
• It mitigates CLIP’s limitations with long and detailed captions and small batch sizes, yielding consistent improvements across zero-shot, retrieval, and visual benchmarks.
• The approach achieves significant performance gains with minimal computational overhead, without the need for additional annotated data.

SuperCLIP is an extension of the standard Contrastive Language-Image Pretraining (CLIP) framework that augments the traditional contrastive loss with a token-level classification supervision in order to leverage fine-grained textual cues during vision-language alignment. It achieves this by introducing a lightweight linear head atop the vision encoder, facilitating explicit supervision over all caption tokens with minimal computational overhead—specifically, a +0.077% increase in FLOPs. This approach addresses CLIP’s documented deficiencies in absorbing rich token-level signals, particularly when presented with long or detailed captions, and mitigates the significant performance drop of CLIP under small batch sizes. SuperCLIP achieves consistent improvements across zero-shot classification, image-text retrieval, and purely visual benchmarks, without the need for any additional annotated data (Zhao et al., 16 Dec 2025).

1. Motivation and Problem Formulation

CLIP’s prevailing objective,

$$\mathcal{L}_{\rm CLIP} = -\frac{1}{2N}\sum_{i=1}^N \left[ \log\frac{\exp(\langle u_i,v_i\rangle/\tau)} {\sum_{k=1}^N\exp(\langle u_i,v_k\rangle/\tau)} + \log\frac{\exp(\langle u_i,v_i\rangle/\tau)} {\sum_{k=1}^N\exp(\langle u_k,v_i\rangle/\tau)} \right],$$

focuses exclusively on maximizing global image-text similarity between the embeddings $u_i=f_\theta(I_i)$ and $v_i=g_\phi(T_i)$. This implicitly overlooks fine-grained cues in captions—e.g., distinctions such as “statue” vs. “real” or “inside” vs. “outside”—unless they directly alter global image–text alignment. Empirical results demonstrate that when captions are automatically enriched to be longer and more detailed (via an LLaMA-3 re-captioner), conventional CLIP models actually suffer a decrease in zero-shot accuracy (from 32.7% to 23.6% on COCO image retrieval in the “long-only” setting), establishing that CLIP’s contrastive loss is incapable of absorbing richer token-level semantics.

Moreover, CLIP’s reliance on large batch sizes for negative sampling causes substantial performance drops at reduced batch scales. Reducing the batch size from 16 k to 1 k results in a decrease of over 10 points on ImageNet zero-shot accuracy. This sensitivity arises from the quadratic scaling of the contrastive loss in batch size $N$, resulting in limited exposure to hard negatives when batches are small.

2. SuperCLIP Architecture and Training Formulation

SuperCLIP retains the standard CLIP vision encoder $f_\theta(\cdot)$ (e.g., ViT-L/16) and incorporates a lightweight linear head for classification. Specifically, given the average-pooled image feature $z_i \in \mathbb{R}^D$, SuperCLIP introduces: $x_i = Wz_i + b \in \mathbb{R}^V,$ where $V$ is the text tokenizer’s vocabulary size (49,408). Each $x_{i,c}$ scores the likelihood of token $c$’s presence in the associated caption $u_i=f_\theta(I_i)$0. This head imposes only 0.051 GFLOPs additional forward cost for L-size models (+0.077% over CLIP’s total).

Token-level supervision is enforced by constructing a K-hot vector $u_i=f_\theta(I_i)$1 for each token $u_i=f_\theta(I_i)$2 present in $u_i=f_\theta(I_i)$3. An inverse document frequency (IDF) weighting is applied: $u_i=f_\theta(I_i)$4 The cross-entropy loss between $u_i=f_\theta(I_i)$5 and the softmax of the classification logits $u_i=f_\theta(I_i)$6 is used: $u_i=f_\theta(I_i)$7 SuperCLIP optimizes the sum of the contrastive and classification objectives: $u_i=f_\theta(I_i)$8 with $u_i=f_\theta(I_i)$9 providing optimal empirical trade-off. Only the vision encoder receives classification supervision in this setup.

3. Computational and Implementation Considerations

The steps above introduce a negligible increase in computational cost. For a batch size of 1:

• Vision encoder: 59.689 GFLOPs
• Text encoder: 6.547 GFLOPs
• Linear head: 0.051 GFLOPs (0.077% increase overall)

The classification loss adds an overhead of 5.666 GFLOPs per batch compared to 412.317 GFLOPs for the contrastive objective, i.e., +1.37% on the loss computation. Importantly, there is no additional inference cost for text queries. No extra annotated data are required; the only supervision comes from the tokenized raw captions.

4. Experimental Protocol and Benchmarks

SuperCLIP is evaluated on the Datacomp-1B dataset (1.3 B image–text pairs) and on Recap-DataComp-1B (containing LLama-3 re-captioned long-image captions) for mixed-caption ablation. Experimental settings match the CLIP baseline for batch size (16 k by default, with ablations from 1 k to 32 k), optimizer (AdamW with cosine LR decay), and total epochs. The evaluation suite includes:

• Zero-shot classification and retrieval tasks on the LAION CLIP Benchmark;
• ImageNet linear probing (per MAE protocol);
• Semantic segmentation (Pascal VOC, ADE20K) and depth estimation (NYUv2), following DINOv2 evaluation;
• Multi-modal LLM integration (LLaVA-1.5 + Vicuna-7B) on VQAv2, GQA, and other benchmarks.

5. Empirical Performance and Analysis

SuperCLIP yields consistent gains across model scales and evaluation settings.

Model & Scale	Metric	CLIP Baseline	SuperCLIP	Absolute Δ
ViT-B/512M	ImageNet-val (%)	60.5	63.5	+3.0
ViT-L/512M	ImageNet-val (%)	66.1	70.1	+4.0
ViT-B/512M	COCO R@1 (%)	38.4	40.7	+2.3
ViT-L/512M	Flickr R@1 (%)	47.4	52.8	+5.4

• On mixed-caption data, SuperCLIP recovers fine-grained textual supervision: on “long-only” recaptioned data, average accuracy increases from 27.8% (CLIP-B) to 31.4% (+3.6%). The “dual” setup (classification loss for long, contrastive loss for short) further increases performance to 45.1% from 42.8%, without tuning the mix.
• Generalization is observed across frameworks: applying SuperCLIP’s classification head to SigLIP and FLIP yields +3.7% and +3.4% image classification gains, and up to +5.3% in text retrieval (vs. original frameworks).
• Purely visual tasks benefit: linear-probe accuracy (ImageNet) increases by +1.5% with B and +1.3% with L; segmentation mIoU improves by +7.7% (Pascal) and +4.1% (ADE20K); and NYUv2 depth RMSE drops by 0.022.
• The classification loss is insensitive to batch size. At 1 k batch, CLIP’s zero-shot accuracy drops to ~30%, whereas SuperCLIP remains at ~36% (+6%). For linear-probe ImageNet, SuperCLIP shows <$v_i=g_\phi(T_i)$0 variance across batch sizes 1 k–32 k, while CLIP drops >4% at small N.
• Ablations on $v_i=g_\phi(T_i)$1 confirm best aggregate results at $v_i=g_\phi(T_i)$2; IDF weighting further improves top-1 accuracy by +2.3% (ImageNet-val) and COCO retrieval by +1.6 points.

6. Contributions, Limitations, and Future Directions

SuperCLIP demonstrates that introducing a token-classification head to the vision encoder enables models to absorb fine-grained caption information systematically omitted by the global contrastive strategy. This addition imposes a marginal computational tax (+0.077% FLOPs), requires no external annotation, and notably stabilizes performance under smaller batch sizes by decoupling supervision strength from $v_i=g_\phi(T_i)$3.

Empirically, SuperCLIP extends improvements to a variety of modalities and downstream tasks, including vision-language and purely visual segmentation and depth estimation. However, the current architecture only supervises the vision encoder; extending classification supervision to the text encoder presents a plausible avenue for enhanced alignment. Structural refinement, such as grouping tokens into entity categories, or using lighter-weight or adapter-style heads, may provide further efficiency for resource-constrained deployments.

In summary, SuperCLIP is a simple, computationally negligible augmentation to CLIP that unlocks fine-grained textual signals, robustifies small-batch training, and achieves broad-based performance gains without external annotation or infringing on inference efficiency (Zhao et al., 16 Dec 2025).