CLIP-IN: Enhancing Fine-Grained Visual Perception

• CLIP-IN is a framework that augments contrastive learning with instruction-editing data and long descriptive captions to capture subtle visual differences.
• It introduces a symmetric hard-negative contrastive loss and rotary positional encoding to extend text processing up to 300 tokens.
• Empirical results show improved retrieval, classification, and compositional reasoning, enhancing both fine-grained and zero-shot visual tasks.

CLIP-IN, expanded as “CLIP with INstruction editing data and INformative long data,” is a framework for enhancing fine-grained visual understanding in CLIP by combining instruction-editing data with long descriptive captions (Wang et al., 4 Aug 2025). It targets two stated limitations of standard CLIP: the global image–text contrastive objective optimizes only for coarse alignment, so subtle changes in color, object count, or spatial relations tend to be ignored; and the text encoder uses absolute positional embeddings with a 77-token context, so richer captions are truncated. CLIP-IN addresses these issues with a symmetric hard-negative contrastive loss over real image edits and a rotary-positional-encoding text encoder that can process captions of approximately 300 tokens. Reported results show gains on MMVP, compositional reasoning benchmarks, retrieval, and downstream Multimodal LLM integration, while preserving robust zero-shot performance on broader tasks (Wang et al., 4 Aug 2025).

1. Problem setting and motivation

Standard CLIP is presented as strong at aligning vision and language, but limited in detailed visual-semantic discrimination. The stated failure mode is not generic recognition, but fine-grained perception: minimally differing image–text pairs such as “black cat with a yellow bow tie” versus “black cat with a red bow tie” are difficult under a purely global contrastive objective. In the same formulation, text-side capacity is also a bottleneck, because the original encoder relies on absolute positional embeddings and a 77-token context window, which truncates captions that contain the details needed for attribute parsing, object count, or spatial relations (Wang et al., 4 Aug 2025).

CLIP-IN is designed around two explicit challenges. The first is to teach CLIP to distinguish minimally differing image–text pairs. The second is to extend the text encoder to process long descriptive captions without losing performance on standard short-text tasks. The framework therefore does not treat fine-grained understanding as a separate downstream adaptation problem; instead, it modifies pretraining data and objectives so that subtle distinctions become part of the core contrastive learning signal.

This positioning matters because the reported improvements are not limited to one narrow benchmark. The same intervention is evaluated on zero-shot classification, short-text retrieval, long-text retrieval, fine-grained perception, compositional reasoning, and Multimodal LLM backbones. A plausible implication is that the paper treats fine-grained perception as a structural weakness of CLIP-style pretraining rather than as a dataset-specific deficiency.

2. Data innovations: instruction edits and informative long captions

The first data innovation is the use of instruction-editing data as hard negatives. CLIP-IN uses UltraEdit (Zhao et al. 2024), described as approximately 4 million tuples $(I^s, T^s, T^e, I^t, T^t)$ in which $I^s \rightarrow I^t$ is a real fine-grained image edit (Wang et al., 4 Aug 2025). Positive pairs are $(I^s, T^s)$ and $(I^t, T^t)$. Hard negatives are $(I^s, T^t)$ and $(I^t, T^s)$. Because the edited image and caption differ only in a small attribute or spatial relation, these negatives are described as extremely challenging. The key methodological point is that the negatives are not synthetic mismatches sampled at random; they are minimally perturbed, semantically adjacent alternatives.

The second data innovation is long-caption augmentation. CLIP-IN generates approximately 300-token captions with InternVL (Chen et al. 2024) for 18 million images sampled from CC3M, CC12M, COYO, and LAION (Wang et al., 4 Aug 2025). The captions average approximately 300 tokens. In the paper’s phrasing, these long captions provide rich semantic context that standard CLIP ignores. They are intended to supply broad “what” and “why” context, complementing the “where” and “how” supervision induced by instruction edits.

Taken together, the two data sources are complementary. Instruction-editing data creates dense pressure on subtle discriminations between nearly matched pairs. Long captions add descriptive breadth that is normally absent from short web captions. This suggests that CLIP-IN combines local discriminability with expanded textual context rather than relying on only one of the two.

3. Model formulation and training procedure

The core model formulation begins with the usual CLIP-style image and text embeddings, $f(x) \in \mathbb{R}^d$ and $g(y) \in \mathbb{R}^d$, and the similarity score

$s(i,j)=\langle f(x_i), g(y_j)\rangle.$

The standard image-to-text InfoNCE loss is

$$L_{I2T} = -\frac{1}{N} \sum_{i=1}^N \log \frac{\exp(s(i,i)/\tau)} {\sum_{k=1}^N \exp(s(i,k)/\tau)}.$$

The symmetric CLIP loss is

$I^s \rightarrow I^t$0

CLIP-IN extends this with a hard-negative objective. Given anchor images $I^s \rightarrow I^t$1, positive captions $I^s \rightarrow I^t$2, and hard negatives $I^s \rightarrow I^t$3, the hard-negative contrastive loss is

$I^s \rightarrow I^t$4

Applied to instruction edits, the image-to-text and text-to-image hard-negative terms are

$I^s \rightarrow I^t$5

$I^s \rightarrow I^t$6

with final symmetric hard-negative loss

$I^s \rightarrow I^t$7

This construction makes the minimally differing edited counterpart enter the denominator explicitly as a hard negative (Wang et al., 4 Aug 2025).

For long text, CLIP-IN replaces CLIP’s absolute position embeddings with rotary positional encodings, RoPE, in the text encoder. The stated purpose is to attend over sequences much longer than 77 tokens and generalize to unseen lengths. To preserve pre-trained short-text behavior, the method uses knowledge distillation from the original CLIP text encoder $I^s \rightarrow I^t$8 to a student encoder $I^s \rightarrow I^t$9 with RoPE and extended context. On truncated captions $(I^s, T^s)$0 of length $(I^s, T^s)$1, the distillation loss is

$(I^s, T^s)$2

After distillation, the student can process up to approximately 300 tokens while matching original behavior on shorter inputs (Wang et al., 4 Aug 2025).

Training proceeds in two stages. Stage 1 distills the RoPE-enabled text encoder with $(I^s, T^s)$3 on long captions. Stage 2 performs joint contrastive training with three losses: standard short-caption contrastive $(I^s, T^s)$4, long-caption contrastive $(I^s, T^s)$5, and hard-negative loss $(I^s, T^s)$6 on instruction-editing data. The overall objective is

$(I^s, T^s)$7

with $(I^s, T^s)$8, $(I^s, T^s)$9, and $(I^t, T^t)$0 (Wang et al., 4 Aug 2025).

4. Long-caption integration and representation behavior

Long-caption integration in CLIP-IN is not a simple increase in tokenizer length. InternVL generates approximately 300-token captions for each image, and these captions are encoded by the RoPE-equipped student encoder trained in the first stage (Wang et al., 4 Aug 2025). The intended effect is twofold: to retain compatibility with standard short captions and to expose the model to descriptive context that standard CLIP would truncate.

The paper explicitly frames the long captions as complementary to instruction edits. Long captions provide broad “what” and “why” context. Instruction edits provide “where” and “how” supervision through minimally differing alternatives. This division is conceptually useful because it distinguishes two mechanisms of improvement: contextual enrichment on the text side and discriminative sharpening on the contrastive side.

The ablations reinforce that distinction. With ViT-L/14 @336, instruction data only improves retrieval average from 62.5 to 72.0 while leaving ImageNet-1K at 76.6 to 76.6. Long-caption only improves retrieval from 62.5 to 71.0 and ImageNet-1K from 76.6 to 76.8. Both combined improve retrieval from 62.5 to 73.1 and ImageNet-1K to 77.0 (Wang et al., 4 Aug 2025). A plausible implication is that long captions contribute more than simple language-side redundancy, while instruction edits contribute more than mere retrieval augmentation; their combination produces the strongest aggregate effect.

5. Empirical performance across benchmarks

The evaluation suite includes zero-shot classification on ImageNet-1K; short-text retrieval on Flickr30K and MS-COCO; long-text retrieval on DCI and ShareGPT4V; fine-grained perception on MMVP; compositional reasoning on Winoground, SugarCrepe, and SPEC; and Multimodal LLM integration by replacing the visual backbone in LLaVA-1.5 and evaluating on MMVP, POPE, MME, MMBench (EN/CN), and LLaVA-Wild (Wang et al., 4 Aug 2025).

For zero-shot classification and retrieval with ViT-L/14 @224, ImageNet-1K Top-1 increases from 75.5 to 76.3, short-caption retrieval average from 60.7 to 72.9, and long-caption retrieval average from 64.3 to 76.8. On MMVP with ViT-L/14 @224, average accuracy increases from 18.5% to 30.4%, with the biggest gains in State/Condition at +33.3 pp and Feature Detection at +20 pp. For compositional reasoning with ViT-L/14 @336, Winoground text-to-image increases from 28.3 to 35.5 and group accuracy from 7.5 to 9.5, SugarCrepe from 73.8 to 79.4, and SPEC from 32.0 to 34.8 (Wang et al., 4 Aug 2025).

Setting	Metric	Reported change
ImageNet-1K	Top-1	75.5 → 76.3
Flickr30K + MS-COCO	Short-caption retrieval avg	60.7 → 72.9
ShareGPT4V + DCI	Long-caption retrieval avg	64.3 → 76.8
MMVP	Average accuracy	18.5% → 30.4%
Winoground	T→I	28.3 → 35.5
SugarCrepe	Accuracy	73.8 → 79.4
SPEC	Accuracy	32.0 → 34.8

These results are notable because the paper explicitly presents them as simultaneous improvements in fine-grained perception and broad zero-shot capability, rather than a trade-off between the two. A common concern in this area is that specialized supervision for attributes or spatial relations will overfit narrow distinctions and weaken open-domain retrieval or classification. The reported ImageNet-1K and retrieval results do not support that concern in this case (Wang et al., 4 Aug 2025).

6. Multimodal LLM integration, applications, and limitations

When CLIP-IN replaces CLIP’s visual encoder in LLaVA-1.5, the reported gains are MMVP 24.7 to 28.0, POPE rand/pop/adv 87.3 to 88.5, 86.1 to 87.2, and 84.2 to 85.2, MME 1 510.7 to 1 709.0, MMBench EN 64.3 to 72.9, MMBench CN 58.3 to 70.3, and LLaVA-Wild 65.4 to 68.5 (Wang et al., 4 Aug 2025). The paper interprets these changes as fewer object hallucinations, stronger fine-grained reasoning, and higher multimodal QA accuracy. Qualitative inspection is reported to confirm more precise grounding for descriptions such as “red bow tie” and “three apples,” together with resistance to spurious attribute insertion.

The paper also reports an MLLM ablation: only instruction data or only long captions provide modest MMVP gains, whereas both together give the full +3.3 pp improvement. This suggests that MLLM behavior benefits from both kinds of supervision: local attribute discrimination and richer caption-level semantics.

Potential applications listed for the framework include open-vocabulary object detection and segmentation requiring attribute parsing, fine-grained visual question answering, robotics instruction following, and downstream systems in which subtle visual differences matter, including e-commerce and medical imaging (Wang et al., 4 Aug 2025). These applications are presented as prospective rather than directly benchmarked deployments.

The stated limitations are dependency on large-scale instruction editing data with compute and storage cost, dependence on the quality and diversity of generated long captions in unusual domains, and a RoPE extension fixed at approximately 300 tokens. Future directions include semi-supervised mining of hard negatives from web images, improving caption generation quality via feedback loops with CLIP-IN, extending positional encoding research to arbitrarily long contexts, and adapting the framework to video–language or 3D multimodal tasks (Wang et al., 4 Aug 2025).

7. Naming ambiguity and related uses of “CLIP-IN”

The label “CLIP-IN” is not unique in the literature, and this is a significant source of confusion. In the present usage, it denotes the fine-grained CLIP framework based on instruction-editing data and informative long captions (Wang et al., 4 Aug 2025). However, unrelated works use the same or a closely similar name for different mechanisms.

“Clip-In Clip-Out” refers to CiCo for one-stage video instance segmentation, which extends a frame-in frame-out pipeline to a clip-in clip-out pipeline through 3D convolutional clip-level prediction and mask heads (Li et al., 2022). A different paper uses “CLIP-IN” for incremental object detection with CLIP, where CLIP text embeddings, super-classes, and pseudo-annotations are introduced to improve forward compatibility in continuous learning (Huang et al., 2023). Another work uses “CLIP-IN” for a training-free feedback-driven self-adaptive attention framework that adapts output-based patch-level correspondences back to intermediate attention in open-vocabulary segmentation (Chi et al., 27 Aug 2025). “Implicit Inversion turns CLIP into a Decoder” uses “CLIP-IN” for implicit CLIP inversion via an implicit neural representation, robust initialization, Orthogonal Procrustes projection, and blending loss (D'Orazio et al., 29 May 2025). In “DINORANKCLIP,” “CLIP-IN” denotes the conflict-aware injection of a frozen DINOv3 teacher into a CLIP-style contrastive trunk together with high-order ranking consistency (Jiang et al., 7 May 2026). “CLIPin” has also been introduced as a non-contrastive plug-in with shared pre-projectors and a momentum encoder for multimodal semantic alignment (Yang et al., 8 Aug 2025).

Accordingly, precise identification by title or arXiv id is essential. Within current vision–language research, “CLIP-IN” is a many-to-one label rather than a uniquely resolved method name.