FIX-CLIP: Long-Text Visual-Language Alignment

• The paper introduces FIX-CLIP, a CLIP extension that overcomes token limitations to enable detailed long-text image retrieval.
• It integrates multiple learnable regional prompts with unidirectional masks to extract localized features from images.
• A hierarchical feature alignment module is implemented to maintain compatibility with short-text performance while improving long-text understanding.

Searching arXiv for the FIX-CLIP paper and closely related CLIP-extension work on long-text understanding. FIX-CLIP is a CLIP extension for long-text visual-language alignment that targets a specific limitation of standard CLIP: strong zero-shot performance on short captions, but weak performance on long-text inputs because the original text encoder is capped at 77 tokens and because final-layer global alignment is not well suited to detailed, compositional descriptions (Wang et al., 14 Jul 2025). The method is designed to improve long-text retrieval while preserving short-text retrieval and zero-shot classification, and it does so through three coupled components: a dual-branch training pipeline, multiple learnable regional prompts with unidirectional masks in the image Transformer, and a hierarchical feature alignment module over intermediate encoder layers. Training relies on synthetic long captions generated for a 30M-image corpus with Llama3-LLaVA-NeXT-8b, while short-caption supervision is also retained to stabilize compatibility with pretrained CLIP representations (Wang et al., 14 Jul 2025).

1. Definition and problem scope

FIX-CLIP addresses a long-standing asymmetry in CLIP-style models: they are highly effective for short text-image matching, yet degrade when the text becomes long, descriptive, and semantically dense. The paper defines the practical regime as long-text inputs $>77$ tokens, a range in which standard CLIP suffers from both hard truncation and a representational bias toward short global summaries (Wang et al., 14 Jul 2025).

The motivating use cases are long-text image-text retrieval, multimodal LLMs, and text-to-image generation. In these settings, the text may contain object counts, materials, colors, positions, scene layout, and interactions that exceed the capacity of the original CLIP text pathway. FIX-CLIP is therefore not presented as a generic retraining of CLIP, but as an incremental extension intended to preserve CLIP’s short-text strengths while improving paragraph-level alignment.

A useful distinction emerges when FIX-CLIP is placed alongside other CLIP “fixes.” Some works diagnose failures of CLIP geometry or composition in general and propose alternative scoring mechanisms or subspace filtering (Kang et al., 10 Mar 2025), whereas FIX-CLIP is narrowly focused on long-text understanding and explicitly aims to avoid the short-text degradation observed in prior long-context CLIP variants (Wang et al., 14 Jul 2025). This suggests that FIX-CLIP is best understood as a task-specific extension rather than a universal repair of CLIP semantics.

2. Architectural design

FIX-CLIP consists of three modules that operate jointly during training. The first is a dual-branch training pipeline. One branch aligns short text with masked images, while the other aligns long text with raw images. The second inserts regional prompts into the image Transformer, together with unidirectional masks that make those prompts act as localized feature extractors. The third aligns intermediate image and text features through hierarchical feature alignment rather than relying only on the final embedding (Wang et al., 14 Jul 2025).

On the text side, the encoder remains a vanilla Transformer, but its positional embeddings are extended from 77 to 248 tokens. The paper keeps pretrained Transformer block parameters and preserves the original CLIP positional structure for short texts. For long texts, the first 20 positional embeddings are frozen, and the remaining positions are interpolated by a factor of 4: $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$ with

$$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$

The paper’s interpretation is that the first roughly 20 tokens often summarize the image’s main content, so preserving them helps maintain compatibility with pretrained CLIP behavior (Wang et al., 14 Jul 2025).

The image encoder is enriched with multiple learnable regional prompts. At Transformer layer $l$, the token sequence becomes

$([CLS], R^l_1,\cdots, R^l_M, P_1,\cdots, P_N),$

where $R_j^l$ are regional prompts and $P_i$ are patch embeddings. After multi-head self-attention, the prompts are replaced before the next layer: $$\mathbf{X}^l[1:1+M] = (R^{l+1}_1,\cdots,R^{l+1}_M).$$ This per-layer replacement is meant to prevent prompt states from accumulating interference across depth and to keep them focused on local features in the current layer (Wang et al., 14 Jul 2025).

The paper identifies $M=4$ prompts as the best setting in its ablation. This suggests that a small number of regional extractors is sufficient to improve long-text grounding without excessively fragmenting the representation.

3. Dual-branch training and regional prompt mechanics

The dual-branch pipeline is the core training strategy. In the short branch, image patch embeddings $I \in \mathbb{R}^{N \times D}$ are partially masked: $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$0 patches are replaced by learnable parameters initialized to zero, producing masked image patches $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$1, with initial mask ratio $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$2. These masked images are aligned with short text $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$3 (Wang et al., 14 Jul 2025).

The rationale is that short captions usually encode coarse global semantics and do not require full visual detail. The short branch therefore acts both as supervision and as a regularizer that preserves the original CLIP-like global alignment space. By contrast, the long branch aligns raw images $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$4 with long text $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$5, since detailed captions require access to fine-grained visual evidence.

The regional prompts are coupled to a unidirectional mask. The paper states that the [CLS] token attends to everything; each regional prompt $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$6 attends only to itself and to the patch embeddings in its assigned image region; and patch embeddings focus on the non-regional-prompt partition. The regional mask for prompt $PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$7 is

$PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$8

where

$PE_l = Concat(PE[:20], Intpol(PE[20:],4)),$9

The masked self-attention is written as

$$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$0

The equation has a formatting issue in the paper, but the intended mechanism is explicit: attention is constrained so that prompts summarize local regions without corrupting the patch stream (Wang et al., 14 Jul 2025).

This design is important because it differs from generic prompt-token insertion. The prompts do not behave as free global adapters; they are localized, masked, and refreshed layer by layer. A plausible implication is that FIX-CLIP treats long-text understanding as a grounding problem, not merely a context-length problem.

4. Hierarchical feature alignment

FIX-CLIP also modifies the supervision depth. Instead of aligning only the final image and text embeddings, it aligns grouped intermediate features from deeper portions of both encoders. Let $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$1 denote the image [CLS] token at layer $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$2. If there are $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$3 Transformer blocks, the layers are divided into $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$4 groups, each of size

$$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$5

Within each group $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$6, Group Tokens Aggregation is defined as

$$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$7

The resulting aggregated feature is projected and normalized: $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$8 These Group Middle Features are then aligned between image and text (Wang et al., 14 Jul 2025).

The group-level alignment uses a symmetric InfoNCE objective. For group $$Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.$$9,

$l$0

The total loss is

$l$1

The paper reports that with 6 groups, the best setting aligns groups 4 through 6, with weights 0.2, 0.4, and 0.8 respectively. It also states that deeper groups are more semantically stable. This indicates that FIX-CLIP does not attempt to enforce shallow cross-modal correspondence uniformly; rather, it emphasizes later intermediate abstractions where image and text representations are more compatible (Wang et al., 14 Jul 2025).

This hierarchical module places FIX-CLIP in a broader family of methods that try to recover structure lost by purely global CLIP alignment. Related work such as DCSM retains token-patch topology by replacing global cosine scoring with dense similarity maps (Kang et al., 10 Mar 2025), whereas FIX-CLIP retains the CLIP-style retrieval objective but injects additional intermediate consistency. The two approaches address different bottlenecks: one changes the scoring geometry, the other changes the training supervision depth.

5. Synthetic caption corpus and training configuration

A central component of FIX-CLIP is synthetic data construction. The paper builds three synthetic long-caption datasets: 5M, 15M, and 30M images. The 30M corpus consists of cleaned subsets from CC3M, VisualGenome, ShareGPT4V, SBU, CC12M, and YFCC15M, with exact cleaned counts reported as 2,760,314, 107,653, 1,246,901, 835,333, 8,523,767, and 14,994,664 respectively (Wang et al., 14 Jul 2025).

Long captions are synthesized with Llama3-LLaVA-NeXT-8b using 20 different prompting templates. The paper explicitly avoids a single generic “describe the image in detail” formulation and instead uses prompts encouraging spatial relationships, composition, atmosphere, foreground/background, lighting, weather, object sizes, direction and movement, emotion and narrative, texture and material, and perspective. The reported average caption lengths show the scale of the long-text regime: raw captions average around 18 tokens, while synthesized long captions average around 120 tokens. Dataset-specific averages include 116.96 for CC3M, 117.68 for VisualGenome, 172.94 for ShareGPT4V, 110.33 for SBU, 131.13 for CC12M, and 115.38 for YFCC15M (Wang et al., 14 Jul 2025).

For the short branch, FIX-CLIP uses synthetic short captions generated by Shikra. The paper reports that these synthetic short captions outperform raw short captions in training. On 5M training, for example, ViT-B/16 long-text average retrieval improves from 87.5 to 87.9, ViT-B/16 short-text average R@1 improves from 68.0 to 69.2, and ViT-L/14 short-text average R@1 improves from 71.2 to 72.1 (Wang et al., 14 Jul 2025).

Implementation details are also explicit. FIX-CLIP uses ViT-B/16 and ViT-L/14, image resolution 224 × 224, text length 248 tokens, batch size 2048, 6 epochs, AdamW, learning rate 1e-6, warmup steps 200, cosine scheduler, $l$2, $l$3, $l$4, and weight decay 1e-2, trained on 16 × A800 GPUs (Wang et al., 14 Jul 2025).

6. Empirical performance, downstream use, and relation to neighboring work

The main empirical claim is that FIX-CLIP improves both long-text and short-text retrieval. On 1M ShareGPT4V training, ViT-B/16 improves from 76.8 average retrieval for Long-CLIP to 82.6 for FIX-CLIP, and ViT-L/14 improves from 79.2 to 85.8 (Wang et al., 14 Jul 2025). At larger scale, ViT-B/16 trained on 30M reaches average 89.8, and ViT-L/14 reaches 91.2, with dataset-specific results including DCI 72.0 / 74.2, IIW 97.9 / 98.2, ShareGPT4V-1k 99.0 / 98.3, and Urban-1k 93.7 / 96.3 for the larger backbone (Wang et al., 14 Jul 2025).

On short-text retrieval, the paper emphasizes non-degradation. For ViT-B/16, 1M, Long-CLIP average R@1 is 64.6, whereas FIX-CLIP reaches 67.9; for ViT-L/14, 1M, Long-CLIP is 68.3 and FIX-CLIP is 70.5. At 30M, average R@1 becomes 70.4 for ViT-B/16 and 73.2 for ViT-L/14 (Wang et al., 14 Jul 2025). Zero-shot classification also improves: average performance rises from 61.5 for CLIP and 62.7 for Long-CLIP to 64.4 for FIX-CLIP on ViT-B/16, and from 66.0 and 68.3 to 71.4 on ViT-L/14 (Wang et al., 14 Jul 2025).

The ablations attribute the largest gains to the dual-branch pipeline. On ViT-L/14 with ShareGPT4V, the DB only variant already yields DCI 60.5 / 61.4 and IIW 94.0 / 95.2, indicating that the short/long separation is foundational. The best overall performance comes from combining the dual branch, hierarchical alignment, and regional prompts with the proposed mask. The prompt ablation shows that 4 prompts with UM reach DCI 65.1 / 66.7 and COCO 63.4 / 46.5, outperforming both no-prompt and higher-prompt settings (Wang et al., 14 Jul 2025).

The paper also reports a downstream application: replacing the CLIP-L text encoder in Stable Diffusion XL with Fix-CLIP-L as a plug-and-play text encoder. Qualitatively, FIX-CLIP is said to better capture long-prompt details such as object orientation, material, color, background elements, interactions, positions, quantity, and shooting angle. The paper notes that an image-to-image refiner is used because the new text encoder differs from the original one (Wang et al., 14 Jul 2025).

FIX-CLIP should also be distinguished from unrelated CLIP modifications that share the vocabulary of “fixing CLIP.” For example, IsoCLIP improves intra-modal tasks such as image-image and text-text retrieval by filtering projector directions through a shared aligned subspace (Magistri et al., 20 Mar 2026), while FIX-CLIP improves long-text image-text understanding through longer positional embeddings, dual-branch training, and regional-hierarchical alignment (Wang et al., 14 Jul 2025). A plausible implication is that these methods target orthogonal failure modes of CLIP and could, in principle, be complementary rather than competing.

In summary, FIX-CLIP is defined by a specific synthesis of architectural and data-centric interventions: it extends the text context window to 248 tokens, preserves short-text behavior through masked-image short-text training, injects localized visual summaries via regional prompts with unidirectional masks, aligns deeper intermediate representations with hierarchical losses, and scales training through synthetic long captions over 30M images. Within the scope defined by the paper—long-text retrieval and related downstream uses—it functions as a CLIP variant optimized for detailed, high-token, semantically dense multimodal alignment (Wang et al., 14 Jul 2025).