---
title: 'FIX-CLIP: Long-Text Visual-Language Alignment'
url: https://www.emergentmind.com/topics/fix-clip
type: topic
---

# FIX-CLIP: Long-Text Visual-Language Alignment

Searching arXiv for the FIX-CLIP paper and closely related CLIP-extension work on long-text understanding.
FIX-CLIP is a CLIP extension for long-text visual-language alignment that targets a specific limitation of standard CLIP: strong zero-shot performance on short captions, but weak performance on long-text inputs because the original text encoder is capped at **77 tokens** and because final-layer global alignment is not well suited to detailed, compositional descriptions [2507.10095]. The method is designed to improve long-text retrieval while preserving short-text retrieval and zero-shot classification, and it does so through three coupled components: a **dual-branch training pipeline**, **multiple learnable regional prompts with unidirectional masks** in the image Transformer, and a **hierarchical feature alignment module** over intermediate encoder layers. Training relies on synthetic long captions generated for a **30M-image corpus** with **Llama3-LLaVA-NeXT-8b**, while short-caption supervision is also retained to stabilize compatibility with pretrained CLIP representations [2507.10095].

## 1. Definition and problem scope

FIX-CLIP addresses a long-standing asymmetry in CLIP-style models: they are highly effective for short text-image matching, yet degrade when the text becomes long, descriptive, and semantically dense. The paper defines the practical regime as long-text inputs **\(>77\) tokens**, a range in which standard CLIP suffers from both hard truncation and a representational bias toward short global summaries [2507.10095].

The motivating use cases are long-text image-text retrieval, multimodal large language models, and text-to-image generation. In these settings, the text may contain object counts, materials, colors, positions, scene layout, and interactions that exceed the capacity of the original CLIP text pathway. FIX-CLIP is therefore not presented as a generic retraining of CLIP, but as an incremental extension intended to preserve CLIP’s short-text strengths while improving paragraph-level alignment.

A useful distinction emerges when FIX-CLIP is placed alongside other CLIP “fixes.” Some works diagnose failures of CLIP geometry or composition in general and propose alternative scoring mechanisms or subspace filtering [2503.08723], whereas FIX-CLIP is narrowly focused on long-text understanding and explicitly aims to avoid the short-text degradation observed in prior long-context CLIP variants [2507.10095]. This suggests that FIX-CLIP is best understood as a task-specific extension rather than a universal repair of CLIP semantics.

## 2. Architectural design

FIX-CLIP consists of three modules that operate jointly during training. The first is a **dual-branch training pipeline**. One branch aligns **short text** with **masked images**, while the other aligns **long text** with **raw images**. The second inserts **regional prompts** into the image Transformer, together with **unidirectional masks** that make those prompts act as localized feature extractors. The third aligns intermediate image and text features through **hierarchical feature alignment** rather than relying only on the final embedding [2507.10095].

On the text side, the encoder remains a vanilla Transformer, but its positional embeddings are extended from **77** to **248** tokens. The paper keeps pretrained Transformer block parameters and preserves the original CLIP positional structure for short texts. For long texts, the first **20 positional embeddings** are frozen, and the remaining positions are interpolated by a factor of **4**:
\[
PE_l = Concat(PE[:20], Intpol(PE[20:],4)),
\]
with
\[
Intpol(PE, q)[i] =(1-\lambda) * PE[\lfloor \frac{i}{q} \rfloor] + \lambda * PE[\lfloor \frac{i}{q} \rfloor +1], \quad \lambda = \frac{i\%q}{q}.
\]
The paper’s interpretation is that the first roughly 20 tokens often summarize the image’s main content, so preserving them helps maintain compatibility with pretrained CLIP behavior [2507.10095].

The image encoder is enriched with **multiple learnable regional prompts**. At Transformer layer \(l\), the token sequence becomes
\[
([CLS], R^l_1,\cdots, R^l_M, P_1,\cdots, P_N),
\]
where \(R_j^l\) are regional prompts and \(P_i\) are patch embeddings. After multi-head self-attention, the prompts are replaced before the next layer:
\[
\mathbf{X}^l[1:1+M] = (R^{l+1}_1,\cdots,R^{l+1}_M).
\]
This per-layer replacement is meant to prevent prompt states from accumulating interference across depth and to keep them focused on local features in the current layer [2507.10095].

The paper identifies **\(M=4\)** prompts as the best setting in its ablation. This suggests that a small number of regional extractors is sufficient to improve long-text grounding without excessively fragmenting the representation.

## 3. Dual-branch training and regional prompt mechanics

The dual-branch pipeline is the core training strategy. In the short branch, image patch embeddings \(I \in \mathbb{R}^{N \times D}\) are partially masked: \(\alpha \times N\) patches are replaced by learnable parameters initialized to zero, producing masked image patches \(I_m\), with initial mask ratio **\(\alpha = 0.75\)**. These masked images are aligned with **short text** \(T_s\) [2507.10095].

The rationale is that short captions usually encode coarse global semantics and do not require full visual detail. The short branch therefore acts both as supervision and as a regularizer that preserves the original CLIP-like global alignment space. By contrast, the long branch aligns **raw images** \(I\) with **long text** \(T_l\), since detailed captions require access to fine-grained visual evidence.

The regional prompts are coupled to a **unidirectional mask**. The paper states that the **[CLS] token** attends to everything; each **regional prompt \(R_j\)** attends only to itself and to the patch embeddings in its assigned image region; and **patch embeddings** focus on the non-regional-prompt partition. The regional mask for prompt \(R_j\) is
\[
\mathbf{Mask}[R_j] = \mathbbm{1}(j,b_j,\cdots,b_j+\lfloor \frac{N}{M} \rfloor -1),
\]
where
\[
b_j = 1 + M + j \times \lfloor \frac{N}{M} \rfloor.
\]
The masked self-attention is written as
\[
\mathbf{X}^{l+1} = \mathcal{T}^{l+1}_{\text{MHSA}(\mathbf{X}^l) = \text{softmax}(\frac{QK^T}{\sqrt{d} \odot \mathbf{Mask})V.
\]
The equation has a formatting issue in the paper, but the intended mechanism is explicit: attention is constrained so that prompts summarize local regions without corrupting the patch stream [2507.10095].

This design is important because it differs from generic prompt-token insertion. The prompts do not behave as free global adapters; they are localized, masked, and refreshed layer by layer. A plausible implication is that FIX-CLIP treats long-text understanding as a grounding problem, not merely a context-length problem.

## 4. Hierarchical feature alignment

FIX-CLIP also modifies the supervision depth. Instead of aligning only the final image and text embeddings, it aligns grouped intermediate features from deeper portions of both encoders. Let \(T_l = \mathbf{X}^l[0]\) denote the image [CLS] token at layer \(l\). If there are \(L\) Transformer blocks, the layers are divided into \(G\) groups, each of size
\[
S = L/G.
\]
Within each group \(\mathbf{T}_g\), Group Tokens Aggregation is defined as
\[
GTA(\mathbf{T}_g) =\sum_{j=1}^{S}  \mathbf{Gaussian}(j;S,1) * \mathbf{T}_g[j].
\]
The resulting aggregated feature is projected and normalized:
\[
GMF_g= LN(Proj(GTA(\mathbf{T}_g))).
\]
These **Group Middle Features** are then aligned between image and text [2507.10095].

The group-level alignment uses a symmetric InfoNCE objective. For group \(i\),
\[
\begin{aligned}
L_{m_i} =& -\sum_{j=1}^B \text{log} \frac{\text{exp}(\text{cos} \langle v_i^j, t_i^j \rangle / \tau)}{\sum_{k=1}^{B}\text{exp}(\text{cos} \langle v_i^j, t_i^k \rangle / \tau)} \\
&-\sum_{j=1}^B \text{log} \frac{\text{exp}(\text{cos} \langle t_i^j, v_i^j \rangle / \tau)}{\sum_{k=1}^{B}\text{exp}(\text{cos} \langle t_i^j, v_i^k \rangle / \tau)}.
\end{aligned}
\]
The total loss is
\[
L = \sum_{i=K}^{G} \omega_i L_{m_i} + L_{short} + L_{long}.
\]

The paper reports that with **6 groups**, the best setting aligns **groups 4 through 6**, with weights **0.2**, **0.4**, and **0.8** respectively. It also states that deeper groups are more semantically stable. This indicates that FIX-CLIP does not attempt to enforce shallow cross-modal correspondence uniformly; rather, it emphasizes later intermediate abstractions where image and text representations are more compatible [2507.10095].

This hierarchical module places FIX-CLIP in a broader family of methods that try to recover structure lost by purely global CLIP alignment. Related work such as DCSM retains token-patch topology by replacing global cosine scoring with dense similarity maps [2503.08723], whereas FIX-CLIP retains the CLIP-style retrieval objective but injects additional intermediate consistency. The two approaches address different bottlenecks: one changes the scoring geometry, the other changes the training supervision depth.

## 5. Synthetic caption corpus and training configuration

A central component of FIX-CLIP is synthetic data construction. The paper builds three synthetic long-caption datasets: **5M**, **15M**, and **30M** images. The **30M** corpus consists of cleaned subsets from **CC3M**, **VisualGenome**, **ShareGPT4V**, **SBU**, **CC12M**, and **YFCC15M**, with exact cleaned counts reported as **2,760,314**, **107,653**, **1,246,901**, **835,333**, **8,523,767**, and **14,994,664** respectively [2507.10095].

Long captions are synthesized with **Llama3-LLaVA-NeXT-8b** using **20 different prompting templates**. The paper explicitly avoids a single generic “describe the image in detail” formulation and instead uses prompts encouraging spatial relationships, composition, atmosphere, foreground/background, lighting, weather, object sizes, direction and movement, emotion and narrative, texture and material, and perspective. The reported average caption lengths show the scale of the long-text regime: raw captions average around **18 tokens**, while synthesized long captions average around **120 tokens**. Dataset-specific averages include **116.96** for CC3M, **117.68** for VisualGenome, **172.94** for ShareGPT4V, **110.33** for SBU, **131.13** for CC12M, and **115.38** for YFCC15M [2507.10095].

For the short branch, FIX-CLIP uses synthetic short captions generated by **Shikra**. The paper reports that these synthetic short captions outperform raw short captions in training. On **5M** training, for example, **ViT-B/16** long-text average retrieval improves from **87.5** to **87.9**, **ViT-B/16** short-text average **R@1** improves from **68.0** to **69.2**, and **ViT-L/14** short-text average **R@1** improves from **71.2** to **72.1** [2507.10095].

Implementation details are also explicit. FIX-CLIP uses **ViT-B/16** and **ViT-L/14**, image resolution **224 × 224**, text length **248 tokens**, batch size **2048**, **6 epochs**, **AdamW**, learning rate **1e-6**, warmup steps **200**, cosine scheduler, \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), \(\epsilon = 1e{-8}\), and weight decay **1e-2**, trained on **16 × A800 GPUs** [2507.10095].

## 6. Empirical performance, downstream use, and relation to neighboring work

The main empirical claim is that FIX-CLIP improves both long-text and short-text retrieval. On **1M ShareGPT4V** training, **ViT-B/16** improves from **76.8** average retrieval for Long-CLIP to **82.6** for FIX-CLIP, and **ViT-L/14** improves from **79.2** to **85.8** [2507.10095]. At larger scale, **ViT-B/16** trained on **30M** reaches average **89.8**, and **ViT-L/14** reaches **91.2**, with dataset-specific results including **DCI 72.0 / 74.2**, **IIW 97.9 / 98.2**, **ShareGPT4V-1k 99.0 / 98.3**, and **Urban-1k 93.7 / 96.3** for the larger backbone [2507.10095].

On short-text retrieval, the paper emphasizes non-degradation. For **ViT-B/16, 1M**, Long-CLIP average **R@1** is **64.6**, whereas FIX-CLIP reaches **67.9**; for **ViT-L/14, 1M**, Long-CLIP is **68.3** and FIX-CLIP is **70.5**. At **30M**, average **R@1** becomes **70.4** for **ViT-B/16** and **73.2** for **ViT-L/14** [2507.10095]. Zero-shot classification also improves: average performance rises from **61.5** for CLIP and **62.7** for Long-CLIP to **64.4** for FIX-CLIP on **ViT-B/16**, and from **66.0** and **68.3** to **71.4** on **ViT-L/14** [2507.10095].

The ablations attribute the largest gains to the **dual-branch pipeline**. On **ViT-L/14** with ShareGPT4V, the **DB only** variant already yields **DCI 60.5 / 61.4** and **IIW 94.0 / 95.2**, indicating that the short/long separation is foundational. The best overall performance comes from combining the dual branch, hierarchical alignment, and regional prompts with the proposed mask. The prompt ablation shows that **4 prompts with UM** reach **DCI 65.1 / 66.7** and **COCO 63.4 / 46.5**, outperforming both no-prompt and higher-prompt settings [2507.10095].

The paper also reports a downstream application: replacing the CLIP-L text encoder in **Stable Diffusion XL** with **Fix-CLIP-L** as a plug-and-play text encoder. Qualitatively, FIX-CLIP is said to better capture long-prompt details such as object orientation, material, color, background elements, interactions, positions, quantity, and shooting angle. The paper notes that an image-to-image refiner is used because the new text encoder differs from the original one [2507.10095].

FIX-CLIP should also be distinguished from unrelated CLIP modifications that share the vocabulary of “fixing CLIP.” For example, IsoCLIP improves **intra-modal** tasks such as image-image and text-text retrieval by filtering projector directions through a shared aligned subspace [2603.19862], while FIX-CLIP improves **long-text** image-text understanding through longer positional embeddings, dual-branch training, and regional-hierarchical alignment [2507.10095]. A plausible implication is that these methods target orthogonal failure modes of CLIP and could, in principle, be complementary rather than competing.

In summary, FIX-CLIP is defined by a specific synthesis of architectural and data-centric interventions: it extends the text context window to **248 tokens**, preserves short-text behavior through masked-image short-text training, injects localized visual summaries via regional prompts with unidirectional masks, aligns deeper intermediate representations with hierarchical losses, and scales training through synthetic long captions over **30M** images. Within the scope defined by the paper—long-text retrieval and related downstream uses—it functions as a CLIP variant optimized for detailed, high-token, semantically dense multimodal alignment [2507.10095].

Source: https://www.emergentmind.com/topics/fix-clip