GR-CLIP: Calibrated Mixed-Modality Retrieval

• GR-CLIP is a lightweight post-hoc calibration method that removes modality-specific offsets from CLIP embeddings to enhance mixed modality search.
• It improves NDCG@10 by up to 26 percentage points over baseline CLIP by employing simple mean-centering of image and text embeddings without retraining.
• The technique mitigates intra-modal ranking bias and fusion failures in heterogeneous retrieval, offering a practical solution for text, image, and multimodal documents.

GR-CLIP, short for Gap-Removed CLIP, is a lightweight post-hoc calibration method for mixed modality search, a retrieval setting in which a corpus contains text-only documents, image-only documents, and multimodal documents. It was introduced to address a pronounced modality gap in contrastive vision-LLMs such as CLIP: image and text embeddings occupy distinct regions of the embedding space, which produces intra-modal ranking bias and causes simple image-text fusion to fail in heterogeneous retrieval. GR-CLIP removes this offset by modality-specific mean-centering at inference time, without retraining the base encoder. Evaluated on MixBench, it improves NDCG@10 by up to 26 percentage points over CLIP, exceeds recent vision-language generative embedding models by 4 percentage points, and does so with 75x less compute (Li et al., 25 Jul 2025).

1. Retrieval setting and formal problem definition

Mixed modality search is retrieval over a heterogeneous corpus in which each document $d_i$ has a modality set $m_i \subseteq \mathcal{M}$, with $\mathcal{M}$ typically containing text and image. Some documents are unimodal, with $|m_i|=1$, and others are multimodal, with $|m_i|>1$. Queries may in principle be multimodal as well, although the reported setting is mostly text-query retrieval. The core requirement is that ranking be meaningful across document modalities rather than merely within each modality class (Li et al., 25 Jul 2025).

In a standard CLIP-style system, text and image encoders map inputs into a nominally shared space: $$f^T: \text{text} \to \mathbb{R}^d, \qquad f^I: \text{image} \to \mathbb{R}^d.$$ A text-only document is embedded as $e_i=f^T(d_i)$, an image-only document as $e_i=f^I(d_i)$, and a multimodal document through linear fusion,

$$e_i=\alpha f^T(d_i^T)+(1-\alpha)f^I(d_i^I), \qquad \alpha \in [0,1].$$

Queries are encoded analogously; in the primary experimental setting, $e_q=f^T(q)$. Retrieval then uses cosine similarity,

$m_i \subseteq \mathcal{M}$0

This formulation is natural for web search, product search, and news retrieval, where relevant items may be textual pages, standalone images, or combined image-text artifacts. A central premise behind GR-CLIP is that the apparent simplicity of this shared-space formulation conceals a geometric defect in the embedding space that systematically distorts ranking when corpora mix modalities.

2. Modality gap and failure modes in CLIP retrieval

The motivating diagnosis is that CLIP’s embedding space is not fully unified. Image embeddings and text embeddings form distinct clusters that lie in approximately parallel subspaces separated by a nearly constant offset vector. For a paired image-text example $m_i \subseteq \mathcal{M}$1, with embeddings

$m_i \subseteq \mathcal{M}$2

the paper adopts the approximation

$m_i \subseteq \mathcal{M}$3

where $m_i \subseteq \mathcal{M}$4 is a modality-gap vector that is approximately orthogonal to the shared semantic subspace (Li et al., 25 Jul 2025).

This offset has two direct consequences. The first is intra-modal ranking bias. Because same-modality similarities are systematically larger than cross-modality similarities, irrelevant text can outrank relevant images for text queries, and vice versa. The paper illustrates this with a synthetic screenshot benchmark, where varying the proportion $m_i \subseteq \mathcal{M}$5 of screenshots in the corpus yields a characteristic U-shaped performance curve: retrieval quality degrades in the mixed regime, collapses when only a few text items remain among mostly images, and recovers again when the corpus becomes entirely image-based. The same analysis is reproduced by a “push-down simulation” that artificially suppresses image-document similarities, which supports the claim that the dominant failure is geometric rather than semantic.

The second consequence is inter-modal fusion failure for multimodal documents. If a multimodal embedding is formed by linear interpolation between text and image embeddings,

$m_i \subseteq \mathcal{M}$6

then interpolation traverses a region poorly supported by training samples when the two modalities lie in offset subspaces. Empirically, vanilla CLIP typically reaches its best retrieval performance at $m_i \subseteq \mathcal{M}$7 or $m_i \subseteq \mathcal{M}$8, meaning that fused representations fail to improve over the best unimodal endpoint. A common misconception is therefore that CLIP’s contrastive training alone guarantees reliable mixed-modality retrieval; GR-CLIP is motivated precisely by the observation that this is false in heterogeneous corpora.

3. GR-CLIP as post-hoc geometric realignment

GR-CLIP addresses the modality gap through a pure inference-time translation of embeddings. The method does not retrain CLIP, alter the encoder architecture, or introduce a new scoring function. Instead, it computes modality- and role-specific mean vectors over calibration data and subtracts them from query and document embeddings before cosine similarity is evaluated (Li et al., 25 Jul 2025).

The paper first gives the conceptual version. If

$m_i \subseteq \mathcal{M}$9

then calibrated embeddings can be defined as

$\mathcal{M}$0

Under the constant-offset model,

$\mathcal{M}$1

The modality gap is thus removed by centering each modality around its own empirical mean.

The implemented version distinguishes queries from documents, because query texts and document texts follow different distributions. Three mean vectors are therefore estimated: $\mathcal{M}$2 The calibrated embeddings are

$\mathcal{M}$3

for a text query,

$\mathcal{M}$4

for a text-only document,

$\mathcal{M}$5

for an image-only document, and

$\mathcal{M}$6

for a multimodal document. Retrieval then uses

$\mathcal{M}$7

The required supervision is minimal. GR-CLIP needs only a pretrained CLIP-like model and unlabeled calibration sets $\mathcal{M}$8 and $\mathcal{M}$9; no relevance judgments or task-specific contrastive retraining are used. In the reported setup, mean vectors are estimated from roughly 10k queries, 10k document texts, and 10k document images drawn from multiple training splits, with additional small calibration sets in some modalities. Once computed, the mean vectors are fixed and reused, so the incremental inference cost is a single vector subtraction per embedding.

4. Benchmarking, empirical behavior, and computational profile

To evaluate mixed-modality retrieval directly, the paper introduces MixBench, which is built from Google WIT, MSCOCO, OVEN, and VisualNews. Documents are sampled in a 1:1:1 ratio across text-only, image-only, and image-plus-text forms, and queries are mostly textual. The principal evaluation metric is NDCG@10, with NDCG@100 and Recall@1 reported in the appendix (Li et al., 25 Jul 2025).

Across OpenAI CLIP, OpenCLIP, and SigLIP variants, GR-CLIP consistently improves retrieval quality. The reported headline result is an improvement of up to 26 percentage points in NDCG@10 over the corresponding base CLIP model. The geometric interpretation is corroborated by several empirical signatures. First, the U-shaped performance curve observed under varying modality mixtures becomes approximately flat after calibration, which indicates that cross-modal items are no longer systematically penalized. Second, for multimodal documents, the best fusion weight $|m_i|=1$0 typically moves from the unimodal endpoints toward the interior of $|m_i|=1$1, which means that combined image-text evidence becomes useful after gap removal.

The method is also compared with VLM2Vec, a generative embedding framework that produces embeddings from large vision-LLMs through autoregressive decoding. Averaged across MixBench subsets, GR-CLIP surpasses VLM2Vec by about 4 NDCG@10 points while using 75x less compute. The computational asymmetry is structural: VLM2Vec requires autoregressive token generation, whereas GR-CLIP uses an ordinary CLIP forward pass followed by mean subtraction.

The reported behavior is not confined to image-text retrieval. The paper further examines analogous modality-mixing phenomena in text-video retrieval with ViCLIP on MSVD, text-audio retrieval with CLAP on Clotho, and additional settings such as Nights. In each case, the original model exhibits a modality-dependent bias and the same mean-centering strategy flattens the mixture curve and improves retrieval, which suggests that the calibration principle is not specific to one CLIP checkpoint or one benchmark.

5. Position within the CLIP literature and naming ambiguities

GR-CLIP addresses a specific problem: the geometric misalignment of modality clusters in mixed modality search. It is therefore distinct from other CLIP-derived methods that happen to involve robustness, generalization, or modality-gap language but target different tasks.

In domain generalization, GOPro combines prompt learning with self-supervised learning, using learnable image and text projectors, a visual contrastive loss, and a prompt consistency loss to make a frozen CLIP backbone more invariant across augmentations and domains (Singha et al., 2023). In open-vocabulary detection, GridCLIP learns grid-level CLIP representations for one-stage object detection and explicitly notes that its official abbreviation is GridCLIP, not GR-CLIP (Lin et al., 2023). In class-incremental learning, GR4CIL studies modality-gap-induced bias in a different sense, introducing task-specific visual LoRA modules, an orthogonal compensation mechanism, and gap-compensated routing for CLIP-based CIL (Wang et al., 20 Apr 2026).

A related misconception is that any method named with “GR” and “CLIP” concerns the same object. In fact, the acronym is overloaded across subfields. Within mixed-modality retrieval, GR-CLIP denotes Gap-Removed CLIP; it is neither a prompt-learning adaptation method nor a dense detection architecture nor a class-incremental routing framework. Its defining operation is the removal of a constant, modality-specific offset through role-aware mean-centering.

6. Limitations, scope conditions, and future directions

The paper identifies several limitations. The document model is restricted to at most one image and one text segment per item, whereas real web pages, PDFs, and scientific documents can contain multiple images and interleaved text blocks. The method is also designed for contrastive dual-encoder architectures such as CLIP, OpenCLIP, SigLIP, ViCLIP, and CLAP; generative embedding models may exhibit different geometric pathologies and may require different corrections (Li et al., 25 Jul 2025).

GR-CLIP also does not change CLIP’s intrinsic cross-modal reasoning capacity. It removes a geometric deficiency in the retrieval space, but it does not add token-level or region-level interaction between modalities. Consequently, it is best understood as a calibration method rather than an architectural alternative to multimodal transformers or generative retrievers. The expected gains are largest in genuinely mixed-modality corpora; for single-modality retrieval, the modality gap is less consequential.

Future directions proposed or implied in the paper include extending calibration and evaluation to richer interleaved documents, analyzing whether analogous modality gaps exist in generative embedding models, and studying the training dynamics that create the offset in the first place. A broader implication is that a substantial fraction of CLIP’s failure in heterogeneous retrieval may be attributable not to missing semantics, but to a removable global translation of modality-specific clusters. On that interpretation, GR-CLIP is significant less as a new encoder than as a demonstration that mixed-modality retrieval quality can be strongly constrained by simple embedding geometry.