---
title: 'GR-CLIP: Calibrated Mixed-Modality Retrieval'
url: https://www.emergentmind.com/topics/gr-clip
type: topic
---

# GR-CLIP: Calibrated Mixed-Modality Retrieval

GR-CLIP, short for **Gap-Removed CLIP**, is a lightweight post-hoc calibration method for **mixed modality search**, a retrieval setting in which a corpus contains text-only documents, image-only documents, and multimodal documents. It was introduced to address a pronounced **modality gap** in contrastive vision-language models such as CLIP: image and text embeddings occupy distinct regions of the embedding space, which produces intra-modal ranking bias and causes simple image-text fusion to fail in heterogeneous retrieval. GR-CLIP removes this offset by modality-specific mean-centering at inference time, without retraining the base encoder. Evaluated on **MixBench**, it improves **NDCG@10 by up to 26 percentage points** over CLIP, exceeds recent vision-language generative embedding models by **4 percentage points**, and does so with **75x less compute** [2507.19054].

## 1. Retrieval setting and formal problem definition

Mixed modality search is retrieval over a heterogeneous corpus in which each document \(d_i\) has a modality set \(m_i \subseteq \mathcal{M}\), with \(\mathcal{M}\) typically containing text and image. Some documents are unimodal, with \(|m_i|=1\), and others are multimodal, with \(|m_i|>1\). Queries may in principle be multimodal as well, although the reported setting is mostly text-query retrieval. The core requirement is that ranking be meaningful **across** document modalities rather than merely within each modality class [2507.19054].

In a standard CLIP-style system, text and image encoders map inputs into a nominally shared space:
\[
f^T: \text{text} \to \mathbb{R}^d, \qquad f^I: \text{image} \to \mathbb{R}^d.
\]
A text-only document is embedded as \(e_i=f^T(d_i)\), an image-only document as \(e_i=f^I(d_i)\), and a multimodal document through linear fusion,
\[
e_i=\alpha f^T(d_i^T)+(1-\alpha)f^I(d_i^I), \qquad \alpha \in [0,1].
\]
Queries are encoded analogously; in the primary experimental setting, \(e_q=f^T(q)\). Retrieval then uses cosine similarity,
\[
s(q,d_i)=\frac{e_q \cdot e_i}{\|e_q\|\,\|e_i\|}.
\]

This formulation is natural for web search, product search, and news retrieval, where relevant items may be textual pages, standalone images, or combined image-text artifacts. A central premise behind GR-CLIP is that the apparent simplicity of this shared-space formulation conceals a geometric defect in the embedding space that systematically distorts ranking when corpora mix modalities.

## 2. Modality gap and failure modes in CLIP retrieval

The motivating diagnosis is that CLIP’s embedding space is not fully unified. Image embeddings and text embeddings form distinct clusters that lie in approximately parallel subspaces separated by a nearly constant offset vector. For a paired image-text example \((I_i,T_i)\), with embeddings
\[
e_i^I=f^I(I_i), \qquad e_i^T=f^T(T_i),
\]
the paper adopts the approximation
\[
e_i^T-e_i^I \approx c_\perp,
\]
where \(c_\perp\) is a modality-gap vector that is approximately orthogonal to the shared semantic subspace [2507.19054].

This offset has two direct consequences. The first is **intra-modal ranking bias**. Because same-modality similarities are systematically larger than cross-modality similarities, irrelevant text can outrank relevant images for text queries, and vice versa. The paper illustrates this with a synthetic screenshot benchmark, where varying the proportion \(p\) of screenshots in the corpus yields a characteristic **U-shaped** performance curve: retrieval quality degrades in the mixed regime, collapses when only a few text items remain among mostly images, and recovers again when the corpus becomes entirely image-based. The same analysis is reproduced by a “push-down simulation” that artificially suppresses image-document similarities, which supports the claim that the dominant failure is geometric rather than semantic.

The second consequence is **inter-modal fusion failure** for multimodal documents. If a multimodal embedding is formed by linear interpolation between text and image embeddings,
\[
e_i=\alpha e_i^T + (1-\alpha)e_i^I,
\]
then interpolation traverses a region poorly supported by training samples when the two modalities lie in offset subspaces. Empirically, vanilla CLIP typically reaches its best retrieval performance at \(\alpha=0\) or \(\alpha=1\), meaning that fused representations fail to improve over the best unimodal endpoint. A common misconception is therefore that CLIP’s contrastive training alone guarantees reliable mixed-modality retrieval; GR-CLIP is motivated precisely by the observation that this is false in heterogeneous corpora.

## 3. GR-CLIP as post-hoc geometric realignment

GR-CLIP addresses the modality gap through a pure inference-time translation of embeddings. The method does not retrain CLIP, alter the encoder architecture, or introduce a new scoring function. Instead, it computes modality- and role-specific mean vectors over calibration data and subtracts them from query and document embeddings before cosine similarity is evaluated [2507.19054].

The paper first gives the conceptual version. If
\[
\mu_T=\mathbb{E}_i[e_i^T], \qquad \mu_I=\mathbb{E}_i[e_i^I],
\]
then calibrated embeddings can be defined as
\[
e_i^{\prime T}=e_i^T-\mathbb{E}_i[e_i^T], \qquad e_i^{\prime I}=e_i^I-\mathbb{E}_i[e_i^I].
\]
Under the constant-offset model,
\[
e_i^{\prime T}-e_i^{\prime I}
=(e_i^T-e_i^I)-(\mathbb{E}_i[e_i^T]-\mathbb{E}_i[e_i^I])
\approx c_\perp-c_\perp=0.
\]
The modality gap is thus removed by centering each modality around its own empirical mean.

The implemented version distinguishes **queries** from **documents**, because query texts and document texts follow different distributions. Three mean vectors are therefore estimated:
\[
\bar e_q=\mathbb{E}_{q\sim \mathcal Q'}[f^T(q)], \qquad
\bar e^T=\mathbb{E}_{d^T\sim \mathcal D'_{\text{text}}}[f^T(d^T)], \qquad
\bar e^I=\mathbb{E}_{d^I\sim \mathcal D'_{\text{image}}}[f^I(d^I)].
\]
The calibrated embeddings are
\[
e_q' = f^T(q)-\bar e_q
\]
for a text query,
\[
e_d' = f^T(d)-\bar e^T
\]
for a text-only document,
\[
e_d' = f^I(d)-\bar e^I
\]
for an image-only document, and
\[
e_d' = \alpha f^T(d^T)+(1-\alpha)f^I(d^I)-[\alpha \bar e^T + (1-\alpha)\bar e^I]
\]
for a multimodal document. Retrieval then uses
\[
s(q,d)=\frac{e_q' \cdot e_d'}{\|e_q'\|\,\|e_d'\|}.
\]

The required supervision is minimal. GR-CLIP needs only a pretrained CLIP-like model and unlabeled calibration sets \(\mathcal Q'\) and \(\mathcal D'\); no relevance judgments or task-specific contrastive retraining are used. In the reported setup, mean vectors are estimated from roughly **10k queries**, **10k document texts**, and **10k document images** drawn from multiple training splits, with additional small calibration sets in some modalities. Once computed, the mean vectors are fixed and reused, so the incremental inference cost is a single vector subtraction per embedding.

## 4. Benchmarking, empirical behavior, and computational profile

To evaluate mixed-modality retrieval directly, the paper introduces **MixBench**, which is built from **Google WIT**, **MSCOCO**, **OVEN**, and **VisualNews**. Documents are sampled in a **1:1:1 ratio** across text-only, image-only, and image-plus-text forms, and queries are mostly textual. The principal evaluation metric is **NDCG@10**, with **NDCG@100** and **Recall@1** reported in the appendix [2507.19054].

Across OpenAI CLIP, OpenCLIP, and SigLIP variants, GR-CLIP consistently improves retrieval quality. The reported headline result is an improvement of **up to 26 percentage points** in NDCG@10 over the corresponding base CLIP model. The geometric interpretation is corroborated by several empirical signatures. First, the U-shaped performance curve observed under varying modality mixtures becomes approximately flat after calibration, which indicates that cross-modal items are no longer systematically penalized. Second, for multimodal documents, the best fusion weight \(\alpha\) typically moves from the unimodal endpoints toward the interior of \([0,1]\), which means that combined image-text evidence becomes useful after gap removal.

The method is also compared with **VLM2Vec**, a generative embedding framework that produces embeddings from large vision-language models through autoregressive decoding. Averaged across MixBench subsets, GR-CLIP surpasses VLM2Vec by about **4 NDCG@10 points** while using **75x less compute**. The computational asymmetry is structural: VLM2Vec requires autoregressive token generation, whereas GR-CLIP uses an ordinary CLIP forward pass followed by mean subtraction.

The reported behavior is not confined to image-text retrieval. The paper further examines analogous modality-mixing phenomena in **text-video** retrieval with ViCLIP on **MSVD**, **text-audio** retrieval with CLAP on **Clotho**, and additional settings such as **Nights**. In each case, the original model exhibits a modality-dependent bias and the same mean-centering strategy flattens the mixture curve and improves retrieval, which suggests that the calibration principle is not specific to one CLIP checkpoint or one benchmark.

## 5. Position within the CLIP literature and naming ambiguities

GR-CLIP addresses a specific problem: the geometric misalignment of modality clusters in **mixed modality search**. It is therefore distinct from other CLIP-derived methods that happen to involve robustness, generalization, or modality-gap language but target different tasks.

In domain generalization, **GOPro** combines prompt learning with self-supervised learning, using learnable image and text projectors, a visual contrastive loss, and a prompt consistency loss to make a frozen CLIP backbone more invariant across augmentations and domains [2308.11605]. In open-vocabulary detection, **GridCLIP** learns grid-level CLIP representations for one-stage object detection and explicitly notes that its official abbreviation is **GridCLIP**, not GR-CLIP [2303.09252]. In class-incremental learning, **GR4CIL** studies modality-gap-induced bias in a different sense, introducing task-specific visual LoRA modules, an orthogonal compensation mechanism, and gap-compensated routing for CLIP-based CIL [2604.17822].

A related misconception is that any method named with “GR” and “CLIP” concerns the same object. In fact, the acronym is overloaded across subfields. Within mixed-modality retrieval, **GR-CLIP** denotes **Gap-Removed CLIP**; it is neither a prompt-learning adaptation method nor a dense detection architecture nor a class-incremental routing framework. Its defining operation is the removal of a constant, modality-specific offset through role-aware mean-centering.

## 6. Limitations, scope conditions, and future directions

The paper identifies several limitations. The document model is restricted to at most **one image and one text segment** per item, whereas real web pages, PDFs, and scientific documents can contain multiple images and interleaved text blocks. The method is also designed for **contrastive dual-encoder architectures** such as CLIP, OpenCLIP, SigLIP, ViCLIP, and CLAP; generative embedding models may exhibit different geometric pathologies and may require different corrections [2507.19054].

GR-CLIP also does not change CLIP’s intrinsic cross-modal reasoning capacity. It removes a **geometric deficiency** in the retrieval space, but it does not add token-level or region-level interaction between modalities. Consequently, it is best understood as a calibration method rather than an architectural alternative to multimodal transformers or generative retrievers. The expected gains are largest in genuinely mixed-modality corpora; for single-modality retrieval, the modality gap is less consequential.

Future directions proposed or implied in the paper include extending calibration and evaluation to richer interleaved documents, analyzing whether analogous modality gaps exist in generative embedding models, and studying the training dynamics that create the offset in the first place. A broader implication is that a substantial fraction of CLIP’s failure in heterogeneous retrieval may be attributable not to missing semantics, but to a removable global translation of modality-specific clusters. On that interpretation, GR-CLIP is significant less as a new encoder than as a demonstration that mixed-modality retrieval quality can be strongly constrained by simple embedding geometry.

Source: https://www.emergentmind.com/topics/gr-clip