---
title: 'MixBench: Benchmark for Mixed Modality Search'
url: https://www.emergentmind.com/topics/mixbench
type: topic
---

# MixBench: Benchmark for Mixed Modality Search

Searching arXiv for MixBench-related papers to ground the article.
MixBench is a benchmark for **mixed modality search**, defined as retrieving semantically relevant information from a heterogeneous corpus in which documents may be **text-only, image-only, or multimodal**, and in which the query itself may also differ in modality from the document. In formal terms, the corpus is written as $\mathcal{C}=\{d_i\}_{i=1}^N$, where each document has a modality set $m_i \subseteq \mathcal{M}$, and the retrieval system must compute $s(q,d_i)$ and rank by semantic relevance “regardless of how the modalities are distributed across queries and documents” [2507.19054]. The benchmark was introduced as “the first benchmark specifically designed for mixed modality search,” and serves both as an evaluation suite and as a diagnostic setting for exposing modality-dependent failures in contrastive vision-language retrieval models [2507.19054].

## 1. Problem formulation and retrieval setting

MixBench targets a retrieval regime that is more general than ordinary text-to-text or text-to-image retrieval. The motivating scenario is a realistic search engine in which a query such as “Mountain Fuji” may need to retrieve text pages, standalone images, and pages that combine image and text, all within the same ranked list [2507.19054]. The benchmark therefore treats mixed modality retrieval as a setting that requires both **cross-modal alignment** and **multimodal fusion** in a single system.

The retrieval model embeds queries and documents into a shared space and scores them with cosine similarity:

$$
s(q, d_i) = \frac{e_q \cdot e_i}{\|e_q\| \cdot \|e_i\|}.
$$

This formulation is standard, but MixBench uses it to test a nonstandard corpus condition: the retrieval space contains heterogeneous document types simultaneously rather than a single clean modality partition [2507.19054]. A plausible implication is that benchmark difficulty arises not only from semantic matching, but also from whether the representation space remains well-calibrated across modality boundaries.

## 2. Benchmark construction and data sources

MixBench is built from four real-world multimodal datasets: **Google WIT**, **MSCOCO**, **OVEN**, and **VisualNews**. Its documents can be **image-only**, **text-only**, or **image-text pairs**, and the document types are sampled in a **$1\!:\!1\!:\!1$ ratio** to keep the distribution balanced [2507.19054]. In the appendix, the evaluation sets are listed as Google WIT, OVEN, MSCOCO, and VisualNews, with queries and documents spanning $T$, $I$, and $I+T$.

| Dataset | Query construction | Document construction |
|---|---|---|
| Google WIT | Page title + image reference description | Page description as text; associated image as image document |
| MSCOCO | One caption | Longer text document from remaining captions via GPT; plus image |
| VisualNews | Original caption | GPT-expanded descriptive paragraph as text; plus image |
| OVEN | Image-text pair | Both image and caption treated as valid standalone documents |

This construction is designed to represent the full mixed-modality setting rather than a single subproblem. The benchmark does not restrict evaluation to text-query/image-document or image-query/text-document cases in isolation; instead, it places heterogeneous query and document types into a common retrieval task [2507.19054]. This suggests that MixBench is structured to test whether a system can rank across modality boundaries without collapsing into modality-specific preference patterns.

## 3. Evaluation protocol and relevance metrics

MixBench uses **NDCG@10** as its primary evaluation metric. The appendix defines it in the standard form:

$$
\text{NDCG@}K = \frac{1}{\text{IDCG@}K} \sum_{i=1}^{K} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)},
$$

with

$$
\text{IDCG@}K = \sum_{i=1}^{K} \frac{2^{\text{rel}_i^\star} - 1}{\log_2(i + 1)}.
$$

The paper also reports robustness to **NDCG@100** and **Recall@1** in the appendix, but NDCG@10 is the main metric used throughout [2507.19054].

The emphasis on ranking quality rather than only top-1 retrieval is consequential for this setting. Mixed modality search can fail even when a model retrieves some relevant items, if modality bias systematically pushes semantically superior cross-modal matches below weaker same-modality items. MixBench is therefore structured to make ranking distortions visible, not merely to test pairwise matching.

## 4. Modality gap diagnosis

A major contribution associated with MixBench is the diagnosis of a **modality gap** in CLIP-style contrastive vision-language models. Although CLIP is trained to align paired images and texts, the paper reports that image embeddings and text embeddings still form distinct clusters in the embedding space and remain far apart [2507.19054]. On MixBench, this gap becomes visible through two concrete failure modes.

The first is **intra-modal ranking bias**. Similarities between items of the same modality are systematically higher than similarities across modalities, so a text query may prefer unrelated text documents over relevant images simply because they are text. The paper illustrates this with a “Mountain Fuji” text query that can rank an unrelated text snippet above an image that actually depicts Mount Fuji [2507.19054].

The second is **inter-modal fusion failure**. When multimodal documents are represented by linearly combining image and text features, the fused embedding can land in a suboptimal region of the space and become worse than using either modality alone. In the multimodal-document experiments, intermediate fusion weights often peak at the unimodal endpoints $\alpha=0$ or $\alpha=1$, which is presented as evidence that naïve fusion is harmed by the modality gap [2507.19054].

MixBench also supports a more striking behavioral diagnosis through heterogeneous-corpus perturbations. The paper replaces text documents with screenshots or with paired images while preserving semantic content. If cross-modal alignment were perfect, retrieval performance should remain stable. Instead, CLIP exhibits a **U-shaped curve**: performance drops as more text documents are replaced with screenshots, reaches a minimum near $p=0.99$, and rises again when the corpus becomes entirely screenshots at $p=1$. One reported curve falls from **0.22** at $p=0$ to **0.02** at $p=0.99$, then rises to **0.36** at $p=1$ [2507.19054]. The paper further validates this interpretation with a push-down simulation that artificially assigns zero similarity to screenshots and reproduces the same U-shape. The benchmark therefore exposes the modality gap through ranking behavior, not only through embedding geometry.

## 5. GR-CLIP and empirical results on MixBench

To address the modality gap revealed on MixBench, the paper proposes **GR-CLIP**, where GR stands for **gap-removed**. It is explicitly presented as a **lightweight post-hoc calibration method** rather than a new encoder or a retrained model [2507.19054]. The key premise, drawn from prior work cited in the paper, is that the modality gap can be approximated as a constant vector orthogonal to the shared embedding subspace, with paired image and text embeddings satisfying

$$
e_i^T - e_i^I \approx c_\perp.
$$

GR-CLIP estimates and removes this gap by mean-centering modality-specific embeddings:

$$
e_i^{\prime T} = e_i^T - \mathbb{E}_i[e_i^T], \quad e_i^{\prime I} = e_i^I - \mathbb{E}_i[e_i^I],
$$

so that

$$
e_i^{\prime T} - e_i^{\prime I} = (e_i^T - e_i^I) - (\mathbb{E}_i[e_i^T] - \mathbb{E}_i[e_i^I]) \approx c_\perp - c_\perp = 0.
$$

For multimodal documents, the same interpolation used by CLIP is applied to calibrated embeddings:

$$
e_{d_j} \gets \alpha f^T(d_j^T) +(1{-}\alpha) f^I(d_j^I) - [\alpha \bar{e}^T + (1{-}\alpha)\bar{e}^I].
$$

The method requires only a single pass to estimate means and adds negligible inference overhead [2507.19054].

On MixBench, the reported gains are substantial. The abstract and main text state that GR-CLIP improves **NDCG@10 by up to 26 percentage points over CLIP**, and that it surpasses the vision-language generative embedding baseline **VLM2Vec** by **4 percentage points overall** while using **75$\times$ less compute** [2507.19054]. The appendix provides the following CLIP-L/14 versus GR-CLIP-L/14 comparison:

| Subset | CLIP-L/14 | GR-CLIP-L/14 |
|---|---:|---:|
| Google WIT | 0.505 | 0.648 |
| MSCOCO | 0.426 | 0.656 |
| OVEN | 0.389 | 0.465 |
| VisualNews | 0.596 | 0.754 |

Similar gains are reported for **OpenCLIP** and **SigLIP** variants. The appendix notes one exception: **MSCOCO**, where VLM2Vec was trained on the dataset, which helps it there. All experiments were inference-only on a **single NVIDIA A100 GPU** [2507.19054]. A plausible implication is that MixBench is useful not only for ranking model quality, but also for comparing efficiency-oriented calibration methods against heavier generative embedding approaches.

## 6. Scope, significance, and nomenclature

The broader implication drawn from MixBench is that **mixed modality retrieval should be treated as its own retrieval setting**, not as a simple extension of text-image matching. The benchmark shows that a unified retrieval system must handle **heterogeneous corpora** and **multimodal documents** simultaneously, and that success depends on more than alignment on paired image-text data during pretraining [2507.19054]. In this sense, MixBench functions both as a diagnostic benchmark and as a practical evaluation suite for future work on unified multimodal embeddings.

The name **MixBench** is, however, not unique across contemporary arXiv usage. In a separate context, “mixbench” appears as a **GPU microbenchmark** used to evaluate “the performance limits of GPUs under mixed operational intensity kernels” in a case study of the NVIDIA **CMP 170HX**. There it reports metrics such as **Compute iters**, **Flops/byte**, **Execution time**, **GFLOPS**, **GB/sec**, and **Iops/byte**, and is used to compare default compilation with **FMA-disabled** compilation [2505.03782]. That benchmark is unrelated to mixed modality search.

It should also be distinguished from **CodeMixBench**, a benchmark for **code generation from code-mixed prompts** built on **BigCodeBench**, with **1,140 unique BigCodeBench tasks** expanded into **6,840 total prompts** across **Hinglish**, **Spanish-English**, and **Chinese Pinyin-English** settings [2505.05063]. Likewise, other papers discuss **MixBench-style** environments or methodologies without naming their systems MixBench: **Bencher** is presented as a modular benchmarking framework for black-box optimization that could conceptually serve as a MixBench-style benchmark environment [2505.21321], while **Mess** is described as very close in spirit to a MixBench-style memory benchmark but is centered on a family of bandwidth–latency curves for memory-system characterization [2405.10170].

A common misconception is therefore to treat “MixBench” as a single benchmark family across domains. The literature instead uses the label in multiple, domain-specific ways. In current retrieval research, the term refers most specifically to the heterogeneous-corpus search benchmark introduced for studying modality gaps and calibration in mixed modality search [2507.19054].

Source: https://www.emergentmind.com/topics/mixbench