MixBench: Benchmark for Mixed Modality Search

• MixBench is a benchmark for mixed modality search that evaluates retrieval performance across text-only, image-only, and multimodal documents.
• It constructs a balanced retrieval setting using datasets like Google WIT, MSCOCO, OVEN, and VisualNews to test cross-modal alignment and fusion.
• The benchmark reveals modality gaps in vision-language models and motivates calibration improvements through methods such as GR-CLIP.

Searching arXiv for MixBench-related papers to ground the article. MixBench is a benchmark for mixed modality search, defined as retrieving semantically relevant information from a heterogeneous corpus in which documents may be text-only, image-only, or multimodal, and in which the query itself may also differ in modality from the document. In formal terms, the corpus is written as $\mathcal{C}=\{d_i\}_{i=1}^N$, where each document has a modality set $m_i \subseteq \mathcal{M}$, and the retrieval system must compute $s(q,d_i)$ and rank by semantic relevance “regardless of how the modalities are distributed across queries and documents” (Li et al., 25 Jul 2025). The benchmark was introduced as “the first benchmark specifically designed for mixed modality search,” and serves both as an evaluation suite and as a diagnostic setting for exposing modality-dependent failures in contrastive vision-language retrieval models (Li et al., 25 Jul 2025).

1. Problem formulation and retrieval setting

MixBench targets a retrieval regime that is more general than ordinary text-to-text or text-to-image retrieval. The motivating scenario is a realistic search engine in which a query such as “Mountain Fuji” may need to retrieve text pages, standalone images, and pages that combine image and text, all within the same ranked list (Li et al., 25 Jul 2025). The benchmark therefore treats mixed modality retrieval as a setting that requires both cross-modal alignment and multimodal fusion in a single system.

The retrieval model embeds queries and documents into a shared space and scores them with cosine similarity:

$$s(q, d_i) = \frac{e_q \cdot e_i}{\|e_q\| \cdot \|e_i\|}.$$

This formulation is standard, but MixBench uses it to test a nonstandard corpus condition: the retrieval space contains heterogeneous document types simultaneously rather than a single clean modality partition (Li et al., 25 Jul 2025). A plausible implication is that benchmark difficulty arises not only from semantic matching, but also from whether the representation space remains well-calibrated across modality boundaries.

2. Benchmark construction and data sources

MixBench is built from four real-world multimodal datasets: Google WIT, MSCOCO, OVEN, and VisualNews. Its documents can be image-only, text-only, or image-text pairs, and the document types are sampled in a $1\!:\!1\!:\!1$ ratio to keep the distribution balanced (Li et al., 25 Jul 2025). In the appendix, the evaluation sets are listed as Google WIT, OVEN, MSCOCO, and VisualNews, with queries and documents spanning $T$, $I$, and $I+T$.

Dataset	Query construction	Document construction
Google WIT	Page title + image reference description	Page description as text; associated image as image document
MSCOCO	One caption	Longer text document from remaining captions via GPT; plus image
VisualNews	Original caption	GPT-expanded descriptive paragraph as text; plus image
OVEN	Image-text pair	Both image and caption treated as valid standalone documents

This construction is designed to represent the full mixed-modality setting rather than a single subproblem. The benchmark does not restrict evaluation to text-query/image-document or image-query/text-document cases in isolation; instead, it places heterogeneous query and document types into a common retrieval task (Li et al., 25 Jul 2025). This suggests that MixBench is structured to test whether a system can rank across modality boundaries without collapsing into modality-specific preference patterns.

3. Evaluation protocol and relevance metrics

MixBench uses NDCG@10 as its primary evaluation metric. The appendix defines it in the standard form:

$$\text{NDCG@}K = \frac{1}{\text{IDCG@}K} \sum_{i=1}^{K} \frac{2^{\text{rel}_i} - 1}{\log_2(i + 1)},$$

with

$$\text{IDCG@}K = \sum_{i=1}^{K} \frac{2^{\text{rel}_i^\star} - 1}{\log_2(i + 1)}.$$

The paper also reports robustness to NDCG@100 and Recall@1 in the appendix, but NDCG@10 is the main metric used throughout (Li et al., 25 Jul 2025).

The emphasis on ranking quality rather than only top-1 retrieval is consequential for this setting. Mixed modality search can fail even when a model retrieves some relevant items, if modality bias systematically pushes semantically superior cross-modal matches below weaker same-modality items. MixBench is therefore structured to make ranking distortions visible, not merely to test pairwise matching.

4. Modality gap diagnosis

A major contribution associated with MixBench is the diagnosis of a modality gap in CLIP-style contrastive vision-LLMs. Although CLIP is trained to align paired images and texts, the paper reports that image embeddings and text embeddings still form distinct clusters in the embedding space and remain far apart (Li et al., 25 Jul 2025). On MixBench, this gap becomes visible through two concrete failure modes.

The first is intra-modal ranking bias. Similarities between items of the same modality are systematically higher than similarities across modalities, so a text query may prefer unrelated text documents over relevant images simply because they are text. The paper illustrates this with a “Mountain Fuji” text query that can rank an unrelated text snippet above an image that actually depicts Mount Fuji (Li et al., 25 Jul 2025).

The second is inter-modal fusion failure. When multimodal documents are represented by linearly combining image and text features, the fused embedding can land in a suboptimal region of the space and become worse than using either modality alone. In the multimodal-document experiments, intermediate fusion weights often peak at the unimodal endpoints $m_i \subseteq \mathcal{M}$0 or $m_i \subseteq \mathcal{M}$1, which is presented as evidence that naïve fusion is harmed by the modality gap (Li et al., 25 Jul 2025).

MixBench also supports a more striking behavioral diagnosis through heterogeneous-corpus perturbations. The paper replaces text documents with screenshots or with paired images while preserving semantic content. If cross-modal alignment were perfect, retrieval performance should remain stable. Instead, CLIP exhibits a U-shaped curve: performance drops as more text documents are replaced with screenshots, reaches a minimum near $m_i \subseteq \mathcal{M}$2, and rises again when the corpus becomes entirely screenshots at $m_i \subseteq \mathcal{M}$3. One reported curve falls from 0.22 at $m_i \subseteq \mathcal{M}$4 to 0.02 at $m_i \subseteq \mathcal{M}$5, then rises to 0.36 at $m_i \subseteq \mathcal{M}$6 (Li et al., 25 Jul 2025). The paper further validates this interpretation with a push-down simulation that artificially assigns zero similarity to screenshots and reproduces the same U-shape. The benchmark therefore exposes the modality gap through ranking behavior, not only through embedding geometry.

5. GR-CLIP and empirical results on MixBench

To address the modality gap revealed on MixBench, the paper proposes GR-CLIP, where GR stands for gap-removed. It is explicitly presented as a lightweight post-hoc calibration method rather than a new encoder or a retrained model (Li et al., 25 Jul 2025). The key premise, drawn from prior work cited in the paper, is that the modality gap can be approximated as a constant vector orthogonal to the shared embedding subspace, with paired image and text embeddings satisfying

$m_i \subseteq \mathcal{M}$7

GR-CLIP estimates and removes this gap by mean-centering modality-specific embeddings:

$m_i \subseteq \mathcal{M}$8

so that

$m_i \subseteq \mathcal{M}$9

For multimodal documents, the same interpolation used by CLIP is applied to calibrated embeddings:

$s(q,d_i)$0

The method requires only a single pass to estimate means and adds negligible inference overhead (Li et al., 25 Jul 2025).

On MixBench, the reported gains are substantial. The abstract and main text state that GR-CLIP improves NDCG@10 by up to 26 percentage points over CLIP, and that it surpasses the vision-language generative embedding baseline VLM2Vec by 4 percentage points overall while using 75$s(q,d_i)$1 less compute (Li et al., 25 Jul 2025). The appendix provides the following CLIP-L/14 versus GR-CLIP-L/14 comparison:

Subset	CLIP-L/14	GR-CLIP-L/14
Google WIT	0.505	0.648
MSCOCO	0.426	0.656
OVEN	0.389	0.465
VisualNews	0.596	0.754

Similar gains are reported for OpenCLIP and SigLIP variants. The appendix notes one exception: MSCOCO, where VLM2Vec was trained on the dataset, which helps it there. All experiments were inference-only on a single NVIDIA A100 GPU (Li et al., 25 Jul 2025). A plausible implication is that MixBench is useful not only for ranking model quality, but also for comparing efficiency-oriented calibration methods against heavier generative embedding approaches.

6. Scope, significance, and nomenclature

The broader implication drawn from MixBench is that mixed modality retrieval should be treated as its own retrieval setting, not as a simple extension of text-image matching. The benchmark shows that a unified retrieval system must handle heterogeneous corpora and multimodal documents simultaneously, and that success depends on more than alignment on paired image-text data during pretraining (Li et al., 25 Jul 2025). In this sense, MixBench functions both as a diagnostic benchmark and as a practical evaluation suite for future work on unified multimodal embeddings.

The name MixBench is, however, not unique across contemporary arXiv usage. In a separate context, “mixbench” appears as a GPU microbenchmark used to evaluate “the performance limits of GPUs under mixed operational intensity kernels” in a case study of the NVIDIA CMP 170HX. There it reports metrics such as Compute iters, Flops/byte, Execution time, GFLOPS, GB/sec, and Iops/byte, and is used to compare default compilation with FMA-disabled compilation (Kangwei, 30 Apr 2025). That benchmark is unrelated to mixed modality search.

It should also be distinguished from CodeMixBench, a benchmark for code generation from code-mixed prompts built on BigCodeBench, with 1,140 unique BigCodeBench tasks expanded into 6,840 total prompts across Hinglish, Spanish-English, and Chinese Pinyin-English settings (Sheokand et al., 8 May 2025). Likewise, other papers discuss MixBench-style environments or methodologies without naming their systems MixBench: Bencher is presented as a modular benchmarking framework for black-box optimization that could conceptually serve as a MixBench-style benchmark environment (Papenmeier et al., 27 May 2025), while Mess is described as very close in spirit to a MixBench-style memory benchmark but is centered on a family of bandwidth–latency curves for memory-system characterization (Esmaili-Dokht et al., 2024).

A common misconception is therefore to treat “MixBench” as a single benchmark family across domains. The literature instead uses the label in multiple, domain-specific ways. In current retrieval research, the term refers most specifically to the heterogeneous-corpus search benchmark introduced for studying modality gaps and calibration in mixed modality search (Li et al., 25 Jul 2025).