UniVer: Universal Vision-Language Retriever

Updated 5 July 2026

UniVer is a universal vision-language retrieval model that embeds queries, texts, and images into a single shared space, replacing separate retrievers.
It uses a balanced contrastive objective with modality-balanced hard negatives and image verbalization to bridge the gap between visual and textual data.
Leveraging CLIP and VinVL, UniVer achieves state-of-the-art multimodal retrieval with a unified indexing strategy and efficient nearest-neighbor search.

UniVer, in the vision–language retrieval sense, denotes a Universal Vision-Language Retriever: a dense retrieval model that learns a single shared embedding space in which natural-language queries, text passages, and images all reside together, so that one unified nearest-neighbor search can return text documents, image documents, or both. In the formulation instantiated by UniVL-DR, the system is designed to replace the conventional divide-and-conquer pipeline of separate text and image retrievers plus late fusion with a unified model optimized by modality-balanced hard negatives and an image verbalization method that bridges the modality gap in raw data space (Liu et al., 2022).

1. Definition and retrieval formulation

UniVer is motivated by the limitations of traditional multi-media search systems that run a text retriever over a text index, run an image retriever or cross-modal retriever over an image index, and then fuse the ranked lists. In that design, text and image relevance are modeled by different systems with incompatible scores; one modality, often text, tends to dominate; and the overall pipeline depends on multiple models, separate indices, and a fusion layer. UniVer instead aims at a single universal retriever that encodes queries and all multi-modal documents into one unified representation space and performs one KNN search (Liu et al., 2022).

The document collection is a mixed-modality corpus

$\mathcal{D} = \{T_1, \dots, T_n, I_1, \dots, I_m\},$

where $T_i$ are text passages and $I_j$ are images with captions. A textual query $q$ , a text document $d^{\,i}_{\text{Text}}$ , and an image document $d^{\,j}_{\text{Image}}$ are embedded as vectors $\vec q$ , $\vec d^{\,i}_{\text{Text}}$ , and $\vec d^{\,j}_{\text{Image}} \in \mathbb{R}^d$ such that relevant query–document pairs are close, irrelevant pairs are far, and the geometry of the space implicitly performs modality routing. The retrieval score is cosine similarity,

$f(q,d)=\cos(\vec q,\vec d),$

and the index is searched with FAISS (Liu et al., 2022).

A central implication of this formulation is that relevance modeling, cross-modality alignment, and modality selection are delegated to the same embedding geometry. This suggests that UniVer is not merely a convenience layer over separate retrievers, but a redefinition of retrieval as unified nearest-neighbor search over heterogeneous resources.

2. Architecture and embedding space

The UniVer blueprint builds on CLIP, specifically the ViT-B/32 version, and uses two encoders with one shared text backbone: CLIP’s text transformer serves as the TextEncoder for queries, text passages, and image captions, while CLIP’s image encoder serves as the ImgEncoder for raw images (Liu et al., 2022).

Query encoding is

$T_i$ 0

and text-document encoding is

$T_i$ 1

For an image document $T_i$ 2, UniVer encodes the raw image and its caption separately and then sums the two embeddings: $T_i$ 3 The paper reports that this sum is empirically superior to concatenation or outer product for the retrieval setting, and interprets the operation as forcing image and caption signals into the same semantic subspace, analogously to how BERT sums token and position embeddings (Liu et al., 2022).

The textual side uses CLIP’s transformer text encoder with a maximum length of 77 tokens. Inputs include queries, text passages, and image captions, including both manual captions and verbalized expansions. The visual side uses CLIP’s image encoder, while VinVL is additionally employed for image verbalization to produce object tags, region features, and generated captions or queries (Liu et al., 2022).

Conceptually, the architecture consists of four stages: input ingestion; modality-specific encoding; projection into a shared embedding space; and top- $T_i$ 4 retrieval by cosine similarity. Because text passages and image documents occupy the same index, the model can surface either modality without a separate routing classifier.

3. Universal embedding optimization

The core training problem is to refine CLIP’s pre-trained alignment space into one that supports open-domain question-to-answer relevance across both text and images. UniVer uses a contrastive objective over a query $T_i$ 5, a positive document $T_i$ 6, and a hard-negative set $T_i$ 7: $T_i$ 8 where $T_i$ 9 and $I_j$ 0 is a temperature coefficient. The loss is described as containing an alignment term $I_j$ 1 together with modality-specific negative components $I_j$ 2 and $I_j$ 3 (Liu et al., 2022).

The distinctive feature is modality-balanced hard negative sampling. If the number of image negatives $I_j$ 4 greatly exceeds the number of text negatives $I_j$ 5, or vice versa, the model can reduce loss by escaping one modality’s region of the space, thereby becoming modality-discriminative rather than modality-universal. UniVer therefore enforces

$I_j$ 6

and in the reported setting mines one hard text negative and one hard image negative per query from the top-100 candidates retrieved by the current CLIP-DPR model (Liu et al., 2022).

This balance is directly tied to modality selection ability. Training with only text hard negatives improves text-text metrics but harms image retrieval and makes top-10 results almost all text; only image negatives induce the opposite bias; unbalanced ratios such as $I_j$ 7 or $I_j$ 8 still show skew; balanced hard negatives yield the best multi-modal MRR@10 of 62.29 and a nearly 50–50 image ratio in top-10 results (Liu et al., 2022).

Hard-negative regime	Observed effect	Multi-modal MRR@10
Only text hard negatives	Text-biased top-10 results	Suffers
Only image hard negatives	Image-biased top-10 results	Suffers
Balanced hard negatives	Nearly 50–50 image ratio in top-10	62.29

Embedding visualizations reported as t-SNE plots reinforce the same point: with balanced hard negatives, text documents and image documents occupy distinct regions and queries are routed toward the appropriate region according to their semantics. A common misconception is that a universal space should collapse modalities into one indistinguishable cluster; the reported behavior is instead structured separation with query-dependent routing (Liu et al., 2022).

4. Image verbalization as modality bridging

UniVer supplements latent-space alignment with image verbalization, a method that translates visual content into additional text so that the TextEncoder can absorb entity- and attribute-level information that may be weakly represented in short captions. The method uses an object-based representation derived from VinVL: an image $I_j$ 9 is decomposed into detected objects $q$ 0, each with a region feature $q$ 1 and predicted class label $q$ 2 (Liu et al., 2022).

For caption verbalization, the generation model is trained on sequences of the form

$q$ 3

using Masked Language Modeling, with initialization from a VinVL model trained on MSCOCO captioning. For query verbalization, the input is

$q$ 4

and the model learns to reconstruct masked tokens in the pseudo-query $q$ 5 (Liu et al., 2022).

The verbalized outputs are appended to the original caption: $q$ 6 and the image-document embedding becomes

$q$ 7

The paper emphasizes that verbalized queries are especially effective because they are more entity-rich and question-like than verbalized captions, often using names such as “Durham Cathedral”, “Minnetonka Rhododendron”, or “Bakery Sattin”, and therefore resemble the downstream retrieval queries more closely (Liu et al., 2022).

The ablations are explicit. Baseline UniVL-DR with in-batch training yields multi-modal MRR@10 of 48.83. Adding verbalized captions raises it to 49.49. Adding verbalized queries raises it to 50.21, and after hard-negative training the best MRR@10 reaches 62.40. The gains are largest when original captions are short or medium length, which the paper interprets as generated questions filling in missing semantic detail (Liu et al., 2022).

A related misconception is that raw image features alone should suffice for cross-modal retrieval when the base encoder is strong. The reported ablation points in the opposite direction: VinVL-DPR without captions obtains image-retrieval MRR@10 of 3.67, whereas adding captions raises it to 48.68, indicating that textual description of images is crucial in this setting (Liu et al., 2022).

5. Data, training protocol, and inference pipeline

UniVer is evaluated on WebQA, converted into an open-domain retrieval task. The collection contains 389,750 images, 787,697 texts, and 1,177,447 total multi-modal documents. The query split is 31,766 for training, 5,000 for development, and 4,966 for testing. Each query has one or more relevant text and/or image documents (Liu et al., 2022).

Three retrieval tasks are considered: text-text retrieval, text-image retrieval, and multi-modal retrieval over the full mixed corpus. Training begins from CLIP encoders, first with in-batch negatives and then with modality-balanced hard negatives. The main training hyperparameters are a text truncation length of 77 tokens, batch size 64, learning rate $q$ 8, maximum 20 epochs, temperature $q$ 9, AdamW optimization, and a cosine annealing learning-rate schedule. Hard negatives are mined from the top-100 CLIP-DPR results with $d^{\,i}_{\text{Text}}$ 0 per query (Liu et al., 2022).

For the VinVL-based verbalization models, the system is initialized from a VinVL captioning model trained on MSCOCO, generates up to 20 tokens with beam size 5, uses input truncation of query length up to 40, caption up to 30, and region features up to 50, and sets MLM mask probability to 0.15 (Liu et al., 2022).

Training element	Reported configuration
Collection size	389,750 images; 787,697 texts; 1,177,447 total
Query split	31,766 train; 5,000 dev; 4,966 test
Retriever training	Batch size 64; LR $d^{\,i}_{\text{Text}}$ 1; 20 epochs; $d^{\,i}_{\text{Text}}$ 2
Hard negatives	Top-100 by CLIP-DPR; one text + one image per query

Inference is a single-pipeline procedure. The query is encoded with the shared TextEncoder. All text documents are encoded once offline with the TextEncoder, and all image documents are encoded once offline with the sum of ImgEncoder output and TextEncoder output on $d^{\,i}_{\text{Text}}$ 3. All document vectors are stored in a FAISS index, and approximate nearest neighbors are retrieved by cosine similarity (Liu et al., 2022).

This one-index design is the operational expression of universality: the query does not first decide whether to search text or images. The decision is implicit in nearest-neighbor structure.

6. Empirical performance, significance, and nomenclature

On WebQA multi-modal retrieval, UniVer outperforms both universal baselines and divide-and-conquer systems. BM25, treated as text-only, reaches multi-modal MRR@10 of 53.75. The best divide-and-conquer oracle-modality system, combining BM25 and CLIP-DPR, reaches 61.05. CLIP-DPR as a universal multi-modal retriever reaches 48.83. UniVL-DR, corresponding to the UniVer design, reaches 62.40 (Liu et al., 2022).

System	Multi-modal MRR@10
BM25	53.75
BM25 + CLIP-DPR, oracle modality	61.05
CLIP-DPR as universal retriever	48.83
UniVL-DR / UniVer	62.40

The gains are not confined to mixed retrieval. On text-text retrieval, UniVer reaches MRR@10 of 60.72, compared with 52.57 for CLIP-DPR in single/cross mode and 51.75 for CLIP-DPR in universal mode. On text-image retrieval, UniVer reaches 65.57, compared with 59.95 and 60.61 for the corresponding CLIP-DPR baselines. Recall@20 also improves by about 10.5 points over CLIP-DPR in multi-modal retrieval, from 69.84 to 80.37. The reported conclusion is that joint multi-modal training benefits single-modality and cross-modality tasks rather than merely trading one for the other (Liu et al., 2022).

The main limitations stated or implied are also clear. The framework addresses only text and images; extending it to video, audio, tables, or structured knowledge would require new encoders and possibly new verbalization strategies. Evaluation is centered on WebQA, so performance on other domains is not extensively studied. Training is relatively heavy because it combines object detection, VinVL, and CLIP, even though FAISS supports billion-scale retrieval at inference (Liu et al., 2022). Future directions proposed in the source include adding more modalities, improving verbalization models, and pretraining universal retrievers on broader multi-modal corpora before task-specific fine-tuning (Liu et al., 2022).

The term “UniVer” is not unique across the literature. In this retrieval context, it is a convenient designation for the Universal Vision-Language Retriever instantiated by UniVL-DR (Liu et al., 2022). Elsewhere, the exact title “UniVer” refers to a verification algorithm for speculative decoding in LLMs (Weng et al., 6 May 2026). The term is also described as an informal abbreviation for Uni-VERSA, a unified network for speech assessment (Shi et al., 27 May 2025), and as a likely mis-typing or shorthand for UniVSE in structured visual-semantic embedding work (Wu et al., 2019). In vision–language dense retrieval, however, UniVer most precisely denotes the single-space, single-index, universal retriever defined by modality-balanced contrastive optimization and image verbalization.