RzenEmbed: Unified Multimodal Embedding

• RzenEmbed is a unified multimodal embedding framework that represents text, images, videos, and visual documents in a shared vector space for retrieval.
• It employs a two-stage training process with an improved InfoNCE loss featuring hardness weighting and false-negative masking to enhance discrimination.
• Benchmark results on MMEB and EDIR highlight its state-of-the-art performance and suggest further improvements for fine-grained compositional reasoning.

Searching arXiv for the cited works to ground the article and resolve the naming ambiguity around “RzenEmbed.” RzenEmbed is a unified multimodal embedding framework built on a Multimodal LLM backbone, specifically Qwen2-VL, to produce a single instruction-following vector representation for text, images, videos, and visual documents in a shared embedding space for retrieval (Jian et al., 31 Oct 2025). In the recent literature, the name is also a source of ambiguity: “RzenEmbed” may be confused with the CTR-oriented “res-embedding” mechanism introduced for deep click-through-rate modeling (Zhou et al., 2019), and with the remote sensing library rs-embed, whose name is explicitly given as “rs-embed” rather than “RzenEmbed” (Ye et al., 27 Feb 2026). In its specific multimodal-retrieval sense, however, RzenEmbed denotes the 2025 framework “RzenEmbed: Towards Comprehensive Multimodal Retrieval,” which emphasizes cross-modality coverage beyond natural images, improved contrastive training, and strong performance on MMEB and composed image retrieval evaluation (Jian et al., 31 Oct 2025).

1. Nomenclature and problem setting

RzenEmbed addresses a concrete limitation in prior embedding systems: many CLIP-style and MLLM-derived embedders are oriented primarily toward image-text retrieval, while offering limited support for videos and visual documents. The framework is therefore positioned as a unified embedder for comprehensive retrieval across four modalities: text, images, videos, and visual documents (Jian et al., 31 Oct 2025).

The central task formulation is retrieval with a shared embedding space. Queries and candidates, whether textual or multimodal, are encoded into a single vector space, and cosine similarity is used for retrieval. This design makes the model applicable to heterogeneous task families spanning image classification, image QA, image retrieval, image grounding, document retrieval, video retrieval, and video QA, which are the task families explicitly used in its learnable-temperature scheme (Jian et al., 31 Oct 2025).

A recurring source of confusion is that the string “RzenEmbed” has appeared in unrelated contexts. In CTR modeling, it is described as an alias for the “res-embedding” method, where each item embedding is decomposed into a graph-derived central component and a small residual component to control intra-domain spread (Zhou et al., 2019). Separately, in remote sensing, “RzenEmbed” is identified as a misspelling or variant of rs-embed, a Python library for ROI-centric access to remote sensing foundation-model embeddings (Ye et al., 27 Feb 2026). These are distinct systems with different objectives, architectures, and evaluation regimes.

2. Architecture and embedding interface

RzenEmbed uses Qwen2-VL as its backbone. The choice is motivated in the paper by three properties: native dynamic resolution for flexible visual input sizes, multimodal RoPE (M-RoPE) for temporal structure in videos, and strong instruction-following and generalization (Jian et al., 31 Oct 2025). The visual encoder produces visual tokens; for videos, a sequence of frames sampled at a fixed interval is encoded into a sequence of visual tokens. These visual features are projected into the LLM token space via a projection layer before fusion with text tokens.

The joint encoder processes concatenated visual and textual tokens in a single sequence. The embedding is taken as the final hidden state of the last token of the fused sequence, yielding one vector per input, whether text-only or multimodal. There is no separate frame-level pooling stage for video: temporal handling is delegated to the LLM sequence model together with M-RoPE, and the final representation is again the last-token hidden state (Jian et al., 31 Oct 2025).

Visual documents are treated as images, potentially at high resolution. This means document retrieval is handled through the same visual pathway rather than through a separate OCR-centric pipeline. The paper explicitly notes that future work could integrate explicit layout or OCR tokenization while preserving a unified embedding interface, which suggests that current document performance is achieved without such explicit symbolic structure (Jian et al., 31 Oct 2025).

Input formatting is prompt-based. The system prompt is: “Given an image, summarize the provided image in one word. Given only text, describe the text in one word.” The representation prompts are modality-specific: “Represent the given text in one word” for text-only inputs and “Represent the given image in one word” for multimodal inputs. During training, the input concatenates system prompt, user content, and representation prompt; the same pattern is used at inference for queries and candidates (Jian et al., 31 Oct 2025).

3. Training recipe and improved InfoNCE

The training procedure is two-stage. Stage 1 is multimodal continual training for foundational alignment. Its data mixture includes approximately 0.3M text-to-text pairs from MS MARCO, NQ, HotpotQA, TriviaQA, SQuAD, FEVER, and AllNLI; 2M text-to-image samples from LAION-2B; 0.25M text-to-video-description samples from ShareGPT4V; and 2.5M image+text-to-image pairs from MegaPairs, for about 5M pairs in total (Jian et al., 31 Oct 2025). LAION images are re-captioned with CogVLM-19B, and noisy or duplicated items are filtered.

Stage 2 is instruction-centric fine-tuning. It uses the MMEB-V2 training set together with public multimodal retrieval and QA datasets, mmE5 synthetic data, and 400k video clips from VideoChat-Flash. Each dataset is capped at 100k samples, except that classification datasets are merged into a single larger dataset to reduce false negatives caused by small label spaces. Except for classification, each batch is sampled from a single dataset, which deliberately creates semantically similar in-batch items and therefore many hard negatives as well as potential false negatives (Jian et al., 31 Oct 2025).

The key objective is an improved InfoNCE loss. Standard InfoNCE is written as

$$L = -\log \frac{\exp(\text{sim}(q, k^+)/\tau)}{\exp(\text{sim}(q, k^+)/\tau) + \sum_{i=1}^{N} \exp(\text{sim}(q, k_i^-)/\tau)}.$$

RzenEmbed modifies this in two ways. First, it introduces hardness weighting, with

$w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$

using $\alpha = 9$, so that harder negatives receive larger weights. Second, it introduces false-negative masking, with

$$M_i = \mathbb{1}[\text{sim}(k_i^-, k^+) \le \delta],$$

using $\delta = 0.95$, so likely false negatives are removed from the denominator. The resulting loss is

$$L = -\log \frac{\exp(\text{sim}(q, k^+)/\tau)}{\exp(\text{sim}(q, k^+)/\tau) + \sum_{i=1}^{N} M_i \cdot w_i \cdot \exp(\text{sim}(q, k_i^-)/\tau)}.$$

This is described as simultaneously emphasizing discriminative negatives and mitigating noise from semantically correct but unlabeled in-batch neighbors (Jian et al., 31 Oct 2025).

A further modification is task-specific learnable temperature. For each task family $t$, the temperature is parameterized as

$\tau_t = \exp(\theta_t),$

with $\theta_t$ learned jointly with the rest of the model. The paper states that smaller $\tau_t$ sharpens the softmax for tasks requiring finer discrimination, while larger $w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$0 smooths it for coarser tasks (Jian et al., 31 Oct 2025).

The training recipe also includes model souping over LoRA adapters. Several adapters are trained, and a final adapter is obtained by weighted aggregation of the low-rank parameters:

$w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$1

This is presented as a way to capture complementary strengths while keeping deployment to a single adapter (Jian et al., 31 Oct 2025).

4. Benchmarks and measured performance

RzenEmbed is evaluated principally on MMEB-V1 and MMEB-V2. MMEB-V2 spans 78 datasets across Image, Video, and VisDoc evaluation tracks. On MMEB-V2, the 2.21B model reports an All score of 67.2, while the 8.29B model reports 71.6. The 8.29B model is described as state of the art among all reported models, narrowly exceeding the closed-source Seed-1.6-Embedding on the overall All score and outperforming it on the Video and VisDoc tracks (Jian et al., 31 Oct 2025).

Model	MMEB-V2 All	Video Overall	VisDoc Overall
RzenEmbed 2.21B	67.2	47.3	74.5
RzenEmbed 8.29B	71.6	55.7	77.1

On MMEB-V1, the 2.21B variant reports Classification 68.5, VQA 66.3, Retrieval 74.5, Grounding 90.3, with IND 76.1, OOD 67.4, and Overall 72.3. The 8.29B variant reports Classification 70.6, VQA 71.7, Retrieval 78.5, Grounding 92.1, with IND 78.5, OOD 72.7, and Overall 75.9 (Jian et al., 31 Oct 2025). These results place it above a range of open baselines including Ops-MM-embedding-v1, B3, LLaVE, UniME-V2, and VLM2Vec-V2 at comparable scale.

Ablation results attribute the gains to a cumulative recipe rather than to a single modification. On MMEB-V2 All, the baseline with plain InfoNCE is 65.7; merging classification datasets yields 66.3; adding learnable temperature yields 66.4; and combining the full recipe with dataset resampling yields 67.2 for the 2B-scale configuration. Model souping further improves the best single-mixture result from roughly 71.18–71.16 to 71.61, with per-modality scores of 75.92 for Image, 55.73 for Video, and 77.06 for VisDoc (Jian et al., 31 Oct 2025).

Implementation details are specified with unusual precision for an embedding paper. Stage-2 fine-tuning uses AdamW, a single epoch, LoRA rank 64 on all linear layers of both the vision encoder and the LLM, learning rate $w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$2 with cosine decay, weight decay $w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$3, global batch size 768, bf16 mixed precision, gradient checkpointing, a maximum of 1280 input tokens for images and videos, and 16 NVIDIA A800 (80 GB) GPUs (Jian et al., 31 Oct 2025).

5. Fine-grained behavior in composed image retrieval

A separate perspective on RzenEmbed is provided by EDIR, a fine-grained composed image retrieval benchmark constructed from image editing. In EDIR, a query is a pair $w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$4 consisting of a reference image and a natural-language modification, and retrieval is performed over a 178,645-image corpus containing the target and multiple hard negatives derived from the same source image (Song et al., 22 Jan 2026). The benchmark comprises 5,000 queries across 5 categories and 15 subcategories, with Recall@1 as the main metric.

Within this evaluation, RzenEmbed-7B reports 47.2 Recall@1, tying Ops-embedding for best overall performance among the general models evaluated. The same benchmark reports that the average of MLLM-based baselines, excluding EDIR-MLLM, is 36.9, while the non-MLLM average is 18.4. RzenEmbed therefore improves by +10.3 points over the MLLM-based average and by +28.8 points over the non-MLLM average (Song et al., 22 Jan 2026).

Its strengths and weaknesses are highly nonuniform across subcategories. The strongest Recall@1 scores are Addition 74.0, Replace 71.0, Action 60.7, Time 58.0, and Weather 50.7. The weakest are Viewpoint 24.0, Remove 28.0, Shape 35.7, Texture 36.7, and Material 37.3. The paper interprets these results as evidence that strong overall retrieval does not imply uniformly strong multimodal compositional reasoning, especially for negation, viewpoint shifts, counting, and subtle local visual attributes (Song et al., 22 Jan 2026).

EDIR also includes an in-domain training experiment with EDIR-MLLM, which reaches 59.9 Recall@1 and outperforms RzenEmbed by substantial margins on many fine-grained categories, including Material, Texture, Style, Weather, and Time. This suggests that some of RzenEmbed’s observed deficits are data-sensitive rather than purely architectural. At the same time, Spatial, Count, and Viewpoint remain comparatively difficult even after in-domain training, which the benchmark authors interpret as evidence of persistent architectural limits in global spatial reasoning, 3D or viewpoint understanding, and explicit counting (Song et al., 22 Jan 2026).

6. Relation to adjacent methods, limitations, and open directions

RzenEmbed is distinct from both residual connections in deep networks and from the CTR-specific res-embedding mechanism. In res-embedding for CTR, the objective is to improve generalization in Embedding+MLP models by decomposing item embeddings as

$w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$5

with graph-derived central embeddings and small residuals, and by linking a smaller domain-wise aggregation radius $w_i = \exp(\alpha \cdot \text{sim}(q, k_i^-)),$6 to a tighter generalization bound (Zhou et al., 2019). That method is not a multimodal retrieval system, does not use Qwen2-VL, and is evaluated on AUC for CTR rather than on multimodal retrieval benchmarks.

RzenEmbed is also distinct from rs-embed in remote sensing. rs-embed is a Python library for obtaining remote sensing foundation-model embeddings through a unified ROI-centric interface, with abstractions for providers such as Google Earth Engine, sensor specifications, standardized preprocessing, and batch export. Its central problem is interoperability and benchmarking across heterogeneous remote sensing foundation models, not unified multimodal retrieval across text, images, videos, and visual documents (Ye et al., 27 Feb 2026).

The limitations reported for RzenEmbed are specific. The paper notes slight underperformance on a few video meta-tasks, indicating room for improving long-range temporal modeling beyond 1–3 minute clips. It also notes that the method relies on large-scale recaptioned and synthetic data, so residual domain bias and noise may persist despite cleaning. For visual documents, pages are treated as images; this leaves open the possibility that explicit layout or OCR tokenization could improve document understanding while preserving a unified embedding interface (Jian et al., 31 Oct 2025).

The EDIR results sharpen these limitations from a diagnostic angle. They show weak performance on removal, negation, viewpoint, and some fine-grained attributes, even when overall retrieval is strong (Song et al., 22 Jan 2026). A plausible implication is that RzenEmbed’s training recipe yields a robust general-purpose embedding model, but not one that uniformly solves the hardest forms of multimodal composition. The benchmark’s authors recommend stronger hard-negative design, better localized grounding, improved relational reasoning, and targeted training on negation, counting, and viewpoint-sensitive examples as promising directions for models in this class (Song et al., 22 Jan 2026).