Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings

Abstract: Multimodal LLMs (MLLMs) have emerged as a promising foundation for universal multimodal embeddings. Recent studies have shown that reasoning-driven generative multimodal embeddings can outperform discriminative embeddings on several embedding tasks. However, Chain-of-Thought (CoT) reasoning tends to generate redundant thinking steps and introduce semantic ambiguity in the summarized answers in broader retrieval scenarios. To address this limitation, we propose Rewrite-driven Multimodal Embedding (RIME), a unified framework that jointly optimizes generation and embedding through a retrieval-friendly rewrite. Meanwhile, we present the Cross-Mode Alignment (CMA) to bridge the generative and discriminative embedding spaces, enabling flexible mutual retrieval to trade off efficiency and accuracy. Based on this, we also introduce Refine Reinforcement Learning (Refine-RL) that treats discriminative embeddings as stable semantic anchors to guide the rewrite optimization. Extensive experiments on MMEB-V2, MRMR and UVRB demonstrate that RIME substantially outperforms prior generative embedding models while significantly reducing the length of thinking.

• The paper presents RIME, a rewriting-driven framework that replaces CoT to reduce reasoning redundancy in multimodal embeddings.
• It uses a three-part strategy combining supervised fine-tuning, cross-mode alignment, and reinforcement learning to enhance semantic fidelity.
• Empirical results demonstrate significant improvements in retrieval performance and inference efficiency across text, images, videos, and documents.

Rewrite as a Universal Interface: RIME for Generative Multimodal Embeddings

Introduction

The proliferation of Multimodal LLMs (MLLMs) has transformed unified representation learning across text, image, video, and document domains. While recent work on reasoning-driven generative embeddings (most notably via Chain-of-Thought [CoT] prompting) has improved performance over traditional discriminative methods, it introduces significant artifacts: redundant reasoning steps, semantic ambiguity, and suboptimal granularity for general retrieval. The paper "Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings" (2604.22280) introduces RIME (Rewrite-drIven Multimodal Embedding), presenting structured rewriting as a retrieval-friendly alternative to CoT for generative multimodal embeddings. RIME augments the generative embedding pipeline with cross-mode alignment and reinforcement learning, achieving SOTA performance with reduced reasoning length.

Limitations of Chain-of-Thought for Multimodal Embeddings

CoT-informed generative embedding methods have dominated recent advances in leveraging MLLMs for retrieval and alignment tasks. However, the CoT paradigm incurs critical challenges:

• Reasoning Redundancy: Generation of stepwise rationales often exceeds what is needed for retrieval, causing unnecessary computation and noise.
• Semantic Target Ambiguity and Bias: The requirement to summarize answers can lead to embeddings that mismatch the heterogeneous, multi-granularity targets common in general retrieval.
• Inference Inefficiency: The long autoregressive sequences mandated by CoT inflate inference latency, impeding scalability.

Recent studies have also established that joint optimization of contrastive and generative (language modeling) objectives can degrade discriminative capacity, even in models decoupling rationales from embeddings [cui2025think, yu2025cafe, chen2025moca].

RIME: Rewrite-Driven Generative Multimodal Embedding

RIME is structured as a modular training and inference framework comprising three pivotal components:

1. Rewrite-Driven Joint SFT (Supervised Fine-Tuning)
2. Cross-Mode Alignment (CMA)
3. Refine Reinforcement Learning (Refine-RL)

The entire pipeline is constructed to avoid deterministic answer summarization, focusing instead on structure-preserving semantic rewrites across modalities.

Figure 1: Three core components of the RIME framework: joint SFT with rewriting, cross-mode InfoNCE-based alignment, and Refine RL with discriminative anchors.

Rewrite-Driven Joint SFT

The framework replaces stepwise CoT rationales with retrieval-oriented, structure-aware rewriting of queries and targets. Rewriting decomposes multimodal context (text, images, VQA, documents, video) into constituent elements, refining the description to maximize semantic overlap without imposing CoT’s goal of singular summarization.

Figure 2: Multimodal rewrite template and prompt demonstration for SFT data construction.

Rewrite SFT is jointly supervised with contrastive InfoNCE loss to produce an embedding space well-adapted to retrieval, with the following objective:

$$\mathcal{L}_{\text{Joint}} = \lambda \cdot \mathcal{L}_{\text{Rewrite}} + \mathcal{L}_{\text{CM\_InfoNCE}}$$

where $\lambda$ is a hyperparameter tuning the contribution of rewrite LM loss versus contrastive retrieval performance.

Cross-Mode Alignment (CMA)

Given the persistent trade-off between discriminative (token-level) and generative (autoregressive) embeddings, RIME’s CMA explicitly regularizes both spaces with mutual InfoNCE loss terms. This incentivizes mutual retrievability between pure discriminative and rewrite-driven generative embeddings, enabling flexible query/target encoding with minimal semantic drift.

CMA achieves a semantic coupling of the two embedding spaces, fundamentally improving test-time deployment flexibility: low-latency applications can rely on discriminative encoding, while generative rewriting is reserved for deeper, context-aware retrieval.

Refine Reinforcement Learning (Refine-RL)

Refine-RL leverages stable discriminative embeddings as semantic anchors and formulates a multi-reward signal that jointly maximizes rewriting quality (format adherence), embedding discriminability (gap reward), and auxiliary process signals (process reward). GRPO (Group Relative Policy Optimization) is used for robust policy learning under this reward function.

This RL formulation systematically encourages rewrites to enhance semantic richness and separability of the generative embedding over its discriminative counterpart.

Data Construction and Modality-Aware Rewriting

High-recall, high-diversity data construction is essential for robust rewrite training. The SFT corpus spans images, videos, visual documents, and VQA, including datasets such as MSCOCO, N24News, LLaVA-Hound, ViDoRe, and VisRAG. Queries and targets are both rewritten, preventing misalignment during retrieval.

Prompt templates for SFT are explicitly designed to be modality-aware, capturing context, object relations (image), reasoning (VQA), and temporal/layout structure (video, document):

Figure 3: Example prompts for Text-to-Rewrite (T2R) and Image-to-Rewrite (I2R).

Figure 4: Example prompts for Image-Text VQA to Rewrite (IT2R-VQA) and Description to Rewrite (IT2R-DESC).

Empirical Results

Benchmark Performance

RIME is evaluated comprehensively on MMEB-V2, MRMR, UVRB, and MR^2 benchmarks, establishing new state-of-the-art (open-source) results. Notable findings include:

• MMEB-V2 (7B Backbone): Overall 68.6, outperforming UME-R1 by 4.1 and VLM2Vec by 16.3. Gains are observed across all modalities (image, video, visual document) with substantial increases in visual document retrieval NDCG@5.
• MRMR: 50.2 average, exceeding prior best models by 2+ points in all subcategories (Knowledge, Theorem, Contradiction); robustness is demonstrated on expert-level, reasoning-intensive retrieval.
• UVRB (Video Retrieval): 55.6 average, outperforming Unite-7B by 1.8 and GME-7B by 2.6, particularly excelling in long-context and coarse-grained video understanding.

Ablation demonstrates that rewriting without summarization is strictly superior to chain-of-thought, both in retrieval performance (+4.4) and inference efficiency (52% sequence reduction vs. UME-R1).

Ablation and Analysis

RIME’s core components each yield incremental gains. Rewrite joint training is the largest contributor to overall improvement, with CMA regularization and Refine-RL policy optimization providing further refinement. Importantly, cross-mode alignment not only enables mutual retrieval but regularizes the discriminative embedding for lower-latency deployments.

Figures 5–9 provide real instance demonstrations of modality-aware rewriting across tasks, illustrating the diversity and robustness of generated rewrites.

Figure 5: Example of modality-aware rewrite for VQA.

Figure 6: Example of structured rewrite for document-based retrieval.

Implications and Future Directions

RIME establishes that retrieval-oriented rewriting is a strict improvement over CoT for generative multimodal embeddings—reducing redundancy and task bias while enhancing semantic alignment, particularly for retrieval across diverse scenarios and granularity. The rewrite paradigm is natively modality-agnostic and enables a universal interface for embedding models spanning image, video, and visual document modalities.

The unification and bidirectional alignment of discriminative and generative representations provide new flexibility for retrieval systems, supporting dynamic switching between efficiency and semantic depth at query/deployment time.

Future research may address the nontrivial inference latency of autoregressive rewriting by compressing explicit reasoning steps into latent, non-autoregressive processes, perhaps via amortized or latent-variable models, to enable scalable deployment without relinquishing semantic fidelity.

Conclusion

RIME provides a systematic alternative to chain-of-thought reasoning for multimodal generative embedding by introducing a structured, retrieval-friendly rewriting interface. Empirical evidence demonstrates consistent improvements across universal retrieval, reasoning-intensive benchmarks, and domain transfer. The cross-modal alignment and reinforcement optimization mechanisms further enhance representation discriminability and enable industry-relevant low-latency applications. Future work should focus on reducing the inherent computational cost of generative embedding while retaining the generality and semantic robustness demonstrated by rewriting-based paradigms.

---

Reference:

"Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings" (2604.22280)