Unified Multimodal Embedding Space for Direct Cross-Modal Search

Construct a unified embedding space spanning text, images, audio, and video that enables direct multimodal search without intermediary conversion modules (e.g., automatic speech recognition), thereby improving alignment and retrieval in multimodal retrieval-augmented generation.

Background

The paper argues that compositional reasoning and alignment across modalities are difficult and that current retrieval pipelines often depend on conversion modules (such as ASR) rather than native cross-modal embeddings.

It identifies building a unified embedding space for all modalities as an open and high-potential direction to enable direct multimodal search.