Modality Embedder: Multimodal Alignment

• Modality Embedder is a mapping that converts heterogeneous inputs (text, image, audio, video) into a shared, normalized space for semantic comparison.
• It includes diverse architectures—from dual encoders to unified generative models—that address challenges like projector drift and modality collapse through contrastive and specialized loss functions.
• Applications span multimodal retrieval, emotion recognition, and recommendation systems, while tackling issues like modality-specific distortions and calibration under domain shifts.

Searching arXiv for papers on modality embedders and multimodal embedding models. A modality embedder is a model, or a pair or family of encoders plus a joint representation space, that maps heterogeneous inputs into a shared embedding space in which semantic proximity can be compared across modalities. In current usage, the term covers systems that align text, image, video, audio, visual documents, and fused multimodal inputs for retrieval, classification, grounding, and related tasks; in broader formulations it also includes structured or implicit modalities such as scene graphs, knowledge-graph concepts, and entity representations extracted from text (Li et al., 8 Jun 2026). The central requirement is not merely cross-modal similarity, but stable geometry: semantically matched inputs should be close, modality-specific distortions should be controlled, and the embedding space should remain usable under missing modalities, instruction constraints, and downstream domain shifts (Ye et al., 2024).

1. Formal definition and semantic role

A standard formulation defines a modality embedder as a mapping $f_\theta$ such that, for any modality $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$, an input $x_m$ is mapped to a shared embedding

$z = \frac{f_\theta(x_m)}{\|f_\theta(x_m)\|},$

with retrieval performed by inner product or cosine similarity on normalized vectors (Li et al., 8 Jun 2026). A closely related vision–language formulation treats the modality embedder as the pair of encoders and the joint space itself: a text encoder and an image encoder produce vectors in a common latent space where matched pairs have small distance and mismatched pairs are separated (Ye et al., 2024).

This shared-space view generalizes in several directions. In label-anchored systems for multimodal emotion recognition, the embedder is not only an encoder of inputs but also a mechanism that transfers textual emotion prototypes into acoustic and visual spaces, then scores an input by dot product against those prototypes (Dai et al., 2020). In post-training semantic compression, the embedder is the precondition for replacing multiple modality-specific vectors that express the same semantics with a single centroid, because sufficiently reduced modality gap implies that cross-modal instances share a common semantic region of the space (Grassucci et al., 29 Sep 2025).

A recurring implication is that a modality embedder is defined less by a specific architecture than by a contract: it must preserve semantic equivalence across modalities while remaining numerically comparable under a common metric. This suggests that “embedding model” and “alignment mechanism” are inseparable in practice.

2. Architectural regimes

Current modality embedders fall into several distinct architectural regimes.

Regime	Mechanism	Representative system
Specialist composition	Decoupled Specialist Fusion, Projector Recovery	Conan-embedding-v3 (Li et al., 8 Jun 2026)
Training-free instruction guidance	Explicit One-word Limitation, semantic SVG rewriting	mEOL (Kim et al., 18 Apr 2026)
Unified generative-embedding stack	One LLM for embeddings and generation, PoolAggregator	MM-GEM (Ma et al., 2024)
Modality completion	Text-to-pseudo-visual tokens, pseudo-visual encoder	UniMoCo (Qin et al., 17 May 2025)
Sparse subset fusion	Modal Channel Attention with channelized masks	MCA (Bjorgaard, 2024)

In specialist-composition systems, modality experts are trained independently and later fused in parameter space. Conan-embedding-v3 exemplifies this regime with a decouple–fuse–recover pipeline: image, video, document, and audio specialists are trained from the same initialization, their task vectors are fused into a single dense backbone, and projector-based modalities are recalibrated afterward (Li et al., 8 Jun 2026). This design treats the shared backbone as the principal semantic substrate while preserving modality-specific interfaces such as external audio encoders and projectors.

A different regime avoids parameter training almost entirely. mEOL uses a single Multimodal LLM to summarize text, raster images, and SVG code into one generated token, then uses the hidden state of that token at the penultimate layer as the embedding. Alignment is achieved by modality-specific instructions rather than learned projection heads or contrastive objectives (Kim et al., 18 Apr 2026). The semantic SVG rewriting module adds structure by assigning meaningful identifiers and simplifying SVG hierarchy, so the embedder can exploit geometry and part relations directly.

Unified generative-embedding architectures collapse encoder and decoder roles into one text backbone. MM-GEM uses one LLM as both text encoder and autoregressive decoder, with image embeddings obtained from pooled visual features and text embeddings extracted from a special [EMB] token. The same model also supports captioning and region-level generation, with PoolAggregator supplying either global mean pooling or RoIAlign-based local pooling (Ma et al., 2024). This architecture makes the modality embedder simultaneously a retriever and a generator.

Other designs explicitly target incomplete modality availability. UniMoCo completes missing visual input by generating pseudo visual tokens from text, encodes them with a separate pseudo-visual encoder, and then feeds both text and visual streams through a unified LVLM interface. The resulting embedding is intended to be robust across combinations such as $(T+I,T)$, $(T,T+I)$, and $(T+I,T+I)$ (Qin et al., 17 May 2025). In sparse multimodal fusion, MCA instead builds distinct fusion channels for modality subsets in a single forward pass and uses attention masks so unimodal blocks self-attend while fusion tokens mediate subset-specific interactions (Bjorgaard, 2024).

3. Alignment objectives and optimization geometry

InfoNCE is the most common retrieval objective in this literature. Conan-embedding-v3 uses pairwise contrastive loss over modality pairs such as $(\text{text},\text{image})$, $(\text{text},\text{video})$, $(\text{text},\text{doc})$, and $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$0, with $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$1 and in-batch negatives (Li et al., 8 Jun 2026). e5-omni likewise adopts a contrastive formulation but adds three explicit alignment components: modality-aware temperature calibration, a controllable negative curriculum with debiasing, and batch whitening with covariance regularization (Chen et al., 7 Jan 2026).

These additions address a central geometric problem: embeddings can occupy a single nominal space while still having modality-dependent sharpness, hardness distributions, and first- and second-order statistics. e5-omni introduces a learnable per-modality temperature vector so that the similarity logit for a pair is scaled by a symmetric pairwise temperature, then applies masked Debiased Contrastive Learning and a CORAL-style covariance penalty (Chen et al., 7 Jan 2026). This makes the modality embedder explicitly calibrated rather than merely co-trained.

Other systems intervene at the batch-construction level. UNITE identifies competitive relationships among heterogeneous target modalities and introduces Modal-Aware Masked Contrastive Learning, which masks out negatives that do not share the same target modality combination as the query’s positive. In Stage 1 it uses bidirectional InfoNCE over text–text, text–image, and text–video data; in Stage 2 it applies MAMCL to instruction data so image-text and video-text instruction samples do not interfere destructively (Kong et al., 26 May 2025).

In settings where one modality dominates optimization, the issue is not only alignment but collapse. VLM2Rec shows that standard contrastive supervised fine-tuning can make the fused update direction align increasingly with text while vision degrades. It addresses this with Weak-modality Penalized Contrastive Learning, which computes a per-user modality gap and dynamically penalizes negatives, and with Cross-Modal Relational Topology Regularization, which aligns row-wise similarity distributions across modalities by symmetric KL divergence (Kim et al., 18 Mar 2026). This suggests that modality embedders require not just a shared objective, but active control of gradient asymmetry.

A different but related line integrates acoustic information into contextual lexical embeddings before contextualization. In “Multimodal Embeddings from LLMs,” the input token embedding is modulated by an acoustic gate,

$$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$2

so paralinguistic information alters semantics before the bidirectional LLM processes context (Tseng et al., 2019). This is a modality embedder at the token level rather than only at the instance level.

4. Fusion, completion, and missing-modality handling

Fusion in modality embedders is not a single operation but a family of strategies for composing heterogeneous evidence.

Conan-embedding-v3 performs fusion in parameter space. Shared parameters are combined by task arithmetic, audio-only parameters are copied, and the resulting backbone is then repaired through projector-only tuning and balanced rehearsal. The need for this recovery stage follows from Projector Drift: an audio projector trained against the audio-specialist backbone remains calibrated to that backbone after fusion, so audio retrieval regresses even if audio-specific modules are copied unchanged (Li et al., 8 Jun 2026). This is an architectural failure mode specific to projector-based modalities.

UniMoCo addresses a different failure mode: heterogeneous modality combinations at inference. Its modality-completion module generates pseudo visual tokens directly from text, and a dedicated pseudo-visual encoder plus padding strategy makes the completed stream compatible with the real-image pathway (Qin et al., 17 May 2025). The method then aligns original and completed views so the model does not learn separate subspaces for text-only and text-plus-image examples. A plausible implication is that modality completion serves not only as imputation but as combination-space regularization.

Sparse-fusion methods treat missingness structurally. MCA defines fusion channels corresponding to modality subsets and uses masks so that loss terms are dropped when the relevant modalities are absent, while unimodal embeddings remain invariant to the presence or absence of other modalities (Bjorgaard, 2024). Earlier constrained-latent approaches achieve independent conditional inference by introducing separate modality-specific latent variables and forcing them to be close, rather than requiring a single shared latent estimated jointly from all modalities (Chaudhury et al., 2017).

At the decision level, fusion can also remain late. Omni-Embed-Nemotron reports that simple early fusion of audio and video underperformed on FineVideo, whereas late score fusion of separately encoded audio and video performed best for joint-modal retrieval (Xu et al., 3 Oct 2025). This suggests that a modality embedder need not always collapse modalities into one vector before scoring; modality-preserving score combination can be preferable when streams have different temporal or noise characteristics.

5. Evaluation, diagnostics, and empirically observed failure modes

Evaluation has expanded from image–text retrieval to full-modality and agent-centric settings. MMEB-V3 covers 190 tasks across image, video, visual documents, audio, text, and agent scenarios, and introduces OmniSET, in which semantically equivalent instances are represented across text, image, video, and audio so that modality effects can be disentangled from semantic similarity (Huang et al., 25 Apr 2026). This benchmark identifies three recurrent findings: models often fail to retrieve the intended target modality, cross-modal retrieval is highly asymmetric and dominated by query-modality bias, and instruction-induced shifts are either insufficient or misaligned with the target modality (Huang et al., 25 Apr 2026).

This diagnostic picture reframes several earlier results. Conan-embedding-v3 reports 74.96 overall on MMEB-V2, with Image 77.20, Video 65.10, and VisDoc 79.00, while also achieving 55.61 on the 30-task MAEB audio suite and a task-type average of 59.32 (Li et al., 8 Jun 2026). e5-omni reports 66.4 overall on MMEB-V2 and 37.7 Recall@1 on AudioCaps, with ablations showing that removing modality-aware temperature or whitening degrades both MMEB-V2 and AudioCaps performance (Chen et al., 7 Jan 2026). UNITE 7B reports 70.3 overall on MMEB and 72.5 Recall@1 on WebVid-CoVR, with MAMCL improving both MMEB and CoVR when image-text and video-text instruction data are jointly trained (Kong et al., 26 May 2025).

Training-free prompting can also be competitive. On the repurposed VGBench text-to-image retrieval task, mEOL with Qwen2.5-VL-7B reports $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$3, $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$4, $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$5, $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$6, and $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$7, outperforming encoder-based and training-based baselines in the reported setting (Kim et al., 18 Apr 2026). The improvement from image+generated SVG over image only further indicates that structural information can materially change embedding distinctiveness (Kim et al., 18 Apr 2026).

Several failure modes recur across papers. Projector Drift is a calibration mismatch after specialist fusion (Li et al., 8 Jun 2026). Combination bias arises when training data overrepresents certain modality pairings and models generalize poorly to others; UniMoCo explicitly identifies and quantifies this bias (Qin et al., 17 May 2025). Modality collapse occurs when optimization is dominated by one modality and the weaker one loses separability, as in VLM2Rec (Kim et al., 18 Mar 2026). Query-modality bias and directional asymmetry are benchmark-level symptoms: OmniSET shows examples such as $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$8 versus $$m \in \{\text{text}, \text{image}, \text{video}, \text{document}, \text{audio}\}$$9 for some models, and instruction shifts that change embeddings without reliably moving them closer to the target modality (Huang et al., 25 Apr 2026).

Interactive diagnostics form a complementary strand. ModalChorus introduces Modal Fusion Map, a parametric dimensionality reduction method with metric and nonmetric terms, and reports inter-modal $x_m$0 and $x_m$1 on COCO, outperforming DCM, t-SNE, and MDS in the reported comparison (Ye et al., 2024). This suggests that modality embedder evaluation increasingly includes visualization of neighborhood trustworthiness rather than only retrieval metrics.

6. Applications, limitations, and emerging directions

The primary application remains multimodal retrieval, but the scope has widened. Omni-Embed-Nemotron uses a Thinker-based bi-encoder to support text, image, audio, and video retrieval for heterogeneous RAG corpora, including layout-preserving PDF and slide retrieval and late-fused video+audio search (Xu et al., 3 Oct 2025). Agent-centric benchmarks now test retrieval of tools, GUI trajectories, and memories under explicit structured instructions rather than ordinary semantic matching (Huang et al., 25 Apr 2026). In recommendation, VLM2Rec adapts VLMs into CF-aware multimodal embedders for sequential recommendation rather than general retrieval (Kim et al., 18 Mar 2026).

The modality set itself is also expanding. “Entity Embeddings” proposes treating conceptual entities extracted from text as implicit modalities that can be mapped into the same linguistic space as images or audio, making places, corporations, dates, or numbers candidates for modality-specific encoding and token interleaving (Unlu et al., 2023). Structured annotations offer an intermediate route: embeddings learned from Visual Genome scene graphs provide complementary grounded information to text-only embeddings while remaining far lighter than image-based pipelines (Verő et al., 2021). Knowledge-fusion work similarly combines text, knowledge-graph, and image embeddings into concept representations using normalization, weighting, and linear fusion such as SVD or PCA (Thoma et al., 2017).

A further direction is storage-aware embedding design. Semantic Compression via Multimodal Representation Learning argues that reducing modality gap makes post-training centroid compression feasible, allowing multiple modality-specific embeddings expressing the same concept to be replaced by a single centroid and then further compressed by Random Feature Selection (Grassucci et al., 29 Sep 2025). This suggests that alignment is not only a retrieval objective but also a systems objective.

Several misconceptions are now difficult to sustain. A shared embedding space does not, by itself, guarantee modality-aware retrieval behavior; MMEB-V3 shows that current models still often fail to obey explicit target-modality instructions (Huang et al., 25 Apr 2026). Naïve parameter fusion does not automatically preserve grafted modalities; Conan-embedding-v3 shows that projector-based paths can regress severely without post-fusion recovery (Li et al., 8 Jun 2026). Conversely, a modality embedder need not always rely on contrastive training or learned projection heads; mEOL demonstrates that instruction-level control over an MLLM’s hidden state can yield competitive retrieval on structure-aware data (Kim et al., 18 Apr 2026).

Taken together, the literature depicts the modality embedder as a general alignment substrate rather than a fixed model class. Its contemporary forms range from dual encoders and VLM-based bi-encoders to LLM-centered hidden-state extractors, prototype systems, channelized sparse-fusion models, and implicit-entity encoders. The common research frontier is no longer simply whether modalities can share a space, but whether that space can remain calibrated, compositional, modality-aware, and operational under the full heterogeneity of real-world information systems.