MoEMB: Sparse Mixture-Of-Experts Framework for Universal Multimodal Embeddings

• The MoEMB framework is a sparse Mixture-of-Experts designed for universal multimodal embeddings (UME), enabling efficient representation and retrieval across text, images, videos, and visual documents. The model uses a larger expert pool and activates only a subset to maintain lower computational costs, integrated into tasks like classification, visual question answering, and retrieval. It leverages native pre-trained sparse multimodal models forbetter performance on tasks like MMEB-V2 and MRMR, outconsistent with dense models complete tasks.
• MoEMB is designed to expand encoding capacity through a large expert bank while keeping active computations low, thus providing a more efficient single-vector representation relative per item index. Testing used MoEMB-A versions with various expert counts and routing strategies, improving accuracy from 32 to 256 experts by approximately 10.4 points at the cost of quality.
• MoEMB's architecture incorporates Gated DeltaNet and sparse routing mechanisms, allowing for dynamic token-level modifications and selective task prioritization. Its training leverages multimodal pretraining data without auxiliary objectives, employing a contrastive loss that emphasizes the batch size for better negative sampling.

MoEMB is a sparse mixture-of-experts framework for universal multimodal embeddings (UME), introduced to increase encoder capacity without proportionally increasing active computation, representation dimensionality, retrieval cost, or autoregressive generation. It produces a single normalized vector for instruction-conditioned text, images, videos, and visual documents, while activating only a small subset of a much larger expert bank. The framework is designed for classification, visual question answering, retrieval, grounding, and moment retrieval, and reports state-of-the-art performance on MMEB-V2 and MRMR among models trained on public MMEB-family data (Cui et al., 8 Sep 2026).

1. Motivation and design objectives

UME models must represent heterogeneous modalities and support diverse objectives within a single embedding space. Conventional scaling strategies enlarge the representation, store multiple vectors, or increase dense encoder size. Matryoshka representations vary embedding dimensionality; multi-vector and late-interaction methods retain several vectors per input; and patch-level document embeddings preserve fine-grained visual representations. These approaches can improve retrieval but increase storage per item, index size, similarity-computation cost, and retrieval latency.

Dense parameter scaling also conflicts with the large training batches required by contrastive learning. For a batch of $N$ positive query-target pairs, the other examples provide in-batch negatives. Increasing dense model size raises memory use and activations, thereby reducing batch size and the number of negatives. In the reported study, reducing the global batch from 512 to 256 lowers MMEB-V2 performance by 4.52 points.

Reasoning-token approaches introduce a different cost. Think-then-embed systems generate intermediate reasoning tokens before producing an embedding, converting encoding into an autoregressive generation problem. The reported cost comparison on the text portion of MMEB-V3 gives approximately 1.7 days for a regular 2B MLLM encoder, compared with approximately 7.8 H200-days for queries and 431 H200-days for candidates for UME-R1-2B with TTE.

MoEMB instead scales along the expert axis. Its central design objective is to expand the total capacity available to the encoder while keeping per-token active computation comparatively small. The resulting embedding remains single-vector and non-autoregressive, preserving ordinary dense-vector indexing and fixed-size candidate representations.

2. Architecture and sparse routing

MoEMB is based on native sparse multimodal Qwen3.5 backbones. Two principal configurations are reported:

Model	Active parameters	Routed experts	Top-$k$	Hidden size	Layers	Expert FFN dimension
MoEMB-A3B	3.1B	256	8	2048	40	512
MoEMB-A10B	10.2B	256	8	3072	48	1024

MoEMB-A3B contains 30 Gated DeltaNet layers and 10 full-attention layers. Each MoE layer has 256 routed experts and one shared expert, while only eight routed experts are selected for each token. Total parameters describe the capacity distributed across the expert bank; active parameters describe the computation used during one forward pass.

For a token representation $h\in\mathbb{R}^{d}$, the router computes logits

$r=W_rh,$

where $W_r\in\mathbb{R}^{E\times d}$. The selected expert set is

$\mathcal{S}=\operatorname{TopK}(W_rh,k).$

The selected routing probabilities are renormalized:

$$\alpha_j= \frac{\exp((W_rh)_j)} {\sum_{u\in\mathcal{S}}\exp((W_rh)_u)}, \qquad j\in\mathcal{S}.$$

A shared expert is retained and controlled by a sigmoid gate,

$\gamma=\sigma(w_{\mathrm{shared}}^\top h).$

The MoE output is

$$\operatorname{MoE}(h)= \gamma\,\operatorname{FFN}_{\mathrm{shared}}(h) +\sum_{j\in\mathcal{S}} \alpha_j\,\operatorname{FFN}_j(h).$$

Routing is token-level rather than exclusively modality-level: different tokens from the same input can select different experts. This permits specialization finer than a fixed assignment of all image, video, document, or text tokens to predetermined modality experts.

After encoding, MoEMB pools a designated embedding representation and applies normalization. The reported routing and pooling ablation favors a dedicated <emb> token over last-token pooling: last-token pooling obtains 69.50 overall, whereas <emb> pooling obtains 70.34–70.38.

3. Multimodal specialization and training formulation

MoEMB is trained on 24 streams consisting of 17 image-text tasks from MMEB, four visual-document streams from ViDoRe and VisRAG, and three video streams derived from LLaVA-Hound. The task mixture covers images, videos, visual documents, classification, question answering, retrieval, grounding, and moment retrieval.

The main training objective is a one-directional contrastive loss. For normalized query and target embeddings $q_i$ and $k$0,

$k$1

The large global batch is essential because each nonmatching target supplies an in-batch negative. The standard training configuration uses global batch size 512, maximum sequence length 8192, BF16, AdamW, $k$2, $k$3, weight decay 0.01, ten warmup steps, cosine decay from $k$4 toward $k$5, two epochs, and 64 H200 GPUs with expert parallelism through Megatron Core MoE. Generated chain-of-thought sequences are excluded from training targets.

The preferred initialization uses native pretrained sparse multimodal models:

• Qwen3.5-35B-A3B for MoEMB-A3B;
• Qwen3.5-122B-A10B for MoEMB-A10B.

The router is retained and preferably frozen. LoRA is applied to the vision encoder, attention projections, and expert feed-forward layers, with rank $k$6 and scale $k$7. The reported trainable parameter counts are approximately 73.6M for MoEMB-A3B and 196.6M for MoEMB-A10B.

The design study indicates that native sparse MoE initialization is substantially more effective than constructing an MoE from a dense model during contrastive adaptation. Dense-to-MoE upcycling performs poorly at equal active compute, while modality-level MoE routing provides only a marginal improvement over dense computation. The reported analysis therefore attributes much of MoEMB’s effectiveness to routing patterns learned during large-scale multimodal pretraining.

Routing distributions exhibit modality specialization. The average Jensen–Shannon distance is 0.114 within modalities and 0.323 between modalities. The number of strongly modality-preferring experts declines from 172 at an early layer to 43 at layer 38, suggesting stronger modality specialization in early layers and more shared multimodal processing in deeper layers.

4. Expert scaling and routing adaptation

MoEMB evaluates expert counts $k$8 while maintaining top-8 routing. Increasing the total expert count from 32 to 256 improves accuracy by 10.4 points for MoEMB-A3B, while QPS changes from 34.2 to 29.3. This establishes the principal scaling behavior: total capacity increases substantially while active routed computation remains approximately bounded.

The paper also investigates router adaptation. Compared strategies include a frozen pretrained router, a trainable router, load-balancing and $k$9-losses, conflict-aware routing, task-ID priors, and instruction priors. The best default retains the pretrained router, freezes it, introduces no auxiliary routing objective, and uses <emb> pooling. This configuration reaches 70.38 in the reported ablation.

Load-balancing and $h\in\mathbb{R}^{d}$0-losses reduce performance in the evaluated setting. Their objective is represented as

$h\in\mathbb{R}^{d}$1

with $h\in\mathbb{R}^{d}$2 and $h\in\mathbb{R}^{d}$3. The reported interpretation is that auxiliary balancing disrupts useful pretrained expert assignments. Conflict-aware routing produces only a negligible improvement, while task-ID and instruction priors also underperform the frozen-router default.

The study reports that global batch size is the most important hyperparameter. Halving LoRA rank costs 2.91 points; reducing logit scale from 50 to 20 costs 1.10 points; lowering the learning rate from $h\in\mathbb{R}^{d}$4 to $h\in\mathbb{R}^{d}$5 improves accuracy by 1.02 points; and a second epoch at the lower learning rate adds 0.55 points.

5. Adaptive computation

MoEMB studies both training-based and inference-only methods for reducing computation. The evaluated approaches include token pruning, expert skipping, adaptive top-$h\in\mathbb{R}^{d}$6, layer skipping, early exit, and combinations of these methods.

Training-based token pruning removes tokens early in the network. Gradual pruning maintains an alive mass,

$h\in\mathbb{R}^{d}$7

Expert skipping learns a gate that bypasses the routed-expert branch while retaining the shared branch and other layer operations. Adaptive top-$h\in\mathbb{R}^{d}$8 introduces null experts and dynamically varies the number of physical experts selected. The trained policy converges to an average of 2.94 physical experts per token, corresponding to a 63.3% reduction in routed expert execution, but its quality is weaker than inference-only routing.

Inference-only token pruning uses the Gated DeltaNet write-strength coefficient $h\in\mathbb{R}^{d}$9 as a token-importance signal. The recurrence contains

$r=W_rh,$0

where $r=W_rh,$1 controls how strongly the current token writes to recurrent memory. Visual tokens with lower ranked $r=W_rh,$2 values are discarded after an early block, while text tokens are preserved. The reported mean tokenwise coefficient of variation is 0.226 for $r=W_rh,$3 compared with 2.316 for attention. Cross-layer cosine similarity is 0.989 for $r=W_rh,$4 compared with 0.752 for attention, and cross-frame video similarity is 0.992 for $r=W_rh,$5 compared with 0.744 for attention.

Inference-only adaptive top-$r=W_rh,$6 retains expert $r=W_rh,$7 when

$r=W_rh,$8

where $r=W_rh,$9 is a layer scaling factor and $W_r\in\mathbb{R}^{E\times d}$0 is a global threshold. This approach uses the pretrained router’s ranking without retraining.

The unreduced reference obtains 70.3 overall with normalized QPS of 1.00. Selected results are:

Method	Budget	Overall	FLOPs reduction	QPS
No reduction	Top-8, all tokens	70.3	—	1.00
Uniform top-4	50% expert reduction	69.8	11.6%	1.19×
MoDES	50% skip	69.8	13.8%	1.18×
GDN-$W_r\in\mathbb{R}^{E\times d}$1 pruning	Keep 50% visual tokens	69.2	22.8%	1.39×
MoDES + GDN-$W_r\in\mathbb{R}^{E\times d}$2	Combined	69.1	28.9%	1.44×
One-shot trained token drop	Drop 70% visual tokens	69.3	32%	—
Learned expert skip	Dynamic	69.4	12%	—
Trained layer skip	Skip 20%	69.5	About 20%	About 1.40×
GRPO gradual drop + skip	Dynamic	69.6	22.5%	1.46×

The general result is that approximately half of logical computation can be removed with less than one point of quality loss, although the tradeoff depends on the method and budget. Inference-only adaptive top-$W_r\in\mathbb{R}^{E\times d}$3 obtains 69.8, whereas trained AdaMoE obtains 67.2 despite averaging 2.94 physical experts per token.

The router’s ordering is important. At an identical 11.6% FLOP reduction, selecting the router’s top four experts costs 0.5 points, whereas selecting four random experts costs 6.6 points. This indicates that computation reduction depends not merely on the number of experts executed but on preserving the pretrained routing preference.

6. Evaluation, efficiency, and limitations

MoEMB is evaluated on MMEB-V2 and MRMR. MMEB-V2 contains 78 tasks: 36 image tasks, 18 video tasks, and 24 visual-document tasks. MRMR contains 11 reasoning-intensive subtasks covering knowledge, theorem, and contradiction categories. Its metrics are NDCG@10 for most subtasks, Hit@1 for Negation, and the mean across all 11 subtasks.

MMEB-V2

Model	Active parameters	Overall
VLM2Vec-V2-2B	2.2B	59.1
UME-R1-2B	2.2B	61.2
PLUME	2.1B	61.6
BToks	2.2B	59.0
MoEMB-A3B	3.1B	70.4
Embed-RL-2B	10.9B	66.8
Embed-RL-4B	13.2B	68.1
UME-R1-7B	8.3B	65.6
UniME-V2	8.0B	60.6
MoEMB-A10B	10.2B	72.4

MoEMB-A3B obtains 70.4 overall with 3.1B active parameters, exceeding Embed-RL-4B at 68.1 with 13.2B active parameters and UME-R1-7B at 65.6 with 8.3B active parameters. MoEMB-A10B obtains 72.4. MoEMB-A3B reports category scores of 71.8 for image, 53.5 for video, and 81.0 for visual documents. MoEMB-A10B obtains 74.0, 56.3, and 82.2 for the same categories.

MRMR

Model	Active parameters	MRMR average
VLM2Vec-V2	2.2B	28.1
PLUME	2.1B	39.9
UME-R1-2B	2.2B	40.0
BToks	2.2B	43.5
MoEMB-A3B	3.1B	51.6
UniME-V2-7B	8.0B	42.0
UME-R1-7B	8.3B	48.2
LaME-7B	8.3B	48.9
Embed-RL-4B	13.2B	46.8
Ops-MM-Embedding	7.0B	48.1

MoEMB-A3B obtains 51.6 on MRMR, exceeding PLUME by 11.6 points, UME-R1-7B by 3.4 points, and Embed-RL-4B by 4.8 points. The result supports the reported conclusion that increasing encoder capacity through sparse experts can be more effective for reasoning-intensive retrieval than adding autoregressive reasoning tokens.

Serving efficiency

FLOP reduction does not necessarily translate directly into wall-clock speed. Retaining the original tensor shape while masking tokens provides approximately 1.02× speedup, whereas physical compaction with packed variable-length sequences provides larger gains:

Keep ratio	FLOPs reduction	QPS speedup
75%	9.9%	1.088×
50%	19.7%	1.194×
25%	29.6%	1.228×

Packed variable-length kernels convert approximately 63–82% of removed FLOPs into actual wall-clock savings.

For expert execution, a naive sequential expert loop is 5.7–8.0 times slower than fused grouped GEMM. At one H200, SonicMoE improves QPS from 53.6 at top-8 relative to standard grouped GEMM’s 38.1, and reaches 66.1 at a mean active $W_r\in\mathbb{R}^{E\times d}$4 of 1.30. Further reducing $W_r\in\mathbb{R}^{E\times d}$5 eventually saturates because attention, shared experts, token dispatch, and other dense operations dominate.

Limitations

MoEMB’s reported limitations include dependence on native pretrained sparse MoE backbones, since dense-to-MoE upcycling is ineffective under the available contrastive training budget. The current model supports image, video, visual-document, and text inputs but not audio. Training-based adaptive computation can be unstable because discrete decisions have delayed quality effects, high-variance gradients, and competing token and expert gates.

Aggressive pruning can degrade performance on complex visual documents, reasoning-intensive tasks, and video tasks requiring temporal context. Visual documents require preservation of layout and text tokens, while the final layer is important for constructing the embedding and is rarely skipped. The model also requires specialized sparse-MoE infrastructure, including effective grouped-GEMM kernels and packed variable-length execution.

MoEMB therefore does not eliminate the costs of large multimodal encoders. The complete expert bank still consumes memory, expert parallelism increases systems complexity, and total capacity does not imply proportional latency reduction. Its principal contribution is the demonstration that native sparse MoE capacity, combined with a pretrained router and selective computation, can improve universal multimodal embeddings while preserving single-vector, non-autoregressive retrieval.