Explain the underperformance of dense-to-MoE upcycling for multimodal embeddings

Determine whether the limited scale of contrastive training on the MMEB-V2 dataset prevents effective router training in dense-to-MoE upcycling, thereby explaining why upcycled dense multimodal encoders underperform native pretrained sparse MoE backbones.

Background

The paper compares three approaches for constructing sparse MoE universal multimodal embedding models: adapting a native pretrained sparse MoE backbone, converting a dense multimodal LLM into an MoE through upcycling, and using modality-level experts. In the reported experiments, dense-to-MoE upcycling fails to outperform the dense baseline, whereas adapting a native sparse MoE backbone produces a substantial improvement.

The authors offer a conjectural explanation rather than an established result: contrastive training on MMEB-V2 may be insufficiently large to learn a strong router from scratch, especially compared with the routing functions obtained through large-scale language-model pretraining. Establishing whether training-data scale is the decisive factor would clarify when expert capacity can be synthesized during downstream contrastive adaptation instead of inherited from pretrained sparse MoE models.

References

We conjecture this is due to that the limited scale for contrastive training over the MMEB V2 dataset is not able to train a strong router, compared to the native obtained routers from massive language pretraining.

— MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models  (2609.08663 - Cui et al., 8 Sep 2026) in Section 4, subsection “Results and Ablations on MoE Design and Training Dynamics for MoE-UME,” paragraph “Ablations on MoE Design”