---
title: 'MoMo: Shared Encoder for Multimodal Fusion'
url: https://www.emergentmind.com/topics/momo-a-shared-encoder-model
type: topic
---

# MoMo: Shared Encoder for Multimodal Fusion

A shared encoder model, often referred to as "MoMo", denotes a class of architectures in which a single parameterized Transformer encoder is used to process multiple input modalities (notably text and image) jointly. The MoMo paradigm contrasts with classical dual-encoder systems that use independent networks per modality, instead seeking parameter efficiency, improved cross-modal generalization, and robust scaling behavior in data-limited regimes. Several variants have been proposed, distinguished by their modality-integration method, pretraining protocols, and task scope [2503.01654][2304.05523][2111.02358].

## 1. Unified Architecture and Modality Integration

A MoMo model is defined by its commitment to a single, shared encoder backbone. For multimodal input $x_M$ associated with modality $M$ (e.g., $M \in \{\text{text}, \text{image}\}$), the data is first tokenized—text is segmented as word pieces, images as non-overlapping patches [2503.01654][2304.05523]. Each token is projected to a vector embedding, typically $e_M \in \mathbb{R}^{s \times d}$.

To inform the encoder about modality, two main mechanisms are used:

- **Modality Feature Vectors**: A learnable vector $v_M \in \mathbb{R}^{d_\text{mod}}$ is concatenated to each token's embedding, producing $h_M^0 = [ [e_M^1 ; v_M], \ldots, [e_M^s ; v_M] ]$, yielding sequence inputs of shape $s \times (d + d_\text{mod})$ [2503.01654].
- **Modality Tokens**: A special token $m_M \in \mathbb{R}^d$ is prepended, resulting in $h_M^0 = [m_M, e_M^1, \ldots, e_M^s]$ and input of length $s+1$ [2503.01654][2304.05523].

The shared Transformer encoder stack $E$ with $L$ identical layers processes $h_M^0$ regardless of modality. The output of the "[CLS]" position, $z_M = E^L(h_M^0)[\text{CLS}]$, is used as a pooled representation for downstream objectives.

An optional architectural refinement is to wrap the shared encoder with shallow modality-specific layers (either before or after the shared stack), introducing a limited degree of specialization which can further improve retrieval or classification accuracy [2503.01654].

A third approach—embodied in the Mixture-of-Modality-Experts (MoME/MoMo) block—routes each token through a modality-specific feedforward sub-network (expert), with hard-gating based on the sequence type and Transformer depth. Lower layers use pure vision or language experts depending on token origin; upper layers can switch to a cross-modal expert for deep fusion [2111.02358].

## 2. Pretraining Objectives and Optimization

Shared encoder models employ a composite training regime integrating several losses:

- **Masked Modeling Losses**: Masked image modeling (MIM) with $\ell_2$ pixel reconstruction over random patch subsets, and masked language modeling (MLM) with cross-entropy over deleted word pieces. These objectives promote unimodal representation robustness [2304.05523][2111.02358].
- **Cross-Modal Masked Modeling**: In the context of concatenated image and text, tokens or patches from both modalities are masked simultaneously and reconstructed, enforcing joint-contextual learning [2304.05523].
- **Contrastive Loss**: With paired (image, text) data, dual projections $z_I$, $z_T$ are trained via a symmetric contrastive loss. The canonical form is:
  $$
  \mathcal{L}_\text{con} = -\frac{1}{N}\sum_{i=1}^N \left[
   \log\frac{
       \exp(\langle z_I^i, z_T^i \rangle / \tau)
   }{
       \sum_{j=1}^N \exp(\langle z_I^i, z_T^j \rangle / \tau)
   }
   +
   \log\frac{
       \exp(\langle z_T^i, z_I^i \rangle / \tau)
   }{
       \sum_{j=1}^N \exp(\langle z_T^i, z_I^j \rangle / \tau)
   }\right]
  $$
  where $\tau$ is a trainable temperature [2503.01654][2304.05523][2111.02358].

- **Image–Text Matching Loss**: A binary classifier is applied over the [CLS] embedding in bimodal input to distinguish matched from hard-negative pairs [2304.05523][2111.02358].

During pretraining, these objectives may be phased in a curriculum (pure vision, then vision+text, then joint multimodal), with a gradient accumulation scheme for simultaneous optimization across modalities to prevent catastrophic forgetting [2304.05523][2111.02358].

Optimization is typically performed with AdamW, using large-scale batches when allowed by compute, heavy data augmentation, mixed-precision arithmetic, and learning rate schedules with warm-up and cosine decay [2503.01654][2304.05523].

## 3. Training Procedures and Stagewise Curriculum

Shared encoder models are pretrained in multiple feedback-coupled stages:

1. **Image-Only Stage**: Masked patch prediction on large image collections (e.g., ImageNet-1K, PMC-OA) for several epochs, solidifying visual representations [2304.05523][2503.01654][2111.02358].
2. **Unimodal Joint Stage**: Simultaneous but modality-separated MIM and MLM, using separate batches or segment embeddings, warm-started from the vision backbone [2304.05523].
3. **Multimodal Joint Stage**: Masked modeling and contrastive/matching losses computed over paired and unpaired text–image data. Random mixing and gradient accumulation are used at each step to preserve knowledge from all modalities [2304.05523][2111.02358].

In variants with expert gating (e.g., VLMo), parameters of the newly-introduced language (or vision) experts are first frozen and only updated for their dedicated modality during their stage. Joint fine-tuning unfreezes all parameters for multimodal learning in later stages [2111.02358].

## 4. Empirical Performance and Generalization

MoMo-type shared encoder models demonstrate superior or competitive empirical performance across several modalities and benchmarks while being data and parameter-efficient.

- **Parameter Efficiency**: MoMo-Base achieves strong multimodal and unimodal results (e.g., 71.3% multimodal, 83.6% visual average, 81.8% language average) on diverse tasks with 110M parameters, outperforming FLAVA (241M parameters) given considerably less pretraining data (27M vs. 70M pairs) [2304.05523].

- **Retrieval and Classification**: On in-domain and cross-domain medical retrieval, shared encoders with modality features surpass CLIP-style dual encoders (both at parity and with more parameters) in 7/8 and 5/8 tasks, respectively. Their advantage grows as labeled data shrinks (e.g., +81% and +94% Recall@200 over modality-specific baselines at 0.66M training pairs) [2503.01654].

- **Ablations**: Early modality-specific specialization (attached before the shared encoder) consistently yields +2–3 points in retrieval; improper or missing modality identification degrades all performance dimensions [2503.01654]. Simultaneous multimodal-and-unimodal training is critical for balanced task retention [2304.05523].

- **Scaling:** Larger shared encoders yield increasing returns, with MoMo-Large (335M parameters) achieving higher VQA, COCO, and ImageNet scores, and comparable scaling seen in VLMo as switchable experts and data size increase [2304.05523][2111.02358].

A summary of key retrieval results across data scales is presented here (metrics: Recall@200; from [2503.01654]):

| Train Size (M) | Model              | Img→Txt@200 | Txt→Img@200 | ΔRel Img2Txt | ΔRel Txt2Img |
|----------------|--------------------|-------------|-------------|--------------|--------------|
| 1.74           | Modality-Specific  | 0.4643      | 0.4663      | –            | –            |
|                | Shared Encoder     | 0.4718      | 0.4721      | +1.62%       | +1.24%       |
| 0.66           | Modality-Specific  | 0.1333      | 0.1220      | –            | –            |
|                | Shared Encoder     | 0.2413      | 0.2369      | +81.02%      | +94.18%      |
| 0.33           | Modality-Specific  | 0.0207      | 0.0186      | –            | –            |
|                | Shared Encoder     | 0.0242      | 0.0241      | +16.91%      | +29.57%      |

## 5. Design Variants and Extensions

Several architectural enhancements have been introduced:

- **Expert Routing**: VLMo’s MoME block deploys separate FFNs for vision, language, and cross-modal inputs, hard-gating tokenwise for efficient yet flexible modality fusion. Bottom layers process tokens by source (vision/language), upper layers converge to joint cross-modal experts for deep interaction [2111.02358].
- **Modular Encoders for Lifelong Learning**: Inspired by advances in modular multilingual NMT, hybrid designs allow fixed shared trunks with plug-and-play per-modality or per-language heads/tails, supporting graceful extension and minimal negative interference [2004.06575].
- **Decoder Usage**: Certain variants employ shallow decoders during pretraining (mask prediction), which are discarded at fine-tuning, emphasizing efficient representation learning in the encoder [2304.05523].

Most tested MoMo frameworks currently handle only text and image, but have “intrinsic extensibility” to other data types by learning additional modality IDs or segment tags [2503.01654][2304.05523].

## 6. Advantages, Limitations, and Outlook

The MoMo shared encoder paradigm presents several advantages:

- **Parameter and Compute Efficiency:** Single-encoder models are substantially smaller and faster than dual/triple-path systems, leading to lower inference cost and feasible large-batch training [2503.01654][2304.05523].
- **Generalization in Low-Data Regimes:** Shared inductive biases and cross-modal co-training confer a significant generalization edge, particularly with limited annotated data [2503.01654].
- **Seamless Modality Integration:** Incorporation of new modalities, in principle, is straightforward via dedicated tokens or vectors, allowing extension to non-vision/text domains [2503.01654][2004.06575].

Limitations are also evident:

- **Underfitting Highly Specialized Features:** Purely shared backbones may insufficiently model exclusive modality dynamics; early modality-specific layers can partially mitigate this [2503.01654].
- **Limited Modal Scope:** Most current instantiations address only bimodal (vision-language) data; extension to audio, video, or structured modalities remains open [2304.05523].
- **Curriculum and Masking Schedules:** Uniform masking ratios and naive training schedules may not be globally optimal per task or modality. Stagewise, curriculum-learned curricula could offer gains [2304.05523].
- **Application Scope:** Generative or multi-task settings with a shared encoder are as yet underexplored [2503.01654].

Empirical results suggest that shared encoder models, if trained with appropriately careful curriculum and regularization, achieve robust cross-modal performance, competitive or superior to much larger modular systems [2304.05523][2111.02358]. The shared encoder research trajectory continues toward richer modality tags, more sophisticated expert gating, and comprehensive multi-modal scalability.

Source: https://www.emergentmind.com/topics/momo-a-shared-encoder-model