---
title: 'VLMo: Unified Vision-Language Model'
url: https://www.emergentmind.com/topics/vlmo
type: topic
---

# VLMo: Unified Vision-Language Model

VLMo is a unified Vision-Language pretrained Model introduced by Bao, Wang, Dong, Liu, et al. that jointly learns a dual encoder and a fusion encoder with a modular Transformer network. Its defining architectural idea is the Mixture-of-Modality-Experts (MoME) Transformer, in which each block contains a pool of modality-specific experts and a shared self-attention layer. Because of the modeling flexibility of MoME, pretrained VLMo can be fine-tuned as a fusion encoder for vision-language classification tasks, or used as a dual encoder for efficient image-text retrieval. The model was presented as a single backbone that can support both efficient retrieval and reasoning-intensive cross-modal understanding, together with a stagewise pre-training strategy that leverages image-only data, text-only data, and image-text pairs [2111.02358].

## 1. Conceptual position and design objective

VLMo was proposed to unify two paradigms that had often been separated in vision-language pre-training: the dual encoder used for efficient image-text retrieval, and the fusion encoder used for classification and reasoning tasks. In VLMo, the same MoME backbone is reused in both modes, with the routing of inputs through modality-specific experts determining whether the model behaves as a dual encoder or a fusion encoder [2111.02358].

This design centers on a modular Transformer rather than on separate model families for separate task classes. The intended benefit is architectural reuse across tasks with different computational profiles. In dual encoder mode, images and texts are encoded separately and compared through embedding similarity, which supports fast retrieval. In fusion encoder mode, image tokens and text tokens are concatenated and processed jointly, which supports deep cross-modal interaction for tasks such as VQA and NLVR2 [2111.02358].

A plausible implication is that VLMo treats retrieval and cross-modal reasoning not as incompatible objectives but as two operating regimes of the same pretrained system. That interpretation is consistent with the paper’s emphasis on “modeling flexibility” and with the fact that the same backbone is fine-tuned differently depending on the downstream task [2111.02358].

## 2. MoME Transformer architecture

The core architectural modification in VLMo is the replacement of the standard Transformer feed-forward network with a Mixture-of-Modality-Experts layer. Each Transformer block contains a shared Multi-Head Self-Attention layer and a pool of modality-specific experts: a vision expert (V-FFN), a language expert (L-FFN), and a vision-language expert (VL-FFN). The self-attention parameters are shared across all input types, whereas the expert used in the FFN path depends on modality and, for paired inputs, on layer position [2111.02358].

Formally, if $\mathbf{h}_{l-1}$ is the input to layer $l$, the block is written as follows:
$$
\begin{align}
\mathbf{h}_l' &= \mathrm{MSA}(\mathrm{LN}(\mathbf{h}_{l-1})) + \mathbf{h}_{l-1} \\
\mathbf{h}_l &= \mathrm{MoME}\text{-}\mathrm{FFN}(\mathrm{LN}(\mathbf{h}_l')) + \mathbf{h}_l'
\end{align}
$$

For image-only inputs, V-FFN is used at all layers. For text-only inputs, L-FFN is used at all layers. For image-text pair inputs, bottom layers use V-FFN for vision tokens and L-FFN for text tokens, while top layers switch both to VL-FFN for cross-modal fusion [2111.02358].

The architecture therefore combines parameter sharing and modality specialization. Shared attention exposes all modalities to the same self-attention mechanism, whereas the expert routing preserves modality-dependent transformations in the FFN path. The ablations reported for VLMo state that shared self-attention outperforms separate self-attention, which suggests that early cross-modal alignment is beneficial within this modular design [2111.02358].

## 3. Input representations and operating modes

VLMo uses standard tokenization schemes adapted to a unified multimodal backbone. Images are represented through patch embeddings, as in ViT or BEiT, together with position and type embeddings. Text is represented through token embeddings with position and type embeddings. For image-text inputs, the two token sequences are concatenated [2111.02358].

The model supports two principal operating modes.

| Mode | Input organization | Primary use |
|---|---|---|
| Dual encoder | Images and texts encoded separately | Efficient image-text retrieval |
| Fusion encoder | Image and text tokens concatenated | Classification and reasoning tasks |

In dual encoder mode, images and texts are processed independently by the same MoME backbone acting as separate encoders. Retrieval is then performed through cosine or dot-product similarity between the output embeddings. The paper emphasizes that this yields linear-time retrieval because image and text embeddings can be pre-computed and indexed [2111.02358].

In fusion encoder mode, the image and text tokens are processed jointly. Bottom layers apply modality-specific experts, and top layers apply the VL-FFN expert for fusion. This regime is used for tasks requiring strong cross-modal interaction, including VQA and NLVR2. The trade-off stated in the paper is that each image-text pair must be encoded together, so retrieval under this regime becomes quadratic time rather than the pre-computable retrieval available in dual encoder mode [2111.02358].

This operational bifurcation is one of VLMo’s most distinctive properties. Rather than selecting between a two-tower architecture and a single-stream cross-encoder at design time, VLMo exposes both behaviors within the same pretrained system.

## 4. Stagewise pre-training and optimization objectives

VLMo uses a three-stage pre-training strategy designed to exploit image-only data, text-only data, and image-text pairs. The first stage is vision pre-training on image-only data with masked image modeling, e.g., BEiT, to train the vision expert and shared self-attention. The second stage is language pre-training on text-only corpora with masked language modeling, while freezing the attention module and vision expert and training the language expert. The third stage is vision-language pre-training on image-text pairs with image-text contrastive loss, image-text matching loss, and masked language modeling [2111.02358].

The stagewise schedule is central to VLMo’s formulation because it explicitly addresses the limited and sometimes poor-quality image-text paired data described in the paper. By first training modality-specific capability on larger unimodal corpora and then aligning modalities on paired data, VLMo attempts to combine scale with cross-modal transfer [2111.02358].

The image-text contrastive loss is defined over projected, normalized image vectors $\{\hat{\mathbf{v}}_i\}$ and text vectors $\{\hat{\mathbf{w}}_i\}$:
$$
\begin{align}
s_{i,j}^{i2t} &= \hat{\mathbf{v}}_i^\top \hat{\mathbf{w}}_j \\
s_{i,j}^{t2i} &= \hat{\mathbf{w}}_i^\top \hat{\mathbf{v}}_j \\
p_{i}^{i2t} &= \frac{\exp\left(s_{i,i}^{i2t}/\sigma\right)}{\sum_{j=1}^N \exp\left(s_{i,j}^{i2t}/\sigma\right)} \\
p_{i}^{t2i} &= \frac{\exp\left(s_{i,i}^{t2i}/\sigma\right)}{\sum_{j=1}^N \exp\left(s_{i,j}^{t2i}/\sigma\right)}
\end{align}
$$

The image-text matching loss is a binary classification objective over whether an image and text match, using the [CLS] token and hard-negative mining with negative pairs sampled by a global in-batch strategy. The masked language modeling objective masks 15% of tokens and predicts them with cross-entropy loss, following the BERT-style formulation [2111.02358].

The implementation details reported for the original model include a Base configuration with 12 layers, 768 hidden dimension, 12 heads, and feedforward dimension 3072, and a Large configuration with 24 layers, 1024 hidden dimension, 16 heads, and feedforward dimension 4096. Images are processed at $224 \times 224$ with patch size $16 \times 16$. Optimization uses AdamW, with learning rate $2e{-}4$ or $5e{-}5$ for Base and Large respectively, batch size up to 32k for large-scale training, and vision-language pre-training for up to 200k steps [2111.02358].

## 5. Empirical results and ablation findings

VLMo was reported to achieve state-of-the-art results on various vision-language tasks at the time of publication. On VQA v2.0, VLMo-Large++ achieves **82.78 vqa-score** on test-std, outperforming Florence-Huge (**80.36**), SimVLM-Huge (**80.34**), and UNITER-Large (**74.02**). On NLVR2, VLMo-Large++ achieves **89.54% accuracy** on test-P, surpassing SimVLM-Huge (**85.15**) and VinVL (**83.98**). On COCO retrieval, VLMo-Large++ achieves **83.1/96.0/98.2** for text retrieval Recall@1/5/10 and **65.2/86.5/92.2** for image retrieval Recall@1/5/10, surpassing ALIGN and Florence-Huge according to the summary provided [2111.02358].

| Task | Metric | Reported VLMo-Large++ result |
|---|---|---|
| VQA v2.0 | test-std vqa-score | 82.78 |
| NLVR2 | test-P accuracy | 89.54% |
| COCO retrieval | TR@1/5/10 | 83.1/96.0/98.2 |
| COCO retrieval | IR@1/5/10 | 65.2/86.5/92.2 |

The ablation results summarized for VLMo identify three especially important factors. First, MoME outperforms a standard Transformer for both retrieval and reasoning tasks. Second, stagewise pre-training with both image-only and text-only initialization yields gains. Third, shared self-attention outperforms separate self-attention, supporting the claim that shared attention contributes meaningfully to multimodal alignment [2111.02358].

These results define VLMo’s place in the 2021 wave of large-scale vision-language pre-training: it is neither simply a retrieval model nor simply a fusion model, but an attempt to make one pretrained architecture competitive in both categories.

## 6. Later adaptations, downstream use, and interpretation

Subsequent work adapted VLMo beyond the canonical vision-language benchmarks. In recommendation, the paper “Large Multi-modal Encoders for Recommendation” describes VLMo as a unified (single-stream) large multi-modal encoder based on the MoME Transformer architecture, and tailors VLMo and CLIP for recommendation through plug-in use of pre-trained encoders, further fine-tuning on recommendation data, and end-to-end training with recommendation loss. Across three Amazon Review datasets—Sports, Clothing, and Baby—the study reports that pre-trained VLMo or CLIP significantly outperforms CNN+Sentence-BERT features in 79% of 120 tested cases, and that fine-tuning further improves performance in about 70% of cases. The same study also reports that end-to-end training does not always boost performance for VLMo and may even decrease it, while pre-trained or fine-tuned VLMo is the better regime for that application [2310.20343].

That downstream evidence is informative because it highlights what later authors regarded as VLMo’s distinctive transferable property: deep alignment of modalities through joint processing, rather than shallow alignment from separately extracted modality-specific features. In the recommendation setting, the paper explicitly contrasts VLMo’s native cross-modal interaction with dual-stream models that process modalities separately and fuse at the top [2310.20343].

This suggests a broader interpretation of VLMo’s legacy. VLMo’s importance lies not only in its benchmark results, but also in its proposal that modality-specific specialization and shared multimodal computation can coexist within one Transformer through expert routing. Later work on multimodal systems and multimodal mixture-of-experts often revisits related questions—how much to share, how much to specialize, and at which depth fusion should occur. VLMo provided one early, explicit answer to that design space by assigning shared self-attention to all inputs while placing specialization in the FFN path [2111.02358].

The original paper states that code and pretrained models are available at **https://aka.ms/vlmo**, which contributed to VLMo’s role as a reusable backbone rather than only a benchmark submission [2111.02358].

Source: https://www.emergentmind.com/topics/vlmo