---
title: 'BEiT-3: Multimodal Transformer Models'
url: https://www.emergentmind.com/topics/beit-3-models
type: topic
---

# BEiT-3: Multimodal Transformer Models

BEiT-3 Models are general-purpose multimodal foundation models based on a unified Multiway Transformer architecture, enabling deep fusion of vision and language modalities for a broad spectrum of vision, language, and vision-language tasks. Leveraging masked "language" modeling objectives across images, text, and image-text pairs, BEiT-3 achieves state-of-the-art transfer performance and forms the backbone of several leading models in open-vocabulary segmentation and referring expression understanding [2208.10442, 2412.18917, 2406.20076].

## 1. Multiway Transformer Backbone and Architectural Innovations

BEiT-3 introduces a Multiway Transformer backbone that generalizes and synthesizes the architectural principles of both vision and language Transformer encoders. Each layer comprises a shared multi-head self-attention (MHSA) module and a modality-specific pool of feed-forward experts (FFNs):

- **Input Embeddings**: Images $I\in\mathbb{R}^{H\times W\times 3}$ are split into non-overlapping patches (e.g., $14\times14$ or $16\times16$), linearly projected to obtain $V\in\mathbb{R}^{N_p\times d}$. Text is tokenized (e.g., XLM-RoBERTa tokenizer) to $L\in\mathbb{R}^{N_w\times d}$.
- **Token Routing and Fusion**: For joint tasks, visual and language tokens are concatenated; MHSA operates on the combined sequence, followed by routing to the modality-specific FFN (V-FFN, L-FFN, VL-FFN in top layers).
- **Deep Fusion**: Cross-modal alignment and interaction occur at every block via shared attention weights, with earlier layers preserving modality–specific pathways and top layers enabling modality fusion [2208.10442].

The base version uses $L=12$ layers, $d=768$ hidden size, and $h=12$ heads, while the large model employs $L=24$ and $d=1024$; the full-scale release scales to 40 layers with a hidden size of 1408 and 16 attention heads.

## 2. Unified Masked "Language" Modeling Pretraining

BEiT-3 conducts self-supervised pretraining by treating both images and texts as token sequences within a shared space, introducing three primary objectives:

- **Imglish (Masked Image Modeling, MIM)**: Images are discretized via a VQ tokenizer; $n$ patches are randomly masked, and reconstruction is supervised with a cross-entropy loss over VQ tokens.
- **English (Masked Text Modeling, MTM)**: Standard BERT/RoBERTa-style masking and prediction over text.
- **Masked Parallel Image-Text (MM)**: Jointly masks both vision and language tokens in aligned image–text pairs, with losses for both modalities:
  $$
  \mathcal{L}_{\mathrm{mm}} = -\mathbb{E}_{(x,t)\sim\mathcal{D}_{\mathrm{mm}}}\left[
    \sum_{i\in\mathcal{M}_{\mathrm{img}}}\log p(v_i|\widetilde{x},\widetilde{t}) +
    \sum_{j\in\mathcal{M}_{\mathrm{txt}}}\log p(w_j|\widetilde{x},\widetilde{t})
  \right]
  $$
  
The aggregate objective is $\mathcal{L}_{\mathrm{pretrain}} = \mathcal{L}_{\mathrm{img}} + \mathcal{L}_{\mathrm{txt}} + \mathcal{L}_{\mathrm{mm}}$ [2208.10442]. All objectives share a single backbone, driving unified, cross-modal pretraining.

## 3. Model Scaling and Parameterization

BEiT-3 adopts a ViT-giant recipe for scaling, maintaining balance between capacity and efficiency:

| Model   | Layers | Hidden | MLP   | V-FFN   | L-FFN   | VL-FFN | Shared Attn | Total Params |
|---------|:------:|:------:|:-----:|:-------:|:-------:|:------:|:-----------:|:------------:|
| BEiT-3  | 40     | 1408   | 6144  | 692 M   | 692 M   | 52 M   | 317 M       | 1.9 B        |

Further breakdown:
- **Patch size**: $14\times14$ (for 224×224 images)
- **Expert FFNs**: Substantial capacity in both vision ($\sim$692 M) and language ($\sim$692 M) paths
- **VL-FFN**: Fused in the top three layers for maximal cross-modal information exchange

This scaling approach yields consistent gains, with deeper, wider models outperforming shallower ones across downstream benchmarks [2208.10442, 2406.20076].

## 4. Downstream Applications: Open-Vocabulary and Referring Segmentation

BEiT-3 constitutes the core of several state-of-the-art segmentation and vision-language models.

### OMTSeg: Open-Vocabulary Panoptic Segmentation

OMTSeg exploits the BEiT-3-Large backbone (frozen) for panoptic segmentation, with key structural choices:

- **Multiway Fusion Encoder**: Stacked "fusion" Transformer layers enable vision–language mixing at every block.
- **Adapters and Prompt Tuning**: Dense prediction adapts via a spatial "visual adapter" (SFI) and prompt tuning, modifying only selected prompt embeddings (e.g., category token deltas).
- **Segmentation Head**: Adapts Mask2Former’s multi-decoder; performs cross-attention to both visual $\mathbf{F}_V$ and linguistic $\mathbf{F}_L$ streams, followed by mask prediction, contrastive alignment, and objectness scoring.

Empirically, OMTSeg achieves state-of-the-art open-vocabulary semantic segmentation mIoU on multiple datasets and panoptic quality (PQ), while being highly parameter-efficient (0.72B) compared to CLIP-based models:

| Method         | Params | ADE20K PQ/AP/mIoU | COCO PQ/AP/mIoU   |
|----------------|--------|-------------------|-------------------|
| ODISE          | 1.5 B  | 22.6 / 14.4 / 29.9| 55.4 / 46.0 / 65.2|
| OMTSeg (Ours)  | 0.72 B | 27.5 / 17.4 / 34.8| 54.9 / 45.0 / 64.0|

Ablative analysis highlights large performance drops when omitting cross-modal attention (PQ 27.5→18.4) or the visual adapter (PQ 27.5→8.8) [2412.18917].

### EVF-SAM: Referring Expression Segmentation

EVF-SAM demonstrates the impact of early-fusion BEiT-3 backbones for text-prompted Segment Anything (SAM):

- **Early Fusion**: Runs self-attention over concatenated image and text streams at every layer, outperforming late-fusion or unimodal text encoders.
- **Prompt Generation**: A [CLS] token from the fused stream is projected as a prompt token for SAM’s sparse prompt interface.

Reported RefCOCO (testA) cIoU metrics:

| Model                           | cIoU    |
|----------------------------------|---------|
| CLIP (text+image, late fuse)     | 67.9    |
| BEiT-3 (early fuse, 1–24 layers) | 83.7    |
| ViLT (early fusion)              | 75.3    |

EVF-SAM achieves state-of-the-art results with fewer parameters (1.32B vs. 7.7B for LISA), and ablation studies confirm that BEiT-3-generated prompts, even with minimal fine-tuning, are sufficient for high segmentation accuracy [2406.20076].

## 5. Pretraining Data, Protocols, and Hyperparameters

BEiT-3 pretraining leverages only publicly available data:

- **Image–Text pairs**: 21M (CC12M, CC3M, SBU, COCO, VG)
- **Images**: 14M (ImageNet-21K)
- **Text**: 160 GB diverse corpora

Key pretraining choices include:
- **Optimizer**: AdamW ($\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=1\text{e}^{-6}$)
- **Weight decay**: 0.05
- **Learning rate schedule**: cosine decay, peak $1\text{e}^{-3}$, 10k warmup steps
- **Augmentations**: random crop, flip, color jitter
- **Batch**: 6144 (2048 each image, text, pair)
- **Resolution**: 224×224 for pretrain; up to 896 or 1280 for various finetuning tasks

Fine-tuning protocols adjust schedules, drop rates, and data augmentation for task specialization across segmentation, detection, VQA, and retrieval. Mask2Former adapters and prompt tuning are commonly used for dense prediction tasks [2208.10442, 2412.18917].

## 6. Quantitative Performance and Comparative Analysis

Across a spectrum of benchmarks, BEiT-3 models reach or surpass prior state of the art in both vision and vision-language domains:

### Vision Tasks (No Extra Data):

| Task                   | Dataset   | Metric           | Previous SOTA    | BEiT-3                  |
|------------------------|-----------|------------------|------------------|-------------------------|
| Semantic Segmentation  | ADE20K    | mIoU (+MS)       | 61.4 (FD-SwinV2) | **62.8** (+1.4)         |
| Object Detection       | COCO      | AP               | 63.3 (DINO)      | **63.7** (+0.4)         |
| Instance Segmentation  | COCO      | AP<sup>mask</sup>| 54.7 (Mask DINO) | **54.8** (+0.1)         |
| Classification         | ImageNet  | Top-1 Acc.       | 89.0 (FD-CLIP)   | **89.6** (+0.6)         |

### Vision–Language Tasks:

- **Visual Reasoning (NLVR2)**: 92.6 test [CoCa: 87.0]
- **VQA (VQAv2)**: 84.0 [CoCa: 82.3]
- **Image Captioning (COCO Karpathy)**: CIDEr 147.6 [OFA: 145.3]
- **Retrieval (Flickr30K, COCO)**: 98.0/94.9 img→txt R@1, 90.3/81.5 txt→img R@1 in finetuned/zero-shot [2208.10442]

Performance gains are attributed to the unique early and deep fusion properties of the architecture, lightweight adapters, and prompt tuning mechanisms, as evidenced in OMTSeg and EVF-SAM deployments.

## 7. Architectural Advantages, Limitations, and Research Outlook

BEiT-3’s design offers several documented advantages:

- **Layer-wise vision-language cross-attention** fosters rich regional and contextual feature extraction, exceeding the capability of two-stream CLIP-based systems [2412.18917].
- **Prompt specialization** via trainable tokens enables superior open-vocabulary generalization [2412.18917].
- **Early cross-modal attention** enhances fine-grained visual-linguistic alignment, especially beneficial for referring segmentation [2406.20076].
- **Parameter efficiency** allows high-quality dense tasks with reduced backbone size (0.7–1.3B vs. 7–13B for LLM alternatives).

Empirical ablations demonstrate that omitting cross-modal attention or the spatial adapter leads to dramatic performance reduction. In masked segmentation, excluding BEiT-3-generated prompts slashes accuracy by over 60 points (cIoU). This suggests cross-modal interaction is essential for BEiT-3's success [2412.18917, 2406.20076].

A plausible implication is that future models may benefit from scaling BEiT-3’s early-fusion backbones further and exploring more sophisticated, task-specific adapters within the Multiway Transformer framework. The modular, expert-driven design supports extensibility and domain adaptation across the vision-language spectrum.

Source: https://www.emergentmind.com/topics/beit-3-models