ONE-PEACE: Unified Multimodal Transformer

• ONE-PEACE is a unified 4-billion parameter Transformer model that integrates vision, audio, and language modalities via dedicated adapters and a shared self-attention encoder.
• It employs modality-agnostic contrastive pretraining with intra- and cross-modal losses to achieve state-of-the-art performance on diverse uni-modal and multi-modal tasks.
• The model’s modular design facilitates extensibility by allowing seamless integration of new modalities through plug-and-play adapters and modality-specific feed-forward networks.

ONE-PEACE is a 4-billion parameter, Transformer-based general representation model designed to align and integrate vision, audio, and language modalities in a unified, extensible architecture. The model is structured for scalability, enabling seamless inclusion of additional modalities through modular components. ONE-PEACE employs modality-agnostic pretraining objectives for both intra- and cross-modal alignment, and is pretrained from scratch (i.e., without initialization from specialized vision or LLMs), demonstrating state-of-the-art performance on a diverse range of uni-modal and multi-modal tasks (Wang et al., 2023).

1. Architectural Design

The core of ONE-PEACE is a modular, “sharing-separated” architecture in which modality-specific processing is decoupled from modality-agnostic attention:

• Modality Adapters: Each modality (vision, audio, language) is equipped with a dedicated adapter converting raw input into token sequences with a unified embedding dimension $D$.
  • Vision adapter uses a hierarchical MLP stem to patchify and embed images.
  • Audio adapter processes normalized 16 kHz waveforms via a CNN extractor, producing convolutional relative positional encodings.
  • Language adapter embeds BPE subwords with absolute 1D positional encodings.
• Modality Fusion Encoder: A stack of Transformer blocks, each comprising:
  • A single multi-head self-attention sublayer, shared across all modalities and incorporating relative positional bias (2D for vision, 1D for audio/text).
  • $K$ modality-specific feed-forward sublayers (FFNs) — one per modality — each using GeGLU activations and LayerScale.
  • Sub-LayerNorm normalizes inputs/outputs of the attention and FFN layers for training stability.
• Extensibility: New modalities are incorporated by adding a corresponding adapter and FFN per block, with no modification to the shared attention mechanism or training code.
• Lightweight Decoder: Used exclusively during self-supervised masked-unit pretraining for intra-modal contrastive tasks.

The following table summarizes the adapter and FFN organization:

Component	Purpose	Per-Modality Instance?
Adapter	Preprocess raw input to $D$-dim tokens	Yes
Self-Attention	Modality fusion, token exchange	No (shared)
FFN	Modality-specific transformation	Yes

2. Data Flow and Modality Fusion

ONE-PEACE supports both uni-modal and multi-modal inference and training regimes:

• Uni-modal Tasks (e.g., image classification):
  • Input is processed by the relevant adapter.
  • Tokens are passed through the Modality Fusion Encoder, applying the corresponding FFN per modality.
  • [CLS] output is projected to target spaces such as class logits.
• Multi-modal Tasks (e.g., image-text, audio-text retrieval):
  • Inputs from each modality are processed by their adapters.
  • Token sequences are concatenated and fused in the encoder.
  • Cross-modal attention is facilitated through shared self-attention, while each token passes through its own modality’s FFN.
  • Multiple [CLS] tokens (one per modality) are used for contrastive learning or as heads for downstream tasks.

This enables unified representation learning while respecting intra- and inter-modal distinctions.

3. Pretraining Objectives

Pretraining leverages two modality-agnostic objectives:

• Cross-Modal Aligning Contrastive Loss ($\mathcal{L}_{CL}$):
  • Paired samples (e.g., image–text, audio–text) are mapped to global [CLS] vectors, normalized, and aligned using a symmetric NT-Xent contrastive loss with a learnable temperature. Negatives are aggregated across GPUs for large-batch contrastive optimization.
  • Separate losses are computed for vision–language ($\mathcal{L}_{CL\text{-}VL}$) and audio–language ($\mathcal{L}_{CL\text{-}AL}$) pairs.
• Intra-Modal Denoising Contrastive Loss ($\mathcal{L}_{DCL}$):
  • For each sample, a random subset of units is masked. The unmasked tokens pass through the encoder, a lightweight 2-layer decoder reconstructs the masked representations, and the loss is computed via contrastive alignment with stop-gradient on the targets.
  • This is applied to single-modality samples ($\mathcal{L}_{DCL\text{-}V}$, $\mathcal{L}_{DCL\text{-}A}$, $\mathcal{L}_{DCL\text{-}L}$) and synchronously to paired-multimodal samples ($K$0, $K$1).
• Two-Stage Pretraining:
  • Stage 1: Vision–language, updating V/L adapters and FFNs, as well as shared attention.
  • Stage 2: Audio–language, freezing all but the audio branch.

The total pretraining loss is a weighted sum of all modality-specific and cross-modal loss components.

4. Training Regimen and Implementation

ONE-PEACE is pretrained on large-scale, heterogeneous multimodal corpora:

• Image–text: LAION-2B, cleaned to 1.5B pairs (English only, $K$2 length $K$3).
• Audio–text: 2.4M pairs ($K$4K hours) from AudioSet, AudioCaps, Freesound, and others.

Key training settings include:

• Memory-efficient attention (XFormers/FlashAttention), fused LayerNorm, nvFuser operator fusion, gradient checkpointing.
• Mixed-precision (BFloat16), cross-GPU negative gathering.
• Batch size of 32,768 for vision-language stage (200k steps, image size $K$5), and 3,072 for audio-language (10 epochs, audio $K$6).

5. Empirical Performance

ONE-PEACE achieves leading results across extensive uni-modal and multi-modal benchmarks without using pretrained initialization for vision or language components.

• ImageNet-1K Top-1 accuracy: 89.8% (1.52B parameters, $K$7 patch, $K$8 resolution).
• ESC-50 Zero-shot Audio Classification: 91.8%.
• Audio–Text Retrieval: R@1 = 42.5% (AudioCaps, text$K$9audio), 51.0% (audio$D$0text). Exceeds previous SOTA (36.1%/39.6%).
• Image–Text Retrieval (MSCOCO, 5K): Zero-shot R@1 = 64.7% (image$D$1text), 48.0% (text$D$2image). Fine-tuned R@1 = 84.1%/65.4%, outperforming prior best (BEiT-3: 82.7%/65.1%).
• Semantic Segmentation (ADE20K mIoU): 63.0% (1.52B params).
• Visual Grounding (RefCOCO/+/g test-u): RefCOCO: 89.26%, RefCOCO+: 83.23%, RefCOCOg: 89.27%.

Ablation studies confirm that combining shared self-attention with separate FFNs provides superior modality transfer and alignment. Denoising contrastive loss produces improvements over L2, cosine, or Smooth L1 for masked unit training. Applying intra-modal loss across all modalities delivers gains in retrieval and ImageNet performance.

6. Extensibility and Modality Expansion

ONE-PEACE’s architecture is structured for direct scalability to new modalities. Extension follows a consistent procedure:

1. A new modality $D$3 receives a dedicated adapter mapping raw $D$4 signals to $D$5-dimensional tokens (plus [CLS]).
2. Each Transformer block incorporates an $D$6-FFN.
3. Shared self-attention and all training protocols remain unchanged.

This separation ensures “plug-and-play” capability and computational scalability. The model employs language as a universal anchor to align additional modalities in a shared semantic embedding space.

7. Limitations and Future Directions

Noted limitations include:

• Fewer image-text pairs observed in pretraining (6.4B) relative to some competitors ($D$7B), which has a marginal effect on zero-shot retrieval.
• Vision and language modules are trained entirely from scratch without dedicated pure-text pretraining, suggesting possible improvements by leveraging pretrained LLMs or larger text corpora.

Future research directions include expanding pretraining to additional modalities (e.g., video, point clouds, sensor data), integrating LLMs for enhanced multimodal generation and reasoning, and exploring active prompting or task-specific adapters within this unified framework (Wang et al., 2023).