---
title: 'ERNIE 5.0: Unified Multimodal Autoregressive Model'
url: https://www.emergentmind.com/topics/ernie-5-0
type: topic
---

# ERNIE 5.0: Unified Multimodal Autoregressive Model

ERNIE 5.0 is a trillion-parameter, natively autoregressive foundation model for unified multimodal understanding and generation across text, image, video, and audio. Trained from scratch without late-fusion modules, it casts all modalities as a single next-group-of-tokens prediction task, employs an ultra-sparse mixture-of-experts (MoE) Transformer backbone with modality-agnostic routing, and introduces a novel elastic training regime. ERNIE 5.0 further develops a robust framework for reinforcement learning post-training under ultra-sparse MoE, ensuring state-of-the-art performance and deployment flexibility for diverse resource constraints. It is the first publicly disclosed production-scale autoregressive model supporting both multimodal understanding and generation at the trillion-parameter scale [2602.04705].

## 1. Unified Autoregressive Model and Design Rationale

ERNIE 5.0 is architected to natively support all four major modalities—text, image, video, and audio—under a single autoregressive objective. Distinct from late-fusion “backbone + decoder” designs, it converts each modality’s modeling challenge into the uniform task of predicting the next group of tokens, facilitating deep cross-modal token-level interactions and eradicating the “ability seesaw” typically observed in modular approaches.

Key motivations include:

- Native autoregression for all modalities, eliminating separate decoder architectures.
- A unified learning signal via next-group-of-tokens prediction, ensuring coherent optimization as scale increases.
- Ultra-sparse MoE for scalable capacity, with modality-agnostic routing enabling both expert specialization and cross-modal knowledge sharing at less than 3% activation rate.
- Elastic pre-training to simultaneously train a "super-network" and a spectrum of sub-networks of varying depths, expert counts, and sparsity.
- Multimodal RL approaches that stabilize post-training for ultra-sparse MoE architectures across diverse modalities.

## 2. Next-Group-of-Tokens Prediction Objective

ERNIE 5.0’s learning framework treats every instance—text sequence, image, video, or audio—as a sequence of non-overlapping token groups:

Given $\mathbf{x} = (x_1, \ldots, x_T)$ organized into groups $G_1, G_2, \ldots$, the model optimizes

$$
\mathcal{L} = -\sum_{g=1}^{G} \log P(G_g \mid \mathbf{x}_{<g}) = -\sum_{g=1}^{G}\sum_{i=1}^{|G_g|} \log P(x_{t_{g-1}+i} \mid \mathbf{x}_{<t_{g-1}+i}).
$$

Token group definitions are modality-specific:

- **Text:** Uses multi-token prediction (MTP) and occasional rewinds for parallelism.
- **Vision:** Cascading multi-scale tokenizers with bit-quantized codes. Generation employs next-frame-and-scale prediction (NFSP), causing intra-scale bidirectionality and inter-scale/temporal causality.
- **Audio:** Hierarchical residual vector quantization (RVQ) generates codec tokens, with next-code prediction (NCP) enabling depth-wise semantic-to-residual modeling via teacher-forced feedback.

This approach enforces a single learning signal for all modalities and enables shared optimization trajectories.

## 3. Ultra-Sparse Mixture-of-Experts Architecture

The backbone utilizes a Transformer stack in which every few layers include an MoE block instead of a standard feed-forward network. Defining features:

- **Modality-agnostic expert routing:** A linear projection computes gating scores for each token’s hidden state $h\in\mathbb{R}^d$:
  $$
  s = W_g h + b_g, \quad \pi = \text{softmax}(s) \in \mathbb{R}^E
  $$
  Tokens are dispatched to their top-$k$ experts ($k=2$ in typical configurations).
- **Ultra-sparse activation:** Fewer than 3% of tokens activate any given expert, significantly reducing FLOPs.
- **Auxiliary-loss-free balancing:** Expert load balancing is achieved via direct gate bias updates without extra loss terms or hyperparameters.
- **Sparse expert aggregation:** Each token receives outputs only from $k$ selected experts:
  $$
  \text{MoE}(h) = \sum_{j\in \text{TopK}(\pi)} \pi_j E_j(h)
  $$
  where each $E_j$ is a 2-layer MLP expert.

The architecture enables scalable and efficient multimodal capacity sharing while maintaining low activation and computational loads.

## 4. Elastic Training and Extractable Sub-networks

ERNIE 5.0 advances an elastic training procedure designed to foster a “super-network,” from which a family of sub-models with different residence in the architecture space can be extracted post-training without any further tuning. For each mini-batch, the training pipeline samples:

1. **Depth $L'$:** $L' \in [L_{min}, L_{max}]$ (75% full, 25% random shallower).
2. **Expert count $E'$:** $E' \in \{E_{min}, E_{max}\}$ (80% full, 20% half).
3. **Routing sparsity $k'$:** $k' \in [k_{min}, k_{max}]$ (80% default, 20% lower).

The corresponding sampled sub-network is trained for the batch. Post-training, any configuration $\mathcal{M}(L',E',k')$ can be selected directly.

Empirical ablation indicates:

| Parameter Varied | Reference Loss | Altered Loss | Performance Impact                        |
|---------------------|:-------------:|:-----------:|:------------------------------------------|
| Full depth vs. 12L  | 1.941         | 2.137       | Shallower architectures slightly worse    |
| Full vs. half experts| 1.957        | 2.218       | Reduced expert width modestly worse       |
| Top-$k$ 4/2/1       | 1.971/2.003/2.175 | –       | Lower sparsity graceful drop, 0.15% loss at inference with 25% $k$ |

Fully elastic sub-models with 53.7% activation and 35.8% parameters achieve 99.5% average performance of the full model on text and vision tasks.

## 5. Reinforcement Learning Techniques for Multimodal Post-Training

Post-supervised fine-tuning, ERNIE 5.0 is further optimized by Unified Multimodal RL (UMRL), making RL tractable on ultra-sparse MoE backbones. The pipeline incorporates:

- **Unbiased Replay Buffer (U-RB):** Extends partial rollouts, enforcing data ordering for efficient and unbiased sequence sampling.
- **Multi-granularity Importance Sampling Clipping (MISC):** Based on GRPO/GSPO, applies per-token double-sided IS clipping to avoid entropy collapse and stabilize policy optimization:
  $$
  s(y) = \exp\left(\frac{1}{|y|}\sum_j \ln\frac{\pi_{\text{train}}(y_j|\cdot)}{\pi_{\text{old}}(y_j|\cdot)}\right)
  $$
  Out-of-bounds ratios are masked.
- **Well-learned Positive Sample Mask (WPSM):** Tracks success rate and entropy; once a task is “mastered,” positive gradients are masked to focus on harder queries.
- **Adaptive Hint-based RL (AHRL):** Prepends partial chain-of-thought hints for sparse-reward problems, with hint length decreasing as training progresses, accelerating convergence on challenging reasoning tasks.

This strategy ensures stable and efficient optimization for unified multimodal objectives.

## 6. Benchmarks and Empirical Evaluation

ERNIE 5.0 achieves strong and balanced multimodal performance pre- and post-RL post-training:

| Modality / Task  | Notable Benchmarks                         | Performance                                            |
|------------------|--------------------------------------------|--------------------------------------------------------|
| **Text**         | MMLU, BBH, HotPotQA, HumanEval+, LiveCodeBench, MMMLU, INCLUDE | Outperforms DeepSeek V3.2 and Kimi K2 (pre-train); matches/exceeds GPT-5 High, Gemini 3-Pro post-train |
| **Vision/Video** | ChartQA, DocVQA, OCRBench, SimpleVQA, CountBench, VideoMME, MMVU, GenEval, VBench | Comparable to Qwen-Image, Veo3; strong multimodal alignment |
| **Audio**        | AISHELL-1, LibriSpeech, VoiceBench, MMAU, TUT2017, CochlScene, SEED-TTS | ASR WER: 0.31% (AISHELL-1), 1.16%/2.61% (LibriSpeech), competitive with whisper-based and Qwen-3-Omni |
| **Ablations**    | Elasticity, routing                         | Modality-agnostic routing yields 1–2% gains; elastic $k$ yields 15% decode speedup with <0.2% loss      |

Shallow sub-networks (60% depth) incur less than 10% average performance loss while halving latency.

## 7. Routing Structure Visualization and Insights

Visualization of expert routing across layers yields the following findings:

- **Expert activation histograms:** Reveal a small core set of experts universally shared, a medium set for text + audio, and extensive tails for modality-specialized experts, particularly in vision.
- **Intersection-over-Union (IoU):** Text–audio IoU is highest in early layers (semantic overlap), image–video IoU peaks in mid-depths (spatial-feature sharing), and overall semantic alignment increases with depth.
- **Normalized entropy ($\mathrm{NE}$):** Text routing is consistently balanced ($\mathrm{NE}\approx0.95$). Visual and audio routing show alternating specialization and re-integration phases as depth increases.

Best practices emerging from the analysis include the sufficiency of modality-agnostic routing (no manual partitioning required), organic emergence of shared experts for generic reasoning, elastic training as a regularizer, and the efficacy of “corrupt-and-correct” training within NFSP to mitigate error accumulation during long visual generation runs.

ERNIE 5.0 establishes that a unified autoregressive Transformer, augmented with ultra-sparse MoE, elastic “once-for-all” training, and stabilized RL, is capable of state-of-the-art performance across all modalities and tasks, while supporting adaptable deployments under diverse operational constraints [2602.04705].

Source: https://www.emergentmind.com/topics/ernie-5-0