---
title: 'LLaMA-Omni 2: Real-Time SpeechLM'
url: https://www.emergentmind.com/topics/llama-omni-2
type: topic
---

# LLaMA-Omni 2: Real-Time SpeechLM

LLaMA-Omni 2 is a series of Speech Language Models (SpeechLMs) designed for real-time, high-fidelity spoken chatbot applications. Integrating a speech encoder, an autoregressive streaming speech decoder, and a large language model (LLM) backbone, LLaMA-Omni 2 achieves state-of-the-art performance on spoken question answering and speech instruction following tasks. Despite being trained on only 200,000 multi-turn speech dialogue samples, it surpasses existing benchmarks established by models trained on much larger datasets, such as GLM-4-Voice [2505.02625].

## 1. Model Architecture and Dataflow

LLaMA-Omni 2 utilizes a modular architecture structured around the following components:

- **Speech Encoder:** Inputs speech $X$ as 80-dimensional log-mel features and encodes via a frozen Whisper-large-v3 (≈1.5B parameters).
- **Speech Adapter:** Applies 5× frame downsampling followed by a feedforward network (FFN), producing representations for the LLM.
- **LLM Backbone:** Qwen2.5-Instruct (0.5B–14B parameters) acts as a decoder-only Transformer for speech-instruction understanding.
- **Gated Fusion:** Combines LLM hidden states and sampled text tokens, providing context to the speech generation module.
- **Text-to-Speech Language Model ($\mathcal{M}_\mathrm{TTS}$):** An autoregressive Transformer LM, initialized from Qwen2.5-0.5B, with vocabulary expanded by 6,561 discrete speech tokens.
- **Chunk-aware Causal Flow Matching + Vocoder:** Converts discrete speech tokens into mel-spectrograms using a frozen, pretrained CosyVoice 2 model and then synthesizes audio waveforms via a streaming HiFi-GAN vocoder (≈50M parameters).

**High-level dataflow:**  
Input speech $X$ → Whisper encoder → speech adapter → LLM → gate fusion → TTS LM → flow matching → mel -> HiFi-GAN vocoder → output waveform $Y^S$.

## 2. Component Specifications

### Speech Encoder

- **Whisper-large-v3**: Processes 80-dimension log-mel features (25 ms window, 10 ms shift); Transformer architecture; 1.5B parameters; weights frozen during LLaMA-Omni 2 training.

### Speech Adapter

- **Downsampling:** Concatenates every $k=5$ frames, reducing input sequence length by 5×.
- **FFN:** Single-layer, intermediate dimension 2048, output matching LLM input dimension.

### LLM Backbone

- **Qwen2.5-Instruct:** Model sizes are 0.5B, 1.5B, 3B, 7B, 14B; decoder-only, standard self-attention; trained by cross-entropy loss mapping speech instructions to text responses.

### Text-to-Speech LM ($\mathcal{M}_\mathrm{TTS}$)

- Same architecture as Qwen2.5-0.5B; vocabulary expanded by 6,561 new discrete speech tokens.
- All weights initialized from Qwen2.5-0.5B, new token embeddings randomly initialized ($\approx$0.5B parameters).

### Flow Matching Model and Vocoder

- **Chunk-aware Flow Matching:** Pretrained CosyVoice 2 (frozen); performs streaming synthesis every $W$ tokens.
- **HiFi-GAN Vocoder:** Streams 2$W$ mel frames per chunk; ≈50M parameters.

## 3. Training Methodology

### Dataset Construction

- **200,000 Multi-turn Dialogues:** Derived by rewriting Alpaca and UltraChat corpora into dialogues ($N \sim \mathrm{Pois}(\lambda=2)$, truncated to 1–5 turns).
- **Instruction Synthesis:** "fish-speech-1.5" produces random-voice prompts, voice cloned per-dialogue via CosyVoice2-0.5B.
- **Response Synthesis:** Uniform voice output generated by CosyVoice2-0.5B.

### Training Stages

1. **Speech-to-Text (Stage I(a)):**  
   - Freeze speech encoder; train adapter and LLM with pairs $\langle X, Y^T \rangle$.
   - Loss:  
     $$
     \mathcal{L}_\mathrm{ST} = -\sum_{t=1}^N \log P(y^T_t \mid X,\;y^T_{<t})
     $$

2. **TTS LM Pretraining (Stage I(b)):**  
   - Train $\mathcal{M}_\mathrm{TTS}$ on $\langle Y^T, Y^U \rangle$, gate fusion disabled.
   - Loss:  
     $$
     \mathcal{L}_\mathrm{TTS}^\mathrm{offline} = -\sum_{i=1}^M \log P(y^U_i \mid \mathbf{e}_{\leq i}^\mathrm{emb}, Y^U_{<i})
     $$

3. **End-to-end Speech-to-Speech (Stage II):**  
   - Freeze encoder, adapter, LLM; train gate fusion and $\mathcal{M}_\mathrm{TTS}$ on $\langle X, Y^S \rangle$.
   - Streaming TTS loss (read–write):
     $$
     \mathbf{C} = [c_1,\dots,c_N],\quad
     \mathcal{L}_\mathrm{TTS} = -\sum_{i=1}^M
     \log P\Bigl(y^U_i \mid
     \mathbf{C}_{\leq \min\!\left(\lfloor\frac{i-1}{\mathcal{W}+1}\rfloor\!\cdot\!\mathcal{R},N\right)},\,
     Y^U_{<i}
     \Bigr)
     $$

### Hyperparameters

- Batch size: 32; Stage I(a) – 3 epochs, learning rate $5 \times 10^{-5}$; Stage I(b) – 5 epochs, $5 \times 10^{-4}$; Stage II – 1 epoch, $1 \times 10^{-3}$.
- 3% warmup, cosine-annealing schedule.
- 4×H800 GPUs (14B), 4×L40 for smaller models.

## 4. Real-Time Streaming and Decoding

### Read–Write Streaming

For every $R$ LLM tokens (default $R=3$), $W$ speech tokens ($W=10$) are synthesized. After the LLM completes, remaining speech tokens are generated in an autoregressive manner.

**Latency Calculation:**
$$
\mathcal{T}_{\text{total}} = \mathcal{T}_{\mathrm{LLM}}(R)
+ \mathcal{T}_{\mathrm{TTS}}(W)
+ \mathcal{T}_{\mathrm{FM}}(W)
+ \mathcal{T}_{\mathrm{Voc}}(2W)
$$

- On NVIDIA L40, Omni2-7B with $(R,W) = (3,10)$ achieves $\approx$583 ms end-to-first-chunk latency.

### Decoding Algorithms

- **LLM:** Greedy decoding for stable generation.
- **TTS LM:** Sampling with temperature 1.0 to minimize repetition.
- **Flow Matching and Vocoder:** Streaming synthesis per chunk.

### Gate Fusion

The gate fusion mechanism computes the fused representation $\mathbf{c}_i$ as:
$$
\begin{aligned}
\mathbf{e}_i^{\rm hidden} &= \mathrm{FFN}(\mathbf{h}_i), \\
\mathbf{e}_i^{\rm emb}   &= \mathrm{Emb}(y^T_i), \\
\mathbf{g}_i             &= \sigma\left(W_g[\mathbf{e}_i^{\rm hidden}\!\parallel\!\mathbf{e}_i^{\rm emb}]+b_g\right), \\
\mathbf{c}_i             &= \mathbf{g}_i \odot \mathbf{e}_i^{\rm hidden} + (1-\mathbf{g}_i)\odot \mathbf{e}_i^{\rm emb}.
\end{aligned}
$$

**Read–Write Streaming Algorithm (pseudocode)**
```python
initialize i_text=1, i_speech=1
while not end_of_llm_output:
  read R new text tokens y^T_{i_text…i_text+R-1}
  compute hidden_states h_{…}
  compute fused C_{…}
  for w in 1..W:
    predict y^U_{i_speech} = TTS_LM(C_{≤⌊(i_speech-1)/(W+1)⌋·R}, Y^U_{<i_speech})
    i_speech += 1
continue generating remaining Y^U in AR fashion
```

## 5. Performance and Benchmarking

LLaMA-Omni 2 demonstrates strong performance in both speech-to-text (S2T) and speech-to-speech (S2S) tasks across various metrics. Key results are summarized below.

| Model             | Llama Qs S2T | S2S  | Web Qs S2T | S2S  | GPT Score S2T | S2S  | ASR-WER↓ | UTMOS↑ | Latency (ms)↓ |
|-------------------|-------------:|-----:|-----------:|-----:|--------------:|-----:|---------:|-------:|--------------:|
| GLM-4-Voice (9B)  |        64.7  | 50.7 |     32.2   | 15.9 |         4.16  | 4.09 |     9.02 |  3.48 |       1562.8  |
| LLaMA-Omni (8B)   |        67.7  | 49.0 |     33.4   | 23.7 |         3.99  | 3.52 |     5.95 |  3.67 |        346.7  |
| Omni2-7B          |        70.3  | 60.7 |     34.5   | 31.3 |         4.28  | 4.15 |     3.26 |  4.19 |        582.9  |
| Omni2-14B         |        73.0  | 62.7 |     40.4   | 37.1 |         4.56  | 4.35 |     3.89 |  4.20 |        663.3  |

**Key observations:**
- Omni 2 markedly improves both S2T and S2S accuracy versus GLM-4-Voice, sharply reducing the S2T→S2S drop (Web Qs: $-3.2$ for Omni 2 versus $-16.3$ for GLM-4-Voice).
- Instruction following (GPT-4o scores), ASR-WER, and naturalness (UTMOS $\approx 4.2$) are superior under real-time streaming synthesis.
- End-to-first-chunk latency ($\sim$600 ms) is about 2× faster than GLM-4-Voice and meets real-time deployment requirements.

## 6. Significance, Context, and Implications

LLaMA-Omni 2 establishes that integration of a lightweight Qwen2.5 LLM with frozen open speech components (Whisper, CosyVoice 2) and end-to-end streaming AR TTS modeling enables efficient, high-quality real-time spoken dialogue systems. The fact that Omni 2 achieves superior performance relative to models trained on orders of magnitude more audio (e.g., millions of hours for GLM-4-Voice) with only 200,000 synthetic multi-turn dialogues suggests a shift in the relative importance of data quantity versus architectural modularity and pretraining alignment in SpeechLMs.

A plausible implication is that, for real-time high-fidelity spoken interaction, exhaustive supervised speech data may be less critical than previously assumed, provided that pre-trained components and data-efficient fusion strategies are employed. This suggests new avenues for multimodal chatbot training that emphasize modular integration, parameter efficiency, and low-latency streaming generation [2505.02625].

Source: https://www.emergentmind.com/topics/llama-omni-2