---
title: 'Covo-Audio: 7B Audio Language Model'
url: https://www.emergentmind.com/topics/covo-audio
type: topic
---

# Covo-Audio: 7B Audio Language Model

Covo-Audio is a 7-billion-parameter end-to-end Large Audio Language Model (LALM) capable of processing continuous audio inputs and generating speech outputs within a single, unified Transformer-based architecture. Designed to address a broad range of speech and audio-centric tasks—encompassing speech-text modeling, spoken dialogue, speech and audio understanding, and full-duplex voice interaction—Covo-Audio exhibits strong semantic reasoning and conversational abilities. Evaluation demonstrates state-of-the-art or competitive performance among models of comparable scale across rigorous multilingual and multimodal benchmarks [2602.09823]. Distinctive variants include Covo-Audio-Chat, tailored for spoken conversational applications, and Covo-Audio-Chat-FD, which supports robust full-duplex voice interaction. The model introduces an intelligence–speaker decoupling strategy to enable flexible voice customization at reduced data cost without degrading dialogue proficiency.

## 1. Model Architecture

Covo-Audio employs a modular “speech-adapter + LLM + speech-decoder” design. Key architectural components are as follows:

- **Audio Encoder:** Utilizes a frozen Whisper-large-v3 model (1.5B parameters) to transform raw waveforms into 80-dimensional log-Mel spectrograms at 50 Hz.
- **Audio Adapter:** Implements three stacked down-sampling blocks (each Linear→ReLU→Linear, followed by a 1D convolution with stride 2), thereby reducing the 50 Hz feature rate to 6.25 Hz and aligning it with the LLM’s hidden dimensionality.
- **LLM Backbone:** Based on Qwen2.5-7B-Base (32 Transformer layers, 4096 hidden units, 32 attention heads, MLP expansion by 4×), extended to handle a unified vocabulary comprising both text tokens and discrete speech tokens. The architecture features interleaved cross-modal self-attention to integrate continuous audio embeddings and tokenized text/audio signals.
- **Speech Tokenizer & Decoder:** Uses a VQ-VAE with Gumbel-softmax and a 1024-token codebook for discrete speech tokenization. Decoding proceeds via a two-stage head: (1) Flow-Matching-based latent reconstructor for converting discrete tokens to continuous latent vectors; (2) BigVGAN vocoder generating final 24 kHz audio waveforms.

The overall parameter breakdown is approximately: 1.5B (encoder), ~50M (adapter), ~6.5B (LLM backbone), and ~50M (decoder/vocoder) [2602.09823].

## 2. Pre-training and Post-training Regimes

### 2.1 Two-Stage Pre-training

Pre-training leverages nearly 2 trillion task tokens, with progressive task diversity and difficulty:

- **Stage 1 (Adapter Only, 30B tokens):** ASR objective ($a_c \rightarrow t$), aligning continuous audio features to text via next-token cross-entropy loss ($L_{CE}$).
- **Stage 2 (Joint Adapter + LLM, 800B–900B tokens per task):** 
  - *ASR:* $a_c \rightarrow t$ (80B tokens)
  - *TTS:* $t \rightarrow a_d$ (160B tokens)
  - *Audio-only:* $a_c \rightarrow a_d$, $a_d \rightarrow a_d$ (240B tokens)
  - *Speech continuation:* $a_c \rightarrow t$ (160B)
  - *Hierarchical Tri-modal Interleaving:* Sequential ($a_c \rightarrow t \rightarrow a_d$) and parallel ($a_c \rightarrow (t | a_d)$) patterns (540B)
  - *Text-only:* $t \rightarrow t$ (800B)
  - *Full-Duplex:* $(a_c \rightarrow t|a_d)$ (5B)

Cross-entropy is the universal learning objective: 
$$
L_{CE} = -\sum_{(x,y)} \sum_{i=1}^{|y|} \log p(y_i \mid x, y_{<i})
$$

### 2.2 Alignment-Focused Fine-Tuning

Post-training involves 50k steps with clustering of training objectives weighted by task group:

| Group                | Tasks                                 | Sampling Ratio |
|----------------------|---------------------------------------|:--------------:|
| General Intelligence | $t\rightarrow t$, $t\rightarrow a_d$, $a_c\rightarrow t$, $a_c\rightarrow t|a_d$ | 40% |
| Spoken Dialogue      | $a_c\rightarrow t|a_d$ (multi-turn)   | 30%            |
| Speech Understanding | $a_c\rightarrow t$, $a_c|t\rightarrow t$, $a_c|a_c\rightarrow t$, $a_c|a_c\rightarrow t|a_d$ | 10% |
| Speech Generation    | $t\rightarrow a_d$                    | 10%            |
| Audio Understanding  | $t|a_c\rightarrow t$                  | 10%            |

Weighted multi-task loss is applied: $L_{total} = \sum_k w_k L_{CE}^k$, with $\sum_k w_k=1$ [2602.09823].

## 3. Training Objectives and Learning Paradigms

The primary optimization target is autoregressive cross-entropy; for instruction-tuning in text-to-text (T2T) mode, a KL-distillation auxiliary is added with $\alpha=0.1$:
$$
L_{T2T} = (1-\alpha) L_{CE} + \alpha \textrm{KL}(p_{student} \| p_{teacher})
$$

For multi-modal reasoning, particularly on MMAU/MMSU evaluation, a composite reinforcement learning (RL) reward is employed:
$$
R_{total} = R_{accuracy} + R_{format} + R_{consistency} + R_{thinking}
$$
Each sub-reward in $[0,1]$ targets answer accuracy, format compliance, reasoning coherence, and quality-of-thought, combined in gradient-based RL for model alignment.

## 4. Evaluation Benchmarks and Quantitative Results

Extensive evaluation spans speech-text, dialogue, full-duplex, paralinguistics, and audio understanding tasks.

### 4.1 Speech–Text Foundation

Signal comprehension, text generation, and grammar generalization (Table 3):

| Metric           | Score     |
|------------------|-----------|
| A2A-tSC          | 83.3      |
| A2T              | 95.7      |
| T2T              | 99.4      |
| sBLIMP           | 61.6      |
| sWUGGY           | 74.9      |

ASR and TTS (Table 4): WER on Aishell-1 = 1.96%, LS-clean = 1.96%, LS-other = 4.55%. TTS MOS: English = 2.44, Mandarin = 1.73.

### 4.2 Speech-to-Speech Dialogue

Half-duplex (Covo-Audio-Chat) on URO-Bench:

- Chinese: SQuAD = 77.34, OpenbookQA = 83.60, APE = 68.42, MLC = 80.69
- English: Gsm8k = 85.68
- Oral Evaluation: AlpacaEval = 90.02, Wildchat = 90.41

VCB-Bench: Instruction Following TIF = 93.07 (ZH), 89.94 (EN); Multiturn Dialogue (MTD) = 87.70; Robustness (SV = 88.94, EV = 87.13, CV = 90.37).

### 4.3 Empathy & Paralinguistics

Mandarin empathy (VStyle table): scores $≈4.8$–$5.0/5$ across four emotional dimensions; AIR-Bench paralinguistics average = 80.86% (emotion, gender, age).

### 4.4 Full-Duplex Interaction

Covo-Audio-Chat-FD demonstrates:

- URO-Bench: Chinese Repeat = 98.35, SQuAD = 75.16; English Repeat = 94.64, Gsm8k = 80.47
- Behavioral metrics: Turn-taking = 99.7%, Pause Handling = 97.6%, Backchanneling = 93.9%, Interruption = 96.8%

### 4.5 Audio Understanding

- MMAU average: 75.30% (Sound: 78.68, Music: 76.05, Speech: 71.17)
- MMSU average: 66.64% (Perception: 58.95, Reasoning: 74.83), the highest among all open/closed models evaluated (Table 6) [2602.09823].

## 5. Conversational and Full-Duplex Variants

### 5.1 Covo-Audio-Chat

This variant is post-trained on 10M text-instructions with parallel TTS, ~100k multi-speaker dialogues, and emotional data, without increasing parameter count. It targets high-quality spoken dialogue, instruction following, and empathetic response generation.

### 5.2 Covo-Audio-Chat-FD

Supports full-duplex voice interaction by modifying input streaming: chunk-streaming encoder with a 1:4 user:output chunk interleaving (0.16s granularity). Special tokens (THINK, SHIFT, BREAK) enable explicit modeling of conversational turn-taking and overlap, trained end-to-end on half- and full-duplex dialogues. Tabled behavioral metrics confirm robust turn-management.

## 6. Intelligence–Speaker Decoupling Strategy

To address the scarcity of high-quality, speaker-specific dialogue data versus the abundance of TTS voice samples, Covo-Audio implements an “intelligence–speaker decoupling” pipeline. The training strategy proceeds as follows:

1. **Pseudo-Conversation Pretraining:** The LALM is trained using multi-speaker TTS data formatted as “pseudo-conversations,” masking text response loss ($L_{resp} = 0$) to focus adaptation on audio output.
2. **Joint Fine-Tuning:** The model is further trained with genuine dialogue data from other speakers, then contextual TTS adaptation is applied to transfer speaker style.

Audio loss remains standard CE on audio output ($L_{audio} = L_{CE}(a_d|…,h)$), ensuring accurate voice style transplantation while preserving core dialogue intelligence. Empirical validation shows that the Covo-Audio-Chat-TTS variant, trained with only lightweight TTS data, suffers less than 1% performance drop across core spoken dialogue benchmarks (Table “URO-All”).

## 7. Insights, Limitations, and Future Perspectives

Covo-Audio demonstrates that a 7B-scale LALM—if trained with large-scale, tri-modal (audio, text, speech) data, hierarchical objectives, and targeted fine-tuning—can match or exceed much larger models on a diverse set of speech, text, and audio benchmarks. Critical aspects contributing to performance include:
- Model–data co-design minimizing parameter bloat
- Sophisticated pretraining/post-training curricula with explicit multimodal alignment
- Robust cross-modal fusion architecture

Limitations include the context window restriction (currently 8k tokens), motivating future work on longer-context architectural variants. Planned extensions include reinforcement learning (e.g., RLHF) for granular conversational act separation, sparse/MoE scaling for larger models at tractable cost, and unified benchmark development for multi-round, full-duplex evaluation (e.g., MTR-Duplex-Bench).

These developments signify a scalable trajectory toward highly capable, multi-purpose audio language models tightly integrating audio intelligence, semantic reasoning, and customized voice rendering [2602.09823].

Source: https://www.emergentmind.com/topics/covo-audio