Covo-Audio: 7B Audio Language Model

• Covo-Audio is a unified audio language model with 7B parameters that processes continuous audio inputs and generates speech using a single Transformer architecture.
• Its modular design combines an audio encoder, adapter, LLM backbone, and advanced speech tokenizer-decoder to seamlessly fuse multimodal information.
• The training regime employs two-stage pre-training and alignment-focused fine-tuning to reach competitive multilingual and full-duplex dialogue benchmarks.

Covo-Audio is a 7-billion-parameter end-to-end Large Audio LLM (LALM) capable of processing continuous audio inputs and generating speech outputs within a single, unified Transformer-based architecture. Designed to address a broad range of speech and audio-centric tasks—encompassing speech-text modeling, spoken dialogue, speech and audio understanding, and full-duplex voice interaction—Covo-Audio exhibits strong semantic reasoning and conversational abilities. Evaluation demonstrates state-of-the-art or competitive performance among models of comparable scale across rigorous multilingual and multimodal benchmarks (Wang et al., 10 Feb 2026). Distinctive variants include Covo-Audio-Chat, tailored for spoken conversational applications, and Covo-Audio-Chat-FD, which supports robust full-duplex voice interaction. The model introduces an intelligence–speaker decoupling strategy to enable flexible voice customization at reduced data cost without degrading dialogue proficiency.

1. Model Architecture

Covo-Audio employs a modular “speech-adapter + LLM + speech-decoder” design. Key architectural components are as follows:

• Audio Encoder: Utilizes a frozen Whisper-large-v3 model (1.5B parameters) to transform raw waveforms into 80-dimensional log-Mel spectrograms at 50 Hz.
• Audio Adapter: Implements three stacked down-sampling blocks (each Linear→ReLU→Linear, followed by a 1D convolution with stride 2), thereby reducing the 50 Hz feature rate to 6.25 Hz and aligning it with the LLM’s hidden dimensionality.
• LLM Backbone: Based on Qwen2.5-7B-Base (32 Transformer layers, 4096 hidden units, 32 attention heads, MLP expansion by 4×), extended to handle a unified vocabulary comprising both text tokens and discrete speech tokens. The architecture features interleaved cross-modal self-attention to integrate continuous audio embeddings and tokenized text/audio signals.
• Speech Tokenizer & Decoder: Uses a VQ-VAE with Gumbel-softmax and a 1024-token codebook for discrete speech tokenization. Decoding proceeds via a two-stage head: (1) Flow-Matching-based latent reconstructor for converting discrete tokens to continuous latent vectors; (2) BigVGAN vocoder generating final 24 kHz audio waveforms.

The overall parameter breakdown is approximately: 1.5B (encoder), ~50M (adapter), ~6.5B (LLM backbone), and ~50M (decoder/vocoder) (Wang et al., 10 Feb 2026).

2. Pre-training and Post-training Regimes

2.1 Two-Stage Pre-training

Pre-training leverages nearly 2 trillion task tokens, with progressive task diversity and difficulty:

• Stage 1 (Adapter Only, 30B tokens): ASR objective ($a_c \rightarrow t$), aligning continuous audio features to text via next-token cross-entropy loss ($L_{CE}$).
• Stage 2 (Joint Adapter + LLM, 800B–900B tokens per task):
  • ASR: $a_c \rightarrow t$ (80B tokens)
  • TTS: $t \rightarrow a_d$ (160B tokens)
  • Audio-only: $a_c \rightarrow a_d$, $a_d \rightarrow a_d$ (240B tokens)
  • Speech continuation: $a_c \rightarrow t$ (160B)
  • Hierarchical Tri-modal Interleaving: Sequential ($a_c \rightarrow t \rightarrow a_d$) and parallel ($a_c \rightarrow (t | a_d)$) patterns (540B)
  • Text-only: $t \rightarrow t$ (800B)
  • Full-Duplex: $L_{CE}$0 (5B)

Cross-entropy is the universal learning objective:

$L_{CE}$1

2.2 Alignment-Focused Fine-Tuning

Post-training involves 50k steps with clustering of training objectives weighted by task group:

Group	Tasks	Sampling Ratio
General Intelligence	$L_{CE}$2, $L_{CE}$3, $L_{CE}$4, $L_{CE}$5	40%
Spoken Dialogue	$L_{CE}$6 (multi-turn)	30%
Speech Understanding	$L_{CE}$7, $L_{CE}$8, $L_{CE}$9, $a_c \rightarrow t$0	10%
Speech Generation	$a_c \rightarrow t$1	10%
Audio Understanding	$a_c \rightarrow t$2	10%

Weighted multi-task loss is applied: $a_c \rightarrow t$3, with $a_c \rightarrow t$4 (Wang et al., 10 Feb 2026).

3. Training Objectives and Learning Paradigms

The primary optimization target is autoregressive cross-entropy; for instruction-tuning in text-to-text (T2T) mode, a KL-distillation auxiliary is added with $a_c \rightarrow t$5:

$a_c \rightarrow t$6

For multi-modal reasoning, particularly on MMAU/MMSU evaluation, a composite reinforcement learning (RL) reward is employed:

$a_c \rightarrow t$7

Each sub-reward in $a_c \rightarrow t$8 targets answer accuracy, format compliance, reasoning coherence, and quality-of-thought, combined in gradient-based RL for model alignment.

4. Evaluation Benchmarks and Quantitative Results

Extensive evaluation spans speech-text, dialogue, full-duplex, paralinguistics, and audio understanding tasks.

4.1 Speech–Text Foundation

Signal comprehension, text generation, and grammar generalization (Table 3):

Metric	Score
A2A-tSC	83.3
A2T	95.7
T2T	99.4
sBLIMP	61.6
sWUGGY	74.9

ASR and TTS (Table 4): WER on Aishell-1 = 1.96%, LS-clean = 1.96%, LS-other = 4.55%. TTS MOS: English = 2.44, Mandarin = 1.73.

4.2 Speech-to-Speech Dialogue

Half-duplex (Covo-Audio-Chat) on URO-Bench:

• Chinese: SQuAD = 77.34, OpenbookQA = 83.60, APE = 68.42, MLC = 80.69
• English: Gsm8k = 85.68
• Oral Evaluation: AlpacaEval = 90.02, Wildchat = 90.41

VCB-Bench: Instruction Following TIF = 93.07 (ZH), 89.94 (EN); Multiturn Dialogue (MTD) = 87.70; Robustness (SV = 88.94, EV = 87.13, CV = 90.37).

4.3 Empathy & Paralinguistics

Mandarin empathy (VStyle table): scores $a_c \rightarrow t$9–$t \rightarrow a_d$0 across four emotional dimensions; AIR-Bench paralinguistics average = 80.86% (emotion, gender, age).

4.4 Full-Duplex Interaction

Covo-Audio-Chat-FD demonstrates:

• URO-Bench: Chinese Repeat = 98.35, SQuAD = 75.16; English Repeat = 94.64, Gsm8k = 80.47
• Behavioral metrics: Turn-taking = 99.7%, Pause Handling = 97.6%, Backchanneling = 93.9%, Interruption = 96.8%

4.5 Audio Understanding

• MMAU average: 75.30% (Sound: 78.68, Music: 76.05, Speech: 71.17)
• MMSU average: 66.64% (Perception: 58.95, Reasoning: 74.83), the highest among all open/closed models evaluated (Table 6) (Wang et al., 10 Feb 2026).

5. Conversational and Full-Duplex Variants

5.1 Covo-Audio-Chat

This variant is post-trained on 10M text-instructions with parallel TTS, ~100k multi-speaker dialogues, and emotional data, without increasing parameter count. It targets high-quality spoken dialogue, instruction following, and empathetic response generation.

5.2 Covo-Audio-Chat-FD

Supports full-duplex voice interaction by modifying input streaming: chunk-streaming encoder with a 1:4 user:output chunk interleaving (0.16s granularity). Special tokens (THINK, SHIFT, BREAK) enable explicit modeling of conversational turn-taking and overlap, trained end-to-end on half- and full-duplex dialogues. Tabled behavioral metrics confirm robust turn-management.

6. Intelligence–Speaker Decoupling Strategy

To address the scarcity of high-quality, speaker-specific dialogue data versus the abundance of TTS voice samples, Covo-Audio implements an “intelligence–speaker decoupling” pipeline. The training strategy proceeds as follows:

1. Pseudo-Conversation Pretraining: The LALM is trained using multi-speaker TTS data formatted as “pseudo-conversations,” masking text response loss ($t \rightarrow a_d$1) to focus adaptation on audio output.
2. Joint Fine-Tuning: The model is further trained with genuine dialogue data from other speakers, then contextual TTS adaptation is applied to transfer speaker style.

Audio loss remains standard CE on audio output ($t \rightarrow a_d$2), ensuring accurate voice style transplantation while preserving core dialogue intelligence. Empirical validation shows that the Covo-Audio-Chat-TTS variant, trained with only lightweight TTS data, suffers less than 1% performance drop across core spoken dialogue benchmarks (Table “URO-All”).

7. Insights, Limitations, and Future Perspectives

Covo-Audio demonstrates that a 7B-scale LALM—if trained with large-scale, tri-modal (audio, text, speech) data, hierarchical objectives, and targeted fine-tuning—can match or exceed much larger models on a diverse set of speech, text, and audio benchmarks. Critical aspects contributing to performance include:

• Model–data co-design minimizing parameter bloat
• Sophisticated pretraining/post-training curricula with explicit multimodal alignment
• Robust cross-modal fusion architecture

Limitations include the context window restriction (currently 8k tokens), motivating future work on longer-context architectural variants. Planned extensions include reinforcement learning (e.g., RLHF) for granular conversational act separation, sparse/MoE scaling for larger models at tractable cost, and unified benchmark development for multi-round, full-duplex evaluation (e.g., MTR-Duplex-Bench).

These developments signify a scalable trajectory toward highly capable, multi-purpose audio LLMs tightly integrating audio intelligence, semantic reasoning, and customized voice rendering (Wang et al., 10 Feb 2026).