Emotion Omni: Empathetic Speech LLM

• Emotion Omni is a speech large language model architecture that integrates dual speech encoders, an instruction-tuned LLM, and emotion-conditioned speech synthesis for empathetic dialogue.
• It employs a synthetic TTS-based data generation pipeline to construct a 200,000-pair emotional dialogue corpus, enabling low-data and computation-conscious training.
• The system achieves competitive voice QA performance by blending semantic understanding with dynamic emotional supervision, enhancing both linguistic and acoustic expressiveness.

Emotion Omni is a speech LLM architecture for empathetic spoken dialogue that is designed to infer emotional content from user speech and to generate emotionally aligned speech responses. Its central premise is that speech interaction cannot be reduced to semantic transcription followed by neutral speech rendering, because identical lexical content may carry different meanings when emotional and paralinguistic cues change. To address that gap, Emotion Omni combines dual speech encoders, an instruction-tuned LLM backbone, an emotion prediction pathway, and an emotion-conditioned speech decoder, while also introducing a TTS-based data generation pipeline that produces a 200,000-pair emotional dialogue corpus for training an empathetic speech assistant (Wang et al., 26 Aug 2025).

1. Problem setting and design objective

Emotion Omni is situated within speech LLM research that seeks direct speech-to-speech interaction rather than a cascade of ASR, text-only reasoning, and neutral TTS. The model is motivated by two related claims: most existing speech LLMs convert response content into speech without fully understanding the emotional and paralinguistic cues embedded in the user query, and many empathetic systems depend on massive labeled datasets and substantial computational budgets. Emotion Omni therefore frames empathetic response generation as a low-data, computation-conscious modeling problem in which emotional understanding must be learned jointly with response generation (Wang et al., 26 Aug 2025).

A key conceptual distinction in the system is between semantic understanding and emotional understanding. The model is not described as merely producing expressive speech; instead, it aims to condition the response itself on the user’s affective state. This makes the task closer to emotionally aligned dialogue generation than to conventional emotional TTS. A common misconception is to treat such systems as TTS pipelines with an added style token. Emotion Omni’s formulation is broader: emotional cues are extracted from the input speech before text response generation, then propagated into the speech decoder so that emotional alignment is present at both linguistic and acoustic levels (Wang et al., 26 Aug 2025).

2. Core architecture

The architecture has four main components: a dual-encoder front end, an LLM backbone, an emotion prediction head, and a speech decoder. The dual-encoder design separates semantic and emotional feature extraction. Semantic content is encoded by Whisper-large-v3, while emotional and paralinguistic cues are encoded by Emotion2Vec. Their outputs are downsampled, concatenated, and projected to the input dimensionality required by the LLM. The representation is defined as

$$\mathcal{H}_{sem/emo} = \mathcal{D}_{sem/emo}\big(E_{sem/emo}(S), k\big),$$

where $E_{sem/emo}$ denotes the semantic or emotion encoder, $\mathcal{D}_{sem/emo}$ the corresponding downsampling adapter, $S$ the speech input, and $k$ the downsampling factor (Wang et al., 26 Aug 2025).

The LLM backbone is Qwen2.5-7B-Instruct, adapted through LoRA for empathetic dialogue alignment. It receives the projected speech representation and generates text autoregressively:

$$P(y_t \mid y_{<t}, \mathcal{H}_{\text{speech}}) = \text{softmax}(\mathcal{LLM}(\mathcal{H}_{\text{speech}}, y_{<t})).$$

Beyond text generation, the model includes an emotion prediction head that maps LLM hidden states to target emotional features derived from the response speech. The target emotion representation is obtained by window averaging over the response-side emotion features, and training uses a combined MSE and cosine-similarity objective. The overall objective is written as

$$\mathcal{L}_{total} = \mathcal{L}_{LLM} + \lambda \mathcal{L}_{emo}.$$

This design indicates that emotional supervision is applied not only at the final acoustic stage but also within the language-model latent trajectory, so that hidden states carry emotion-relevant information before decoding to speech (Wang et al., 26 Aug 2025).

3. Data generation pipeline and corpus construction

A defining contribution of Emotion Omni is its synthetic data strategy. Because large-scale, high-quality emotional speech dialogue corpora are scarce, the system constructs training data through a pipeline that combines GPT-4o and an open-source TTS framework. GPT-4o is used to generate textual dialogue pairs with explicit emotion labels across 20 domains. CosyVoice2 then synthesizes emotional speech for both user queries and assistant responses. The synthesis setup uses ten speakers, consisting of five male and five female voices, to increase diversity (Wang et al., 26 Aug 2025).

The resulting corpus has 200,000 emotional dialogue pairs. Of these, 150,000 pairs are generated from GPT-4o prompts, and 50,000 additional samples are drawn from the VoiceAssistant dataset after filtering and relabeling. The paper characterizes this pipeline as low-cost and precise in emotional control. In methodological terms, the corpus is not simply a large generic speech set with post hoc emotion annotations; it is generated so that emotional attributes are structurally available during training. This suggests that the model’s data efficiency depends not only on architecture but also on the controllability and consistency of the synthesized supervision signal (Wang et al., 26 Aug 2025).

The corpus is intended to support an empathetic speech assistant rather than only isolated emotion classification. That distinction matters because the paired data couples user affect, assistant text response, and assistant speech realization. A plausible implication is that the training signal jointly regularizes emotional perception, pragmatic response selection, and speech rendering, which differs from datasets built only for recognition or only for style-conditioned TTS.

4. Training procedure and emotion-conditioned speech synthesis

Training proceeds in two stages. Stage 1 performs LoRA fine-tuning on the LLM for speech-text alignment and emotion prediction, while the adapters remain frozen for stability. This stage establishes the mapping from dual-encoder speech representations to empathetic text and to predicted emotional features. Stage 2 trains the speech generation subsystem. It is itself split into two sub-stages: Stage 2.1 trains the decoder using text embeddings alone for basic semantic alignment, and Stage 2.2 activates full emotional and contextual conditioning through gated fusion and Adaptive Layer Normalization (AdaLN) (Wang et al., 26 Aug 2025).

The speech decoder uses CosyVoice2’s tokenizer with Finite Scalar Quantization to convert speech into discrete units while retaining emotional content. Context integration is handled by a gated fusion mechanism that combines text embeddings with LLM hidden states:

$$\text{GateFusion}(\mathcal{H}_{\text{text}}, \mathcal{H}_{\text{out}}) = \alpha \odot \mathcal{H}_{\text{text}} + (1-\alpha) \odot \mathcal{H}_{\text{out}},$$

with $$\alpha = \sigma(W_g[\mathcal{H}_{\text{text}}, \mathcal{H}_{\text{out}}] + b_g)$$ (Wang et al., 26 Aug 2025).

Emotion control is introduced through AdaLN, which modulates decoder hidden states using the predicted emotional features:

$$\text{AdaLN}(\mathbf{h}_i, \mathbf{e}_i) = \boldsymbol{\gamma}(\mathbf{e}_i) \odot \frac{\mathbf{h}_i - \boldsymbol{\mu}(\mathbf{h}_i)}{\boldsymbol{\sigma}(\mathbf{h}_i)} + \boldsymbol{\beta}(\mathbf{e}_i).$$

The final adapted representation is synthesized into waveform output by the CosyVoice2 vocoder. Architecturally, the important point is that emotion is not appended as a static label but used as a dynamic conditioning variable during normalization, which allows the decoder to modulate prosody and delivery in a feature-dependent manner rather than by discrete style switching alone (Wang et al., 26 Aug 2025).

5. Evaluation and reported empirical performance

Emotion Omni is evaluated on VoiceBench, which includes AlpacaEval, CommonEval, IFEval, WildVoice, and SD-QA, with speech quality measured by UTMOS. The reported results place the model alongside Qwen2-Audio, Moshi, GLM-4-Voice, and LLaMA-Omni. Emotion Omni records 3.84 on Alpaca, 3.47 on Common, 27.89 on IFEval, 3.13 on WildVoice, 36.87 on SD-QA, and 4.41 on UTMOS. In the accompanying comparison, Qwen2-Audio scores 3.74, 3.43, 26.33, 3.01, and 35.71 on the five VoiceBench tasks; Moshi scores 2.01, 1.60, 10.12, 1.30, and 15.64, with UTMOS 3.81; GLM-4-Voice scores 3.97, 3.42, 25.92, 3.18, and 25.92, with UTMOS 3.48; and LLaMA-Omni scores 3.70, 3.46, 14.87, 2.92, and 39.69, with UTMOS 3.98 (Wang et al., 26 Aug 2025).

For empathetic response quality, the paper reports a GPT Emotion Score of 3.97, a MOS Emotion Score of 4.43, and an ASR-WER of 5.61 on “QA Pairs 1000.” The reported interpretation is that Emotion Omni matches or outperforms state-of-the-art models on core voice QA and speech quality, while human and automated evaluations support strong emotional expressiveness and empathetic appropriateness (Wang et al., 26 Aug 2025).

Evaluation setting	Reported result	Interpretation
VoiceBench + UTMOS	Alpaca 3.84, Common 3.47, IFEval 27.89, WildVoice 3.13, SD-QA 36.87, UTMOS 4.41	Joint voice QA and speech quality
QA Pairs 1000	GPT Emotion 3.97, MOS Emotion 4.43, ASR-WER 5.61	Emotional alignment, perceptual quality, intelligibility

A practical reading of these numbers is that the model’s performance claim is two-dimensional: it is intended to remain competitive on general spoken QA while also improving emotional alignment in synthesized responses. This is different from systems optimized only for emotion rendering, where linguistic task quality is secondary.

6. Position within omni-modal emotion research and open issues

Emotion Omni occupies a specific niche within a broader line of work on omni-modal emotion understanding and expression. Contemporary systems such as HumanOmni emphasize human-centric audio-visual understanding and report strong emotion-recognition performance on DFEW and MAFW through instruction-driven fusion of face, body, interaction, and audio features (Zhao et al., 25 Jan 2025). Omni-Emotion extends video MLLMs with explicit facial encoding and audio modeling for multimodal emotion analysis, using self-reviewed and human-reviewed datasets for recognition and reasoning tasks (Yang et al., 16 Jan 2025). R1-Omni applies RLVR to explainable omni-multimodal emotion recognition, focusing on reasoning capability, accuracy, and out-of-distribution generalization rather than speech response generation (Zhao et al., 7 Mar 2025). OpenOmni, by contrast, addresses open-source omnimodal alignment and real-time self-aware emotional speech synthesis, reporting non-autoregressive generation with less than 1 second latency and improved emotion classification after direct preference optimization (Luo et al., 8 Jan 2025).

Later work pushes the same research frontier in more explicitly structured directions. EmoOmni introduces an emotional Chain-of-Thought that acts as an explicit interface between understanding and expressive speech generation in a Thinker-Talker architecture (Tian et al., 25 Feb 2026). OPPO focuses on utilization and faithfulness in multimodal emotion reasoning, arguing that omni-modal emotion systems can underuse cues or hallucinate modality-specific evidence (Han et al., 24 Jun 2026). Relative to these systems, Emotion Omni is best understood as a speech-centered architecture whose primary contribution is the coupling of empathetic speech generation with a low-cost synthetic data pipeline, rather than a general-purpose multimodal reasoning framework (Wang et al., 26 Aug 2025).

The paper also states a clear limitation: emotion recognition is less reliable for subtle or low-intensity emotions, and the architecture may over-focus on semantics while missing weak paralinguistic cues. Future work is described as improving sensitivity, rebalancing semantic and emotional modeling, and extending performance to natural, spontaneous conversations (Wang et al., 26 Aug 2025). This suggests that the main unresolved issue is not basic affect conditioning, but robustness when emotional evidence is weak, mixed, or only indirectly expressed in speech.