OSUM-EChat: Empathetic Speech Dialogue

• OSUM-EChat is an open-source, end-to-end system for empathetic speech dialogue that models both linguistic content and paralinguistic cues such as emotion, age, gender, and sound events.
• It employs a three-stage understanding-driven training strategy and a dual thinking mechanism to integrate speech understanding with dialogue generation for improved empathy.
• The architecture utilizes modules like Whisper, Qwen2.5, and CosyVoice to convert speech to tokens and back, enabling robust native speech-to-speech interactions.

OSUM-EChat is an open-source, end-to-end empathetic spoken dialogue system that takes speech as input and produces speech as output while explicitly modeling linguistic content and paralinguistic cues such as emotion, age, gender, and sound events. It was introduced to improve empathetic spoken interaction in resource-limited settings through two central ideas: a three-stage understanding-driven spoken dialogue training strategy, and a linguistic-paralinguistic dual thinking mechanism that integrates paralinguistic understanding through a chain of thought with dialogue generation. The same work also introduced the EChat-200K speech-to-speech empathetic dialogue corpus and the EChat-eval benchmark for multi-dimensional empathy evaluation (Geng et al., 13 Aug 2025).

1. Conceptual setting and research lineage

OSUM-EChat is motivated by the claim that empathy in spoken dialogue is not purely semantic. A user utterance carries not only lexical content, but also vocal evidence such as emotional tone, speaker traits, and sound events, and the paper argues that these cues are essential for an empathetic response (Geng et al., 13 Aug 2025). In that framing, a spoken chatbot should react not only to what was said, but also to how it was said.

The system is positioned against both cascaded and end-to-end alternatives. A classic cascaded system—ASR $\rightarrow$ LLM $\rightarrow$ TTS—transcribes speech to text, reasons over text, and synthesizes new speech, but tends to discard much of the original signal’s paralinguistic information. The paper also contrasts OSUM-EChat with modular aligned multimodal systems such as Freeze-Omni and Qwen2.5-Omni, and with native multimodal systems such as Mini-Omni, GLM-4-Voice, Moshi, and DeepTalk, arguing that existing systems either model understanding and generation somewhat independently or rely on large-scale industrial spoken dialogue data to implicitly learn empathy-relevant associations (Geng et al., 13 Aug 2025).

Its Stage 1 is explicitly inherited from the OSUM framework and uses an ASR+P strategy (Geng et al., 13 Aug 2025). OSUM itself is an open academic speech-understanding LLM that combines a Whisper encoder with a Qwen2 LLM, supports eight speech tasks including speech-to-text chat, and is trained with an ASR+X strategy to stabilize multi-task learning under constrained academic resources (Geng et al., 23 Jan 2025). A related earlier line of work is E-chat, an emotion-sensitive spoken dialogue system that used a HuBERT speech encoder, an emotion embedding path, a connection module, and a frozen Atom-7B decoder to generate text responses conditioned on emotional speech (Xue et al., 2023). OSUM-EChat extends this research space from speech-to-text emotional response generation to native speech-to-speech empathetic dialogue (Geng et al., 13 Aug 2025).

2. Architecture and end-to-end pipeline

OSUM-EChat has four modules: a Speech Encoder, an Adapter, an LLM Decoder, and a token2wav module (Geng et al., 13 Aug 2025). The speech encoder is a pre-trained Whisper-Medium encoder that consumes an 80-channel log-magnitude Mel spectrogram. The adapter contains a three-layer 1D convolutional module, four transformer encoder layers, and a final linear layer. Its role is to downsample the speech representation by a factor of 4 and map the speech encoder output into the embedding space of the LLM (Geng et al., 13 Aug 2025).

The decoder is based on Qwen2.5-3B-Instruct. To support native speech modeling, the vocabulary is extended by 4097 tokens: 4096 speech tokens from the CosyVoice speech codebook and 1 extra token for the start/end of speech generation. The LLM generates text and speech tokens, including interleaved text-speech generation at a 6:18 ratio for stable streaming generation (Geng et al., 13 Aug 2025).

The token2wav module is based on CosyVoice and includes a flow-matching model for Mel spectrogram estimation and a HiFi-GAN vocoder for waveform reconstruction. It converts the speech-token stream into a 24 kHz, 16-bit PCM waveform (Geng et al., 13 Aug 2025). At inference time, user speech is converted to Mel spectrograms, encoded by Whisper, projected by the adapter into the LLM space, decoded into text and speech tokens by the Qwen-based decoder, and reconstructed as audio by token2wav.

This design is categorized as native multimodal rather than a purely modular cascade, because the LLM directly participates in speech token generation rather than operating only on an intermediate transcript (Geng et al., 13 Aug 2025).

3. Understanding-driven spoken dialogue training

The central training idea is “understanding-driven spoken dialogue”: train speech understanding first, then add generation ability, then integrate empathy explicitly (Geng et al., 13 Aug 2025). The curriculum has three stages—Understanding, Generation, and Empathy—and each stage has its own loss.

Stage 1 trains the model to understand both linguistic content and paralinguistic attributes $P$, including emotion, gender, age, and sound events. The loss is:

$$L_{S1} = \sum_{i=1}^{N} L_{ASR}(x_i, \hat{x_i}) + \sum_{j=1}^{M} L_{P}(p_{j}, \hat{p}_{j}),$$

where $x_i$ and $\hat{x_i}$ are ground-truth and predicted ASR tokens, and $p_j$ and $\hat{p}_j$ are ground-truth and predicted paralinguistic labels (Geng et al., 13 Aug 2025). The purpose of this stage is to build an encoder-adapter front end that already understands empathy-relevant cues before spoken dialogue generation.

Stage 2 equips the model with generation ability. In the TTS phase, only the LLM is learnable, and the objective is speech-token prediction:

$$L_{S2-TTS} = \sum_{i=1}^{S} L_{speech}(y_i, \hat{y}_i),$$

where $y_i$ and $\rightarrow$0 are ground-truth and predicted speech tokens (Geng et al., 13 Aug 2025). In the subsequent speech-to-speech phase, the speech encoder, adapter, and LLM are all learnable, and the model is trained jointly on text and speech outputs:

$\rightarrow$1

with non-streaming output organized as text tokens first and then speech tokens, and streaming output organized as interleaved text and speech tokens at a 6:18 ratio (Geng et al., 13 Aug 2025).

Stage 3 introduces empathy-specific supervision through the dual-think mechanism. Its loss is:

$\rightarrow$2

where $\rightarrow$3 and $\rightarrow$4 are ground-truth and predicted think tokens (Geng et al., 13 Aug 2025). This stage is trained primarily on EChat-200K and emphasizes reasoning over linguistic and paralinguistic evidence before response generation.

The optimization setup uses WeNet and AdamW with betas $\rightarrow$5, weight decay $\rightarrow$6, initial learning rate $\rightarrow$7, warmup of 8,000 steps, gradient clipping of 5, gradient accumulation over 4 mini-batches, and dynamic batch size with total LLM input capped at 2,800 tokens, on 64 ShengTeng 910B GPUs (Geng et al., 13 Aug 2025).

4. Linguistic-paralinguistic dual thinking

The most distinctive mechanism in OSUM-EChat is the linguistic-paralinguistic dual think mechanism. “Dual think” denotes a two-part intermediate reasoning process over the input speech: first linguistic understanding, then paralinguistic inference (Geng et al., 13 Aug 2025). The model is taught to emit a structured intermediate sequence enclosed between $\rightarrow$8 and $\rightarrow$9 before generating the final answer.

The appendix prompt specifies the intended behavior: “Before responding, first output your reasoning inside > ...</think end>, analyzing the user’s words and vocal cues. Then generate a reply with appropriate text and emotionally matched synthetic speech” (Geng et al., 13 Aug 2025). This mechanism is meant to force the model to surface its analysis of emotion, age, gender, and sound events instead of relying only on latent internal states.

Because EChat-200K contains fully labeled samples, single-label samples, and dialogue data with no paralinguistic labels, the training process uses Structured Sparse Learning. Every sample is cast into a think format with all label slots, unknown labels are filled with unk, and unk positions are masked from loss backpropagation (Geng et al., 13 Aug 2025). This yields a unified supervision format across partially labeled data.

The paper argues that the mechanism supports empathy in two ways. First, it transfers knowledge from the understanding stages into dialogue generation. Second, it creates explicit fusion of semantic and vocal evidence before response generation. This is borne out empirically: removing Dual Think reduces EChat-eval scores from 58.0 to 54.5 on Emotion, from 63.1 to 42.3 on Age, from 87.1 to 61.1 on Event, and from 72.0 to 56.4 on Multi (Geng et al., 13 Aug 2025).

5. EChat-200K and EChat-eval

EChat-200K is a speech-to-speech empathetic dialogue dataset with approximately 200,000 conversations (Geng et al., 13 Aug 2025). It is designed to train models to respond to speech while accounting for multiple paralinguistic dimensions. Each instance includes query text, query speech, paralinguistic labels, response text, and response speech (Geng et al., 13 Aug 2025).

The construction pipeline is synthetic-plus-real. DeepSeek generates empathy-related query texts for specific scenarios; the query texts are annotated with paralinguistic labels; CosyVoice2 synthesizes query speech with controllable voice characteristics; sound-event clips are concatenated to enrich the input; DeepSeek generates an empathetic response text using the query and labels; and CosyVoice2 synthesizes the response audio with appropriate emotional coloring (Geng et al., 13 Aug 2025). The main label dimensions are Emotion, Gender, Age, and Sound events. The emotion label space includes anger, fear, happy, surprise, sad, disgust, confused, sarcasm, embarrassed, curious, worried, shy, sorry, neutral, while the sound-event space includes laugh, cough, cry, scream, sigh, throat clearing, sneeze, other (Geng et al., 13 Aug 2025).

The corpus has both single-label and multi-label components. The single-label portion is organized into emotion-only, gender-only, age-only, and sound-event-only data, while the multi-label portion teaches integration of several cues jointly (Geng et al., 13 Aug 2025). To reduce overfitting to synthetic audio, the dataset includes 107k real labels in the single-label emotion data and 10k real labels in the multi-label data (Geng et al., 13 Aug 2025).

EChat-eval is the benchmark introduced alongside the dataset. It includes Emotion, Age, Gender, Event, and Multi-label tasks, each with 280 entries, and about one-third of the benchmark comes from real recordings (Geng et al., 13 Aug 2025). Automatic evaluation uses ChatGPT-4o as the scoring model, VoiceBench, and emotion2vec-Large to extract emotional labels from generated response speech (Geng et al., 13 Aug 2025). Human evaluation compares OSUM-EChat with Qwen2.5-Omni and Doubao (Geng et al., 13 Aug 2025).

6. Empirical performance and ablation evidence

On EChat-eval automatic evaluation, OSUM-EChat scores 58.0 on Emotion, 63.1 on Age, 62.7 on Gender, 87.1 on Event, and 72.0 on Multi (Geng et al., 13 Aug 2025). The same benchmark reports lower scores for Qwen2.5-Omni, Freeze-Omni, GLM-4-Voice, Baichuan-Audio, Kimi-Audio, and OpenS2S, so OSUM-EChat is the best model on all reported dimensions (Geng et al., 13 Aug 2025).

Human evaluation shows a similar ranking. OSUM-EChat scores 72.0 on Emotion, 68.9 on Age, 65.8 on Gender, 88.0 on Event, and 73.3 on Multi, compared with Qwen2.5-Omni at 54.9, 55.0, 49.2, and 67.3 on the corresponding reported dimensions, and Doubao at 78.0 on Emotion, 42.2 on Age, 39.0 on Gender, and 60.7 on Multi (Geng et al., 13 Aug 2025). The paper notes that OSUM-EChat still trails Doubao on the Emotion dimension alone, but outperforms it on Age, Gender, and Multi (Geng et al., 13 Aug 2025).

The ablation study is central to the paper’s argument. Removing understanding-driven training reduces scores to 53.4 on Emotion, 35.4 on Age, 41.0 on Gender, 52.9 on Event, and 63.1 on Multi. Removing Dual Think yields 54.5, 42.3, 61.3, 61.1, and 56.4. Removing both understanding-driven training and Dual Think reduces performance further to 51.6, 40.3, 39.1, 10.4, and 54.2 (Geng et al., 13 Aug 2025). The drop from 87.1 to 10.4 on Event when both components are removed is the clearest indication that empathy performance is tied to transferred speech understanding and explicit reasoning rather than dialogue generation alone.

The think-token analysis reinforces that interpretation. The $P$0 variant achieves ASR 2.80, Age 84.1, Gender 98.6, Emotion 79.0, and Sound 80.2, showing that substantial understanding competence is retained inside the dialogue model through the think representation (Geng et al., 13 Aug 2025).

7. Limitations and significance

The paper identifies two main limitations. First, dynamic paralinguistic scenarios remain underexplored, including emotion transitions such as sadness to joy and multi-speaker interactions with diverse genders and ages. Second, the automatic scoring framework is imperfect because of inaccurate emotional label extraction and difficulty handling LLM hallucinations in scoring (Geng et al., 13 Aug 2025). The authors therefore present refinement of automated scoring and more complex conversational scenarios as future work.

OSUM-EChat’s significance lies less in proposing a larger backbone than in reorganizing the training problem. Its claim is that empathetic spoken dialogue can be improved by transferring structured speech understanding into generation, rather than relying solely on very large end-to-end dialogue corpora (Geng et al., 13 Aug 2025). A plausible implication is that its main contribution is methodological: it reframes empathy as a problem of jointly modeling lexical content and explicitly supervised paralinguistic understanding within a native speech-to-speech architecture.

In that sense, OSUM-EChat stands at the intersection of the OSUM speech-understanding program and earlier emotion-sensitive spoken dialogue systems such as E-chat, but pushes the field toward fully end-to-end spoken empathy with dedicated data and evaluation resources (Geng et al., 23 Jan 2025, Xue et al., 2023).