OpenS2S: Empathetic End-to-End Speech Model

• OpenS2S is an open-source, transparent end-to-end empathetic speech model that listens to user speech and generates emotionally appropriate responses.
• It adopts a streaming interleaved decoding architecture combining a pre-trained audio encoder, LLM backbone, and speech tokenizer to achieve low-latency, expressive speech synthesis.
• The model’s design supports scalable dialogue construction and empathetic instruction tuning, providing insights for multilingual and multi-speaker research in speech generation.

to=arxiv_search.search 天天中彩票足彩 code {"query":"id:(Wang et al., 7 Jul 2025) OR id:(Lu et al., 2 Jun 2025) OR id:(Wang et al., 2021) OR id:(Wei et al., 2022)", "max_results": 10} OpenS2S most specifically denotes an open-source, transparent, end-to-end empathetic large speech LLM for speech-to-speech interaction, introduced as a system that listens to user speech rich in paralinguistic cues and generates emotionally appropriate, expressive spoken responses. In a broader research sense, the name also sits within a family of open sequence-to-sequence speech efforts spanning speech synthesis, direct speech-to-speech translation, interactive ASR–LLM–TTS pipelines, and generic decoding frameworks. The primary contemporary referent is the empathetic LSLM released with datasets, model weights, pre-training and fine-tuning code, and evaluation scripts, and built around BLSP-Emo-style alignment plus a streaming interleaved decoding architecture for low-latency speech generation (Wang et al., 7 Jul 2025).

1. Terminological scope and lineage

The designation “OpenS2S” is not confined to a single architectural tradition. In current literature, it includes at least four adjacent open research directions. fairseq S$^2$ is an open extension of fairseq for text-to-speech and unit-to-speech synthesis, with autoregressive and non-autoregressive acoustic models, multi-speaker variants, robust preprocessing, and automatic metrics (Wang et al., 2021). Speech2S is a direct speech-to-speech translation model that addresses S2ST data scarcity through joint pre-training on unpaired speech and bilingual text in a shared discrete unit space (Wei et al., 2022). CleanS2S is a single-file framework for proactive speech-to-speech interaction that integrates ASR, LLM, and TTS in a full-duplex websocket pipeline with memory and action judgement modules (Lu et al., 2 Jun 2025). RASR2 uses “OpenS2S” in the ASR sense of open-vocabulary sequence-to-sequence decoding, emphasizing lexicon-free and lexical-constrained search within a generic C++ decoder (Zhou et al., 2023).

Within that landscape, the 2025 model titled “OpenS2S” narrows the term to an end-to-end empathetic speech agent rather than a toolkit, translation model, or decoder. This suggests that “OpenS2S” functions both as a proper model name and as a broader open-research orientation around sequence-to-sequence speech systems, with different projects occupying different points in the design space between modular pipelines and integrated speech-native models (Wang et al., 7 Jul 2025).

2. End-to-end empathetic LSLM architecture

The core OpenS2S model is organized as an end-to-end speech-to-speech stack. User waveform input is encoded by a pretrained audio encoder from Qwen2-Audio, which produces features at 25 Hz. A speech adapter, consisting of two CNN layers for downsampling by $4\times$ plus a feed-forward layer, reduces the sequence to continuous speech representations at 6.25 Hz. These audio embeddings are concatenated with text embeddings into an interleaved input sequence for Qwen3-8B-Instruct, which provides the instruction-following language backbone (Wang et al., 7 Jul 2025).

The system inherits its speech-understanding side from BLSP-Emo-style pretraining. Semantic alignment is used to make the LLM behave consistently across speech and text inputs for the same utterance, while emotion alignment introduces emotion-aware continuation behavior. OpenS2S does not add separate emotion or prosody heads; instead, paralinguistic information is represented implicitly in the learned audio embeddings and reinforced through emotion-aware prompting and alignment. Emotion, gender, and age also appear explicitly as tags in dataset construction and prompt structure (Wang et al., 7 Jul 2025).

Speech generation is handled by a streaming speech decoder rather than a spectrogram predictor. The speech tokenizer is a quantization module inserted into the Whisper-large-v3 encoder and adopted from GLM-4-Voice; it produces a discrete sequence at 12.5 tokens per second with vocabulary size $V = 16{,}384$. The decoder itself is a decoder-only Transformer initialized from Qwen3-1.8B, with its vocabulary extended to include the 16,384 speech tokens. Final hidden states from the LLM are linearly projected into the speech decoder’s embedding space and used as conditioning (Wang et al., 7 Jul 2025).

The defining generation mechanism is streaming interleaved decoding. For each $M$ LLM hidden states consumed, the decoder emits $N$ speech tokens, with implementation values $M = 4$ and $N = 8$. After the LLM hidden states are exhausted, decoding continues autoregressively until the end of response. A token2wav module then incrementally converts token chunks into waveforms using chunk-aware causal flow matching for mel-spectrogram generation and a HiFi-GAN vocoder, both adopted from GLM-4-Voice. The architectural effect is to overlap understanding, planning, and synthesis rather than waiting for a fully completed text response before speech generation begins (Wang et al., 7 Jul 2025).

3. Objectives and staged training

The reported objective set separates speech understanding, emotion alignment, offline speech-token generation, and streaming decoding. With $x_s$ denoting input speech, $x_t$ input transcript text, $e$ an emotion label, $4\times$0 LLM hidden states, and $4\times$1 a discrete speech-token sequence, the losses are given as

$4\times$2

$4\times$3

$4\times$4

$4\times$5

The paper presents a generic decomposition

$4\times$6

while explicitly noting that the coefficients are not provided. For streaming generation, the conditional sequence probability is expressed as

$4\times$7

The paper does not report auxiliary CTC, contrastive, or alignment losses beyond these behavioral-alignment formulations (Wang et al., 7 Jul 2025).

Training proceeds in three stages. Stage 1 performs speech understanding pretraining. English ASR supervision comes from LibriSpeech, CommonVoice 13.0, and GigaSpeech-M, totaling about $4\times$8 million $4\times$9 pairs; Chinese ASR uses a comparable number of samples randomly drawn from WeNetSpeech. Emotion alignment uses IEMOCAP, MELD, CMU-MOSEI, MEAD, and ESD, totaling about 70k utterances in English and Chinese. In this stage, the audio encoder and LLM are frozen, and only the speech adapter is trained (Wang et al., 7 Jul 2025).

Stage 2 performs speech-generation pretraining. Offline TTS pretraining uses 5k h English plus 5k h Chinese sampled from Emilia in order to expand the speech decoder vocabulary and learn text-to-speech-token mapping. The subsequent streaming integration step connects the frozen LLM to the speech decoder; linear projection and speech decoder are trained on 1k h English plus 1k h Chinese sampled from the first stage’s dataset (Wang et al., 7 Jul 2025).

Stage 3 performs empathetic speech instruction tuning. The speech encoder remains frozen, while the audio adapter, LLM, linear projection, and speech decoder are jointly fine-tuned. Training uses 50k English plus 50k Chinese empathetic speech-to-speech pairs, together with 100k bilingual general instruction pairs and text-to-speech samples extracted from the latter so that the model can handle both speech and text inputs (Wang et al., 7 Jul 2025).

4. Automated data construction and paralinguistic supervision

A central feature of OpenS2S is its automated data construction pipeline for empathetic dialogues. The starting point is a seed audio pool drawn from IEMOCAP, MELD, CMU-MOSEI, MEAD, and ESD. From this pool, 1,000 English and 1,000 Chinese seed audio samples are manually annotated with transcript, speaker gender, age, and emotion labels. These labels are then used as scaffolding for scalable synthetic generation (Wang et al., 7 Jul 2025).

Instruction generation is performed by Qwen3-32B-Instruct, which synthesizes paralinguistic-sensitive prompts such as age-sensitive utterances. CosyVoice2 then clones the seed voice to preserve emotion and gender attributes, yielding 50k English plus 50k Chinese speech instructions with diverse paralinguistic characteristics. Response generation is again handled by Qwen3-32B-Instruct in “thinking mode,” which produces concise empathetic responses and infers an appropriate emotional delivery conditioned on the input speech’s emotion, age, gender, and the response content. Response speech is synthesized by CosyVoice2 under instruction prompts, with a consistent reference voice fixed as young female for responses while emotional tone varies (Wang et al., 7 Jul 2025).

The final corpus contains 50k English and 50k Chinese empathetic speech-to-speech pairs. To preserve general-purpose instruction-following capability, the pipeline also adds 50k English instructions from Instruct200K translated to Chinese; neutral-emotion seed audio is used to produce speech, and the same response-generation pipeline yields an additional 100k bilingual speech-to-speech instruction pairs. The system also extracts text-to-speech samples from this general speech-to-speech data. Emotion taxonomy is inherited from the SER datasets’ categories, and the paper reports distributions for query emotion, age, gender, and response emotion, although exact per-class counts are not enumerated (Wang et al., 7 Jul 2025).

The resulting supervision is therefore bilingual and multi-dimensional. Input speech carries emotion, gender, and age tags; speaker variation in user queries is induced by seed voice cloning; output speech uses a fixed young female reference voice with varying emotional delivery. This design makes empathetic behavior partly a learned consequence of structured synthetic supervision and partly a property of the speech-native end-to-end architecture (Wang et al., 7 Jul 2025).

5. Inference, benchmarks, and released artifacts

At inference time, incoming speech is encoded by the Qwen2-Audio encoder at 25 Hz and downsampled to 6.25 Hz by the adapter. Audio embeddings, optionally mixed with text embeddings, are sent to Qwen3-8B-Instruct, whose final-layer hidden states for the response segment condition the speech decoder. Streaming generation then applies the fixed interleave ratio $V = 16{,}384$0, $V = 16{,}384$1: for every four hidden states consumed, eight speech tokens are generated; once the hidden-state stream is exhausted, generation continues autoregressively until EOS. Token chunks are converted incrementally to waveform through chunk-aware causal flow matching and HiFi-GAN (Wang et al., 7 Jul 2025).

The main published evaluation targets speech-to-text chat and empathy-sensitive understanding. On VoiceBench, OpenS2S reports alpacaeval 4.09, commoneval 3.65, ifeval 42.89, and wildvoice 3.66. On the URO-Bench empathy subsets, it reports UnderEmotion-en 46.90 and UnderEmotion-zh 67.68. The paper compares these values with Kimi-Audio, GLM-4-Voice, Qwen-2-Audio, and LLaMA-Omni2-7B-Bilingual. Kimi-Audio achieves higher scores and is noted as being trained on about 13M hours, while OpenS2S is reported as ranking second to Kimi-Audio on VoiceBench and outperforming the other open models listed (Wang et al., 7 Jul 2025).

The release strategy is unusually broad for this subfield. The project provides a demo site, code repository, model weights, and dataset distribution through project and Hugging Face pages. The repository is described as containing scripts for dataset construction, pretraining, fine-tuning, and evaluation. This openness is a central part of the system’s identity, since the paper frames current state-of-the-art empathetic LSLMs as increasingly closed in their architecture, data, and development practices (Wang et al., 7 Jul 2025).

6. Relation to the broader open S2S ecosystem

OpenS2S occupies a distinct position relative to other open speech sequence-to-sequence systems. fairseq S$V = 16{,}384$2 is a research toolkit for text-to-speech and unit-to-speech synthesis rather than an end-to-end interactive agent. It implements Tacotron 2-style, TransformerTTS, and FastSpeech 2 acoustic models, supports characters, phonemes, and learned units, provides multi-speaker conditioning, and integrates preprocessing tools such as denoising, VAD, and CER-based filtering. Its main emphasis is scalable synthesis research within the fairseq ecosystem rather than speech-native dialogue (Wang et al., 2021).

Speech2S, by contrast, is aimed at direct speech-to-speech translation. It uses a three-module architecture with a source speech encoder, shared unit encoder, and unit decoder, operating in a shared discrete “hidden-unit” space derived from HuBERT-style discretization and text-to-unit conversion. Its principal claim is that joint pre-training on unpaired speech and bilingual text improves direct S2ST by about 5 BLEU over encoder-only pre-training baselines and can match or exceed larger unit-based systems despite fewer parameters (Wei et al., 2022).

CleanS2S remains modular rather than end-to-end. It composes ASR, LLM, and TTS through full-duplex websockets, non-blocking I/O, a finite-state machine, a memory module, and a Subjective Action Judgement component that selects among interruption, refusal, deflection, silence, and standard response. Its contribution lies in proactive turn management, barge-in handling, and transparent single-file orchestration rather than unified speech-native representation learning (Lu et al., 2 Jun 2025).

RASR2 addresses yet another layer of the stack: decoding. It is a C++ research decoder for generic sequence-to-sequence ASR that supports AED, RNN-T, CTC, and related families across open-vocabulary and closed-vocabulary settings, with flexible topologies, LM integration, and generalized search. In that setting, “OpenS2S” refers to open-vocabulary sequence-to-sequence ASR rather than empathetic speech generation (Zhou et al., 2023).

Taken together, these systems indicate that open S2S research spans at least four strata: synthesis toolkits, direct translation models, modular conversational pipelines, and decoding frameworks. This suggests that the 2025 OpenS2S model is distinctive less because it is open-source in isolation than because it collapses speech understanding, empathetic instruction following, and speech generation into a single end-to-end LSLM while still releasing the training pipeline and data (Wang et al., 7 Jul 2025).

7. Limitations, safety, and future directions

The paper states several limitations. Language coverage is restricted to English and Chinese. Out of the box, the response voice is a single young female rather than a multi-speaker target space. Robustness to environmental noise, accents, and extreme paralinguistic edge cases is not systematically evaluated. The paper also does not report ablation studies for removing BLSP-Emo alignment or interleaved streaming, and it does not provide latency metrics such as first-token delay, real-time factor, throughput, or hardware-specific performance (Wang et al., 7 Jul 2025).

The data pipeline introduces additional safety and ethics questions. Because synthetic empathetic dialogues are generated by an LLM and TTS system, biases can be inherited from training corpora and prompt design. The paper also notes emotion manipulation risks: empathetic voice agents may unduly influence users, so careful guardrails and content filtering are recommended in deployment. These concerns are especially relevant because the model is explicitly optimized to perceive and respond to prosody, emotion, age, gender, and related paralinguistic attributes (Wang et al., 7 Jul 2025).

Future work is framed around expanding multilingual coverage, enabling multi-speaker response synthesis, adding explicit prosody control, and improving robust streaming under noisy conditions. Public ablations and deeper error analyses are also identified as desirable directions. In that sense, OpenS2S should be understood not as a finalized endpoint but as an openly released research platform for studying how empathetic behavior can be induced in end-to-end speech-native large models through behavioral alignment, streaming token generation, and scalable synthetic dialogue construction (Wang et al., 7 Jul 2025).