BLSP-Emo: Emotion-Aware Speech-Language Model

• The paper introduces BLSP-Emo, which extends bootstrapped language-speech pretraining with a two-stage alignment procedure for semantic and emotion-aware continuation.
• It leverages a Whisper-large-v2 encoder, Qwen-7B-Chat LLM, and a convolutional adapter to process speech inputs directly without cascaded ASR pipelines.
• Empirical results demonstrate robust speech emotion recognition, improved empathetic response generation, and strong zero-shot cross-lingual performance.

Searching arXiv for BLSP-Emo and closely related speech-emotion benchmark papers. BLSP-Emo is a large speech-LLM framework for emotion-aware spoken interaction that extends Bootstrapped Language-Speech Pretraining with explicit emotion support. It is designed to process speech input directly, capture both semantic and paralinguistic information, and generate empathetic textual responses without relying on a cascaded ASR$\rightarrow$LLM or SER+ASR$\rightarrow$LLM pipeline. The method is defined by a two-stage alignment procedure: semantic alignment using ASR corpora, followed by emotion alignment using speech emotion recognition datasets through an emotion-aware continuation objective. In the reported experiments, this design yields strong speech emotion recognition, competitive or superior empathetic response behavior, and zero-shot cross-lingual generalization, while preserving general instruction-following ability better than direct SER fine-tuning (Wang et al., 2024).

1. Definition and research context

BLSP-Emo stands for Bootstrapped Language-Speech Pretraining with Emotion support. It is presented as an end-to-end speech-LLM whose central objective is to understand both what is said and how it is said, then produce responses that are appropriate to the conveyed emotional tone (Wang et al., 2024). The framework is motivated by a limitation in prior speech-LLM systems: some systems are task-oriented audio models that do not inherit strong instruction-following and conversational capabilities, while others inherit instruction-following ability but align primarily to linguistic content rather than emotion (Wang et al., 2024).

The architecture preserves the high-level BLSP formulation: a speech encoder with parameters $\psi$, an instruction-following LLM with parameters $\phi$, and a modality adapter with parameters $\theta$ that bridges speech and text spaces (Wang et al., 2024). The concrete backbone choices are explicitly specified as the encoder of Whisper-large-v2, Qwen-7B-Chat, and a convolutional subsampler used as the modality adapter (Wang et al., 2024). Appendix details describe the adapter as three 1D conv layers + bottleneck hidden size 512, with each convolution using stride 2, kernel size 5, and padding 2, thereby reducing speech sequence length by a factor of 8 (Wang et al., 2024).

The model is end-to-end in the sense of speech input directly to LLM text output, rather than a modular cascade. The paper is explicit that it is not trained to generate speech tokens; its output is text, and its end-to-end property concerns speech understanding and text generation in a single model (Wang et al., 2024). This places BLSP-Emo in a distinct position relative to conventional speech emotion recognition systems, such as biologically inspired reservoir-based SER pipelines, which are optimized for classification rather than open-ended response generation (Lotfidereshgi et al., 2021).

2. Two-stage alignment procedure

The defining feature of BLSP-Emo is its two-stage training pipeline. The first stage is semantic alignment, which uses ASR data to align speech input with transcript-conditioned LLM behavior. The second stage is emotion alignment, which uses SER data to align speech with emotion-sensitive continuation behavior (Wang et al., 2024).

In the semantic stage, each ASR sample consists of a speech-transcript pair $(\mathbf{s}, \mathbf{x})$. A continuation-style instruction is constructed:

$\psi$8

The frozen text LLM generates a continuation $\mathbf{y}$, producing tuples $(\mathbf{s}, \mathbf{x}, \mathbf{y})$ (Wang et al., 2024). The speech-LLM is then trained to generate the same continuation from the speech input, using a knowledge-distillation-style objective:

$$\begin{align} &\ell_\text{Semantic}(\mathbf{s}, \mathbf{x}, \mathbf{y}) = \nonumber \ &\quad -\sum_{j,y} p_\phi(y|\mathbf{x}, \mathbf{y}_{<j})\log p_{\psi, \theta, \phi}(y|\mathbf{s}, \mathbf{y}_{<j}) \end{align}$$

During this stage, the speech encoder $\psi$ is frozen, the LLM $\rightarrow$0 is frozen, and only the adapter $\rightarrow$1 is tuned (Wang et al., 2024). This training strategy is intended to preserve the LLM’s pre-existing instruction-following behavior while making speech a first-class conditioning modality.

The second stage introduces emotion support. Each SER sample has the form $\rightarrow$2, where $\rightarrow$3 is an emotion label (Wang et al., 2024). An emotion-aware transcript prompt is used to generate a continuation with the underlying LLM:

$\psi$9

This yields an emotion-conditioned continuation target $\rightarrow$4 coherent with both the semantic content and the annotated emotion (Wang et al., 2024). After initialization from the semantically aligned model, BLSP-Emo is then trained from speech alone with the prompt:

$\phi$0

The primary generation loss is:

$\rightarrow$5

An auxiliary SER classification loss is also added:

$\rightarrow$6

In this second stage, the speech encoder $\rightarrow$7 is unfrozen, the adapter $\rightarrow$8 remains trainable, the LLM $\rightarrow$9 is adapted, and the SER head $\psi$0 is trainable (Wang et al., 2024). The LLM is adapted with PLoRA, where LoRA is applied to the key/query/value/output projection matrices only for speech tokens (Wang et al., 2024). This selective adaptation is intended to add speech-specific capacity while preserving ordinary text-side capabilities.

A plausible implication is that BLSP-Emo’s core novelty lies less in the backbone components than in the alignment strategy that ties emotional supervision to LLM-style continuation behavior rather than to classification alone.

3. Data resources, label space, and prompt design

BLSP-Emo reuses existing public resources rather than relying on proprietary multimodal corpora. For semantic alignment, the ASR datasets are LibriSpeech, CommonVoice 13.0, GigaSpeech M, and WeNetSpeech (Wang et al., 2024). The paper reports about 1.9M English $\psi$1 pairs, plus a comparable number of Chinese ASR samples randomly selected from WeNetSpeech (Wang et al., 2024).

For emotion alignment, the training SER datasets are IEMOCAP sessions 1–4, MELD train, CMU MOSEI, MEAD, and ESD, for a total of about 70k utterances in English and Chinese (Wang et al., 2024). Evaluation is separated into in-domain, out-of-domain, and cross-lingual settings:

Setting	Datasets
In-domain	IEMOCAP session 5; MELD test
Out-of-domain	RAVDESS; MerBench test1; MerBench test2
Zero-shot cross-lingual	AESDD; CaFE; RESD

The label inventory is normalized to five emotion classes: neutral, happy, sad, angry, and surprise (Wang et al., 2024). Original labels are mapped when necessary, such as joy $\psi$2 happy, sadness $\psi$3 sad, anger $\psi$4 angry, and happiness $\psi$5 happy; labels outside the common set are discarded, including disgust, fear, frustrated, disappointed, and worried, depending on the source dataset (Wang et al., 2024). A specific exception is that IEMOCAP surprise was excluded due to scarcity (Wang et al., 2024).

To ensure sufficient semantic content for continuation, English samples with fewer than 5 words and Chinese samples with fewer than 5 characters are removed (Wang et al., 2024). For CMU MOSEI, only single-label samples are used (Wang et al., 2024).

The paper specifies several prompt templates central to the method. Besides the semantic and emotion-aware continuation prompts, it defines a prompt for SER evaluation:

$\phi$1

and a cascaded baseline prompt that combines transcript and predicted emotion:

$\phi$2

For empathetic response evaluation, the system prompt is modified to:

“You are a helpful assistant. Your response should fulfill requests with empathy toward the user's emotional tone.” (Wang et al., 2024)

These prompt designs make clear that BLSP-Emo operationalizes emotion not as an auxiliary tag alone, but as a factor that should shape downstream continuation behavior.

4. Empirical performance

BLSP-Emo is evaluated on speech emotion recognition, empathetic response generation, multi-turn conversation, and zero-shot cross-lingual SER (Wang et al., 2024). The paper reports accuracy (%) for SER and GPT-4 judged Quality and Empathy (0–10) for response generation (Wang et al., 2024).

On SER, the reported results are as follows:

Method	IEMOCAP	MELD	RAVDESS	MerBench test1	MerBench test2
Text+LLM	54.8	54.0	11.1	n/a	n/a
Whisper+LLM	57.1	53.8	13.7	49.4	46.9
BLSP	52.8	53.1	11.1	44.9	45.3
BLSP-SER	78.6	56.4	70.5	51.5	56.0
BLSP-Emo	76.0	57.3	72.0	60.0	54.7

Encoder-only classifiers are also reported: HuBERT-Large at 64.6 / 53.2 / 70.5 / 55.6 / 45.3, wav2vec2-Large at 69.3 / 54.8 / 64.0 / 41.2 / 40.6, and WavLM-Large at 68.9 / 54.6 / 70.3 / 48.3 / 42.8 across the same evaluation columns (Wang et al., 2024).

The empirical pattern supports several conclusions stated in the paper. BLSP-Emo substantially exceeds transcript-only systems, especially where acoustic emotion is important. It also substantially exceeds BLSP, which indicates that semantic alignment alone is insufficient for robust emotion understanding. At the same time, BLSP-SER attains the best IEMOCAP accuracy, which shows that direct label fine-tuning can optimize classification performance on some datasets, but that is not the whole design target of BLSP-Emo (Wang et al., 2024).

For empathetic response generation, the paper introduces SpeechAlpaca, built from Alpaca-52k. GPT-4 is used to infer plausible emotional tones for text instructions, the pool is restricted to neutral, cheerful, sad, and angry, one plausible emotion is randomly selected, and 100 instructions per emotion are synthesized with the Microsoft TTS API, giving 400 test examples (Wang et al., 2024). Average GPT-4 ambiguity is reported as about 1.4 plausible emotions per utterance (Wang et al., 2024).

The SpeechAlpaca results are:

Method	SER	Quality	Empathy
Text+LLM	40.0	8.9	7.4
Whisper+LLM	40.1	8.9	7.4
BLSP	36.8	8.6	7.1
BLSP-SER	80.3	1.9	2.1
BLSP-Emo	83.8	8.8	7.7
HuBERT+Whisper+LLM	76.3	8.9	7.6
wav2vec2+Whisper+LLM	83.3	9.0	7.7
WavLM+Whisper+LLM	80.8	8.9	7.8

These numbers are central to the interpretation of the method. BLSP-Emo raises BLSP from 36.8 to 83.8 on SpeechAlpaca SER while also improving Quality from 8.6 to 8.8 and Empathy from 7.1 to 7.7 (Wang et al., 2024). By contrast, BLSP-SER attains 80.3 SER but collapses response quality and empathy to 1.9 and 2.1, respectively (Wang et al., 2024). This directly supports the claim that classification-oriented fine-tuning alone is not sufficient for empathetic spoken assistance.

For zero-shot cross-lingual SER, BLSP-Emo is evaluated on AESDD (Greek), CaFE (French), and RESD (Russian) (Wang et al., 2024):

Method	AESDD (Gr)	CaFE (Fr)	RESD (Ru)	Avg.
Whisper+LLM	25.3	16.2	35.4	25.6
BLSP	6.8	17.3	27.2	17.1
BLSP-SER	68.9	76.7	41.4	62.3
BLSP-Emo	68.8	75.3	46.2	63.4
HuBERT-Large	53.9	66.5	43.0	54.5
wav2vec2-Large	31.2	61.7	39.2	44.0
WavLM-Large	47.0	70.7	37.3	51.7

These results indicate strong cross-lingual transfer for a generative speech-LLM trained on English and Chinese emotion resources (Wang et al., 2024). This suggests that the learned emotional representations are not purely dataset-specific.

5. Ablations, comparisons, and relation to adjacent research

The ablation study clarifies which parts of BLSP-Emo are structurally necessary. Without semantic pretraining, results fall from 76.0 to 68.5 on IEMOCAP, from 72.0 to 68.6 on RAVDESS, from 83.8 to 80.3 on SpeechAlpaca SER, and from 8.8 / 7.7 to 6.7 / 7.0 on Quality/Empathy (Wang et al., 2024). This shows that semantic alignment is foundational rather than optional.

Without the auxiliary SER loss, IEMOCAP drops from 76.0 to 72.2, RAVDESS from 72.0 to 66.6, while SpeechAlpaca SER changes only slightly from 83.8 to 83.3, and Quality/Empathy remain 8.8 / 7.7 (Wang et al., 2024). The paper interprets this as evidence that the auxiliary classifier helps standalone SER on natural speech, whereas empathetic behavior is driven primarily by the continuation objective.

Two analysis variants are especially informative. BLSP-ChatGPT, which replaces same-LLM target construction with GPT-3.5-turbo, performs markedly worse: 68.9 on IEMOCAP, 54.2 on RAVDESS, 68.0 on SpeechAlpaca SER, 6.1 Quality, and 6.0 Empathy, compared with 76.0 / 72.0 / 83.8 / 8.8 / 7.7 for BLSP-Emo (Wang et al., 2024). BLSP-MultiTask, which uses plain continuation + SER instead of emotion-aware continuation, also underperforms BLSP-Emo: 75.3 vs 76.0 on IEMOCAP, 71.5 vs 72.0 on RAVDESS, 77.8 vs 83.8 on SpeechAlpaca SER, 8.3 vs 8.8 Quality, and 7.2 vs 7.7 Empathy (Wang et al., 2024). These results support the claim that the emotion-aware continuation task is not reducible to generic multitask training.

Within the broader SER landscape, BLSP-Emo occupies a different methodological regime from classic classification systems. For example, the biologically inspired SER framework later referred to as BLSP-Emo in the query context of older work is a two-branch liquid state machine operating on source-filter auditory representations and optimized for categorical recognition on Emo-DB, achieving 82.35% accuracy under 50-fold cross-validation (Lotfidereshgi et al., 2021). Despite the acronym collision in informal usage, this is a distinct research object from the 2024 BLSP-Emo speech-LLM (Lotfidereshgi et al., 2021). The 2024 model is instead aligned more closely with the contemporary movement toward speech-native LLMs that integrate semantic and affective cues in a single generation model (Wang et al., 2024).

A further point of comparison comes from large-scale SER benchmarking. EMO-SUPERB shows that modern SSL upstreams such as XLS-R-1B, WavLM, and HuBERT are strong frozen-encoder baselines, with XLS-R-1B achieving the best average macro-F1 of 0.38352 across benchmark settings (Wu et al., 2024). BLSP-Emo differs by optimizing a generative speech-language pathway rather than a frozen-encoder classification head, yet its comparison to encoder-only classifiers in the paper serves a similar benchmarking role (Wang et al., 2024, Wu et al., 2024).

6. Limitations, misconceptions, and broader significance

Several limitations are explicit. SpeechAlpaca uses synthetic speech, so it lacks realistic variation in speakers and emotional expression (Wang et al., 2024). The multi-turn evaluation checks only one response turn within dialogue context rather than long-horizon conversational behavior (Wang et al., 2024). The emotion ontology is restricted to five discrete classes, which is much narrower than human affective space (Wang et al., 2024). The work also focuses on emotion rather than other paralinguistic and pragmatic dimensions such as intention, sarcasm, or broader tone beyond emotion (Wang et al., 2024). Finally, the model does not generate speech, so it does not constitute a full speech-to-speech empathetic assistant (Wang et al., 2024).

One potential misconception is that BLSP-Emo is primarily a better SER classifier. The reported results do not support that simplification. BLSP-SER can outperform BLSP-Emo on particular SER test sets such as IEMOCAP (78.6 vs 76.0) (Wang et al., 2024). The central claim of BLSP-Emo is instead that emotion-aware continuation preserves and extends instruction-following behavior while enabling emotionally appropriate generation, something direct SER fine-tuning fails to do, as evidenced by the catastrophic 1.9 / 2.1 Quality/Empathy scores of BLSP-SER on SpeechAlpaca (Wang et al., 2024).

Another important contextual issue is evaluation robustness. Recent large-scale SER analysis shows that modern systems can exhibit substantial subgroup bias, with females obtaining slightly better overall SER performance on average, large male-female gaps at the individual-emotion level, and especially strong valence-linked gender bias in Mandarin corpora (Lin et al., 2024). This suggests that future evaluations of BLSP-Emo-style systems would benefit from fairness-aware reporting rather than aggregate scores alone. A plausible implication is that empathetic large speech-LLMs may need the same bias auditing now expected of standalone SER systems.

More broadly, BLSP-Emo demonstrates a practical route toward emotion-aware speech-language modeling using public datasets and moderate adaptation cost. The paper reports semantic alignment at 1 epoch, batch size 768, requiring about 2.5 days on 4$\psi$6A100, and emotion alignment at 3 epochs, batch size 128, requiring about 3 hours on 4$\psi$7A100 (Wang et al., 2024). This supports the authors’ position that the method is an accessible alternative to opaque, high-compute proprietary systems (Wang et al., 2024).

In that sense, BLSP-Emo is significant less as a terminal architecture than as a training recipe: align speech with transcript-conditioned LLM behavior first, then align speech with emotion-sensitive continuation behavior. This suggests a generalizable pattern for future speech-language systems that aim to incorporate richer paralinguistic signals beyond semantics alone (Wang et al., 2024).