---
title: 'EmoQ: Speech Emotion Recognition Framework'
url: https://www.emergentmind.com/topics/emoq
type: topic
---

# EmoQ: Speech Emotion Recognition Framework

EmoQ is an MLLM-based framework for speech emotion recognition (SER) that couples a frozen speech encoder, a speech-aware Q-Former, and a frozen Large Language Model to fuse audio and ASR transcripts, inject the fused representation into the LLM as a soft prompt, and predict emotion through constrained generation. It was introduced to address three specific limitations: insufficient emotion information in unimodal systems, feature-alignment difficulty in multimodal systems, and hallucination and misclassification in complex emotion reasoning. In the reported configuration, EmoQ uses HuBERT-large as the speech encoder, Qwen2.5-7B-Instruct as the LLM, and LoRA for parameter-efficient adaptation, and it achieves state-of-the-art performance on IEMOCAP and MELD [2509.15775].

## 1. Problem setting and scope

EmoQ is formulated for SER rather than generic multimodal affect analysis. Its input space consists of speech audio and an ASR transcript, and its central objective is to produce an emotion posterior over a fixed label set while preserving the representational advantages of an LLM. The framework is therefore positioned against both unimodal SER systems, which are described as having insufficient emotion information, and multimodal systems, which are described as suffering from feature alignment difficulties.

The design assumes that acoustic prosody and transcript semantics should be fused before the LLM produces its output. This differs from a naive late-fusion configuration in which audio and text are independently encoded and only loosely combined downstream. EmoQ instead inserts a dedicated intermediate module, the EmoQ-Former, between the frozen speech encoder and the frozen LLM, so that the representation entering the language model is already cross-modally aligned.

A recurring misconception is to treat EmoQ as a general-purpose multimodal emotion-understanding model across arbitrary media. In the strict sense used in the paper, EmoQ is a speech emotion recognition framework. This constraint matters because its architectural choices—HuBERT-large, transcript embeddings, query-based fusion, soft-prompt injection, and final-position label-word scoring—are all optimized around SER rather than image-text or video-language reasoning [2509.15775].

## 2. EmoQ-Former and multimodal fusion

The EmoQ-Former sits between the frozen speech encoder and the frozen LLM. Speech input \(a\) is encoded by HuBERT into embeddings \(\mathbf{E}_a \in \mathbb{R}^{L_a \times d_a}\), which are then linearly projected to
$$
\mathbf{E}'_a = \mathbf{E}_a \mathbf{W}_a \in \mathbb{R}^{L_a \times d_h}.
$$
In parallel, the ASR transcript \(t\) is tokenized into text embeddings \(\mathbf{E}_t \in \mathbb{R}^{L_t \times d_h}\). The module introduces learnable queries \(\mathbf{Q} \in \mathbb{R}^{N_q \times d_h}\), and concatenates queries with transcript embeddings:
$$
\mathbf{X} = [\mathbf{Q}; \mathbf{E}_t] \in \mathbb{R}^{(N_q + L_t) \times d_h}.
$$

Self-attention is applied over \(\mathbf{X}\) so that the queries absorb transcript semantics. The query portion is then extracted as \(\mathbf{Q}' \in \mathbb{R}^{N_q \times d_h}\). A cross-attention stage follows, in which \(\mathbf{Q}'\) attends to the projected speech sequence \(\mathbf{E}'_a\), using a binary mask \(\mathbf{M} \in \{0,1\}^{N_q \times L_a}\) to suppress padded frames. After feed-forward processing, attentive pooling, and \(L_2\) normalization, the model produces the final fused vector
\(\mathbf{e}'_q \in \mathbb{R}^{d_h}\) [2509.15775].

The architectural significance of this sequence is that discrete transcript semantics and continuous acoustic evidence are not merely co-present; they are actively fused through a query bottleneck. The paper characterizes \(\mathbf{e}'_q\) as carrying aligned acoustic-textual cues and making the representation suitable both for auxiliary classification during stage 1 and for soft-prompt injection during stage 2. This suggests that the query mechanism functions not only as a fusion operator but also as a dimensionality and task-interface constraint.

## 3. Multi-objective affective learning

EmoQ trains the fused embedding with multi-objective affective learning (MAL), which combines supervised contrastive learning (SCL) with focal loss. The stated motivation is to make \(\mathbf{e}'_q\) highly discriminative under class imbalance. The supervised contrastive component is an InfoNCE variant:
$$
\mathcal{L}_{\mathrm{SCL}} =
-\frac{1}{N}\sum_{i=1}^{N}
\frac{1}{|P(i)|}
\sum_{j \in P(i)}
\log
\frac{\exp(\mathbf{e}_i \cdot \mathbf{e}_j/\tau)}
{\sum_{k \in A(i)} \exp(\mathbf{e}_i \cdot \mathbf{e}_k/\tau)},
$$
where \(\mathbf{e}_i\) are the \(L_2\)-normalized forms of \(\mathbf{e}'_q\), \(P(i)\) indexes same-class samples, \(A(i)\) indexes all other samples, and \(\tau\) is temperature.

The focal-loss term is
$$
\mathcal{L}_{\mathrm{focal}} =
-\frac{1}{N}\sum_{i=1}^{N}
\alpha_{y_i}(1-p_{i,y_i})^{\gamma}\log p_{i,y_i},
$$
where \(p_{i,y_i}\) is the predicted probability for the true label \(y_i\), \(\alpha_{y_i}\) is a class weight, and \(\gamma\) emphasizes hard examples. MAL combines the two:
$$
\mathcal{L}_{\mathrm{MAL}} = \mathcal{L}_{\mathrm{SCL}} + \lambda \mathcal{L}_{\mathrm{focal}}.
$$

Within the logic of the model, these terms play distinct roles. SCL pulls same-class embeddings together and pushes different-class embeddings apart, while focal loss addresses class imbalance and prioritizes difficult examples. The paper’s interpretation is that this produces more separable emotion embeddings and reduces downstream ambiguity in nuanced classes such as “sadness” versus “neutral” [2509.15775].

## 4. Soft-prompt injection, optimization, and inference

After the fused vector \(\mathbf{e}'_q\) is learned, it is projected into the LLM embedding space:
$$
\mathbf{e}_h = \mathbf{e}'_q \mathbf{W}_p \in \mathbb{R}^{d_\ell}.
$$
The LLM prompt template is given as

> User: Speaker=\{\} Text=\{\} Audio=[\<AUDIO\>] Assistant: Emotion:

The special token \<AUDIO\> is replaced by \(\mathbf{e}_h\), while all other token embeddings remain unchanged. During fine-tuning, only the LoRA adapters in the LLM, the projector \(\mathbf{W}_p\), and the EmoQ-Former are updated; the speech encoder remains frozen, and most LLM weights remain frozen.

Training proceeds in two stages. In stage 1, HuBERT is frozen, the EmoQ-Former is initialized from RoBERTa, and the EmoQ-Former plus a small auxiliary MLP classifier are trained using \(\mathcal{L}_{\mathrm{MAL}}\). After convergence, the auxiliary MLP is discarded and the EmoQ-Former weights are retained. In stage 2, the pretrained EmoQ-Former is connected to the projector and Qwen2.5-7B-Instruct, \(\mathbf{e}_h\) is inserted as a soft prompt, and weighted cross-entropy is optimized on token generation:
$$
\mathcal{L}_{\mathrm{CE}} =
-\frac{1}{N}\sum_{i=1}^N \sum_{j=1}^L
w_{i,j}\log P\bigl(t_{i,j}\mid t_{i,<j}, \mathbf{x}_i\bigr),
$$
where \(w_{i,j}\) is higher for emotion keywords [2509.15775].

Inference is generation-based but label-constrained. After the model autoregressively produces “Emotion: \(X\),” the logits over a fixed set of emotion words \(\{k_1,\dots,k_C\}\) at the final position are collected and normalized:
$$
\mathbf{q}_i =
\operatorname{Softmax}\bigl([\ell_{i,k_1}, \dots, \ell_{i,k_C}]\bigr)
\in \mathbb{R}^C.
$$
This construction is presented as a direct mechanism for reducing hallucination. The paper attributes the reduction to three components: cross-modal alignment in the EmoQ-Former, template-constrained generation with token-level weighting, and the MAL objective’s production of highly separable emotion embeddings. The specific claim is that the model cannot “imagine” audio content beyond \(\mathbf{e}_h\), and that the fixed response prefix “Emotion:” steers the system away from off-topic answers.

## 5. Experimental results and ablations

EmoQ is evaluated on IEMOCAP and MELD. IEMOCAP is treated as a 4-class task under 5-fold speaker-independent cross-validation, while MELD is evaluated in both 4-class and 7-class settings using the provided train/validation/test splits. The reported headline results are as follows [2509.15775].

| Benchmark | Setting | EmoQ result |
|---|---|---|
| IEMOCAP | 4-class WA / UA | 74.4 / 74.5 |
| MELD | 4-class WA / UA / WF1 | 73.6 / 57.2 / 71.6 |
| MELD | 7-class WA / UA / WF1 | 67.6 / 50.8 / 66.5 |

These scores are reported as state of the art relative to the cited baselines ENT (2024), LLM-CL (2025), and SpeechFormer++ (2023). The empirical profile is reinforced by ablations on MELD 7-class. In the modality-and-alignment study, adding text and audio yields \(+3.6\) WA, and adding the EmoQ-Former yields a further \(+3.1\) WA. In the MAL study, focal loss alone yields \(+3.4\) WA, SCL alone yields \(+1.7\) WA, and their combination yields \(+5.3\) WA. In the training-strategy study, the two-stage procedure gives \(+2.7\) WA over naive end-to-end training.

The reported pattern is therefore not limited to a single design choice. Query-based alignment, the joint affective objective, and stage-wise optimization each contribute measurable gains. A plausible implication is that EmoQ’s performance derives from interaction effects among representation alignment, imbalance-aware discrimination, and tightly constrained LLM conditioning rather than from the mere presence of an LLM.

## 6. Nomenclature and position in emotion-AI research

The term “EmoQ” is not fully unambiguous in the 2025 literature. A separate paper uses “EmoQ” as a shorthand for Emotion-Qwen, a distinct multimodal framework for unified emotion and general vision-language understanding rather than SER. That system uses a Facial Emotion Capture module, a frozen CLIP-ViT vision encoder, a Mixture-of-Experts Hybrid Compressor with emotion and general experts, Qwen2.5 as the LLM backbone, a three-stage pre-training pipeline, and the Video Emotion Reasoning dataset with 40,391 bilingual video clips and approximately 80,800 reasoning labels [2505.06685]. In encyclopedia usage, the exact title “EmoQ: Speech Emotion Recognition via Speech-Aware Q-Former and Large Language Model” is therefore the clearest disambiguator for the SER framework.

EmoQ also sits within a broader ecosystem of multimodal emotion research that separates modeling from evaluation. EmoBench-Reddit, for example, is a hierarchical benchmark for multimodal emotion understanding with 350 Reddit image-text samples and seven questions per sample spanning perception and cognition. Its empirical findings show that strong MLLMs can exceed 90% accuracy on low-level perception tasks while dropping to 50–65% on the most demanding cognition task, with sarcasm emerging as the hardest category [2509.11101]. This suggests that high raw perceptual competence does not imply robust emotional reasoning, and it clarifies why frameworks such as EmoQ emphasize explicit alignment and affect-specific learning objectives.

A related but methodologically different direction is the Expansion Quantization Network, which targets text emotion detection and annotation by expanding sparse manual labels into continuous energy-level scores in \([0,10]^C\), using regression rather than generative prompting. EQN is described as the first framework to achieve automatic micro-emotion annotation with energy-level scores, and it improves GoEmotions results through a full-label regression scheme [2411.06160]. Compared with that line of work, EmoQ remains centered on multimodal SER and fixed-label posterior extraction. The contrast highlights a broader division in emotion computing between continuous intensity modeling and prompt-conditioned categorical reasoning.

Source: https://www.emergentmind.com/topics/emoq