---
title: 'EmoQ-Former: Multimodal Fusion in SER'
url: https://www.emergentmind.com/topics/emoq-former
type: topic
---

# EmoQ-Former: Multimodal Fusion in SER

EmoQ-Former is the central fusion module in the EmoQ framework for speech emotion recognition (SER), where it functions as a Q-Former-based Bridge-Net that aligns and fuses continuous acoustic features and discrete textual semantic features before large-language-model decoding [2509.15775]. Within EmoQ, speech is encoded by a pretrained HuBERT model, transcription is embedded as text tokens, and the resulting multimodal representation is passed to Qwen2.5-7B-Instruct through a soft-prompt injection strategy. The module is designed to address two problems identified for MLLM-based SER: hallucination and misclassification in complex emotion reasoning, as well as the broader difficulty of feature alignment in multimodal systems [2509.15775].

## 1. Position within the EmoQ framework

EmoQ comprises three main modules: an audio encoder, the EmoQ-Former, and an LLM decoder [2509.15775]. The audio encoder uses a pretrained HuBERT model to extract frame-wise embeddings from speech, while the text branch tokenizes transcription into embeddings in the LLM hidden space. EmoQ-Former sits between these branches and the decoder, serving as the bridge-net that converts modality-specific representations into a fused, emotion-aware embedding suitable for downstream reasoning.

This placement is consequential because the framework does not treat the LLM as the primary fusion mechanism. Instead, fusion and cross-modal alignment are performed before the LLM receives the information. The paper explicitly attributes to EmoQ-Former the role of overcoming traditional challenges in multimodal SER by effectively aligning and fusing audio and text, thereby providing a compact summary that is better aligned for emotional reasoning than raw features [2509.15775].

## 2. Internal architecture and attention path

The architectural pipeline begins with speech input $a$ and transcription $t$ [2509.15775]. HuBERT produces frame-wise acoustic embeddings $\bm{E}_a \in \mathbb{R}^{L_a \times d_a}$, which are then projected to $\bm{E}'_a \in \mathbb{R}^{L_a \times d_h}$ so that they match the hidden-state size used by the rest of the model. The transcription is tokenized to embeddings $\bm{E}_t \in \mathbb{R}^{L_t \times d_h}$.

EmoQ-Former introduces $N_q$ learnable query vectors $\bm{Q} \in \mathbb{R}^{N_q \times d_h}$ and concatenates them with the text embeddings:
\[
\bm{X} = \text{Concat}(\bm{Q}, \bm{E}_t) \in \mathbb{R}^{(N_q + L_t) \times d_h}.
\]
The first stage is self-attention over the concatenated sequence, allowing the learnable queries and text tokens to interact so that the queries absorb semantic context. The updated query portion $\bm{Q}'$ is then isolated and used in a cross-attention stage over the projected audio embeddings $\bm{E}'_a$.

The cross-attention is modulated by an attention mask $\bm{M} \in \mathbb{R}^{N_q \times L_a}$, which masks out invalid or padded audio frames so that only valid audio content is attended to [2509.15775]. After cross-attention, the resulting vectors are passed through a feed-forward network, followed by attentive pooling to reduce them to a single embedding $\bm{e}_q$. The final fused multimodal embedding is obtained by L2-normalization:
\[
\bm{e}'_q = \frac{\bm{e}_q}{\|\bm{e}_q\|_2}.
\]

The operational logic is therefore sequential: semantic conditioning through self-attention, acoustic enrichment through masked cross-attention, and compression through attentive pooling. The paper characterizes the result as a cross-modally aligned emotional representation [2509.15775].

## 3. Cross-modal alignment and emotion representation

A central motivation for EmoQ-Former is the heterogeneity between modalities: audio is continuous and long, whereas text is discrete and short [2509.15775]. EmoQ-Former addresses this asymmetry by using learnable queries as the intermediate carrier of multimodal information. In the first attention stage, the queries absorb semantic information from transcription tokens; in the second, the same queries attend to the audio stream and incorporate acoustic details.

This two-step procedure is the model’s explicit alignment strategy. Rather than concatenating audio and text features directly, EmoQ-Former uses attention to align the modalities in a shared latent space, while the masking mechanism handles variable-length inputs and padding. The resulting embedding is intended to contain both semantic and acoustic cues relevant to emotion classification [2509.15775].

The paper further states that fusing both acoustic and semantic cues into the final query embedding helps the downstream LLM avoid hallucination and misclassification. A plausible implication is that the fusion module acts as a regularized interface between raw multimodal signals and autoregressive decoding, reducing the burden on the LLM to resolve low-level alignment on its own. The paper’s description of the representation as a compact, cross-modally rich “summary” is consistent with that interpretation [2509.15775].

## 4. Multi-objective affective learning

EmoQ does not rely on fusion alone; it adds multi-objective affective learning (MAL) to make the fused embeddings emotion-discriminative before LLM decoding [2509.15775]. MAL combines supervised contrastive loss and focal loss:
\[
\mathcal{L}_{MAL}= \mathcal{L}_{SCL}+\lambda \mathcal{L}_{focal}.
\]

The supervised contrastive term $\mathcal{L}_{SCL}$ brings embeddings of the same emotion class closer together while separating those from different classes in representation space. The focal term $\mathcal{L}_{focal}$ addresses class imbalance by emphasizing hard or misclassified classes. The stated purpose of the combined objective is to ensure that multimodal fusion yields emotion-aware, class-balanced, and discriminative embeddings before they are fed into the LLM [2509.15775].

This training design gives EmoQ-Former a dual role. Architecturally, it is the bridge-net for audio–text fusion; optimization-wise, it is the object of a dedicated affective objective that shapes the embedding geometry. The paper also states that two-stage training stabilizes optimization and further mitigates mode collapse between modalities. That claim locates MAL not merely as an auxiliary loss but as part of the representation-learning strategy underpinning EmoQ-Former [2509.15775].

## 5. Soft-prompt injection and decoding through the LLM

After MAL-based pretraining, EmoQ-Former is integrated into the LLM pipeline through a soft-prompt injection strategy [2509.15775]. The prompt template contains speaker information, transcription, and a special `<AUDIO>` token placeholder. Before the LLM processes the sequence, the embedding of this placeholder is replaced with the projected EmoQ-Former output, $\bm{e}_h$, which is dimension-matched to the LLM token embedding space.

The LLM decoder is Qwen2.5-7B-Instruct, and the second training stage jointly fine-tunes the EmoQ-Former, projector, and LoRA-adapted LLM using weighted cross-entropy loss with a focus on emotion token prediction [2509.15775]. At inference time, emotion categories are mapped to predefined vocabulary words, and the model computes logits from the last-token hidden state:
\[
\bm{l}_i = \bm{h}_{T,i} \bm{W}_o \in \mathbb{R}^{|V|}.
\]
Posterior probabilities are then extracted over the emotion vocabulary:
\[
\bm{q}_{i} = \text{Softmax}\left([\bm{l}_{i,k_1}, \bm{l}_{i,k_2}, ..., \bm{l}_{i,k_C}]\right) \in \mathbb{R}^C.
\]

This arrangement clarifies an important point about EmoQ-Former: it is not itself the final classifier. It produces a fused embedding that is injected into the LLM’s reasoning path, and the LLM performs the final classification over emotion words. The paper frames this as a way to guide the language model with emotionally rich fused information without architectural surgery on the LLM [2509.15775].

## 6. Training stages, empirical performance, and ablations

The training procedure is explicitly two-stage [2509.15775]. In Stage 1, EmoQ-Former and an auxiliary MLP classifier head are pre-trained using only the MAL objective and without the LLM, with the goal of learning a discriminative, emotion-aligned multimodal embedding $\bm{e}'_q$. In Stage 2, EmoQ-Former is plugged into the LLM pipeline via soft-prompt injection, and the multimodal system is jointly fine-tuned.

On IEMOCAP (4-class), the paper reports that EmoQ achieves WA of 74.4 and UA of 74.5, compared with 72.4 and 73.0 for ENT [2509.15775]. On MELD (4-class), it reports WA of 73.6, UA of 57.2, and WF1 of 71.6, compared with 61.4, 52.6, and 59.9 for LLM-CL. On MELD (7-class), it reports WA of 67.6, UA of 50.8, and WF1 of 66.5, compared with WF1 of 49.3 for DropFormer [2509.15775].

The ablation study isolates the contribution of EmoQ-Former. On MELD 7-class, the multimodal configuration without EmoQ-Former drops to WA 64.5, UA 45.3, and WF1 61.7, whereas the full EmoQ configuration reports 67.6, 50.8, and 66.5. The paper also states that removing MAL causes a significant drop, and that removing Stage-1 pre-training yields further reduction, supporting the claim that both the bridge-net and its training regime are essential components rather than interchangeable implementation details [2509.15775].

## 7. Relation to adjacent Q-Former variants and naming distinctions

EmoQ-Former belongs to a broader family of Q-Former-based modules used in multimodal emotion recognition, but its design target is distinct. In EmoQ, the module is optimized for audio–text alignment in SER and produces a single fused embedding for soft-prompt injection into an LLM [2509.15775]. By contrast, MicroEmo introduces an utterance-aware video Q-Former for video dialogues, where the module captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video, then combining them [2407.16552]. AcoustEmo similarly uses an Utterance-Aware Acoustic Q-Former with a timestamp-synchronized sliding window to extract segment-level audio tokens aligned with utterance boundaries, plus a global token for context [2603.20894].

These systems share the Q-Former lineage, and MicroEmo explicitly situates its mechanism relative to “standard Q-Formers” in models such as BLIP-2 and Video-LLaMA [2407.16552]. However, the operational differences are substantial. EmoQ-Former uses learnable queries to first interact with text and then cross-attend to projected audio embeddings, followed by attentive pooling to a single embedding [2509.15775]. MicroEmo and AcoustEmo instead emphasize utterance-aware temporal segmentation and multi-scale token fusion for dialogue-centered open-vocabulary emotion reasoning [2407.16552][2603.20894].

A common naming confusion arises with “Emoformer,” the module introduced in EmoCaps [2203.13504]. Emoformer is a transformer encoder-based block used to extract modality-specific emotion vectors that are fused into an emotion capsule for conversational emotion recognition, whereas EmoQ-Former is a Q-Former-based Bridge-Net inside an MLLM pipeline for SER. The similarity in naming does not indicate architectural equivalence: EmoCaps builds emotion vectors and capsules for downstream context modeling with Bi-LSTM or optionally DialogueRNN, while EmoQ-Former is tied to learnable queries, cross-modal attention, MAL, and soft-prompt injection into Qwen2.5-7B-Instruct [2203.13504][2509.15775].

Source: https://www.emergentmind.com/topics/emoq-former