EmoQ-Former: Multimodal Fusion in SER

• The paper introduces EmoQ-Former, which overcomes modality alignment issues by using a two-stage attention process with learnable queries and MAL.
• It leverages a HuBERT encoder, self- and cross-attention mechanisms, and attentive pooling to fuse acoustic and textual embeddings into a single emotion-aware representation.
• Empirical evaluations on IEMOCAP and MELD demonstrate that EmoQ-Former improves classification performance and mitigates hallucination and misclassification in SER.

EmoQ-Former is the central fusion module in the EmoQ framework for speech emotion recognition (SER), where it functions as a Q-Former-based Bridge-Net that aligns and fuses continuous acoustic features and discrete textual semantic features before large-language-model decoding (Yang et al., 19 Sep 2025). Within EmoQ, speech is encoded by a pretrained HuBERT model, transcription is embedded as text tokens, and the resulting multimodal representation is passed to Qwen2.5-7B-Instruct through a soft-prompt injection strategy. The module is designed to address two problems identified for MLLM-based SER: hallucination and misclassification in complex emotion reasoning, as well as the broader difficulty of feature alignment in multimodal systems (Yang et al., 19 Sep 2025).

1. Position within the EmoQ framework

EmoQ comprises three main modules: an audio encoder, the EmoQ-Former, and an LLM decoder (Yang et al., 19 Sep 2025). The audio encoder uses a pretrained HuBERT model to extract frame-wise embeddings from speech, while the text branch tokenizes transcription into embeddings in the LLM hidden space. EmoQ-Former sits between these branches and the decoder, serving as the bridge-net that converts modality-specific representations into a fused, emotion-aware embedding suitable for downstream reasoning.

This placement is consequential because the framework does not treat the LLM as the primary fusion mechanism. Instead, fusion and cross-modal alignment are performed before the LLM receives the information. The paper explicitly attributes to EmoQ-Former the role of overcoming traditional challenges in multimodal SER by effectively aligning and fusing audio and text, thereby providing a compact summary that is better aligned for emotional reasoning than raw features (Yang et al., 19 Sep 2025).

2. Internal architecture and attention path

The architectural pipeline begins with speech input $a$ and transcription $t$ (Yang et al., 19 Sep 2025). HuBERT produces frame-wise acoustic embeddings $\bm{E}_a \in \mathbb{R}^{L_a \times d_a}$, which are then projected to $\bm{E}'_a \in \mathbb{R}^{L_a \times d_h}$ so that they match the hidden-state size used by the rest of the model. The transcription is tokenized to embeddings $\bm{E}_t \in \mathbb{R}^{L_t \times d_h}$.

EmoQ-Former introduces $N_q$ learnable query vectors $\bm{Q} \in \mathbb{R}^{N_q \times d_h}$ and concatenates them with the text embeddings: $$\bm{X} = \text{Concat}(\bm{Q}, \bm{E}_t) \in \mathbb{R}^{(N_q + L_t) \times d_h}.$$ The first stage is self-attention over the concatenated sequence, allowing the learnable queries and text tokens to interact so that the queries absorb semantic context. The updated query portion $\bm{Q}'$ is then isolated and used in a cross-attention stage over the projected audio embeddings $\bm{E}'_a$.

The cross-attention is modulated by an attention mask $t$0, which masks out invalid or padded audio frames so that only valid audio content is attended to (Yang et al., 19 Sep 2025). After cross-attention, the resulting vectors are passed through a feed-forward network, followed by attentive pooling to reduce them to a single embedding $t$1. The final fused multimodal embedding is obtained by L2-normalization: $t$2

The operational logic is therefore sequential: semantic conditioning through self-attention, acoustic enrichment through masked cross-attention, and compression through attentive pooling. The paper characterizes the result as a cross-modally aligned emotional representation (Yang et al., 19 Sep 2025).

3. Cross-modal alignment and emotion representation

A central motivation for EmoQ-Former is the heterogeneity between modalities: audio is continuous and long, whereas text is discrete and short (Yang et al., 19 Sep 2025). EmoQ-Former addresses this asymmetry by using learnable queries as the intermediate carrier of multimodal information. In the first attention stage, the queries absorb semantic information from transcription tokens; in the second, the same queries attend to the audio stream and incorporate acoustic details.

This two-step procedure is the model’s explicit alignment strategy. Rather than concatenating audio and text features directly, EmoQ-Former uses attention to align the modalities in a shared latent space, while the masking mechanism handles variable-length inputs and padding. The resulting embedding is intended to contain both semantic and acoustic cues relevant to emotion classification (Yang et al., 19 Sep 2025).

The paper further states that fusing both acoustic and semantic cues into the final query embedding helps the downstream LLM avoid hallucination and misclassification. A plausible implication is that the fusion module acts as a regularized interface between raw multimodal signals and autoregressive decoding, reducing the burden on the LLM to resolve low-level alignment on its own. The paper’s description of the representation as a compact, cross-modally rich “summary” is consistent with that interpretation (Yang et al., 19 Sep 2025).

4. Multi-objective affective learning

EmoQ does not rely on fusion alone; it adds multi-objective affective learning (MAL) to make the fused embeddings emotion-discriminative before LLM decoding (Yang et al., 19 Sep 2025). MAL combines supervised contrastive loss and focal loss: $t$3

The supervised contrastive term $t$4 brings embeddings of the same emotion class closer together while separating those from different classes in representation space. The focal term $t$5 addresses class imbalance by emphasizing hard or misclassified classes. The stated purpose of the combined objective is to ensure that multimodal fusion yields emotion-aware, class-balanced, and discriminative embeddings before they are fed into the LLM (Yang et al., 19 Sep 2025).

This training design gives EmoQ-Former a dual role. Architecturally, it is the bridge-net for audio–text fusion; optimization-wise, it is the object of a dedicated affective objective that shapes the embedding geometry. The paper also states that two-stage training stabilizes optimization and further mitigates mode collapse between modalities. That claim locates MAL not merely as an auxiliary loss but as part of the representation-learning strategy underpinning EmoQ-Former (Yang et al., 19 Sep 2025).

5. Soft-prompt injection and decoding through the LLM

After MAL-based pretraining, EmoQ-Former is integrated into the LLM pipeline through a soft-prompt injection strategy (Yang et al., 19 Sep 2025). The prompt template contains speaker information, transcription, and a special <AUDIO> token placeholder. Before the LLM processes the sequence, the embedding of this placeholder is replaced with the projected EmoQ-Former output, $t$6, which is dimension-matched to the LLM token embedding space.

The LLM decoder is Qwen2.5-7B-Instruct, and the second training stage jointly fine-tunes the EmoQ-Former, projector, and LoRA-adapted LLM using weighted cross-entropy loss with a focus on emotion token prediction (Yang et al., 19 Sep 2025). At inference time, emotion categories are mapped to predefined vocabulary words, and the model computes logits from the last-token hidden state: $t$7 Posterior probabilities are then extracted over the emotion vocabulary: $t$8

This arrangement clarifies an important point about EmoQ-Former: it is not itself the final classifier. It produces a fused embedding that is injected into the LLM’s reasoning path, and the LLM performs the final classification over emotion words. The paper frames this as a way to guide the LLM with emotionally rich fused information without architectural surgery on the LLM (Yang et al., 19 Sep 2025).

6. Training stages, empirical performance, and ablations

The training procedure is explicitly two-stage (Yang et al., 19 Sep 2025). In Stage 1, EmoQ-Former and an auxiliary MLP classifier head are pre-trained using only the MAL objective and without the LLM, with the goal of learning a discriminative, emotion-aligned multimodal embedding $t$9. In Stage 2, EmoQ-Former is plugged into the LLM pipeline via soft-prompt injection, and the multimodal system is jointly fine-tuned.

On IEMOCAP (4-class), the paper reports that EmoQ achieves WA of 74.4 and UA of 74.5, compared with 72.4 and 73.0 for ENT (Yang et al., 19 Sep 2025). On MELD (4-class), it reports WA of 73.6, UA of 57.2, and WF1 of 71.6, compared with 61.4, 52.6, and 59.9 for LLM-CL. On MELD (7-class), it reports WA of 67.6, UA of 50.8, and WF1 of 66.5, compared with WF1 of 49.3 for DropFormer (Yang et al., 19 Sep 2025).

The ablation study isolates the contribution of EmoQ-Former. On MELD 7-class, the multimodal configuration without EmoQ-Former drops to WA 64.5, UA 45.3, and WF1 61.7, whereas the full EmoQ configuration reports 67.6, 50.8, and 66.5. The paper also states that removing MAL causes a significant drop, and that removing Stage-1 pre-training yields further reduction, supporting the claim that both the bridge-net and its training regime are essential components rather than interchangeable implementation details (Yang et al., 19 Sep 2025).

7. Relation to adjacent Q-Former variants and naming distinctions

EmoQ-Former belongs to a broader family of Q-Former-based modules used in multimodal emotion recognition, but its design target is distinct. In EmoQ, the module is optimized for audio–text alignment in SER and produces a single fused embedding for soft-prompt injection into an LLM (Yang et al., 19 Sep 2025). By contrast, MicroEmo introduces an utterance-aware video Q-Former for video dialogues, where the module captures multi-scale and contextual dependencies by generating visual token sequences for each utterance segment and for the entire video, then combining them (Zhang, 2024). AcoustEmo similarly uses an Utterance-Aware Acoustic Q-Former with a timestamp-synchronized sliding window to extract segment-level audio tokens aligned with utterance boundaries, plus a global token for context (Zhang et al., 21 Mar 2026).

These systems share the Q-Former lineage, and MicroEmo explicitly situates its mechanism relative to “standard Q-Formers” in models such as BLIP-2 and Video-LLaMA (Zhang, 2024). However, the operational differences are substantial. EmoQ-Former uses learnable queries to first interact with text and then cross-attend to projected audio embeddings, followed by attentive pooling to a single embedding (Yang et al., 19 Sep 2025). MicroEmo and AcoustEmo instead emphasize utterance-aware temporal segmentation and multi-scale token fusion for dialogue-centered open-vocabulary emotion reasoning (Zhang, 2024, Zhang et al., 21 Mar 2026).

A common naming confusion arises with “Emoformer,” the module introduced in EmoCaps (Li et al., 2022). Emoformer is a transformer encoder-based block used to extract modality-specific emotion vectors that are fused into an emotion capsule for conversational emotion recognition, whereas EmoQ-Former is a Q-Former-based Bridge-Net inside an MLLM pipeline for SER. The similarity in naming does not indicate architectural equivalence: EmoCaps builds emotion vectors and capsules for downstream context modeling with Bi-LSTM or optionally DialogueRNN, while EmoQ-Former is tied to learnable queries, cross-modal attention, MAL, and soft-prompt injection into Qwen2.5-7B-Instruct (Li et al., 2022, Yang et al., 19 Sep 2025).