MECap-R1: Multimodal Speech Emotion Captioning

• MECap-R1 is a two-stage framework that first applies supervised fine-tuning and then refines captioning via reinforcement learning.
• It fuses speech audio, video cues, and textual context to generate natural language descriptions that capture nuanced emotions.
• The model’s emotion-aware reward and Emo-GRPO technique significantly boost accuracy and vocabulary diversity on the EmotionTalk dataset.

MECap-R1 is a two-stage, multimodal speech-emotion-captioning framework for Speech Emotion Captioning (SEC), a task motivated by the observation that the inherent complexity of emotional content in human speech makes traditional discrete classification methods inadequate and that natural-language descriptions can capture affect more effectively. The model combines supervised fine-tuning with reinforcement learning, uses an emotion-aware policy to align generated captions with emotional semantics, and is evaluated on the EmotionTalk dataset, where it is reported to generate emotion descriptions with gains in both accuracy and diversity (Sun et al., 23 Sep 2025).

1. Problem setting and conceptual scope

Speech Emotion Captioning seeks to describe speech emotions in natural language rather than assign them to a fixed label set. In MECap-R1, the target is not only semantic adequacy but also affective faithfulness: the generated caption should reflect emotional nuance that is difficult to express through rigid categorical rules. The paper frames this as a limitation of conventional discrete classification and positions caption generation as a more flexible representation of affect (Sun et al., 23 Sep 2025).

Within that framing, MECap-R1 is explicitly multimodal. Its inputs comprise speech audio, video frames carrying facial and gesture cues, and, optionally, textual context or transcript tokens. This design places the method at the intersection of speech modeling, vision-language fusion, and sequence generation. A plausible implication is that the method treats emotion description as a conditional generation problem over heterogeneous evidence streams rather than as a univariate acoustic inference task.

2. Multimodal architecture and generative policy

The system architecture is a two-stage pipeline. Speech audio is processed by a pre-trained HuBERT acoustic encoder that yields frame-level acoustic embeddings. Video frames are processed by a standard vision encoder, given in the paper as an example of ResNet-50 plus Transformer, producing per-frame visual embeddings. Textual context, when present, is incorporated as token embeddings. These modality-specific embeddings are projected into a shared embedding space, fused, for example via cross-modal attention, and then passed to an autoregressive Transformer decoder denoted as the generative policy network $\pi_\theta$ (Sun et al., 23 Sep 2025).

At each decoding step $t$, the decoder produces

$\pi_\theta(y_t \mid y_{<t}, M),$

where $M = \{\text{audio, video, text embeddings}\}$. Over a full caption sequence $Y$ of length $T$, the policy is

$$\pi_\theta(Y \mid M)=\prod_{t=1}^T\pi_\theta(y_t\mid y_{<t},M).$$

The decoder parameters $\theta$ are first learned by supervised fine-tuning and then refined by reinforcement learning. This division into a supervised cold start followed by policy optimization is central to the framework: the first stage establishes a usable captioning model, while the second stage modifies generation behavior according to an explicit reward.

3. Emotion anchor space and emotion-aware reward

A distinguishing component of MECap-R1 is its explicit emotion-aware policy module, which constructs an “emotion anchor” subspace. A Sentence-BERT semantic encoder $\mathcal{E}:\mathcal{T}\to\mathbb{R}^D$ maps a text sequence $T$ to a latent vector $t$0. For each of $t$1 discrete emotion categories $t$2, a small lexicon $t$3 of prototypical words is collected, and the corresponding anchor vector is defined as the centroid

$t$4

An emotion-coordinate mapping $t$5 then projects any text embedding onto the anchors via cosine similarity:

$t$6

A candidate caption $t$7 is therefore represented in emotion-coordinate space as $t$8 (Sun et al., 23 Sep 2025).

The reinforcement-learning signal uses this space directly. The emotion-aware reward between generated and reference captions is

$t$9

Auxiliary quality metrics $\pi_\theta(y_t \mid y_{<t}, M),$0 and $\pi_\theta(y_t \mid y_{<t}, M),$1 are also computed, and the total reward is

$\pi_\theta(y_t \mid y_{<t}, M),$2

This reward design suggests that the policy is optimized for both emotional alignment and caption quality, rather than for lexical overlap alone. The paper states that this mechanism addresses the shortcomings of rigid rules in handling the dynamic and flexible nature of captions.

4. Emo-GRPO and the two-phase training procedure

Phase 1 is supervised fine-tuning (SFT) on EmotionTalk. Preprocessing extracts 16 kHz waveforms, segments utterances, normalizes loudness, decodes video frames at 25 fps, crops and resizes faces, and tokenizes ground-truth captions with the model’s byte-pair encoding. The SFT objective is the standard autoregressive negative log-likelihood

$\pi_\theta(y_t \mid y_{<t}, M),$3

Optimization uses AdamW with learning rate $\pi_\theta(y_t \mid y_{<t}, M),$4 and batch size $\pi_\theta(y_t \mid y_{<t}, M),$5 with grad-accum $\pi_\theta(y_t \mid y_{<t}, M),$6, and LoRA adapters of rank $\pi_\theta(y_t \mid y_{<t}, M),$7 are used in the decoder layers (Sun et al., 23 Sep 2025).

Phase 2 applies Group Relative Policy Optimization with emotion-aware reward, termed Emo-GRPO. The SFT checkpoint is frozen as $\pi_\theta(y_t \mid y_{<t}, M),$8, and reinforcement learning proceeds with $\pi_\theta(y_t \mid y_{<t}, M),$9 rollouts per example. For sampled captions $M = \{\text{audio, video, text embeddings}\}$0 with rewards $M = \{\text{audio, video, text embeddings}\}$1, the group baseline is

$M = \{\text{audio, video, text embeddings}\}$2

and the gradient estimate is

$M = \{\text{audio, video, text embeddings}\}$3

An optional KL penalty with respect to the pre-update policy can be added to stabilize learning. The optimizer and learning rate remain as in SFT; grad-accum is increased to $M = \{\text{audio, video, text embeddings}\}$4 and a linear LR warm-up of $M = \{\text{audio, video, text embeddings}\}$5 of total steps is used. Additional hyperparameters reported in Table 1 are KL coefficient $M = \{\text{audio, video, text embeddings}\}$6, maximum generation length $M = \{\text{audio, video, text embeddings}\}$7 tokens, and temperature $M = \{\text{audio, video, text embeddings}\}$8. A plausible reading of this setup is that the method uses relative within-group advantage estimation to reduce variance while preserving direct sequence-level optimization.

5. Empirical results, ablations, and qualitative behavior

The evaluation uses BLEU-1 through BLEU-4, ROUGE_L, METEOR, SPICE (termed SPIDER), and unique vocabulary size (“Vocab”). On the EmotionTalk test set, MECap-R1 (full) reports BLEU$M = \{\text{audio, video, text embeddings}\}$9, ROUGE_L $Y$0, METEOR $Y$1, SPIDER $Y$2, and Vocab $Y$3. The listed baselines are BART, GPT-2, Qwen-2, and zero-shot Qwen2.5-Omni* (Sun et al., 23 Sep 2025).

Model	Key scores	Vocab
BART	BLEU$Y$4 1.8, ROUGE_L 46.9, METEOR 23.3, SPIDER 23.0	–
GPT-2	BLEU$Y$5 1.5, ROUGE_L 46.2, METEOR 21.4, SPIDER 22.7	–
Qwen-2	BLEU$Y$6 3.3, ROUGE_L 53.5, METEOR 26.8, SPIDER 12.1	–
Qwen2.5-Omni*	BLEU$Y$7 0.0, ROUGE_L 36.1, METEOR 13.1, SPIDER 10.8	100
MECap-R1 (full)	BLEU$Y$8 7.2, ROUGE_L 54.7, METEOR 29.3, SPIDER 12.8	229

The ablation results clarify the role of each component. Without SFT, BLEU$Y$9, indicating that reinforcement learning is not a substitute for the supervised initialization. Without emo-GRPO, BLEU$T$0 drops from $T$1 to $T$2 and Vocab decreases from $T$3 to $T$4, which supports the claim that the RL phase contributes to both accuracy and diversity. Without the emotion-aware reward, the paper reports a slight metric drop and poorer emotion overlap in human/GPT-4 scoring. Without video, BLEU$T$5, which the paper interprets as confirmation of multimodal gains.

The qualitative examples reinforce that interpretation. For one utterance, the version without emotion-aware reward produces “The man’s voice is loud and clear… with heightened excitement.” with GPT-4 emo-score $T$6, whereas MECap-R1 generates “The tone rises, speed is quick, emotion is agitated and forceful.” with GPT-4 emo-score $T$7. For another utterance, the version without emotion-aware reward gives “The man’s voice is bright… with upward inflection indicating urgency.” with score $T$8, while MECap-R1 outputs “The voice is calm and natural, pace moderate, showing relaxed emotion.” with score $T$9. The paper uses these cases to argue that emotion-aware RL captures fine-grained affect more faithfully.

6. Strengths, limitations, and prospective extensions

The paper identifies three principal strengths. It states that MECap-R1 is the first to integrate video cues into speech emotion captioning; that Emo-GRPO’s anchor-based reward aligns generated captions with human emotional semantics; and that the model yields substantial gains in both accuracy, measured with BLEU and METEOR, and diversity, measured with vocabulary size (Sun et al., 23 Sep 2025).

The limitations are also explicit. The RL stage is computationally expensive because training uses a small batch and multiple rollouts. The method relies on handcrafted emotion lexicons $$\pi_\theta(Y \mid M)=\prod_{t=1}^T\pi_\theta(y_t\mid y_{<t},M).$$0, which may not generalize to new emotions or languages. In addition, the SPICE/SPIDER score remains modest, indicating room for improvement in propositional content. These caveats constrain how the reported improvements should be interpreted: a plausible misconception is that MECap-R1 is only an emotion-conditioning mechanism, whereas the modest SPICE/SPIDER value indicates that affective alignment and proposition-level caption quality are not identical objectives.

The future directions proposed in the paper are automatic discovery of emotion anchors, for example via clustering rather than manual lexica; scaling to larger batch sizes or off-policy RL to reduce compute; extension to cross-lingual or low-resource settings, including few-shot anchor transfer; incorporation of physiological signals such as heart rate; and user studies correlating automated metrics such as $$\pi_\theta(Y \mid M)=\prod_{t=1}^T\pi_\theta(y_t\mid y_{<t},M).$$1 and BLEU with human judgments of empathy and naturalness. Taken together, these directions suggest an agenda in which emotion captioning is treated as a broader multimodal affect-generation problem, with anchor construction, reward design, and modality integration as the main axes of future development.