AffectGPT-R1: RL for Open-Vocab Emotion Recog

• AffectGPT-R1 is a reinforcement learning extension in the AffectGPT lineage that shifts from token-level imitation to reward-driven emotion recognition using multimodal inputs.
• It employs a two-stage training pipeline—supervised cold-start followed by reinforcement learning—to align model outputs with emotion-wheel metrics.
• The framework improves OV-MER benchmark scores by directly optimizing semantic emotion predictions, offering insights into reward design and structured output formatting.

Searching arXiv for the cited AffectGPT papers and related reinforcement-learning OV-MER work. AffectGPT-R1 is a reinforcement learning framework for open-vocabulary multimodal emotion recognition (OV-MER). It was introduced as a later extension of the AffectGPT line, with the specific aim of aligning training with emotion-wheel-based evaluation rather than relying only on token-level supervision. In this formulation, a multimodal LLM receives video-centered affective evidence and generates natural-language emotion outputs, while reinforcement learning uses emotion-wheel metrics as rewards to optimize semantically meaningful emotion prediction in the open-vocabulary regime (Lian, 2 Aug 2025).

1. Naming, lineage, and scope

The name belongs to a broader AffectGPT lineage, but its meaning changed across successive papers. In 2024, “AffectGPT” denoted a two-stage framework for Explainable Multimodal Emotion Recognition (EMER), built around EMER-Coarse, EMER-Fine, and a Video-LLaMA-based architecture trained to generate emotion-related descriptions from audio, video, and subtitles (Lian et al., 2024). In early 2025, “AffectGPT” referred to a different but related framework centered on the MER-Caption dataset, the AffectGPT multimodal model, and the MER-UniBench benchmark for multimodal emotion understanding with natural-language outputs (Lian et al., 27 Jan 2025).

The foundational 2025 AffectGPT paper explicitly stated that it did not use the term “AffectGPT-R1” anywhere, and described only AffectGPT, MER-Caption, MER-Caption+, and MER-UniBench (Lian et al., 27 Jan 2025). The formal designation “AffectGPT-R1” appears later, in the paper titled “AffectGPT-R1: Leveraging Reinforcement Learning for Open-Vocabulary Emotion Recognition,” where it denotes a reinforcement-learning extension of the AffectGPT framework for OV-MER (Lian, 2 Aug 2025).

Within this lineage, AffectGPT-R1 is best understood not as an unrelated architecture, but as an alignment layer over the AffectGPT foundation. It preserves the multimodal large-language-model orientation of AffectGPT while changing the optimization target from token-level imitation to reward-driven optimization of emotion-wheel similarity.

2. Problem formulation and motivation

AffectGPT-R1 is defined for Open-Vocabulary Multimodal Emotion Recognition, a setting in which predicted emotions are not restricted to a predefined label set. This differs from conventional multimodal emotion recognition, where models are trained as discriminative classifiers over small closed taxonomies such as six or seven basic emotions. OV-MER instead allows free-form emotion words or phrases, which makes generative multimodal LLMs a natural fit (Lian, 2 Aug 2025).

The immediate motivation is objective mismatch. AffectGPT, like other generative OV-MER systems, was trained with standard token-level loss on descriptive emotion data. However, OV-MER evaluation is based on emotion wheels and related set-level matching procedures rather than exact textual overlap. The AffectGPT paper had already formalized fine-grained emotion recognition through open-vocabulary prediction on OV-MERD+, with synonym normalization and emotion-wheel grouping, and used the set-level score $F_s$ as the primary metric for fine-grained recognition (Lian et al., 27 Jan 2025). AffectGPT-R1 argues that token-level cross-entropy is only an indirect proxy for these evaluation criteria and cannot directly optimize them because emotion-wheel metrics depend on discrete language outputs and non-differentiable matching procedures (Lian, 2 Aug 2025).

This design responds to a broader pattern in affective computing with LLMs. Earlier work had shown that GPT-style systems could exhibit strong zero-shot affective processing, including Valence–Arousal–Dominance prediction, latent mapping between situations and emotion words, and basic OCC-style appraisal reasoning (Broekens et al., 2023). A wider benchmark study also found that GPT-4 performed strongly on many sentiment-, emotion-, and toxicity-related tasks, but fell short on problems with implicit signals such as engagement measurement and personality assessment (Amin et al., 2023). AffectGPT-R1 can be read as a multimodal and reinforcement-learning response to those limitations: rather than treating affect as ordinary text generation, it optimizes directly for the semantic structure used in OV-MER evaluation.

3. Base architecture and two-stage training pipeline

AffectGPT-R1 uses the same architecture as AffectGPT for its cold-start stage. In the AffectGPT implementation, the visual encoder is frozen CLIP ViT-L, the acoustic encoder is frozen HuBERT-L, and the language backbone is Qwen2.5, frozen except for a LoRA module; trainable components include the LoRA parameters, the audio and video projectors, and the pre-fusion module for audio–video integration (Lian et al., 27 Jan 2025). AffectGPT’s central architectural idea is pre-fusion: cross-modal fusion is performed before projection into LLM token space, rather than being left entirely to the LLM. Two variants were proposed—Q-Former-based pre-fusion and a simpler attention-based pre-fusion—and both improved over the no-pre-fusion baseline, with the attention-based version slightly outperforming Q-Former in the reported setting (Lian et al., 27 Jan 2025).

AffectGPT-R1 then adds a two-stage post-training pipeline. The first stage is a supervised “cold-start” stage on MER-Caption+, where the model learns both multimodal emotion understanding and a structured output format. The second stage is reinforcement learning on MER2025-OV, where labels are available but emotion descriptions are not, and the model is optimized against emotion-wheel-based rewards (Lian, 2 Aug 2025).

Stage	Dataset	Role
Cold start	MER-Caption+	31,327 samples; emotion descriptions and labels
RL fine-tuning	MER2025-OV	1,000 non-overlapping samples; emotion labels only
Evaluation	OV-MERD+	532-sample OV-MER benchmark

The cold-start stage teaches the model to emit two explicitly separated regions:

<think> ... </think> <answer> ... </answer>

In the original formulation, the > segment contains explanatory reasoning or emotion-related description, and the <answer> segment contains the final open-vocabulary emotion label or phrase (Lian, 2 Aug 2025). This format is not merely cosmetic. It provides a parsing boundary for reward computation and separates descriptive reasoning from the final label string used in evaluation.

4. Reinforcement learning, GRPO, and reward design

The reinforcement-learning stage treats OV-MER as a sequence-generation problem with non-differentiable rewards. Given multimodal input and a prompt, the model samples candidate outputs; each output is scored by a reward function that combines format compliance with emotion-wheel accuracy. In the original AffectGPT-R1 paper, the combined reward is

$$R(o, y \mid v, q) = R_{\text{accuracy}}(o, y \mid v, q) + \beta\, R_{\text{format}}(o \mid v, q),$$

where the accuracy term is the official emotion-wheel metric computed from the extracted <answer> content, and the format term checks whether the output strictly follows the required <think><answer></answer> structure (Lian, 2 Aug 2025).

Optimization uses Group Relative Policy Optimization (GRPO), a PPO-style method that removes the critic and computes relative advantages within a sampled group of outputs. For a group of rewards $\{r_1,\dots,r_G\}$, the group-normalized advantage is

$$A_i = \frac{r_i - \mathrm{mean}(r_1,\dots,r_G)}{\mathrm{std}(r_1,\dots,r_G)}.$$

This replacement of critic-based advantages by group-relative normalization is intended to reduce memory overhead while preserving stable policy updates (Lian, 2 Aug 2025).

A later analytical extension, “AffectGPT-RL,” substantially deepened the reward analysis. It introduced additional rewards beyond accuracy and format: an alignment reward for consistency between reasoning and answer, a dual reward comparing extracted reasoning emotions directly against ground-truth labels, and a perception reward based on pairwise judging of reasoning quality. It also documented reward hacking: because emotion-wheel evaluation is set-based and tolerant to synonym expansion, the model can improve reward by outputting overly long lists of emotion words. To counter this, the paper proposed length penalties $P_1$, $P_2$, and $P_3$, and reported that $P_2$ and $P_3$ suppressed overlong outputs without harming performance (Lian et al., 7 May 2026).

One of the most consequential later findings concerns explicit reasoning. Although the original AffectGPT-R1 formulation emphasized <think> and <answer>, AffectGPT-RL reported that RL without a thinking segment outperformed RL with a thinking segment on OV-MERD+, and interpreted this as evidence that noisy or weak reasoning traces can distract optimization away from the core semantic target (Lian et al., 7 May 2026). This does not invalidate the original format, but it reframes the role of explicit reasoning in affective generation: chain-of-thought-style structure is useful only when the reasoning supervision itself is reliable.

5. Empirical performance and benchmark position

The baseline AffectGPT system already reported strong multimodal emotion-understanding performance. On MER-UniBench, which aggregates basic emotion recognition, sentiment analysis, and fine-grained emotion recognition from generated text, AffectGPT achieved a mean score of 74.77 across all tasks and reported an OV-MERD+ fine-grained score of 62.52 (Lian et al., 27 Jan 2025).

AffectGPT-R1 improved the OV-MERD+ score to 66.35, compared with 62.52 for AffectGPT, an absolute gain of 3.83 points on that benchmark (Lian, 2 Aug 2025). The paper interprets this as evidence that directly optimizing emotion-wheel metrics via reinforcement learning is more effective than optimizing token-level likelihood alone when the evaluation target is semantic rather than lexical.

Subsequent work in AffectGPT-RL further analyzed where these gains come from. It reported that accuracy reward and dual reward were the most effective single-reward choices, that combining accuracy with format could improve over accuracy alone, and that combining too many rewards led to reward interference. It also examined generalization beyond OV-MER and reported gains on basic emotion recognition tasks, while noting that optimization for OV-MER and basic emotion prediction could slightly reduce performance on some sentiment-analysis datasets (Lian et al., 7 May 2026).

These results sharpen the distinction between the three major stages in the AffectGPT line. The 2024 EMER-oriented AffectGPT showed that large weakly supervised descriptive datasets could make explainable multimodal emotion recognition feasible at scale (Lian et al., 2024). The 2025 AffectGPT system supplied a stronger multimodal foundation with MER-Caption, MER-Caption+, pre-fusion multimodal integration, and MER-UniBench (Lian et al., 27 Jan 2025). AffectGPT-R1 then reoriented training toward the benchmark’s own semantic geometry by optimizing emotion-wheel reward directly (Lian, 2 Aug 2025).

6. Limitations, interpretations, and broader significance

The original AffectGPT-R1 paper is explicit about several limitations. It reports limited hyperparameter tuning, application only to OV-MER, evaluation restricted to OV-MERD+, and no extensive ablations on reward components or RL configuration (Lian, 2 Aug 2025). It also notes broader concerns typical of affective computing systems, including possible cultural skew in emotion data and the risks inherent in inferring emotions from video and speech (Lian, 2 Aug 2025). The later AffectGPT-RL analysis adds a more technical limitation: not all reasoning-style additions help, and some reward terms—especially perception-style judging—can introduce noise rather than useful supervision (Lian et al., 7 May 2026).

The broader significance of AffectGPT-R1 lies in objective alignment. Earlier affective large-language-model studies had demonstrated that general models could perform meaningful affective processing under prompting alone, including continuous affect prediction and appraisal-style reasoning, but those studies did not solve the mismatch between open-vocabulary generation and semantic emotion evaluation (Broekens et al., 2023). AffectGPT-R1 addresses that mismatch by using reinforcement learning to optimize the same non-differentiable emotion-wheel structure that governs evaluation (Lian, 2 Aug 2025).

A plausible implication is that AffectGPT-R1 marks a transition from descriptive multimodal emotion understanding to metric-aligned multimodal emotion optimization. In that sense, it is less a standalone architecture than a specific training doctrine: start from a multimodal AffectGPT backbone, represent emotions in open vocabulary, and then use reinforcement learning to move the model toward the semantic topology encoded by emotion wheels. The later AffectGPT-RL study suggests that this doctrine extends beyond OV-MER to parts of MER-UniBench, but also that affective reasoning benefits only when rewards, reasoning traces, and output constraints are chosen with unusual care (Lian et al., 7 May 2026).