IERP Challenge 2024: Emotion Recognition Beyond Labels

• The paper introduces a conceptual shift from discrete emotion classification to predicting eight emotion intensity scores that incorporate personality traits.
• The study leverages a fine-tuned DWFormer model with multiple pre-trained speech features, employing data augmentation and score fusion to enhance prediction accuracy.
• The challenge demonstrates the effectiveness of adapting unimodal systems while highlighting future potential for multimodal emotion recognition research.

IERP Challenge 2024 was a competition in emotion recognition that explicitly incorporated personality traits into the research problem, motivated by the premise that human personality is directly related to emotion and that subjective emotional expression varies substantially across individuals. In the documented Track 1 formulation, the task moved beyond conventional discrete emotion classification and instead required prediction of intensity scores for multiple emotional states from provided acoustic and textual features. The public description available in the literature centers on Track 1, where participants predicted eight emotion scores on a $1$–$5$ scale and were evaluated by root mean squared error (RMSE), with lower values indicating better performance (Wang et al., 15 Aug 2025).

1. Research framing and conceptual shift

The challenge was designed to move beyond the dominant orientation of extant emotion recognition models toward improving the precision of discrete emotion label prediction. Its central motivation was that emotional expression is strongly influenced by human personality and varies substantially across speakers. On that basis, the challenge reframed emotion recognition as a score prediction problem over multiple emotional states rather than a single-label classification problem (Wang et al., 15 Aug 2025).

This reframing has methodological consequences. Instead of deciding which single category is present, systems must estimate graded intensity values for several emotions simultaneously. This suggests a closer alignment with continuous or ordinal affect modeling than with conventional closed-set categorical speech emotion recognition, although the documented task still uses a fixed inventory of emotion categories. A common simplification is to describe the challenge as ordinary eight-way classification; the available specification does not support that characterization, because the target is an eight-dimensional score vector rather than one discrete label.

2. Track 1 task specification

Track 1 primarily concerned the recognition of emotions in audio, while also providing text and audio features. The organizers supplied acoustic and textual features, with speech features extracted from large pre-trained self-supervised models, and asked participants to predict intensity scores for eight emotion categories: sadness, happiness, relaxation, surprise, anger, fear, disgust, and neutrality. Each emotion was rated on a $1$–$5$ scale, where $1$ means no emotion and $5$ indicates the strongest emotion (Wang et al., 15 Aug 2025).

The official dataset for the reported Track 1 experiments comprised $2{,}608$ audio features for training and validation and $480$ audio features for testing. These samples came from $163$ subjects who watched $16$ emotion-inducing videos. The documented public account therefore situates the challenge at the intersection of speech-based affect estimation, personality-sensitive modeling, and small-data adaptation.

Component	Track 1 specification	Value
Emotion categories	sadness, happiness, relaxation, surprise, anger, fear, disgust, neutrality	8
Rating scale	no emotion to strongest emotion	1–5
Training/validation set	audio features	2,608
Test set	audio features	480
Subjects	participants in the dataset	163
Stimuli	emotion-inducing videos	16

An important clarification is that Track 1 was not defined as an audio-only benchmark in the narrow sense. Textual features were available to participants, even though at least one leading submission elected not to use them. This distinction matters when interpreting published results.

3. Evaluation protocol and optimization target

The official evaluation metric was RMSE over the eight predicted emotion scores, so lower values correspond to better performance. This choice operationalized the challenge as a multi-output regression problem rather than a label-classification benchmark. In the reported winning submission, training used a mean squared error objective of the form

$5$0

which is conceptually consistent with the challenge’s score-prediction framing (Wang et al., 15 Aug 2025).

The use of RMSE changes both model selection and interpretation. Systems are rewarded for calibrating the magnitude of each emotion score, not merely for ranking or selecting categories. This suggests that threshold-free regression behavior is more central to success than class-boundary sharpness. It also means that errors distributed across several emotions can be consequential even when the dominant emotion is qualitatively plausible.

4. Documented first-place Track 1 system

The published paper on the challenge reports Fosafer’s first-place submission to Track 1. That system used exclusively audio-based features and fine-tuned a pre-trained speech emotion recognition model, DWFormer, through the integration of data augmentation and score fusion strategies. DWFormer is a dynamic window transformer architecture originally pretrained for five-way emotion recognition; it uses a Vanilla Transformer encoder, an Importance Calculation (IC) module, and stacked dynamic local and global window transformer blocks to capture temporal emotion cues at multiple scales (Wang et al., 15 Aug 2025).

For the challenge, the model was adapted to output eight emotion scores instead of five. Only the final fully connected classification layer was trained, while the rest of the network was frozen. This was a consequential design choice in the reported system: rather than training a large model from scratch on a relatively small official dataset, the submission leveraged emotional representations already learned by DWFormer and specialized only the task-specific output layer.

The system used three audio feature types derived from large pre-trained speech representations: WavLM-Large, Chinese-HuBERT-Large, and Chinese-Wav2vec2-Large. These were the sole input modality, despite the availability of text features in the track. The authors explicitly stated that they did not exploit the textual side of Track 1, focusing instead on the speech channel.

5. Augmentation, fusion, and ablation structure

Because the official training set was relatively small and DWFormer has many parameters, the reported system introduced data augmentation using noise from the MUSAN corpus. The augmentation procedure was applied directly to the precomputed speech features. Since the features are two-dimensional, the original feature length was defined as the product of the two leading dimensions. A random noise segment was sampled from MUSAN; if the noise was longer than the target feature sequence, it was truncated, and if it was shorter, it was repeated until the lengths matched. Noise injection was probabilistic: a random value in $5$1 was sampled, and if the preset probability exceeded this value, noise was added. The tested noise-addition probabilities were $5$2, $5$3, and $5$4 (Wang et al., 15 Aug 2025).

A second major component was score fusion. Predictions were fused in two settings: across models trained on different audio features, and across models trained on the same feature but with different training parameter configurations. Three fusion rules were evaluated. The first was simple averaging,

$5$5

The second was weighted averaging,

$5$6

where the weights were chosen according to validation RMSE, with larger weights assigned to better-performing models. In the final feature-based fusion, the weights were $5$7 for Chinese-Wav2vec2-Large, $5$8 for Chinese-HuBERT-Large, and $5$9 for WavLM-Large. The third rule was elementwise maximum selection, where the final score was taken as the maximum among model outputs.

The ablation structure reported a clear progression. Without augmentation, test RMSE values were $1$0 for WavLM-Large, $1$1 for Chinese-HuBERT-Large, and $1$2 for Chinese-Wav2vec2-Large, making Chinese-Wav2vec2-Large the strongest single-feature model. After MUSAN-based noise augmentation, performance improved for all three features at probabilities $1$3 and $1$4, while probability $1$5 degraded results. The best single-feature augmented result was again obtained with Chinese-Wav2vec2-Large, reaching RMSE $1$6 at noise probability $1$7. This suggests that mild augmentation improved generalization, while excessive noise distorted the speech features too strongly.

6. Results, interpretation, and limitations

Score fusion produced additional gains beyond the best single model. For multi-feature fusion, the reported RMSE values were $1$8 for average, $1$9 for weighted average, and $5$0 for max. For fusion of models trained with the same Chinese-Wav2vec2-Large feature but different parameter settings, the results were stronger: $5$1 for average, $5$2 for weighted average, and $5$3 for max. Because of submission limits, the official leaderboard entry used the average-fusion result of $5$4, which secured first place. The authors noted that the internal best system was slightly better, at $5$5, achieved by max fusion, but the submitted result remained the champion entry (Wang et al., 15 Aug 2025).

These results support several technical conclusions documented in the paper. First, a strong pretrained speech emotion recognition backbone could be adapted minimally to an eight-score regression setting. Second, mild feature-level noise augmentation improved robustness. Third, fusion across multiple model variants was more effective than reliance on a single model. The best-performing configuration combined Chinese-Wav2vec2-Large with mild augmentation and test-time fusion.

The same report also identifies limitations. Most importantly, the system did not use the textual features provided by Track 1, leaving multimodal information untapped. It also did not explore pretraining an eight-class speech emotion recognition model on open-source emotion datasets and then fine-tuning on the challenge data, nor converting external sentiment data into challenge-like labels for data expansion. The authors argued that future work should focus on multimodal emotion recognition, since multimodal systems generally outperform unimodal ones, and suggested that additional data sources and label transformation strategies may further improve performance. A plausible implication is that the challenge, while already departing from single-label SER, also exposed a gap between available multimodal inputs and the strongest reported unimodal submission.