2009 INTERSPEECH Emotion Challenge

• The paper established the FAU-AIBO corpus as a benchmark for evaluating spontaneous, noisy child speech emotion recognition.
• It defined two classification tasks with strict train/test splits to ensure speaker independence and realistic domain shifts.
• Recent analyses reveal that modern architectures achieve only marginal improvements over the original baseline, highlighting enduring challenges.

Searching arXiv for the specified paper to ground the article and citation. The 2009 INTERSPEECH Emotion Challenge was the first open benchmark in speech emotion recognition (SER), established to address the lack of a standardised evaluation protocol for spontaneous, “in-the-wild” speech and the absence of comparable baselines for future work. It centered on the FAU-AIBO corpus, a German children’s speech dataset recorded in a Wizard-of-Oz interaction with Sony’s AIBO robot, and defined two supervised classification tasks under a fixed train/test partition. A 2024 re-examination of the benchmark assessed whether contemporary deep learning methods substantially improve on the original challenge systems and found that FAU-AIBO remains a difficult benchmark: most modern architectures perform below or close to the official baseline, and only marginal gains over the original winners are observed after hyperparameter tuning (Triantafyllopoulos et al., 2024).

1. Origin, benchmark design, and corpus characteristics

The challenge was organised around FAU-AIBO, which consists of 18 216 speech “chunks” produced by ten German-speaking children aged approximately 5–8 years. The recordings were collected in a playful Wizard-of-Oz game in which Sony’s AIBO robot served as the interlocutor. Each word was annotated by five raters for one of eleven communicative or emotional states, after which the word-level annotations were aggregated into semantically and prosodically coherent chunks at the turn or phrase level (Triantafyllopoulos et al., 2024).

The corpus is characterised by spontaneous speech, noisy recording conditions including classroom and home background sounds, and high variability in children’s voices. These properties are explicitly identified as factors that make the benchmark particularly challenging. The challenge’s historical role was therefore not merely to provide a dataset, but to instantiate a controlled evaluation setting for spontaneous affective speech under realistic acoustic and interactional conditions.

A plausible implication is that the benchmark occupies a distinct position relative to cleaner acted-emotion corpora: its difficulty arises from the combination of spontaneous elicitation, environmental noise, child speech, and label aggregation rather than from class cardinality alone.

2. Label structure and task formulation

Two classification tasks were defined. The first is a coarse 2-class problem distinguishing “negative” from “non-negative.” “Negative” covers chunks expressing frustration, boredom, or other adverse affect, whereas “non-negative” covers neutral or positive states. The second is a finer 5-class problem with the labels Angry (A), Neutral (N), Motherese/Joyful (P), Emphatic (E), and Rest (R) (Triantafyllopoulos et al., 2024).

These labels were derived heuristically from the original eleven states. The data block gives the example that “bored” and “frustrated” were mapped to Angry. The resulting label set is severely imbalanced: Neutral and non-negative dominate more than 50% of the data, while Angry and Emphatic may each be below 10% (Triantafyllopoulos et al., 2024).

This imbalance is central to the benchmark’s evaluation logic. Because dominant classes would distort ordinary accuracy, the challenge uses Unweighted Average Recall (UAR), which gives equal weight to per-class recall. In the revisited study, UAR over $C$ classes is defined as

$UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$

where $TP_c$ is the number of true positives for class $c$ and $N_c$ its total occurrences (Triantafyllopoulos et al., 2024).

The revisited models are trained with a class-weighted categorical cross-entropy,

$$L = - \sum_{i=1}^N \sum_{c=1}^C w_c y_{ic} \log p_{ic},$$

where $y_{ic}\in\{0,1\}$ is the one-hot ground truth, $p_{ic}$ the predicted probability, and $w_c = 1/N_c$ the inverse-frequency weight (Triantafyllopoulos et al., 2024). This design directly reflects the imbalance profile of the corpus.

3. Official evaluation protocol

The official train/test split is school-based. All utterances from speakers recorded at the “Ohm” school, with utterance count approximately 12 000, form the training set, while all utterances from the “Mont” school, approximately 6 000, form the test set. This guarantees full speaker independence and simulates a real domain shift (Triantafyllopoulos et al., 2024).

In the revisited study, a separate validation split was created by carving out two speakers, Ohm_31 and Ohm_32, corresponding to approximately 700 utterances, from the official training set. This validation subset was used only for early stopping; no results were reported on validation, and all performance figures were computed on the sealed official test set (Triantafyllopoulos et al., 2024).

Preprocessing includes audio resampling, typically to 16 kHz, silence removal as per openSMILE defaults, and feature extraction with the openSMILE/openEAR toolkit. For spectrogram-based models, 64 Mel bands with a 32 ms window and 10 ms hop were used. Three-channel “RGB” inputs were obtained by replicating the single-channel Mel-spectrogram (Triantafyllopoulos et al., 2024).

The protocol is methodologically stringent. Strict adherence to the official split preserves speaker independence and prevents leakage across schools. The held-out validation mechanism serves only optimisation control, not model selection by test-set feedback. The 2024 study explicitly recommends reporting UAR on the sealed test set and using only a small held-out validation set for early stopping (Triantafyllopoulos et al., 2024).

4. Baselines, winners, and evaluation targets

The official baseline compared HMMs on dynamic low-level descriptors (LLDs) treated as sequences with SVMs on static features constructed as functionals over LLDs. Both dynamic-feature HMMs and static-feature SVMs applied SMOTE oversampling to mitigate class imbalance (Triantafyllopoulos et al., 2024).

The static representation comprised 16 LLDs plus 16 deltas with 19 functionals, producing 608-dimensional vectors. This static-feature setup yielded the stronger baseline:

System/result	2-class UAR	5-class UAR
Official static-feature baseline	0.677	0.382
“Classifier performance” winner	0.703	0.417
Late fusion of top 2009 systems	0.712	0.440

Two sub-challenges produced winners. Dumouchel et al. (2009) used GMMs on cepstral plus expert prosodic and vocal-tract features to win the “open performance” track. Lee et al. (2011) used a hierarchical cascade of binary SVMs on the official static features to win the “classifier performance” track, achieving UAR $= 0.703$ for the 2-class task and UAR $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$0 for the 5-class task. A late fusion of the top 2009 systems reached UAR $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$1 (Triantafyllopoulos et al., 2024).

These numbers remain the principal historical reference points for later work on FAU-AIBO. The revisited benchmark is framed explicitly against them rather than against arbitrary contemporary SER datasets. This suggests that the challenge’s enduring value lies in stable comparative evaluation rather than in absolute performance magnitude.

5. Fifteen years of model development in the revisited benchmark

The revisited benchmark evaluated 43 models spanning four development eras (Triantafyllopoulos et al., 2024).

The first group, covering 2009–2016, includes openSMILE static feature sets and LLD-sequence feature sets IS09–IS13, IS16, plus eGeMAPS. Static features were paired with a 3-layer MLP with 64 hidden units, ReLU, and dropout 0.5. Dynamic LLDs were processed with a 2-layer LSTM with 32 units, followed by mean pooling, a 32-unit linear layer, and the output layer (Triantafyllopoulos et al., 2024).

The second group, covering 2012–2023 and associated with ImageNet pre-training, includes AlexNet, ResNet50, VGG11/13/16/19, EfficientNet-B0, ConvNeXt (t/s/b/l), and Swin Transformer (t/s/b), all fine-tuned on Mel-spectrogram “images” (Triantafyllopoulos et al., 2024).

The third group, covering 2016–2023 and associated with end-to-end learning and supervised audio pre-training, includes CRNN^18, CRNN^19, ETDNN, CNN14, AST, and three Whisper variants (tiny/base/small). These operate on raw waveform, log-Mel plus statistics, or other audio-native inputs depending on architecture (Triantafyllopoulos et al., 2024).

The fourth group, covering 2020–2023 and associated with self-supervised audio pre-training, includes wav2vec 2.0 (base/large), HuBERT (base/large), multilingual wav2vec 2.0, robust-wav2vec 2.0, and a pruned wav2vec 2.0 fine-tuned for dimensional SER, denoted W2V2^e. These models feed last-layer hidden representations into a 2-layer MLP head (Triantafyllopoulos et al., 2024).

Inputs were aligned to architecture type: raw feature vectors for static and dynamic LLD models, 64-band Mel-spectrograms with 32 ms window and 10 ms hop for spectrogram/CNN models, and 16 kHz waveform for raw-audio models. Training proceeded in two phases. The exploration phase used Adam with learning rate $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$2, batch size 4, and 30 epochs. The tuning phase used a grid over optimiser $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$3Adam, SGD$UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$4, learning rate $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$5, batch size $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$6, and 50 epochs. Early stopping used the 698-utterance validation split. All models except frozen-backbone SSL models were fine-tuned end-to-end. During training, utterances were randomly cropped or padded to 3 s with fixed seed; during inference, only padding to 2 s if shorter was applied (Triantafyllopoulos et al., 2024).

The breadth of the benchmark is important. It does not compare a single modern architecture against a legacy baseline; rather, it samples the main methodological trajectories in SER, including handcrafted feature pipelines, image-transfer paradigms, end-to-end audio models, and SSL-derived representations.

6. Revisited results and technical interpretation

In the exploration phase with fixed hyperparameters, the best 2-class model was CNN14 with UAR $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$7, and the best 5-class model was ResNet50 with UAR $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$8 (Triantafyllopoulos et al., 2024). Most architectures, including large transformers and end-to-end CRNNs, achieved UAR in the 0.50–0.68 range, often below the 2009 winners and around the original baseline. A late fusion of the top-5 exploration models reached $UAR = (1/C) \sum_{c=1}^C (TP_c / N_c)$9, which was close to the original fusion result of $TP_c$0 (Triantafyllopoulos et al., 2024).

After hyperparameter tuning, the top official-test-set results were as follows:

Model	2-class UAR	5-class UAR
IS10^d (LLD-LSTM)	0.685	0.394
ResNet50	0.690	0.423
CNN14	0.672	0.448
W2V2^e	0.717	0.448
Whisper^t	0.707	0.454

The associated 95% bootstrap confidence intervals were reported as follows: IS10^d, $TP_c$1–$TP_c$2 and $TP_c$3–$TP_c$4; ResNet50, $TP_c$5–$TP_c$6 and $TP_c$7–$TP_c$8; CNN14, $TP_c$9–$c$0 and $c$1–$c$2; W2V2^e, $c$3–$c$4 and $c$5–$c$6; Whisper^t, $c$7–$c$8 and $c$9–$N_c$0 (Triantafyllopoulos et al., 2024).

Under this tuned setting, W2V2^e marginally surpasses the best 2009 winner on the 2-class task, with $N_c$1, and Whisper^t slightly exceeds the best 2009 winner on the 5-class task, with $N_c$2. The revisited study states that these gains are statistically significant because the 2009 baselines, $N_c$3, lie outside the lower bounds of the tuned models’ confidence intervals (Triantafyllopoulos et al., 2024).

The broader interpretation offered by the benchmark is not that modern SER has solved FAU-AIBO. On the contrary, most models remain below or close to the official baseline, and improvements over the original winners are marginal. This directly supports the paper’s conclusion that progress toward “solving” SER is not necessarily monotonic (Triantafyllopoulos et al., 2024).

7. Difficulty profile, model complementarity, and reproducibility

The revisited analysis identifies several reasons why FAU-AIBO remains difficult: spontaneous child speech recorded in noisy homes and classrooms, domain shift between the two schools, a complex elicitation style mixing game, instruction, and complaint, label uncertainty induced by collapsing eleven annotator states into two or five classes, and strong class imbalance in which neutral and non-negative dominate (Triantafyllopoulos et al., 2024).

The study further reports no correlation between model publication year or size, measured via MACs and parameters, and UAR, with Spearman’s $N_c$4 (Triantafyllopoulos et al., 2024). This result directly counters a common assumption that newer or larger models should systematically dominate older or smaller ones on this benchmark. Inter-model agreement on test predictions is only 70% for the 2-class task and 55% for the 5-class task, rising to 80% and 57% for the top-5 models. The paper interprets this as evidence that different architectures make complementary errors (Triantafyllopoulos et al., 2024).

Instance “difficulty,” defined as $N_c$5 minus annotator confidence, correlates only weakly with model disagreement, with $N_c$6, or with per-instance cross-entropy loss, with average $N_c$7 (Triantafyllopoulos et al., 2024). This suggests that annotation ambiguity alone does not fully explain where models fail.

The practical implications stated in the revisited work are precise: adhere strictly to the official train/test splits and speaker independence; report UAR on the sealed test set while using a small held-out validation set solely for early stopping; include the original static and dynamic openSMILE baselines in comparisons; and explore model ensembles or fusions, which remain strong (Triantafyllopoulos et al., 2024). Future work is recommended to focus on robust learning under domain shift, class imbalance, and label noise, and to evaluate across multiple, well-specified benchmarks (Triantafyllopoulos et al., 2024).

Reproducibility is supported by the release of code, model checkpoints, and hyperparameter specifications at the repository listed in the source paper, with fixed random seeds for 3 s padding/cropping and model initialisation. The FAU-AIBO corpus remains publicly accessible under the original INTERSPEECH 2009 Emotion Challenge licence, and the exact speaker-wise train/test partitions are documented in Steidl (2009) (Triantafyllopoulos et al., 2024). In that sense, the challenge continues to function as a durable benchmark for testing whether methodological advances in SER generalise to spontaneous, noisy, child-speech settings rather than merely to easier experimental conditions.