EmoNet-Voice: Fine-Grained Emotion Benchmark

• EmoNet-Voice is a comprehensive synthetic speech dataset offering nuanced emotion annotations across 40 categories and varied intensity levels.
• It utilizes advanced synthetic audio generation with linguistic and prosodic controls to ensure diversity and privacy-compliant data.
• Expert-driven annotation and rigorous regression-based benchmarking protocols enable precise evaluation of SER models with metrics like MAE and Pearson correlation.

EmoNet-Voice is a two-tier resource and benchmark for fine-grained speech emotion detection and recognition. Developed to address the deficits of existing datasets—which often suffer from limited emotional taxonomy, privacy constraints, or over-reliance on acted portrayals—EmoNet-Voice comprises a large-scale synthetic speech corpus with extensive expert-verified annotations. It is explicitly constructed to enable the evaluation and training of speech emotion recognition (SER) models across 40 emotion categories at three notable intensity levels, leveraging both linguistic and acoustic diversity. Its design incorporates synthetic audio generation technology and rigorous psychological validation, establishing a privacy-preserving yet emotionally comprehensive resource suitable for pre-training, benchmarking, and systematic model comparison.

1. Dataset Composition and Structure

EmoNet-Voice consists of two major components: EmoNet-Voice Big and @@@@2@@@@.

• EmoNet-Voice Big:
  • Over 4,500 hours of synthetic speech, generated at 24kHz.
  • 11 synthetic voices (6 female, 5 male) spanning 4 languages (English, German, Spanish, French).
  • Detailed breakdown by accent (e.g., British, Valley Girl, Texan) and voice.
  • 40 distinct emotions, including common states (anger, sadness) and compound/cognitive states (concentration, contemplation, bittersweet).
  • Wide coverage of prosodic and vocal variation.
• EmoNet-Voice Bench:
  • 12,600 unique audio clips (~36.26 hours, avg. 10.36s per clip).
  • Language distribution: ~49% English, ~15% German, ~17% Spanish, ~19% French.
  • Clips are annotated by psychology experts using a 3-level intensity scale: 0 (“not present”), 1 (“mildly present”), 2 (“intensely present”).
  • Ratings normalized to the 0–10 scale (0→0, 1→5, 2→10) for consistent assessment.

The following table summarizes key elements:

Component	Hours / Clips	Voices / Langs	Emotion Categories	Annotation Method
EmoNet-Voice Big	~4,500 h	11 / 4	40	Synthetic, prompt-driven
EmoNet-Voice Bench	~36 h / 12.6k	11 / 4	40	Expert, 3-level intensity (0-2)

The two-part structure allows for broad coverage in pre-training and precise evaluation in benchmarking.

2. Synthetic Audio Generation

All audio data within EmoNet-Voice is generated synthetically using OmniAudio technologies (e.g., GPT-4 OmniAudio). Speech snippets (3–30 seconds) are created as if performed by actors auditioning for a film role, with detailed prompted instructions for emotional nuance, naturalistic flow, and vocal burst control.

• Synthetic generation facilitates inclusion of sensitive emotions absent in natural datasets.
• Privacy requirements are satisfied, permitting open-access usage in model development.
• Prompts target specific emotions and intensities, with linguistic and prosodic controls (intonation, rhythm, tone, etc.).
• Audio is rendered across a diversity of voices and accents (see above), ensuring linguistically and demographically realistic distributions.

This design ensures the resource can be freely used for pre-training and benchmarking without compromising user privacy or ethical principles.

3. Annotation and Expert Verification

EmoNet-Voice Bench undergoes a stringent annotation protocol:

• Psychology experts (min. Bachelor’s degree) independently evaluate each clip, focusing on a single target emotion.
• Each clip receives at least two initial ratings; if both concur that an emotion is present (1 or 2), a third rating is solicited, and occasionally a fourth to mitigate subjectivity.
• The intensity scale (0,1,2) is normalized linearly to 0–10 for regression evaluation.
• Inter-rater reliability (Cronbach’s α ≈ 0.14) is low; this is interpreted as reflecting nuanced, ambiguous emotion perception rather than annotation error.
• The procedure ensures nuanced identification—even for low-arousal or compound states—and robustly documents perceptual ambiguity.

This annotation protocol supports both high-confidence benchmarking for easily agreed emotions (e.g., anger, embarrassment) and provides clear context for challenging categories (e.g., contemplation).

4. Benchmarking and Evaluation Protocols

The SER benchmarking protocol employs regression-based and correlation metrics on the 0–10 intensity scale:

• Evaluation measures include:
  • Mean Absolute Error (MAE):

  $$\text{MAE} = \frac{1}{N} \sum_{i=1}^N |\hat{y}_i - y_i|$$

  ($\hat{y}_i$: model prediction; $y_i$: expert label). - Root Mean Squared Error (RMSE) - Pearson and Spearman correlation coefficients - Refusal rate: percentage of clips on which a model declines to assess emotion.

• Benchmarks compare:

  • Foundational models (Gemini, GPT-4o)
  • EmpathicInsight-Voice models: small (74M parameters) and large (148M parameters).
• The benchmarking setup allows both performance and reliability to be assessed across the full emotion taxonomy and intensity range.

Results demonstrate greater accuracy, lower MAE/RMSE, and higher correlation for the EmpathicInsight-Voice Large model, with near-zero refusal rates and high predictive agreement for easily perceived emotions.

5. EmpathicInsight-Voice Models

EmpathicInsight-Voice models constitute specialized SER architectures trained on EmoNet-Voice Big and additional public emotion audio sets:

• Encoder: Whisper, pre-trained on 4,500h of audio (EmoNet-Voice Big + public data).
• Decoder (“expert heads”): MLP regressors trained to estimate intensity for each of the 40 dimensions.
• Training objective: minimize MAE in predicting expert intensity ratings.
• Parameters: Small (∼74M), Large (∼148M).
• Performance:
  • Large: Pearson correlation ≈ 0.421, MAE ≈ 2.995.
  • Consistently outperform commercial/foundation models on emotional alignment.

Prediction accuracy varies with expert agreement: emotions with strong inter-rater consensus (teasing, embarrassment, anger) yield tight model alignment; low-arousal and cognitive states (concentration, contemplation) remain difficult, mirroring human ambiguity.

6. Findings and Research Implications

Several empirical findings and forward-looking implications emerge from the benchmark:

• Detection is significantly easier for high-arousal emotions (anger, teasing, embarrassment)—strong acoustic features like pitch and loudness facilitate reliable machine recognition.
• Performance bottleneck for cognitive/low-arousal states is primarily attributed to perceptual variability among annotators and inherent subtlety in vocal expression.
• Model accuracy correlates strongly with inter-annotator consensus, implying that upper bounds on machine recognition may reflect human perceptual noise.
• Synthetic data and expert-driven annotation establish an ethical pathway for future emotion detection work, allowing sensitive emotion categories without personal privacy risk.

A plausible implication is that advancements in context integration (e.g., using surrounding dialogue, multimodal input, listener context) will be needed to improve recognition of ambiguous or cognitive emotional states.

7. Context, Limitations, and Future Directions

EmoNet-Voice establishes technical standards and resources for nuanced SER benchmarking. Its synthetic generation approach addresses privacy limitations and expands emotional coverage; expert annotation ensures validity across the 40-dimensional taxonomy. Nonetheless, limitations persist:

• Lower performance for subtle/cognitive emotions highlights the need for deeper context modeling and possible multimodal fusion.
• Methods to further disambiguate annotator disagreement (e.g., greater contextual metadata or hierarchical labeling schemes) are necessary.
• Future research will benefit from leveraging contextual cues, dialogue history, and possibly cross-modal information (e.g., gesture, facial expression), and from richer pre-training using the full diversity of EmoNet-Voice Big.

The resource guides future development of expressive, ethical, and context-sensitive AI systems capable of modeling complex vocal emotion—a fundamental advance for both SER research and practical deployment.