Do Speech Emphasis Models Generalize across Languages and Emotions?

Abstract: Prosodic emphasis varies across languages, emotions, and speaking styles, yet existing emphasis detection models are largely trained and evaluated on monolingual neutral read speech. We introduce MMEE (Multilingual Multi-Emotion Emphasis), a corpus of 10,000 professionally recorded expressive utterances (14.13 hours) across 7 languages and 34 emotion/style categories, with three-level perceptual labels (10 annotations per sample). We benchmark two state-of-the-art architectures under monolingual, cross-lingual, multilingual, cross-emotion, cross-dataset, and data-scale settings. Monolingual models show limited zero-shot transfer, degrading across typologically distant languages, while multilingual training substantially improves robustness. Models transfer robustly between high- and low-arousal emotions; bidirectional transfer between synthetic and perceptual benchmarks suggests shared prosodic structure; and performance stays robust even at smaller training scales.

• The paper shows that multilingual training on the MMEE corpus improves cross-lingual and cross-emotional generalization compared to monolingual models.
• It benchmarks EmphaClass and WhiStress, revealing notable data efficiency with significant performance gains even at low-resource scales.
• The study demonstrates that prosodic emphasis cues remain largely universal, facilitating reliable transfer between synthetic and human-annotated datasets.

Cross-Language and Cross-Emotion Generalization of Speech Emphasis Models: An Analysis of MMEE

Introduction

This work rigorously addresses the question of whether speech emphasis models—critical for expressive TTS, speech translation, and computational paralinguistics—generalize across language boundaries and affective states. Prior benchmarks on prosodic emphasis have almost uniformly focused on English, neutral, and read speech, with label sources often limited to synthetic or expert-marked annotations. In contrast, this study introduces the MMEE (Multilingual Multi-Emotion Emphasis) corpus, a substantial and methodologically robust dataset enabling a comprehensive evaluation of generalization in emphasis modeling. Two leading architectures, EmphaClass and WhiStress, are systematically benchmarked to dissect monolingual, cross-lingual, multilingual, cross-emotion, and cross-dataset generalization, alongside data efficiency.

MMEE Corpus: Design and Annotation Paradigms

The MMEE corpus comprises 10,000 professionally produced utterances in 7 macro-languages covering 10 regional variants. Scripts and emotion/style prompts—generated with LLM assistance—systematically induce a rich variety of prosodic patterns. Each utterance receives 10 word-level, perceptual annotations on a 3-level emphasis scale from demographically-filtered native listeners, generating granular scalar and binary aggregate labels. The annotation interface (Figure 1) operationalizes rigorous perceptual data collection and integrates multi-layered quality controls, including transcript-to-audio alignment checks and detection of labeling anomalies.

Figure 1: Speech emphasis annotation interface capturing graded perceptual labels at the word level via native listeners on Prolific.

Agreement rates confirm high annotation reliability, with pooled pairwise Cohen's $\kappa$ = 0.451 and stable emphasis rates (15–22%) across both languages and emotions. This corpus sharply contrasts with prior benchmarks, which are either synthetic or narrowly monolingual and lack broad affective or dialectal coverage.

Modeling Approaches: EmphaClass and WhiStress

Two distinct SOTA frameworks are benchmarked:

• EmphaClass: Finetunes XLS-R, a large multilingual SSL model, for frame-level binary and scalar word emphasis, utilizing regression or classification heads and tailored loss masking.
• WhiStress: Extends Whisper with an auxiliary FCNN head for token-wise emphasis, supporting multilinguality via language-conditioned decoding.

Both architectures are evaluated under controlled data splits (80/10/10 train/val/test) and across experimental conditions elucidating the boundaries of generalization.

Empirical Evaluation: Transfer and Robustness Across Languages

Monolingual training yields strong in-language performance, but cross-lingual zero-shot transfer sharply degrades with typological distance—for instance, transfer from Germanic/Romance families to Mandarin is markedly weaker, reflecting the divergence in prosodic cue structures such as the confounding of F0 with lexical tone in tonal languages.

Pooled multilingual training drives robust cross-lingual generalization. As shown in Figure 2, models trained on all languages match or exceed monolingual accuracy on most test languages, highlighting the advantage of exposure to a diverse prosodic regime.

Figure 2: Multilingual training enables models to achieve robust and stable generalization across diverse language families and dialects.

Data Efficiency and Scaling

By varying training dataset sizes, the study quantifies the data efficiency of both models. There are pronounced performance gains with the first few thousand utterances, with diminishing marginal returns at dataset scales beyond 20–30% of the full corpus. EmphaClass demonstrates robust performance even at low-resource scales, whereas WhiStress's performance improves more gradually with additional data but attains competitive scores at higher scales.

Figure 3: Model performance (Binary Accuracy, F1, Pearson correlation) exhibits rapid saturation as dataset size increases, indicating strong data efficiency.

Emotion and Arousal Transfer

The corpus enables a direct test of generalization across the arousal spectrum of emotions (high arousal: e.g., happiness, anger; low arousal: e.g., calmness, sadness). Despite substantial acoustic-phonetic variation associated with arousal states, both models generalize robustly across arousal boundaries in both binary and scalar settings. This indicates that the models' representations encode emphasis cues that are not fundamentally confounded with broader affective prosody—supporting the universality of certain types of prominence irrespective of global affective state.

Cross-Dataset Transfer and Annotation Paradigm Robustness

Cross-dataset evaluation reveals strong bi-directional transferability between models trained on MMEE and those trained on existing synthetic or LLM-annotated datasets. For instance, MMEE-trained EmphaClass achieves higher accuracy when transferred to EmphAssess than vice versa, while WhiStress shows near-symmetric transfer with TinyStress-15K. This supports the hypothesis of substantial overlap in the prosodic correlates of emphasis between synthetic and human-labeled data. Notably, synthetic-to-perceptual and perceptual-to-synthetic transfer remain robust, indicating shared underlying prosodic structures across annotation paradigms.

Implications and Future Directions

By systematically isolating cross-lingual and cross-emotional effects, the study demonstrates both the limits and possibilities of prosodic universality in neural speech models. While representations are not entirely language-agnostic—fracturing at significant typological boundaries—multilingual joint training substantially mitigates overfitting to language-specific cues. Practically, this suggests that high-quality, graded perceptual annotations, especially at modest scale, can drive robust emphasis modeling even for under-resourced languages.

On a theoretical level, these results reinforce the notion of partially universal prosodic structure but highlight the need for further research into representing suprasegmental phenomena in a typologically sensitive manner. There is also clear scope for future exploration of domain adaptation methods, meta-learning for rapid adaptation to novel dialects, and deeper disentanglement of linguistic, affective, and paralinguistic cues in unified speech models.

Conclusion

This research demonstrates that while speech emphasis detection models exhibit partial universality, typological distance continues to constrain cross-lingual transfer. Multilingual training on perceptually-annotated, emotionally-diverse data sets leads to robust generalization, and data efficiency is high, lowering resource barriers in many languages. Furthermore, the models encode prominence signals relatively independent from affective arousal and annotation source. These findings inform the deployment of speech technologies in multilingual, expressive contexts and motivate further work on enhancing cross-linguistic and cross-affective generalization in prosodic modeling.