Zero-shot Cross-lingual Speech Emotion Recognition

• Zero-shot cross-lingual SER is a task where emotion models, trained on source languages, operate on target languages without labeled emotion data.
• It employs techniques like emotion-discriminative representation learning and domain adversarial training to overcome language-specific prosodic differences.
• Recent methods report significant gains in UAR and Macro-F1 by combining contrastive, metric, and adversarial strategies for effective cross-lingual transfer.

Zero-shot cross-lingual speech emotion recognition (SER) is the task of training an emotion recognizer on one or more source languages and evaluating it directly on an unseen target language without target-language emotion annotations. The central difficulty is distribution mismatch: prosodic patterns, phonetic realizations, and cultural conventions of emotional prosody differ across languages, so models that depend on source-language cues often generalize poorly to new languages. Recent work addresses this problem through emotion-discriminative representation learning, adversarial removal of speaker or language cues, multilingual self-supervised or ASR-pretrained backbones, continuous metric learning, synthetic multilingual corpora, and, more recently, speech-LLM prompting and distribution-aware evaluation (Mi et al., 4 Jun 2026, Osman et al., 2024, Zhang et al., 9 Mar 2026).

1. Problem formulation and protocol variants

A standard formulation trains on labeled source-language data and evaluates on an unseen target language with no target-language emotion labels. One recent formulation writes the training set as

$D = D_{\text{source}} \cup D_{\text{non-target}},$

where $D_{\text{source}}$ is the labeled source-language set, $D_{\text{non-target}}$ contains auxiliary labeled data from other non-target languages, and $D_{\text{target}}$ is held out entirely during training (Mi et al., 4 Jun 2026). In this strict version, zero-shot transfer excludes any access to the target language during optimization.

The literature also contains a closely related but not identical protocol: unsupervised cross-lingual SER with unlabeled target-language speech available during training. In the domain-adversarial formulation, the source set is $D_s=\{(x_i^s,y_i^s)\}_{1}^{n}$ and the target set is $D_t=\{x_i^t\}_{1}^{m}$, where target emotion labels are never used, but target utterances are passed through a language-classification branch so that the encoder learns language-invariant features (Cai et al., 2020). This distinction is methodologically important: some papers reserve “zero-shot” for strict target holdout, whereas others include unlabeled target exposure under zero-shot or unsupervised adaptation language.

Benchmark design further complicates the notion of transfer. In one large-scale evaluation, training corpora and held-out corpora are separated by a unified eight-class label inventory, and “OOD-eligible” corpora are those whose post-mapping label spaces exactly match that inventory (Osman et al., 2024). In speech-LLM evaluation, cross-lingual SER is framed even more broadly, spanning 35 corpora across 15 languages with alias-normalized labels, speaker-independent partitions, and both hard-label and soft-label protocols (Zhang et al., 9 Mar 2026). This makes direct comparison across studies nontrivial, because “zero-shot cross-lingual SER” may refer to strict target holdout, adversarial adaptation with unlabeled target speech, or prompt-based inference on multilingual benchmarks.

2. Representation objectives and invariance mechanisms

A prominent line of work learns embeddings that are explicitly emotion-discriminative while discouraging nuisance factors. In a wav2vec 2.0-based framework, an utterance $x$ is mapped by a pretrained encoder plus mean pooling to $h \in \mathbb{R}^d$, an emotion classifier is trained with cross-entropy, a supervised contrastive module aligns same-emotion embeddings, and a speaker-adversarial branch with a gradient reversal layer suppresses speaker identity. The total objective is

$$L_{\text{total}} = L_{\text{CE}}(c(h),y) +\alpha L_{\text{SupCLR}} +\beta L_{\text{SpkAdv}},$$

with $\alpha=1.0$ and $D_{\text{source}}$0 in practice (Mi et al., 4 Jun 2026). The supervised contrastive term uses a language-aware positive-pair weight

$D_{\text{source}}$1

with $D_{\text{source}}$2, thereby emphasizing cross-lingual alignment of same-emotion samples. This directly operationalizes the idea that zero-shot transfer should be driven by emotion similarity rather than language similarity.

Domain-adversarial training is an earlier and still influential alternative. In the DANN-based formulation, the encoder $D_{\text{source}}$3 feeds both an emotion classifier $D_{\text{source}}$4 and a language classifier $D_{\text{source}}$5 through a gradient reversal layer. The objective combines source-only emotion loss and source-plus-target language loss,

$D_{\text{source}}$6

so that $D_{\text{source}}$7 is optimized to minimize emotion loss while maximizing language loss (Cai et al., 2020). After training, the language classifier is discarded and the encoder-plus-emotion head is applied directly to target-language utterances. In this setting, invariance is formulated at the language level rather than the speaker level.

A different strategy uses continuous metric learning in a denoising autoencoder. Instead of treating labels as discrete classes only, the latent distances are constrained to linearly reflect pairwise distances in activation or valence space. With latent embeddings $D_{\text{source}}$8 and scalar labels $D_{\text{source}}$9, the model estimates a slope $D_{\text{non-target}}$0 by ordinary least squares and defines

$D_{\text{non-target}}$1

where $D_{\text{non-target}}$2 and $D_{\text{non-target}}$3; the final objective is $D_{\text{non-target}}$4 (Das et al., 2022). This replaces binary push-pull metric learning with a continuous geometry tied to dimensional emotion labels. A plausible implication is that transfer may benefit when the embedding space preserves graded affective structure rather than only categorical separation.

3. Backbones, labels, and data regimes

The architectural range in zero-shot cross-lingual SER is unusually broad. One end of the spectrum uses 88-dimensional eGeMAPS features extracted via openSMILE and a compact denoising autoencoder with a 2-dimensional latent space (Das et al., 2022). Another uses log-Mel spectrograms, a 1-D convolutional encoder, attention pooling, Batch-Normalization, and a shallow emotion and language classifier pair (Cai et al., 2020). More recent systems adopt large pretrained speech models, including wav2vec 2.0 encoders (Mi et al., 4 Jun 2026) and Whisper-medium encoders whose initial convolutional layers are frozen while transformer encoder blocks are fine-tuned (Osman et al., 2023).

Label regimes are likewise heterogeneous. A strict four-class cross-lingual setup uses angry, happy, sad, and neutral across MELD, ESD, EMO-DB, CaFE, and URDU (Mi et al., 4 Jun 2026). Other work studies binary arousal and valence transfer between IEMOCAP and RECOLA (Cai et al., 2020). Large multilingual aggregation maps many corpora into an eight-class space—happiness, sadness, disgust, fear, surprise, anger, neutral, and other—and introduces soft probability targets $D_{\text{non-target}}$5 with neutral smoothing,

$D_{\text{non-target}}$6

followed by soft-target cross-entropy and a small adversarial domain term with $D_{\text{non-target}}$7 (Osman et al., 2023). Benchmarking work standardizes a unified eight-class taxonomy and uses logit adjustment,

$D_{\text{non-target}}$8

to compensate for class-prior mismatch across corpora (Osman et al., 2024).

Data scale has become a central variable. One study amalgamates 16 datasets into 375 hours across languages including English, Chinese, and Japanese, and evaluates zero-shot on held-out German, Mandarin Chinese, Urdu, and Greek corpora (Osman et al., 2023). Another introduces M5SER, a synthetic multilingual SER corpus of approximately 1,009 hours created by translating English emotional speech into French, German, Italian, Mandarin, and Spanish with SeamlessExpressive while preserving emotional prosody (Zou et al., 25 Mar 2025). In speech-LLM work, VoxEmo expands evaluation to 35 corpora across 15 languages and emphasizes that open text generation introduces prompt-sensitive stochasticity absent from conventional closed-set classification (Zhang et al., 9 Mar 2026). This suggests that the field is simultaneously scaling data, strengthening backbones, and broadening the very definition of what constitutes an SER prediction.

4. Empirical findings

The strongest gains in strict target-held-out evaluation are reported by the emotion-discriminative wav2vec 2.0 framework. Across nine source$D_{\text{non-target}}$9target configurations, the full model reaches an average UAR of 82.26% and Macro-F1 of 81.96%, compared with 59.49% and 58.24% for fine-tuning on source only, and 73.21% and 72.58% for fine-tuning on source plus non-target languages. The same study reports that this closes much of the gap to the target-supervised upper bound, with UAR approximately 91.9%. Ablation is also informative: removing $D_{\text{target}}$0 reduces UAR by 5.40% and F1 by 5.26%, while removing $D_{\text{target}}$1 reduces UAR by 2.15% and F1 by 1.82%; t-SNE visualizations show compact, well-separated emotion clusters across all languages (Mi et al., 4 Jun 2026).

Domain-adversarial training yields smaller but consistent improvements in the unsupervised target-access setting. The DANN system improves average UAR by 3.91% over its non-adversarial baseline for binary arousal and valence transfer, with condition-specific gains including 62.49 to 71.99 on RECOLA$D_{\text{target}}$2IEMOCAP arousal and 58.11 to 61.43 on IEMOCAP$D_{\text{target}}$3RECOLA valence. Batch-Normalization is not incidental: removing BN reduces the average DANN-over-baseline gain from 3.91% to 2.61% UAR, and combined source-plus-target batch normalization strategies BN1 and BN2 outperform BN3 and BN4 (Cai et al., 2020).

Large-scale multilingual pretraining also shows substantial zero-shot performance. A Whisper-based multilingual model with soft labeling, aggressive waveform and spectrogram augmentation, and domain-adversarial training reports zero-shot micro-F1 of 70.09 on Z-EmoDB, 45.20 on Z-MASC, 52.49 on Z-URDU, and 61.92 on Z-AESDD, all without fine-tuning on the held-out datasets (Osman et al., 2023). In a broad frozen-backbone benchmark, Whisper variants lead both in-domain and zero-shot OOD macro-F1; for example, Whisper-Large reports OOD macro-F1 of $D_{\text{target}}$4, Whisper-medium $D_{\text{target}}$5, and Whisper-Large-v2 attains the highest weighted average score, while non-Whisper SSL models lag behind (Osman et al., 2024). The benchmark also shows high variability across training and target corpora, indicating that transfer quality depends strongly on corpus-specific properties.

Lower-complexity and LLM-mediated methods enlarge the design space rather than replacing earlier approaches. Continuous metric learning with a tiny denoising autoencoder improves balanced accuracy over an unsupervised autoencoder on several transfer sets, reaching 0.78 on German EMO-DB and 0.60 on URDU for the Metric-Val variant, while remaining comparable to or better than a HuBERT-based SUPERB reference on several zero-shot tasks (Das et al., 2022). An LLM-centered two-stage contrastive framework using frozen Whisper, an Emotion Q-Former, and frozen LLaMA-3 reports zero-shot results on unseen datasets including ShEMO, AESDD, RAVDESS, and RESD, with WA of 71.3%, 42.6%, 41.7%, and 39.2%, respectively (Zou et al., 25 Mar 2025). These results indicate that cross-lingual SER now spans both conventional discriminative classification and generative, connector-based architectures.

5. Evaluation methodology and interpretive cautions

No single metric dominates the literature. Strict cross-lingual classification studies frequently use Unweighted Average Recall and Macro-F1 because of class imbalance (Mi et al., 4 Jun 2026). Binary affect transfer uses UAR for arousal and valence (Cai et al., 2020). Large multilingual zero-shot evaluation has used micro-F1 on held-out datasets (Osman et al., 2023), while broader backbone benchmarks standardize on macro-F1 and explicitly define OOD generalization from a train-test performance matrix $D_{\text{target}}$6 (Osman et al., 2024). LLM-oriented work adds Weighted Accuracy, Unweighted Accuracy, Weighted F1, Micro-F1, Macro-F1, Top-1 Accuracy, and distributional divergences such as KL, JSD, TVD, cosine similarity, and MSE (Zou et al., 25 Mar 2025, Zhang et al., 9 Mar 2026). Because the label inventories also vary—binary arousal/valence, four shared emotions, seven basic emotions, and unified eight-class taxonomies—numerical results from different papers are not directly commensurate.

Prompt-based zero-shot SER introduces an additional evaluation axis absent from classifier-style systems. VoxEmo shows that prompt choice dominates zero-shot performance: single-block prompts such as Direct, +A, or +T outperform multi-block variants in approximately 95% of cases, while complex prompts with transcript, acoustic caption, reasoning, and distribution blocks can raise parse failures to approximately 93%, collapsing F1 toward zero (Zhang et al., 9 Mar 2026). The same work proposes a prompt-ensemble distribution,

$D_{\text{target}}$7

with uniform fallback on parse failure, as a way to emulate annotator disagreement. On IEMOCAP, a Qwen2-Audio ensemble reaches Macro-F1 = 56% while also achieving cosine similarity 0.69 and JSD approximately 0.254 against human vote distributions (Zhang et al., 9 Mar 2026). This reframes evaluation: a system may be weak under hard-label accuracy yet comparatively strong in matching the subjective distribution of human emotion judgments.

A persistent misconception is that zero-shot cross-lingual SER refers to a single protocol. The cited literature instead supports at least three distinct regimes: strict unseen-target evaluation with no target exposure (Mi et al., 4 Jun 2026), adversarial training with unlabeled target speech (Cai et al., 2020), and prompt-based multilingual inference with no task-specific fine-tuning but strong dependence on foundation-model capabilities and prompt format (Zhang et al., 9 Mar 2026). Treating these regimes as interchangeable obscures the source of reported gains.

6. Limitations, controversies, and research directions

Several limitations recur. The emotion-discriminative contrastive framework explicitly notes reliance on a few non-target languages for contrastive anchors and points to stronger alignment methods and multimodal signals such as video or text as future work (Mi et al., 4 Jun 2026). Benchmarking work identifies persistent class imbalance, long-tail emotions, and high variability across corpora, even after logit adjustment (Osman et al., 2024). Continuous metric learning highlights the more difficult relation between acoustics and valence than between acoustics and activation, motivating non-linear metric losses and higher-dimensional embeddings (Das et al., 2022).

The role of pretraining remains an active interpretive issue. One benchmark finds that Whisper, though designed for ASR, outperforms dedicated SSL models in cross-lingual SER and attributes this to massive multilingual weak supervision and a sequence-to-sequence training objective sensitive to acoustic and prosodic contours (Osman et al., 2024). Another line argues that multilingual transfer can be strengthened further by synthetic data generation: M5SER enlarges language coverage through translation with preserved emotional prosody and supports two-stage contrastive alignment into an LLM’s textual emotion space (Zou et al., 25 Mar 2025). A plausible implication is that backbone scale alone is insufficient; performance appears to depend on how pretraining, label harmonization, and downstream alignment objectives interact.

Speech LLMs introduce a separate controversy about what should count as success. VoxEmo reports that zero-shot speech LLMs trail supervised baselines in hard-label accuracy, often with Macro-F1 below 20% on many small acted sets and under 40% on most cross-lingual sets, yet they uniquely align with human subjective distributions (Zhang et al., 9 Mar 2026). This does not negate the value of closed-set classifiers; rather, it indicates that SER now spans two partially different desiderata: maximizing categorical recognition under fixed taxonomies and modeling affective ambiguity under distribution-aware supervision.

Taken together, the literature depicts zero-shot cross-lingual SER as a representation-learning problem shaped by protocol definition. The most successful strict-transfer systems combine explicit emotion alignment with nuisance suppression (Mi et al., 4 Jun 2026), unsupervised adaptation benefits from adversarial language invariance and careful batch normalization (Cai et al., 2020), large multilingual backbones and label harmonization improve held-out-language robustness (Osman et al., 2023, Osman et al., 2024), and emerging LLM-based methods broaden both the modeling interface and the evaluation target (Zou et al., 25 Mar 2025, Zhang et al., 9 Mar 2026). The field’s next advances are likely to depend less on any single architectural motif than on cleaner protocol separation, stronger cross-corpus harmonization, and evaluation frameworks that jointly measure categorical transfer and the subjective distribution of perceived emotion.