Learning Emotion-discriminative Representations for Zero-Shot Cross-lingual Speech Emotion Recognition

Abstract: Zero-shot cross-lingual speech emotion recognition (SER) remains challenging due to distribution mismatches across languages and the lack of emotion annotations in target language. Under such conditions, models trained solely on source-language data frequently suffer from degraded generalization when evaluated on unseen target languages. To address this limitation, we propose an emotion-discriminative representation learning method that integrates supervised contrastive learning and speaker adversarial learning. The contrastive learning promotes cross-lingual emotion alignment, while speaker adversarial learning suppresses speaker-related cues to encourage speaker-invariant representations. Experimental results under a zero-shot cross-lingual SER setting demonstrate that the proposed method significantly improves SER performance over conventional training strategies.

• The paper demonstrates that integrating supervised contrastive and adversarial speaker learning significantly enhances emotion alignment across languages in SER.
• It leverages a pretrained wav2vec 2.0 encoder with a hierarchical cross-lingual batching scheme to enforce language-invariant emotion clustering.
• Experimental results show substantial gains in UAR and Macro-F1 metrics, confirming improved cross-lingual generalization and robust emotion representation.

Emotion-discriminative Representation Learning for Zero-Shot Cross-lingual Speech Emotion Recognition

Problem Formulation and Motivation

Zero-shot cross-lingual Speech Emotion Recognition (SER) addresses the critical challenge of deploying emotion classification models in languages where labeled emotional corpora are sparse or nonexistent. Conventional fine-tuning strategies using SSL models (e.g., wav2vec 2.0, WavLM) rely on supervised adaptation to target languages, which is unattainable in the zero-shot regime due to the absence of target-language emotion labels. Prior work on language-adversarial domain adaptation and multilingual pretraining has mitigated inter-lingual distribution mismatch but failed to enforce explicit structural alignment of emotion categories across languages, resulting in suboptimal cross-lingual generalization.

Proposed Method: Emotion Alignment and Speaker Invariance

The proposed framework integrates two core objectives: supervised contrastive learning for emotion-class alignment across languages and adversarial speaker learning for robust emotion discrimination devoid of speaker artifacts.

The architecture utilizes a pretrained SSL model (wav2vec 2.0) as an encoder to extract contextualized speech representations. Supervised contrastive learning is employed to maximize intra-emotion similarity and inter-emotion dissimilarity across source and auxiliary languages, with cross-lingual samples of the same emotion class assigned amplified weights to encourage language-invariant emotion clustering. This is facilitated by a hierarchical cross-lingual batch sampling scheme, ensuring multi-language, multi-emotion co-occurrence per batch.

Speaker adversarial learning is conducted via a Gradient Reversal Layer (GRL)-based speaker classifier that penalizes the retention of speaker-identifying features in the learned embeddings, thus maximizing the decoupling of emotion and speaker information. The cumulative objective optimizes cross-entropy for emotion classification, the contrastive loss for structural alignment, and the adversarial loss for speaker invariance.

Figure 1: Overview of the architecture showing source, target, and non-target language speech flows through feature extraction, supervised contrastive, adversarial speaker, and emotion classification modules.

Experimental Setup and Baseline Comparison

Empirical evaluations span nine zero-shot transfer settings across five languages (EN, CN, DE, FR, UR). Datasets are harmonized to support four emotion classes (happy, angry, sad, neutral) with rigorous data splits for source, auxiliary, and target-language corpora. Metrics adopted are Unweighted Average Recall (UAR) and Macro-F1, emphasizing balanced performance despite class and corpus imbalance.

Two primary baselines are constructed:

• Baseline 1 trains only on source language, providing minimal cross-lingual transfer.
• Baseline 2 leverages source and auxiliary (non-target) languages, without explicit cross-lingual emotion alignment or speaker disentanglement.

Three variants of the proposed model are evaluated:

• Full model (with supervised contrastive and adversarial speaker objectives)
• Ablation of supervised contrastive objective
• Ablation of adversarial speaker objective

An upper bound (oracle) system is trained with target-language supervision for ceiling reference.

Quantitative Results

The proposed method exhibits substantial performance improvements over baselines across all cross-lingual tasks. The average UAR and F1 reach 82.26% and 81.96%, respectively, representing a 9.05% and 9.38% absolute gain over the best baseline. Both ablated variants retain superiority over baselines, substantiating the synergistic impact of the dual objectives.

Notably, removal of the contrastive objective degrades UAR and F1 by 5.40% and 5.26%, underscoring the primacy of emotion-class structural alignment in enhancing cross-lingual transfer. Speaker adversarial ablation causes 2.15% UAR and 1.82% F1 loss, confirming the necessity of speaker-invariance for robust generalization.

Representation Analysis

t-SNE visualization of hidden embeddings reveals the comparative clustering performance of various systems. The full proposed model achieves the most compact and well-separated emotion clusters across languages, closely approximating the oracle system's target-language clustering while surpassing it in non-target language cluster cohesion.

Figure 2: t-SNE plots demonstrating the clear emotion-based separation and cross-lingual structural consistency achieved by the proposed system as compared to baselines and ablations.

Practical and Theoretical Implications

The proposed architecture establishes an efficient mechanism for zero-shot SER with only minimal source and auxiliary data and no reliance on target-language labels. Practically, this enables scalable deployment of SER systems in low-resource languages, circumventing the bottleneck of manual annotation. Theoretical advances include the explicit modeling of emotion-category structural consistency across languages, which can be extended to other affective computing tasks involving cross-cultural transfer. Speaker-invariant emotion representation learning also provides a robust foundation for multimodal affective modeling, especially in heterogeneous data environments.

Future Directions

Potential avenues for extension include the integration of stronger cross-lingual alignment techniques such as language-aware adversarial feature learning, the incorporation of multimodal cues (e.g., visual or lexical), and adaptation to more fine-grained or hierarchical emotion taxonomies. Further research may also explore parameter-efficient transfer methods for scaling to larger language pools and applications beyond SER.

Conclusion

The paper introduces a formal framework for zero-shot cross-lingual SER based on supervised contrastive learning and speaker adversarial objectives. Results substantiate the claim of improved cross-lingual generalization and emotion-discriminative representation learning, validated quantitatively and visually. The implications span both scalable SER deployment and structural affective modeling, positioning the approach as a foundational methodology for future advances in language-agnostic emotion recognition.