Speaker Identity in Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts Approach

Abstract: As expressive text-to-speech (TTS) and voice conversion (VC) systems increasingly generate non-verbal vocalizations (NVVs) to enhance naturalness, reliable speaker verification (SV) becomes essential to objectively assess identity consistency across both verbal and non-verbal segments. Yet current SV systems generalize poorly to NVVs, and fine-tuning on NVV data causes catastrophic forgetting of speech performance. We present the first systematic study across 10 NVV types and propose a framework combining frozen Data2Vec self-supervised features with ECAPA-TDNN, enhanced by a Mixture of Experts (MoE) module with learned domain-aware routing. A conditional distillation loss on speech inputs via a pretrained teacher retains speech-to-speech accuracy, while a contrastive loss bridges the speech-NVV domain gap. Our method reduces speech-NVV EER from 38.93% to 22.66% over a pretrained baseline, and improves speech EER from 13.17% to 9.24% via distillation.

• The paper presents a novel SV architecture that integrates frozen Data2Vec front-ends with ECAPA-TDNN backends and a Mixture of Experts module to address cross-domain challenges in non-verbal vocalizations.
• It introduces a conditional distillation loss that mitigates catastrophic forgetting, reducing NVV error rates (e.g., lowering NvS EER from 38.93% to 22.66%) while maintaining speech domain performance.
• The study demonstrates that expert specialization and dynamic domain-specific routing via MoE effectively separate speech and NVV pathways, advancing multi-domain speaker verification robustness.

Speaker Identity Verification Across Non-Verbal Vocalizations: Conditional Distillation and Mixture of Experts

Motivation and Problem Formulation

Contemporary expressive TTS and VC systems increasingly synthesize non-verbal vocalizations (NVV)—such as laughter, cough, sigh, and breath—to enhance paralinguistic richness. Preserving speaker identity across both verbal and NVV segments is vital for perceptual consistency and security, necessitating scalable SV solutions capable of handling highly heterogeneous acoustic events. However, prevailing SV paradigms—typically Self-Supervised Learning (SSL) front-ends with discriminative backends (e.g., ECAPA-TDNN)—fail to generalize to NVVs due to their lack of phonemic structure and diverse spectral characteristics. Fine-tuning on NVV data leads to catastrophic forgetting, severely degrading speech-to-speech verification.

Methodological Framework

The paper proposes a domain-aware SV architecture integrating frozen Data2Vec front-end with ECAPA-TDNN backend and a Mixture of Experts (MoE) module for domain-specific routing. The feature extractor (Data2Vec) addresses modality bottlenecks prevalent in cluster-based SSLs by predicting continuous latent representations, empirically superior for NVV. The MoE framework includes two architectural variants: a post-fusion MoE operating after SSL feature aggregation, and an inter-layer residual MoE (IR-MoE) with trainable adapters after each transformer block, enabling progressive domain separation.

Three distinct training objectives synergize to achieve robust multi-domain performance:

• AAM-Softmax Loss: Ensures discriminative embedding efficacy for standard speaker classification.
• Event-Guided MoE Routing Constraints: Enforces expert specialization via batch-level entropy maximization, intra-event KL divergence for routing prototype consistency, and inter-event cosine-margin for separation.
• Conditional Knowledge Distillation Loss: Preserves speech domain embedding fidelity by constraining the student model to match a frozen WavLM-based teacher on speech samples, preventing NVVs from collapsing into the speech manifold.
• Supervised Contrastive Loss: Explicitly bridges speech–NVV domain gap by including cross-domain anchor-positive pairs, while negatives are sampled across speakers.

Experimental Setup and Metrics

Evaluation is conducted on the publicly available NonverbalTTS dataset, comprising 17 hours of audio with 10 NVV types, partitioned across 1,314 training speakers, 46 validation, and 147 test speakers. Each SV trial includes balanced same-speaker and different-speaker pairs for speech-speech (SvS), NVV-speech (NvS), and NVV-NVV (NvN) verification. Cosine similarity is the scoring metric. Performance is quantified via Equal Error Rate (EER) and minimum detection cost (mDCF), following the NIST SV protocol.

Batch composition employs speaker-balanced sampling (16 unique speakers per batch with 8 utterances each), ensuring sufficient cross-domain anchor-positive pairs for supervised contrastive loss. Optimization follows a progressive schedule, initially warming up with AAM-Softmax, then gradually introducing MoE routing and event regularization.

Results and Analysis

Domain Mismatch and Transfer Dilemma

The empirical analysis reveals substantial domain mismatch: baseline wavlm-base-plus-sv achieves only 5.60% SvS EER, but 38.93%/39.13% EER on NvS/NvN, confirming severe overlap between same-speaker and different-speaker NVV score distributions. Fine-tuning on NVV leads to catastrophic forgetting, with SvS EER worsening markedly.

Ablations—Conditional Distillation

Without distillation, IR-MoE models reduce NvS EER to 24.95% (from baseline 38.93%) but degrade SvS EER to 13.17%. Incorporating conditional distillation further decreases NvS EER to 22.66% and recovers SvS EER to 9.24%, a statistically significant improvement. The gap to the zero-shot baseline is largely attributed to differences in training data scale; large-scale joint training remains an open research direction.

Mixture of Experts Effectiveness

Standalone Data2Vec+ECAPA-TDNN baseline yields NvS EER of 23.33%. MoE architectures, especially the 4-expert IR-MoE configuration, achieve the best balance: NvS EER of 22.66%, SvS EER of 9.24%. Increasing the number of experts improves verbal domain EER marginally but risks specialization collapse in NVV with limited data. Expert separation enables dynamic routing, alleviating feature bottlenecks and improving multi-domain robustness.

Practical and Theoretical Implications

This work positions speaker verification for expressive synthesis as a challenging multi-domain problem. Decoupling speech and NVV pathways via MoE, while unifying identity embeddings post-routing, enables SV systems to objectively evaluate synthetic voices with NVVs—a requirement for robust TTS and VC deployment. The conditional distillation mechanism prevents catastrophic forgetting, addressing a prominent continual learning dilemma in audio domain adaptation.

Theoretically, the divide-and-conquer approach—domain separation in intermediate layers with late fusion—suggests new directions for multi-domain representation learning, particularly for diverse, heterogeneous non-verbal signals.

Future Directions

• Scaling SV training with joint large-scale corpora could further close the speech–NVV performance gap.
• Direct evaluation of synthetic NVVs from modern generative pipelines may validate zero-shot adaptation.
• Theoretical analysis of representational dynamics for intermediate separation and final embedding fusion is necessary to understand the limits and advantages of the MoE paradigm in speaker identity modeling.
• Extension to other paralinguistic events (emotional bursts, pathological signals) and multilingual NVVs.

Conclusion

The paper provides the first systematic SV evaluation across a diverse NVV taxonomy, demonstrating the limitations of current SSL-based SV systems, and introduces a conditional distillation and MoE framework that significantly improves cross-domain robustness. By aligning multi-domain embeddings while maintaining domain-specific routing, the approach advances the state-of-the-art for speaker identity preservation in expressive, non-verbal synthetic speech (2606.21215).