Papers
Topics
Authors
Recent
Search
2000 character limit reached

Robust End-to-End Diarization with Domain Adaptive Training and Multi-Task Learning

Published 12 Dec 2023 in cs.SD and eess.AS | (2312.07136v1)

Abstract: Due to the scarcity of publicly available diarization data, the model performance can be improved by training a single model with data from different domains. In this work, we propose to incorporate domain information to train a single end-to-end diarization model for multiple domains. First, we employ domain adaptive training with parameter-efficient adapters for on-the-fly model reconfiguration. Second, we introduce an auxiliary domain classification task to make the diarization model more domain-aware. For seen domains, the combination of our proposed methods reduces the absolute DER from 17.66% to 16.59% when compared with the baseline. During inference, adapters from ground-truth domains are not available for unseen domains. We demonstrate our model exhibits a stronger generalizability to unseen domains when adapters are removed. For two unseen domains, this improves the DER performance from 39.91% to 23.09% and 25.32% to 18.76% over the baseline, respectively.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. “X-vectors: Robust dnn embeddings for speaker recognition,” in 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5329–5333.
  2. “Generalized end-to-end loss for speaker verification,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4879–4883.
  3. Stephen Shum, Unsupervised methods for speaker diarization, Ph.D. thesis, Massachusetts Institute of Technology, 2011.
  4. “Speaker diarization with plda i-vector scoring and unsupervised calibration,” in 2014 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2014, pp. 413–417.
  5. “Speaker diarization with lstm,” in 2018 IEEE International conference on acoustics, speech and signal processing (ICASSP). IEEE, 2018, pp. 5239–5243.
  6. “End-to-end neural speaker diarization with permutation-free objectives,” arXiv preprint arXiv:1909.05952, 2019.
  7. “End-to-end neural speaker diarization with self-attention,” in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2019, pp. 296–303.
  8. “Encoder-decoder based attractors for end-to-end neural diarization,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 1493–1507, 2022.
  9. “Bw-eda-eend: Streaming end-to-end neural speaker diarization for a variable number of speakers,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021, pp. 7193–7197.
  10. “Online streaming end-to-end neural diarization handling overlapping speech and flexible numbers of speakers,” arXiv preprint arXiv:2101.08473, 2021.
  11. “Robust end-to-end speaker diarization with conformer and additive margin penalty.,” in Interspeech, 2021, pp. 3575–3579.
  12. “Improving end-to-end neural diarization using conversational summary representations,” arXiv preprint arXiv:2306.13863, 2023.
  13. “Learning multiple visual domains with residual adapters,” Advances in neural information processing systems, vol. 30, 2017.
  14. “Large-scale multilingual speech recognition with a streaming end-to-end model,” arXiv preprint arXiv:1909.05330, 2019.
  15. “A configurable multilingual model is all you need to recognize all languages,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 6422–6426.
  16. Yu Zhang and Qiang Yang, “A survey on multi-task learning,” IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 12, pp. 5586–5609, 2021.
  17. “Multitask learning in connectionist robust asr using recurrent neural networks,” in Eighth European Conference on Speech Communication and Technology, 2003.
  18. “Multitask learning of context-dependent targets in deep neural network acoustic models,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 2, pp. 238–247, 2016.
  19. “Dnn based speaker embedding using content information for text-dependent speaker verification,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 5344–5348.
  20. “Introducing phonetic information to speaker embedding for speaker verification,” EURASIP Journal on Audio, Speech, and Music Processing, vol. 2019, pp. 1–17, 2019.
  21. “Leveraging speaker attribute information using multi task learning for speaker verification and diarization,” arXiv preprint arXiv:2010.14269, 2020.
  22. “Asr-aware end-to-end neural diarization,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 8092–8096.
  23. “End-to-end speaker diarization conditioned on speech activity and overlap detection,” in 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021, pp. 849–856.
  24. “Eend-ss: Joint end-to-end neural speaker diarization and speech separation for flexible number of speakers,” in 2022 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2023, pp. 480–487.
  25. “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  26. “Conformer: Convolution-augmented transformer for speech recognition,” Proc. Interspeech 2020, pp. 5036–5040, 2020.
  27. “The hitachi-jhu dihard iii system: Competitive end-to-end neural diarization and x-vector clustering systems combined by dover-lap,” arXiv preprint arXiv:2102.01363, 2021.
  28. “End-to-end neural diarization: From transformer to conformer,” arXiv preprint arXiv:2106.07167, 2021.
  29. “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  30. “Librispeech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210.
  31. “The third dihard diarization challenge,” arXiv preprint arXiv:2012.01477, 2020.
  32. “Spot the conversation: speaker diarisation in the wild,” arXiv preprint arXiv:2007.01216, 2020.
  33. “Open source magicdata-ramc: A rich annotated mandarin conversational (ramc) speech dataset,” arXiv preprint arXiv:2203.16844, 2022.
  34. “The ami meeting corpus: A pre-announcement,” in International workshop on machine learning for multimodal interaction. Springer, 2005, pp. 28–39.
  35. “Musan: A music, speech, and noise corpus,” arXiv preprint arXiv:1510.08484, 2015.
  36. “Summary on the icassp 2022 multi-channel multi-party meeting transcription grand challenge,” in ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2022, pp. 9156–9160.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up