Papers
Topics
Authors
Recent
Search
2000 character limit reached

EM-Network: Oracle Guided Self-distillation for Sequence Learning

Published 14 Jun 2023 in cs.LG, cs.CL, and eess.AS | (2306.10058v1)

Abstract: We introduce EM-Network, a novel self-distillation approach that effectively leverages target information for supervised sequence-to-sequence (seq2seq) learning. In contrast to conventional methods, it is trained with oracle guidance, which is derived from the target sequence. Since the oracle guidance compactly represents the target-side context that can assist the sequence model in solving the task, the EM-Network achieves a better prediction compared to using only the source input. To allow the sequence model to inherit the promising capability of the EM-Network, we propose a new self-distillation strategy, where the original sequence model can benefit from the knowledge of the EM-Network in a one-stage manner. We conduct comprehensive experiments on two types of seq2seq models: connectionist temporal classification (CTC) for speech recognition and attention-based encoder-decoder (AED) for machine translation. Experimental results demonstrate that the EM-Network significantly advances the current state-of-the-art approaches, improving over the best prior work on speech recognition and establishing state-of-the-art performance on WMT'14 and IWSLT'14.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Aguilar, G. et al. Knowledge distillation from internal representations. Proc. AAAI, 2020.
  2. Wav2vec 2.0: a framework for self-supervised learning of speech representations. In Proc. NIPS, 2020.
  3. Data2vec: a general framework for self-supervised learning in speech, vision and language. In Proc. ICML, pp.  1298–1312, 2022.
  4. Caron, M. et al. Emerging properties in self-supervised vision transformers. In Proc. ICCV, 2021.
  5. Imputer: sequence modelling via imputation and dynamic programming. In Proc. ICML, 2020.
  6. Attention-based models for speech recognition. In Proc. NIPS, pp.  577–585, 2015.
  7. Clark, K. et al. Bam! born-again multi-task networks for natural language understanding. In Proc. ACL, 2019.
  8. Daniel, S. P. et al. Specaugment: a simple data augmentation method for automatic speech recognition. In Proc. INTERSPEECH, 2019.
  9. Bert: pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, pp.  4171–4186, 2019.
  10. Multi-branch attentive transformer. arXiv preprint arXiv:2006.10270, 2020.
  11. Feng, Y. et al. Guiding teacher forcing with seer forcing for neural machine translation. In Proc. ACL-IJCNLP, 2021.
  12. Furlanello, T. et al. Born again neural networks. In Proc. ICML, 2018.
  13. Ghazvininejad, M. et al. Aligned cross entropy for non-autoregressive machine translation. In Proc. ICML, 2020.
  14. Graves, A. Sequence transduction with recurrent neural networks. In Proc. ICML Workshop on Representation Learning, 2012.
  15. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proc. ICML, pp.  369–376, 2006.
  16. Grill, B. et al. Bootstrap your own latent: a new approach to self-supervised learning. In Proc. NIPS, 2020.
  17. Gulati, A. et al. Conformer: convolution-augmented transformer for speech recognition. In Proc. INTERSPEECH, 2020.
  18. Mask ctc: non-autoregressive end-to-end asr with ctc and mask predict. In Proc. INTERSPEECH, 2020.
  19. Distilling the knowledge in a neural network. In Proc. NIPS Workshop Deep Learn., 2014.
  20. Hubert: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451–3460, 2021.
  21. Kang, Y. et al. Self-supervised audio-and-text pre-training with extremely low-resource parallel data. In Proc. AAAI, 2022.
  22. Glow-tts: a generative flow for text-to-speech via monotonic alignment search. In Proc. NIPS, 2020.
  23. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In Proc. ICML, 2021.
  24. Sequence-level knowledge distillation. In Proc. EMNLP, 2016.
  25. Adam: a method for stochastic optimization. In Proc. ICLR, 2015.
  26. Kong, X. et al. En-compactness: self-distillation embedding & contrastive generation for generalized zero-shot learning. In Proc. CVPR, 2022.
  27. Kuchaiev, O. et al. Nemo: a toolkit for building ai applications using neural modules. arXiv preprint arXiv:1909.09577, 2019.
  28. Guiding ctc posterior spike timings for improved posterior fusion and knowledge distillation. In Proc. INTERSPEECH, 2019.
  29. Liang, X. et al. R-drop: regularized dropout for neural networks. In Proc. NIPS, 2021.
  30. Liu, H. et al. Connectionist temporal classification with maximum entropy regularization. In Proc. NIPS, 2018.
  31. Decoupled weight decay regularization. In Proc. ICLR, 2019.
  32. Understanding and improving transformer from a multi-particle dynamic system point of view. In ICLR Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  33. Miao, Y. et al. Eesen: End-to-end speech recognition using deep rnn models and wfst-based decoding. In Proc. ASRU, 2015.
  34. Apprentice: using knowledge distillation techniques to improve low-precision network accuracy. In Proc. ICLR, 2018.
  35. Scaling neural machine translation. In Proc. Machine Translation: Research Papers, pp.  1–9, 2018.
  36. Ott, M. et al. fairseq: a fast, extensible toolkit for sequence modeling. In Proc. NAACL-HLT: Demonstrations, 2019.
  37. Librispeech: an asr corpus based on public domain audio books. In Proc. ICASSP, pp.  5206–5210, 2015.
  38. Park, D. S. et al. Improved noisy student training for automatic speech recognition. In Proc. INTERSPEECH, 2020.
  39. Pei, Y. et al. Self-boosting for feature distillation. In Proc. IJCAI, 2021.
  40. Leveraging pre-trained checkpoints for sequence generation tasks. Transactions of the Association for Computational Linguistics, 8:264–280, 2020.
  41. Acoustic modelling with cd-ctc-smbr lstm rnns. In Proc. ASRU, pp.  604–609, 2015.
  42. The evolved transformer. In Proc. ICML, pp.  5877–5886, 2019.
  43. Alignart: non-autoregressive neural machine translation by jointly learning to estimate alignment and translate. In Proc. EMNLP, 2021.
  44. Sun, S. et al. Patient knowledge distillation for bert model compression. Proc. EMNLP, 2019.
  45. Suárez, P. et al. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proc. ACL, pp.  1703–1714, 2020.
  46. An investigation of a knowledge distillation method for ctc acoustic models. In Proc. ICASSP, pp.  5809–5813, 2018.
  47. Investigation of sequence-level knowledge distillation methods for ctc acoustic models. In Proc. ICASSP, pp.  6156–6160, 2019.
  48. Thomas, S. et al. Towards reducing the need for speech training data to build spoken language understanding systems. In Proc. ICASSP, 2022.
  49. Vaswani, A. et al. Attention is all you need. In Proc. NIPS, pp.  5998–6008, 2017.
  50. Improving self-supervised learning for speech recognition with intermediate layer supervision. In Proc. ICASSP, 2022.
  51. Improving neural language modeling via adversarial training. In Proc. ICML, pp.  6555–6565, 2019.
  52. Watanabe, S. et al. Hybrid ctc/attention architecture for end-to-end speech recognition. IEEE Journal of Selected Topics in Signal Processing, 2017.
  53. Watanabe, S. et al. ESPnet: End-to-end speech processing toolkit. In Proc. INTERSPEECH, pp.  2207–2211, 2018.
  54. Pay less attention with lightweight and dynamic convolutions. In Proc. ICLR, 2019.
  55. Sequence generation with mixed representations. In Proc. ICML, pp.  10388–10398, 2020.
  56. Unidrop: a simple yet effective technique to improve transformer without extra cost. In Proc. NAACL, pp.  3865–3878, 2021.
  57. Bert, mbert, or bibert? a study on contextualized embeddings for neural machine translation. In Proc. EMNLP, pp.  6663–6675, 2021.
  58. Yao, Z. et al. Tessp: text-enhanced self-supervised speech pre-training. arXiv preprint arXiv:2211.13443v1, 2022.
  59. Tutornet: towards flexible knowledge distillation for end-to-end speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1626–1638, 2021.
  60. Yu, J. et al. Fastemit: low-latency streaming asr with sequence-level emission regularization. In Proc. ICASSP, 2021.
  61. Yue, X. et al. Token2vec: a joint self-supervised pre-training framework using unpaired speech and text. In Proc. ICASSP, 2023.
  62. Zhang, Y. et al. Deep mutual learning. In Proc. CVPR, 2018.
  63. Zhang, Y. et al. Pushing the limits of semi-supervised learning for automatic speech recognition. In Proc. NIPS SAS Workshop, 2020a.
  64. Zhang, Z. et al. Distilling knowledge from well-informed soft labels for neural relation extraction. In Proc. AAAI, 2020b.
  65. Zhang, Z. et al. Speechlm: enhanced speech pre-training with unpaired textual data. arXiv preprint arXiv:2209.15329v1, 2022.
  66. Incorporating bert into neural machine translation. In Proc. ICLR, 2020.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up