Papers
Topics
Authors
Recent
Search
2000 character limit reached

Amphion: An Open-Source Audio, Music and Speech Generation Toolkit

Published 15 Dec 2023 in cs.SD and eess.AS | (2312.09911v3)

Abstract: Amphion is an open-source toolkit for Audio, Music, and Speech Generation, targeting to ease the way for junior researchers and engineers into these fields. It presents a unified framework that includes diverse generation tasks and models, with the added bonus of being easily extendable for new incorporation. The toolkit is designed with beginner-friendly workflows and pre-trained models, allowing both beginners and seasoned researchers to kick-start their projects with relative ease. The initial release of Amphion v0.1 supports a range of tasks including Text to Speech (TTS), Text to Audio (TTA), and Singing Voice Conversion (SVC), supplemented by essential components like data preprocessing, state-of-the-art vocoders, and evaluation metrics. This paper presents a high-level overview of Amphion. Amphion is open-sourced at https://github.com/open-mmlab/Amphion.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (62)
  1. Tacotron: Towards end-to-end speech synthesis. Interspeech, 2017.
  2. Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions. In ICASSP, pages 4779–4783, 2018.
  3. FastSpeech: Fast, robust and controllable text to speech. In NeurIPS, volume 32, 2019.
  4. FastSpeech 2: Fast and high-quality end-to-end text to speech. In ICLR, 2020.
  5. End-to-end adversarial text-to-speech. arXiv preprint arXiv:2006.03575, 2020.
  6. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In ICML, volume 139, pages 5530–5540, 2021.
  7. Neural codec language models are zero-shot text to speech synthesizers. arXiv preprint arXiv:2301.02111, 2023.
  8. Naturalspeech 2: Latent diffusion models are natural and zero-shot speech and singing synthesizers. arXiv preprint arXiv:2304.09116, 2023.
  9. WeNet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. In INTERSPEECH, pages 4054–4058, 2021.
  10. Robust speech recognition via large-scale weak supervision. In ICML, volume 202, pages 28492–28518, 2023.
  11. ContentVec: An improved self-supervised speech representation by disentangling speakers. In ICML, volume 162, pages 18003–18017, 2022.
  12. Leveraging content-based features from multiple acoustic models for singing voice conversion. In Machine Learning for Audio Workshop, NeurIPS 2023, 2023.
  13. WaveNet: A generative model for raw audio. In Speech Synthesis Workshop, page 125, 2016.
  14. DiffWave: A versatile diffusion model for audio synthesis. In ICLR, 2021.
  15. DiffSVC: A diffusion probabilistic model for singing voice conversion. In ASRU, pages 741–748, 2021.
  16. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  17. AudioGen: Textually guided audio generation. In ICLR, 2022.
  18. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661, 2023.
  19. AudioLDM: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023.
  20. AudioLDM 2: Learning holistic audio generation with self-supervised pretraining. arXiv preprint arXiv:2308.05734, 2023.
  21. AUDIT: Audio editing by following instructions with latent diffusion models. arXiv preprint arXiv:2304.00830, 2023.
  22. Text-to-text pre-training for data-to-text tasks. In INLG, pages 97–102, 2020.
  23. Efficient neural audio synthesis. In ICML, volume 80, pages 2415–2424, 2018.
  24. Waveglow: A flow-based generative network for speech synthesis. In ICASSP, pages 3617–3621, 2019.
  25. Waveflow: A compact flow-based model for raw audio. In ICML, volume 119, pages 7706–7716, 2020.
  26. Dspgan: a gan-based universal vocoder for high-fidelity tts by time-frequency domain supervision from dsp. In ICASSP, pages 1–5. IEEE, 2023.
  27. Fre-GAN: Adversarial frequency-consistent audio synthesis. In INTERSPEECH, pages 2197–2201, 2021.
  28. MelGAN: Generative adversarial networks for conditional waveform synthesis. In NeurIPS, pages 14881–14892, 2019.
  29. Bigvgan: A universal neural vocoder with large-scale training. In ICLR, 2023.
  30. Hifi-gan: High-fidelity denoising and dereverberation based on speech deep features in adversarial networks. In INTERSPEECH, pages 4506–4510, 2020.
  31. Source-filter hifi-gan: Fast and pitch controllable high-fidelity neural vocoder. In ICASSP, pages 1–5, 2023.
  32. SingGAN: Generative adversarial network for high-fidelity singing voice generation. In ACM Multimedia, pages 2525–2535, 2022.
  33. Parallel Wavegan: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram. In ICASSP, pages 6199–6203, 2020.
  34. Universal MelGAN: A robust neural vocoder for high-fidelity waveform generation in multiple domains. arXiv, abs/2011.09631, 2020.
  35. APNet: An all-frame-level neural vocoder incorporating direct prediction of amplitude and phase spectra. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:2145–2157, 2023.
  36. WaveGrad: Estimating gradients for waveform generation. In ICLR, 2021.
  37. Neural source-filter-based waveform model for statistical parametric speech synthesis. In ICASSP, pages 5916–5920. IEEE, 2019.
  38. Neural networks fail to learn periodic functions and how to fix it. In NeurIPS, volume 33, pages 1583–1594, 2020.
  39. High fidelity neural audio compression. arXiv, abs/2210.13438, 2022.
  40. Multi-scale sub-band constant-q transform discriminator for high-fidelity vocoder. arXiv preprint arXiv:2311.14957, 2023.
  41. Chunked Autoregressive GAN for conditional waveform synthesis. In ICLR, 2022.
  42. Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs. In ICASSP, pages 749–752, 2001.
  43. A short-time objective intelligibility measure for time-frequency weighted noisy speech. In ICASSP, pages 4214–4217. IEEE, 2010.
  44. Fréchet Audio Distance: A reference-free metric for evaluating music enhancement algorithms. In Gernot Kubin and Zdravko Kacic, editors, Interspeech, pages 2350–2354, 2019.
  45. Robert Kubichek. Mel-cepstral distance measure for objective speech quality assessment. In Proceedings of IEEE Pacific Rim Conference on Communications Computers and Signal Processing, volume 1, pages 125–128, 1993.
  46. Yi Luo and Nima Mesgarani. TaSNet: Time-domain audio separation network for real-time, single-channel speech separation. In ICASSP, pages 696–700, 2018.
  47. SDR - half-baked or well done? In ICASSP, pages 626–630, 2019.
  48. Pushing the limits of raw waveform speaker recognition. In INTERSPEECH, pages 2228–2232, 2022.
  49. Wespeaker: A research and production oriented speaker embedding learning toolkit. In ICASSP, pages 1–5, 2023.
  50. AudioCaps: Generating captions for audios in the wild. In NAACL-HLT, pages 119–132, 2019.
  51. Children’s song dataset for singing voice research. In ISMIR, 2020.
  52. LibriTTS: A corpus derived from librispeech for text-to-speech. In INTERSPEECH, pages 1526–1530, 2019.
  53. M4singer: A multi-style, multi-singer and musical score provided mandarin singing corpus. In NeurIPS, 2022.
  54. The NUS sung and spoken lyrics corpus: A quantitative comparison of singing and speech. In APSIPA, pages 1–9, 2013.
  55. Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis. In INTERSPEECH, pages 4242–4246. ISCA, 2022.
  56. Multi-singer: Fast multi-singer singing voice vocoder with A large-scale corpus. In ACM Multimedia, pages 3945–3954, 2021.
  57. Automatic identification of emotional cues in chinese opera singing. In ICMPC, 2014.
  58. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. In AAAI, pages 11020–11028. AAAI Press, 2022.
  59. PJS: phoneme-balanced japanese singing-voice corpus. In APSIPA, pages 487–491, 2020.
  60. The singing voice conversion challenge 2023. arXiv, abs/2306.14422, 2023.
  61. CSTR VCTK Corpus: English multi-speaker corpus for cstr voice cloning toolkit (version 0.92). University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
  62. PANNs: Large-scale pretrained audio neural networks for audio pattern recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2880–2894, 2020.
Citations (15)
View on Semantic Scholar

Summary

  • The paper introduces Amphion as a unified open-source toolkit for audio, music, and speech generation tasks.
  • It provides a beginner-friendly, stage-wise workflow integrating preprocessing, training, and interactive visualizations.
  • Experiments show competitive performance using models like VITS, FastSpeech2, AudioLDM, and DiffWaveNetSVC across tasks.

Amphion: Advancing Research in Audio, Music, and Speech Generation

The research paper introduces Amphion, an open-source toolkit designed to facilitate and streamline research in the domains of audio, music, and speech generation. Amphion addresses the challenges faced by researchers, particularly those new to the field, in navigating the scattered landscape of model implementations and the varying quality of open-source resources.

Amphion provides a unified framework that supports a wide range of tasks, including Text to Speech (TTS), Text to Audio (TTA), and Singing Voice Conversion (SVC). The toolkit's design integrates end-to-end workflows and pre-trained models aimed at helping both novice and seasoned researchers commence their projects more efficiently. This initiative is intended to address reproducibility challenges in research, promote consistency across various implementations, and provide a platform for fair model comparisons.

Technical Contributions

Unified Framework: Amphion offers a scalable and adaptable framework accommodating disparate tasks related to audio waveforms. It categorizes these into Text to Waveform, Descriptive Text to Waveform, and Waveform to Waveform, thus covering a comprehensive range of applications such as TTS, SVS, TTA, and VC. The system architecture emphasizes fair comparison by standardizing preprocessing procedures, model training frameworks, and vocoders.

Beginner-Friendly Workflow: The toolkit presents a linear, stage-wise workflow, incorporating data preprocessing, feature extraction, model training, inference, evaluation, and visualization. These stages provide a structured environment for the development of models while integrating efficient methods for handling large datasets using on-the-fly feature extraction and dynamic batching.

Models and Datasets: Amphion encompasses a diverse set of models and vast datasets, supported by pre-trained models to assist researchers in building applications and validating experimental results. The paper discusses the defined standards for releasing pre-trained models, including providing metadata, training configurations, evaluation results, and ethical usage considerations.

Visualization and Interactivity: One of Amphion's significant features is its interactive visualization tools designed for educational purposes. Visualizations such as SingVisio provide detailed insights into SVC diffusion models, demonstrating transformations of audio signals throughout the generation process. Online platforms like Gradio are employed to enable users to experiment interactively with different models.

Experimental Evaluation

The authors evaluate Amphion's efficacy by benchmarking its performance across several tasks. In the TTS domain, Amphion's implementation of VITS and FastSpeech2 show competent results compared to popular systems like Coqui TTS and SpeechBrain. For TTA, the AudioLDM model demonstrates performance comparable to state-of-the-art models, while the DiffWaveNetSVC shows improved naturalness and similarity for SVC tasks. The toolkit’s HiFi-GAN vocoder exhibits strong capabilities in reconstructing high-fidelity audio with improved spectrogram distortion metrics.

Implications and Future Directions

Amphion aims to refine reproducible research in the audio generation field by providing an all-encompassing, open-source solution. Its structured approach and extensive resources make it a valuable tool for advancing both theoretical research and practical applications. The toolkit’s future developments include expanding its support for additional audio generation tasks, furthering the scope of large-scale pre-trained models, and maintaining its adaptability to integrate evolving technologies.

In summary, Amphion stands as a comprehensive resource that addresses many of the systemic challenges in audio, music, and speech generation research. By simplifying the entry barrier for new researchers and promoting transparent, reproducible science, Amphion may significantly contribute to the pace and quality of innovation in this multidisciplinary field.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 30 likes about this paper.

Sign Up for Free