Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

APCodec: A Neural Audio Codec with Parallel Amplitude and Phase Spectrum Encoding and Decoding (2402.10533v2)

Published 16 Feb 2024 in cs.SD and eess.AS

Abstract: This paper introduces a novel neural audio codec targeting high waveform sampling rates and low bitrates named APCodec, which seamlessly integrates the strengths of parametric codecs and waveform codecs. The APCodec revolutionizes the process of audio encoding and decoding by concurrently handling the amplitude and phase spectra as audio parametric characteristics like parametric codecs. It is composed of an encoder and a decoder with the modified ConvNeXt v2 network as the backbone, connected by a quantizer based on the residual vector quantization (RVQ) mechanism. The encoder compresses the audio amplitude and phase spectra in parallel, amalgamating them into a continuous latent code at a reduced temporal resolution. This code is subsequently quantized by the quantizer. Ultimately, the decoder reconstructs the audio amplitude and phase spectra in parallel, and the decoded waveform is obtained by inverse short-time Fourier transform. To ensure the fidelity of decoded audio like waveform codecs, spectral-level loss, quantization loss, and generative adversarial network (GAN) based loss are collectively employed for training the APCodec. To support low-latency streamable inference, we employ feed-forward layers and causal deconvolutional layers in APCodec, incorporating a knowledge distillation training strategy to enhance the quality of decoded audio. Experimental results confirm that our proposed APCodec can encode 48 kHz audio at bitrate of just 6 kbps, with no significant degradation in the quality of the decoded audio. At the same bitrate, our proposed APCodec also demonstrates superior decoded audio quality and faster generation speed compared to well-known codecs, such as Encodec, AudioDec and DAC.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. K. Brandenburg and G. Stoll, “ISO/MPEG-1 audio: A generic standard for coding of high-quality digital audio,” Journal of the Audio Engineering Society, vol. 42, no. 10, pp. 780–792, 1994.
  2. T. Tremain, “Linear predictive coding systems,” in Proc. ICASSP, vol. 1, 1976, pp. 474–478.
  3. P. Kroon, E. Deprettere, and R. Sluyter, “Regular-pulse excitation–a novel approach to effective and efficient multipulse coding of speech,” IEEE transactions on acoustics, speech, and signal processing, vol. 34, no. 5, pp. 1054–1063, 1986.
  4. R. Salami, C. Laflamme, J.-P. Adoul, and D. Massaloux, “A toll quality 8 kb/s speech codec for the personal communications system (pcs),” IEEE Transactions on Vehicular Technology, vol. 43, no. 3, pp. 808–816, 1994.
  5. Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi et al., “AudioLM: A language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023.
  6. C. Wang, S. Chen, Y. Wu, Z. Zhang, L. Zhou, S. Liu, Z. Chen, Y. Liu, H. Wang, J. Li et al., “Neural codec language models are zero-shot text to speech synthesizers,” arXiv preprint arXiv:2301.02111, 2023.
  7. X. Zhang, D. Zhang, S. Li, Y. Zhou, and X. Qiu, “SpeechTokenizer: Unified speech tokenizer for speech large language models,” arXiv preprint arXiv:2308.16692, 2023.
  8. Z. Huang, C. Meng, and T. Ko, “RepCodec: A speech representation codec for speech tokenization,” arXiv preprint arXiv:2309.00169, 2023.
  9. Y. Ren, T. Wang, J. Yi, L. Xu, J. Tao, C. Zhang, and J. Zhou, “Fewer-token neural speech codec with time-invariant codes,” arXiv preprint arXiv:2310.00014, 2023.
  10. D. O’Shaughnessy, “Linear predictive coding,” IEEE potentials, vol. 7, no. 1, pp. 29–32, 1988.
  11. J.-M. Valin, G. Maxwell, T. B. Terriberry, and K. Vos, “High-quality, low-delay music coding in the opus codec,” in Audio Engineering Society Convention 135, 2013.
  12. M. Dietz, M. Multrus, V. Eksler, V. Malenovsky, E. Norvell, H. Pobloth, L. Miao, Z. Wang, L. Laaksonen, A. Vasilache et al., “Overview of the EVS codec architecture,” in Proc. ICASSP, 2015, pp. 5698–5702.
  13. W. B. Kleijn, F. S. Lim, A. Luebs, J. Skoglund, F. Stimberg, Q. Wang, and T. C. Walters, “WaveNet based low rate speech coding,” in Proc. ICASSP, 2018, pp. 676–680.
  14. J. Klejsa, P. Hedelin, C. Zhou, R. Fejgin, and L. Villemoes, “High-quality speech coding with sample RNN,” in Proc. ICASSP, 2019, pp. 7155–7159.
  15. J.-M. Valin and J. Skoglund, “A real-time wideband neural vocoder at 1.6 kb/s using LPCNet,” in Proc. Interspeech, 2019, pp. 3406–3410.
  16. A. Mustafa, J. Büthe, S. Korse, K. Gupta, G. Fuchs, and N. Pia, “A streamwise GAN vocoder for wideband speech coding at very low bit rate,” in Proc. WASPAA, 2021, pp. 66–70.
  17. Y. Zheng, L. Xiao, W. Tu, Y. Yang, and X. Xu, “CQNV: A combination of coarsely quantized bitstream and neural vocoder for low rate speech coding,” arXiv preprint arXiv:2307.13295, 2023.
  18. G. Davidson, M. Vinton, P. Ekstrand, C. Zhou, L. Villemoes, and L. Lu, “High quality audio coding with MDCTNet,” in Proc. ICASSP, 2023, pp. 1–5.
  19. H. Lim, J. Lee, B. H. Kim, I. Jang, and H.-G. Kang, “End-to-end neural audio coding in the MDCT domain,” in Proc. ICASSP, 2023, pp. 1–5.
  20. H. S. Black and J. Edson, “Pulse code modulation,” Transactions of the American Institute of Electrical Engineers, vol. 66, no. 1, pp. 895–899, 1947.
  21. S. Kankanahalli, “End-to-end optimized speech coding with deep neural networks,” in Proc. ICASSP, 2018, pp. 2521–2525.
  22. A. Van Den Oord, O. Vinyals et al., “Neural discrete representation learning,” in Proc. NIPS, vol. 30, 2017.
  23. C. Gârbacea, A. van den Oord, Y. Li, F. S. Lim, A. Luebs, O. Vinyals, and T. C. Walters, “Low bit-rate speech coding with VQ-VAE and a WaveNet decoder,” in Proc. ICASSP, 2019, pp. 735–739.
  24. K. Zhen, J. Sung, M. S. Lee, S. Beack, and M. Kim, “Cascaded cross-module residual learning towards lightweight end-to-end speech coding,” 2019, pp. 3396–3400.
  25. N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 495–507, 2021.
  26. A. Défossez, J. Copet, G. Synnaeve, and Y. Adi, “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022.
  27. A. Vasuki and P. Vanathi, “A review of vector quantization techniques,” IEEE Potentials, vol. 25, no. 4, pp. 39–47, 2006.
  28. J. Kong, J. Kim, and J. Bae, “HiFi-GAN: Generative adversarial networks for efficient and high fidelity speech synthesis,” in Proc. NIPS, vol. 33, 2020, pp. 17 022–17 033.
  29. D. Yang, S. Liu, R. Huang, J. Tian, C. Weng, and Y. Zou, “HiFi-Codec: Group-residual vector quantization for high fidelity audio codec,” arXiv preprint arXiv:2305.02765, 2023.
  30. L. Xu, J. Jiang, D. Zhang, X. Xia, L. Chen, Y. Xiao, P. Ding, S. Song, S. Yin, and F. Sohel, “An intra-BRNN and GB-RVQ based end-to-end neural audio codec,” in Proc. Interspeech, 2023, pp. 800–803.
  31. T. Jenrungrot, M. Chinen, W. B. Kleijn, J. Skoglund, Z. Borsos, N. Zeghidour, and M. Tagliasacchi, “LMCodec: A low bitrate speech codec with causal transformer models,” in Proc. ICASSP, 2023, pp. 1–5.
  32. W. Xiao, W. Liu, M. Wang, S. Yang, Y. Shi, Y. Kang, D. Su, S. Shang, and D. Yu, “Multi-mode neural speech coding based on deep generative networks,” in Proc. Interspeech, 2023, pp. 819–823.
  33. Y.-C. Wu, I. D. Gebru, D. Marković, and A. Richard, “AudioDec: An open-source streaming high-fidelity neural audio codec,” in Proc. ICASSP, 2023, pp. 1–5.
  34. J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
  35. S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie, “ConvNeXt v2: Co-designing and scaling convnets with masked autoencoders,” in Proc. CVPR, 2023, pp. 16 133–16 142.
  36. D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” arXiv preprint arXiv:1606.08415, 2016.
  37. Y. Ai and Z.-H. Ling, “Neural speech phase prediction based on parallel estimation architecture and anti-wrapping losses,” in Proc. ICASSP, 2023, pp. 1–5.
  38. Y. Ai and Z.-H. Ling, “APNet: An all-frame-level neural vocoder incorporating direct prediction of amplitude and phase spectra,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2145–2157, 2023.
  39. W. Jang, D. Lim, J. Yoon, B. Kim, and J. Kim, “Univnet: A neural vocoder with multi-resolution spectrogram discriminators for high-fidelity waveform generation,” arXiv preprint arXiv:2106.07889, 2021.
  40. A. L. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities improve neural network acoustic models,” in Proc. ICML, vol. 30, no. 1, 2013, p. 3.
  41. K. Kumar, R. Kumar, T. de Boissiere, L. Gestin, W. Z. Teoh, J. Sotelo, A. de Brébisson, Y. Bengio, and A. C. Courville, “MelGAN: Generative adversarial networks for conditional waveform synthesis,” Advances in neural information processing systems, vol. 32, 2019.
  42. J. Yamagishi, C. Veaux, K. MacDonald et al., “CSTR VCTK corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92),” University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2019.
  43. I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2018.
  44. C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, “A short-time objective intelligibility measure for time-frequency weighted noisy speech,” in Proc. ICASSP, 2010, pp. 4214–4217.
  45. M. Chinen, F. S. Lim, J. Skoglund, N. Gureev, F. O’Gorman, and A. Hines, “ViSQOL v3: An open source production ready objective speech and audio metric,” in Proc. QoMEX, 2020, pp. 1–6.
  46. Y.-X. Lu, Y. Ai, H.-P. Du, and Z.-H. Ling, “Towards high-quality and efficient speech bandwidth extension with parallel amplitude and phase prediction,” arXiv preprint arXiv:2401.06387, 2024.
  47. Y. Wang, X. Wang, P. Zhu, J. Wu, H. Li, H. Xue, Y. Zhang, L. Xie, and M. Bi, “Opencpop: A high-quality open source chinese popular song corpus for singing voice synthesis,” arXiv preprint arXiv:2201.07429, 2022.
  48. E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “FSD50K: An open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 829–852, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yang Ai (41 papers)
  2. Xiao-Hang Jiang (6 papers)
  3. Ye-Xin Lu (17 papers)
  4. Hui-Peng Du (15 papers)
  5. Zhen-Hua Ling (114 papers)
Citations (12)
View on Semantic Scholar