Scaling Transformers for Low-Bitrate High-Quality Speech Coding (2411.19842v1)

Abstract: The tokenization of speech with neural audio codec models is a vital part of modern AI pipelines for the generation or understanding of speech, alone or in a multimodal context. Traditionally such tokenization models have concentrated on low parameter-count architectures using only components with strong inductive biases. In this work we show that by scaling a transformer architecture with large parameter count to this problem, and applying a flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to reach state-of-the-art speech quality at extremely low bit-rates of $400$ or $700$ bits-per-second. The trained models strongly out-perform existing baselines in both objective and subjective tests.

• The paper introduces a transformer-based codec with 1B parameters that achieves high-fidelity speech coding at extremely low bitrates.
• It employs Finite Scalar Quantization to optimize codebook utilization, overcoming limitations of conventional RVQ methods.
• Experimental evaluations demonstrate significant improvements in SI-SDR, Mel distances, and perceptual quality, highlighting its potential for real-time applications.

Overview of "Scaling Transformers for Low-Bitrate High-Quality Speech Coding"

The paper presents a novel approach to speech coding that focuses on leveraging transformer architectures to achieve high-quality audio compression at extremely low bitrates, notably at 400 and 700 bits-per-second (bps). This paper is significant due to its emphasis on scaling the codec model's parameters into the billion range, surpassing typical convolutional neural networks' parameter counts in audio coding.

Key Contributions and Methodology

1. Transformer-Based Codec Architecture: The paper introduces a codec architecture that primarily employs transformers rather than convolutional networks, thereby extending the model's scalability to 1 billion parameters. This shift addresses the limitation of previous codec designs constrained by considerably lower parameter counts.
2. Finite Scalar Quantization (FSQ): The authors implement a bottleneck strategy using FSQ instead of the conventional Residual Vector Quantizer (RVQ). This approach mitigates the issues related to codebook utilization and hierarchical token stream complexities, enhancing both the performance and reliability of the quantization process.
3. Objective and Subjective Performance: Empirical tests demonstrate that the proposed model substantially outperforms existing neural audio codecs (NACs) in both objective metrics and human-perceived quality, as evidenced by MUSHRA listening tests. The model achieves state-of-the-art performance with lower latency and fewer tokens per second, revealing its efficiency and effectiveness in high-fidelity audio reconstruction.
4. Training and Optimization Strategies: The model is trained over 105,000 hours of English speech data, employing sophisticated training procedures, including perceptual loss tuning using WavLM features and a feature-matching loss with a multi-resolution discriminator. These methods ensure robust convergence and high-quality audio output.

Key Numerical Results and Enhancements

• The FSQ-based approach achieves nearly optimum codebook utilization and entropy coding efficiency, as reflected in normalized entropy scores close to 0.97. In comparison, baseline models demonstrate much lower utilization efficiencies.
• Comparatively, the model shows superior performance in terms of Scale-Invariant Signal-to-Distortion Ratios (SI-SDR), Mel distances, and PESQ scores, indicating a considerable enhancement in both reconstruction fidelity and perceived audio quality.

Implications and Future Directions

The implications of deploying such transformer-based architectures for speech codecs are multifaceted:

• Scalability and Efficiency: The findings indicate that transformer architectures, albeit less parameter-efficient than convolutional counterparts, offer a path toward scalable and efficient audio coding solutions. This shift could potentially redefine the paradigm in generative audio modeling, where speech-first applications necessitate robust and scalable codecs.
• Streaming and Real-Time Processing: Despite its large parameter count, the proposed model can be adapted for causal configurations, enabling real-time streaming capabilities with minimal latency. This suitability could be pivotal in applications like live speech transmission and in-device processing.

Moving forward, it is anticipated that training models on more extensive and diverse datasets, including multilingual and higher sampling rate data, will catalyze further improvements in audio coding techniques. Moreover, integrating these codecs with advanced generative models could push the boundaries of what is achievable in neural audio synthesis and compression. Exploring deeper into efficient transformer resource utilization will unlock more practical implementations across various hardware and application environments.

In conclusion, while the proposed approach showcases substantial promise for the future of neural audio coding, it also sets the stage for subsequent research to explore enhanced architectural choices and more comprehensive datasets to achieve even greater advancements in audio processing capabilities.