DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation (2405.13274v2)

Abstract: Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) translations on the CVSS benchmark, while attaining over 14x speedup for En-Es and 5x speedup for En-Fr translations compared to autoregressive baselines.

• The paper introduces DiffNorm, a self-supervised diffusion normalization strategy that simplifies speech feature distributions in non-autoregressive translation systems.
• It employs synthetic noise injection and a denoising process to tackle the multi-modality problem, resulting in coherent, high-quality outputs.
• The approach, enhanced by classifier-free guidance, achieves significant ASR-BLEU improvements and inference speed gains over autoregressive models.

Simplifying Non-Autoregressive Speech Translation with DiffNorm

Introduction

In recent years, Non-Autoregressive Transformers (NATs) have shown promise for direct speech-to-speech translation (S2ST), yielding faster inference and maintaining competitive translation quality compared to their autoregressive counterparts. However, NATs struggle with the "multi-modality problem," which results in incoherent and repetitive outputs due to the complexity of speech data distributions.

To address this, a new strategy known as DiffNorm has been introduced. DiffNorm relies on diffusion-based normalization to simplify these data distributions, thus enhancing the performance of NATs. This article will break down the core concepts behind DiffNorm, its implementation, and the resulting benefits.

DiffNorm: A New Approach to Speech Normalization

The Multi-Modality Problem in NATs

NATs can generate high-quality outputs and offer significant speed advantages over autoregressive models. However, they often produce outputs that are incoherent or repetitive. This issue stems from the assumption of conditional independence during parallel decoding, which struggles to capture the complex variations in speech data, such as acoustic and linguistic differences.

Introducing DiffNorm

DiffNorm is a self-supervised strategy based on Denoising Diffusion Probabilistic Models (DDPM). It works by injecting synthetic noise into speech features and then recovering the original features through a denoising process. The denoising objective helps create a simpler and more consistent data distribution, which is crucial for training NAT models effectively.

Here’s how DiffNorm works in a nutshell:

1. Synthetic Noise Injection: Speech features are injected with noise, which creates a corrupted version of the original data.
2. Denoising Process: Using a diffusion model, the system gradually removes the noise to recover the speech features.

By training the system to denoise synthetically corrupted features, DiffNorm normalizes the data. This eliminates the need for transcription data or manually crafted perturbation functions.

Enhancing NATs with Classifier-Free Guidance

In addition to DiffNorm, the researchers proposed a method called classifier-free guidance to regularize NATs. This involves randomly dropping out source information during training, compelling the model to generate coherent outputs even without full context. By doing so, the model becomes more robust and produces higher-quality translations.

Strong Numerical Results

The benefits of DiffNorm and classifier-free guidance are highlighted through strong numerical results:

• English-Spanish (En-Es) Translation: Around a +7 ASR-BLEU improvement.
• English-French (En-Fr) Translation: Around a +2 ASR-BLEU improvement.
• Inference Speed: Achieving over 14× speedup for En-Es and 5× speedup for En-Fr compared to autoregressive baselines.

Implications and Future Directions

The findings have both practical and theoretical implications. On a practical level, the improvements in speed and accuracy make direct speech-to-speech translation systems more viable for real-world applications. Theoretically, the use of diffusion models and classifier-free guidance for NATs opens up new avenues for future research in AI and machine learning.

In the future, we can anticipate further developments that refine these techniques, making them even more efficient and accurate. This research lays the groundwork for more sophisticated speech translation systems that could become ubiquitous in various communication and accessibility applications.

Conclusion

DiffNorm and classifier-free guidance offer a promising solution to the multi-modality problem in NATs, resulting in significant performance improvements for direct speech-to-speech translation. These advancements not only enhance the efficiency and accuracy of these systems but also pave the way for future innovations in the field. If you’re intrigued and want to delve deeper, you can check out the full research and implementation details on GitHub.