Diffsound: Discrete Diffusion Model for Text-to-sound Generation (2207.09983v2)

Abstract: Generating sound effects that humans want is an important topic. However, there are few studies in this area for sound generation. In this study, we investigate generating sound conditioned on a text prompt and propose a novel text-to-sound generation framework that consists of a text encoder, a Vector Quantized Variational Autoencoder (VQ-VAE), a decoder, and a vocoder. The framework first uses the decoder to transfer the text features extracted from the text encoder to a mel-spectrogram with the help of VQ-VAE, and then the vocoder is used to transform the generated mel-spectrogram into a waveform. We found that the decoder significantly influences the generation performance. Thus, we focus on designing a good decoder in this study. We begin with the traditional autoregressive decoder, which has been proved as a state-of-the-art method in previous sound generation works. However, the AR decoder always predicts the mel-spectrogram tokens one by one in order, which introduces the unidirectional bias and accumulation of errors problems. Moreover, with the AR decoder, the sound generation time increases linearly with the sound duration. To overcome the shortcomings introduced by AR decoders, we propose a non-autoregressive decoder based on the discrete diffusion model, named Diffsound. Specifically, the Diffsound predicts all of the mel-spectrogram tokens in one step and then refines the predicted tokens in the next step, so the best-predicted results can be obtained after several steps. Our experiments show that our proposed Diffsound not only produces better text-to-sound generation results when compared with the AR decoder but also has a faster generation speed, e.g., MOS: 3.56 \textit{v.s} 2.786, and the generation speed is five times faster than the AR decoder.

An Overview of "Diffsound: Discrete Diffusion Model for Text-to-sound Generation"

Recent advances in artificial intelligence have pushed the boundaries in various domains, including computer vision, natural language processing, and audio generation. "Diffsound: Discrete Diffusion Model for Text-to-sound Generation" introduces a novel framework that addresses the challenge of generating sounds from textual input—a task with significant implications for applications such as audio effects in virtual reality or constructing auditory scenes in film and music production.

Framework Overview

The text-to-sound generation framework proposed in this paper comprises several key components: a text encoder, VQ-VAE (Vector Quantized Variational Autoencoder), a token-decoder, and a vocoder. The main task is to transform textual descriptions into mel-spectrogram representations, and subsequently into audio waveforms. The text encoder utilizes pre-trained models like BERT or CLIP to extract relevant features from the text input. These features are then processed through a non-autoregressive token-decoder dubbed “Diffsound,” which utilizes a discrete diffusion model to enhance generation performance by predicting mel-spectrogram tokens efficiently.

A crucial novelty here lies in the non-autoregressive diffusion model. Traditional autoregressive (AR) models, although effective, often suffer from accumulated prediction errors and limited flexibility due to their stepwise prediction nature.

Diffusion Model Advancements

The paper builds on discrete diffusion models, wherein the process of generating mel-spectrogram tokens collaboratively accounts for both past and future context by refining token predictions iteratively. This contrasts with autoregressive models that predict tokens sequentially, potentially limiting generative performance due to structural biases and error propagation.

Significantly, results demonstrate the Diffsound model yields substantial improvements over autoregressive approaches. Quantitative measures include a Mean Opinion Score (MOS) increase to 3.56 compared to 2.786 from the autoregressive baseline. Moreover, the framework accelerates sound generation, achieving up to five-fold faster generation speeds while maintaining or improving the quality of output, noted in objective metrics like FID and KL divergence.

Pre-training Strategies

An innovative aspect of this research is utilizing large-scale datasets such as AudioSet for pre-training. By leveraging a Mask-based Text Generation (MBTG) strategy, the authors simulate text-audio datasets, thus augmenting resources needed for model training. This approach appears to be effective, as evidenced by improved model performance when pre-training is incorporated.

Evaluation and Implications

Evaluation metrics include human-based MOS ratings and objective metrics like SPICE and CIDEr scores, traditionally used in a captioning context, which have been adapted to assess audio diversity and semantic alignment here. This multi-faceted evaluation strongly backs the model's relevance to text prompts and quality of generated sounds.

The theoretical implications of this research extend into potential advancements in discrete diffusion applications beyond text-to-sound generation, such as video-audio translation, multimodal representation learning, and understanding the role of diffusion processes in generative modeling.

In practical terms, the Diffsound model's efficient architecture opens avenues for real-time audio generation applications, an aspect increasingly crucial as VR and AR technologies continue to evolve. By bridging textual and auditory domains more effectively, such frameworks enhance machine interaction capabilities, leading to improved user experiences across interactive mediums.

Moving forward, while the current framework operates in a staged fashion—training VQ-VAE, decoders, and vocoders independently—an end-to-end approach could potentially harmonize these processes, optimizing performance and reducing training overhead. Exploring such architectures represents a promising direction for future research.