MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (2410.02130v2)

Abstract: We introduce MDSGen, a novel framework for vision-guided open-domain sound generation optimized for model parameter size, memory consumption, and inference speed. This framework incorporates two key innovations: (1) a redundant video feature removal module that filters out unnecessary visual information, and (2) a temporal-aware masking strategy that leverages temporal context for enhanced audio generation accuracy. In contrast to existing resource-heavy Unet-based models, \texttt{MDSGen} employs denoising masked diffusion transformers, facilitating efficient generation without reliance on pre-trained diffusion models. Evaluated on the benchmark VGGSound dataset, our smallest model (5M parameters) achieves $97.9$% alignment accuracy, using $172\times$ fewer parameters, $371$% less memory, and offering $36\times$ faster inference than the current 860M-parameter state-of-the-art model ($93.9$% accuracy). The larger model (131M parameters) reaches nearly $99$% accuracy while requiring $6.5\times$ fewer parameters. These results highlight the scalability and effectiveness of our approach. The code is available at https://bit.ly/mdsgen.

• The paper introduces MDSGen, which leverages redundant video feature removal and temporal-aware masking to enhance open-domain sound generation.
• It achieves state-of-the-art performance with a 5M parameter model reaching 97.9% alignment accuracy and 36× faster inference.
• The research demonstrates significant reductions in computational cost and memory usage compared to traditional Unet-based diffusion models.

MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation

The research paper introduces MDSGen, a novel framework for vision-guided open-domain sound generation that emphasizes efficiency in terms of model size, memory consumption, and inference speed. The authors present two primary innovations within this framework: a redundant video feature removal module and a temporal-aware masking strategy. These components collectively enhance the accuracy and efficiency of audio generation without the extensive computational requirements typically associated with Unet-based models leveraging diffusion processes.

Key Innovations

1. Redundant Video Feature Removal: The authors identify that existing methods retain extraneous features throughout the generation process. MDSGen effectively condenses video features using a learnable Reducer module, projecting the original dimension down to a single representative vector. This approach not only reduces redundancy but also ensures that the most salient features are prioritized, improving alignment accuracy.
2. Temporal-Aware Masking (TAM): Differing from the spatial-aware masking approaches tailored for image data, the MDSGen framework implements temporal-aware masking strategies that better capture the temporal context intrinsic to audio data. By focusing on temporal dimensions, TAM results in more effective learning of audio sequences, significantly boosting model performance metrics such as FID and IS.

Performance Evaluation

MDSGen is evaluated on the VGGSound dataset, achieving remarkable results. The smallest model, with merely 5 million parameters, delivers a $97.9$\% alignment accuracy, surpassing the current state-of-the-art model that requires 860 million parameters. Remarkably, the model also achieves a $36\times$ faster inference time and requires $171\times$ fewer parameters, and $371$\% less memory usage. The larger variation of MDSGen, with 131 million parameters, further improves alignment accuracy to nearly $99$\%.

Comparative Insights

The paper comprehensively compares MDSGen to other state-of-the-art approaches such as Diff-Foley and existing models like SpecVQGAN and Im2Wav, which show limitations in efficiency and scalability. By transitioning from traditional Unet architectures to a transformer-based approach, the authors demonstrate substantial gains in performance while dramatically reducing computational overheads.

Implications and Future Directions

The implications of this research extend beyond immediate applications in video-to-audio generation. The successful implementation of masked diffusion transformers hints at expansive future developments in AI, particularly in domains demanding efficient resource utilization and rapid inference capabilities. Further exploration into deploying such transformers across varied audio and video synthesis tasks could prove highly beneficial. Potential future work could address challenges related to fixed input lengths, explore larger-scale data for training, and investigate additional applications where diffusion models may reveal previously untapped efficiencies.

Conclusion

MDSGen stands as a testament to the potential of masked diffusion transformers in sound generation, showcasing a balanced approach to improving performance metrics while maintaining efficiency. This work stimulates further research on the applicability of similar strategies to other data types and generative tasks, embodying a significant step forward in the domain of AI-driven audio synthesis.