FoleyGen: Visually-Guided Audio Generation (2309.10537v1)
Abstract: Recent advancements in audio generation have been spurred by the evolution of large-scale deep learning models and expansive datasets. However, the task of video-to-audio (V2A) generation continues to be a challenge, principally because of the intricate relationship between the high-dimensional visual and auditory data, and the challenges associated with temporal synchronization. In this study, we introduce FoleyGen, an open-domain V2A generation system built on a LLMing paradigm. FoleyGen leverages an off-the-shelf neural audio codec for bidirectional conversion between waveforms and discrete tokens. The generation of audio tokens is facilitated by a single Transformer model, which is conditioned on visual features extracted from a visual encoder. A prevalent problem in V2A generation is the misalignment of generated audio with the visible actions in the video. To address this, we explore three novel visual attention mechanisms. We further undertake an exhaustive evaluation of multiple visual encoders, each pretrained on either single-modal or multi-modal tasks. The experimental results on VGGSound dataset show that our proposed FoleyGen outperforms previous systems across all objective metrics and human evaluations.
- “AudioLDM: Text-to-audio generation with latent diffusion models,” Proceedings of the International Conference on Machine Learning, 2023.
- “AudioGen: Textually guided audio generation,” in The Eleventh International Conference on Learning Representations, 2023.
- “AudioLDM 2: Learning holistic audio generation with self-supervised pretraining,” arXiv preprint arXiv:2308.05734, 2023.
- “Simple and controllable music generation,” arXiv preprint arXiv:2306.05284, 2023.
- “Visually indicated sounds,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
- “Visually indicated sound generation by perceptually optimized classification,” in Proceedings of the European Conference on Computer Vision Workshops, 2018.
- “Visual to sound: Generating natural sound for videos in the wild,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 3550–3558.
- “Generating visually aligned sound from videos,” IEEE Transactions on Image Processing, vol. 29, pp. 8292–8302, 2020.
- “Taming visually guided sound generation,” in British Machine Vision Conference (BMVC), 2021.
- “I hear your true colors: Image guided audio generation,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2023, pp. 1–5.
- “Diff-Foley: Synchronized video-to-audio synthesis with latent diffusion models,” arXiv preprint arXiv:2306.17203, 2023.
- “High fidelity neural audio compression,” arXiv preprint arXiv:2210.13438, 2022.
- “Audiolm: a language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
- “An image is worth 16x16 words: Transformers for image recognition at scale,” in International Conference on Learning Representations, 2021.
- “Learning transferable visual models from natural language supervision,” in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
- “ImageBind: One embedding space to bind them all,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 15180–15190.
- “VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training,” Advances in Neural Information Processing Systems, vol. 35, pp. 10078–10093, 2022.
- “Attention is all you need,” in Advances in Neural Information Processing Systems. 2017, vol. 30, Curran Associates, Inc.
- “VggSound: A large-scale audio-visual dataset,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2020, pp. 721–725.
- “FlashAttention: Fast and memory-efficient exact attention with io-awareness,” Advances in Neural Information Processing Systems, vol. 35, pp. 16344–16359, 2022.
- “Classifier-free diffusion guidance,” arXiv preprint arXiv:2207.12598, 2022.
- “Fréchet audio distance: A metric for evaluating music enhancement algorithms,” arXiv preprint arXiv:1812.08466, 2018.
- “CNN architectures for large-scale audio classification,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2017, pp. 131–135.
- “Efficient training of audio transformers with patchout,” in Interspeech. 2022, pp. 2753–2757, ISCA.
- Xinhao Mei (24 papers)
- Varun Nagaraja (9 papers)
- Zhaoheng Ni (32 papers)
- Ernie Chang (34 papers)
- Yangyang Shi (54 papers)
- Vikas Chandra (75 papers)
- Gael Le Lan (23 papers)