Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SonicVisionLM: Playing Sound with Vision Language Models (2401.04394v3)

Published 9 Jan 2024 in cs.MM, cs.SD, and eess.AS

Abstract: There has been a growing interest in the task of generating sound for silent videos, primarily because of its practicality in streamlining video post-production. However, existing methods for video-sound generation attempt to directly create sound from visual representations, which can be challenging due to the difficulty of aligning visual representations with audio representations. In this paper, we present SonicVisionLM, a novel framework aimed at generating a wide range of sound effects by leveraging vision-LLMs(VLMs). Instead of generating audio directly from video, we use the capabilities of powerful VLMs. When provided with a silent video, our approach first identifies events within the video using a VLM to suggest possible sounds that match the video content. This shift in approach transforms the challenging task of aligning image and audio into more well-studied sub-problems of aligning image-to-text and text-to-audio through the popular diffusion models. To improve the quality of audio recommendations with LLMs, we have collected an extensive dataset that maps text descriptions to specific sound effects and developed a time-controlled audio adapter. Our approach surpasses current state-of-the-art methods for converting video to audio, enhancing synchronization with the visuals, and improving alignment between audio and video components. Project page: https://yusiissy.github.io/SonicVisionLM.github.io/

PDF HTML Abstract

SonicVisionLM: Advancing Video-Sound Generation with Vision-LLMs

The research paper introduces SonicVisionLM, a sophisticated framework designed to generate sound effects for silent videos using Vision-LLMs (VLMs). The primary motivation lies in overcoming the challenges faced by existing methods in aligning visual and audio representations, which is critical for practical applications in video post-production. The proposed framework transforms this alignment challenge by breaking it down into the more manageable tasks of image-to-text and text-to-audio alignment, harnessing the capabilities of LLMs and diffusion models.

Framework Overview

SonicVisionLM distinguishes itself by integrating three pivotal components: a video-to-text converter, a text-based interaction module, and a text-to-audio generation mechanism. This method deviates from direct audio generation from visual data, which often leads to synchronization issues and semantic irrelevance. Instead, by leveraging VLMs, the system first parses a silent video to identify events and suggests possible accompanying sounds. This approach transforms the complex task of video-audio alignment into more tractable subtasks that utilize the strengths of existing natural language processing models.

The key innovation in this framework is the introduction of a time-controlled audio adapter, which facilitates precise synchronization between audio and video events. Additionally, the authors have developed a dataset, CondPromptBank, featuring over ten thousand entries across 23 categories, mapping text descriptions to specific sound effects to enhance the training and performance of their model.

Methodology and Components

  1. Video-to-Text Transformer: This component uses VLMs to produce text descriptions of on-screen events from video inputs. By doing so, it transitions the problem from generating audio directly from video to one of generating text from video.
  2. Timestamp Detection: Trained on a curated dataset, this module employs a ResNet(2+1)-D18 network to predict time-specific information, which is crucial for guiding the audio generation step and ensuring temporal accuracy.
  3. Text-Based Interaction: This module allows for user input to refine or introduce new text-timestamp pairs, effectively adding a level of customizability and adaptability to the sound design process.
  4. Text-to-Audio Generation: Using a Latent Diffusion Model (LDM), this component translates text and timestamps into synchronized, diverse audio outputs. The novel time-controlled adapter is pivotal in aligning the generated audio with the video events accurately.

Experimental Results

SonicVisionLM is rigorously evaluated against state-of-the-art methods, showing superior performance across both conditional and unconditional sound generation tasks. The framework achieves commendable results, notably improving synchronization and semantic accuracy, as evidenced in metrics like CLAP scores and IoU, with notable improvements in conditional generation tasks (e.g., IoU improving from 22.4 to 39.7). These results underscore the model's ability to not only produce audio that matches the targeted visual elements but also maintain precise timing, which is crucial for real-world applications.

Implications and Future Directions

SonicVisionLM highlights the potential of utilizing VLMs in multimedia applications, particularly in automating labor-intensive tasks of video post-production such as sound effect integration. From a practical standpoint, this research offers sound designers a tool for efficiently producing high-quality, synchronized audio, reducing manual effort significantly.

Theoretically, the introduction of time-controlled features into text-audio models opens new avenues for exploring temporal dynamics in multimodal machine learning tasks. Future research could build upon this framework by expanding the complexities of soundscapes it can generate, perhaps integrating AI learning from user edits over time to improve its recommendation engine further.

In summary, SonicVisionLM represents a notable advancement in the domain of video-sound generation, leveraging the capabilities of LLMs to simplify and improve the workflow significantly. The research sets a robust foundation for future explorations in automated audio synthesis from visual content, suggesting exciting possibilities for enhanced multimedia production processes.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (40)
  1. Musiclm: Generating music from text. arXiv preprint arXiv:2301.11325, 2023.
  2. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
  3. Infergrad: Improving diffusion models for vocoder by considering inference in training. In ICASSP, pages 8432–8436. IEEE, 2022a.
  4. Resgrad: Residual denoising diffusion probabilistic models for text to speech. arXiv preprint arXiv:2212.14518, 2022b.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
  6. Simple and controllable music generation. arXiv preprint arXiv:2306.05284, 2023.
  7. Varietysound: Timbre-controllable video to sound generation via unsupervised information disentanglement. In ICASSP, pages 1–5. IEEE, 2023.
  8. Conditional generation of audio from video via foley analogies. In CVPR, pages 2426–2436, 2023.
  9. Text-to-audio generation using instruction-tuned llm and latent diffusion model. arXiv preprint arXiv:2304.13731, 2023.
  10. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NIPS, 30, 2017.
  11. Noise2music: Text-conditioned music generation with diffusion models. arXiv preprint arXiv:2302.03917, 2023a.
  12. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661, 2023b.
  13. Taming visually guided sound generation. arXiv, 2021.
  14. Sparse in space and time: Audio-visual synchronisation with trainable selectors. In The 33st British Machine Vision Virtual Conference. BMVC, 2022.
  15. Fr\\\backslash\’echet audio distance: A metric for evaluating music enhancement algorithms. arXiv preprint arXiv:1812.08466, 2018.
  16. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  17. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  18. Audiogen: Textually guided audio generation. arXiv preprint arXiv:2209.15352, 2022.
  19. Bilateral denoising diffusion models. arXiv preprint arXiv:2108.11514, 2021.
  20. Efficient neural music generation. arXiv preprint arXiv:2305.15719, 2023.
  21. Priorgrad: Improving conditional denoising diffusion models with data-dependent adaptive prior. arXiv preprint arXiv:2106.06406, 2021.
  22. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023.
  23. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12503, 2023.
  24. Diff-foley: Synchronized video-to-audio synthesis with latent diffusion models. arXiv, 2023.
  25. Visually indicated sounds. In CVPR, pages 2405–2413, 2016.
  26. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
  27. Grad-tts: A diffusion probabilistic model for text-to-speech. In ICML, pages 8599–8608. PMLR, 2021.
  28. Fastspeech 2: Fast and high-quality end-to-end text to speech. arXiv preprint arXiv:2006.04558, 2020.
  29. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234–241. Springer, 2015.
  30. Improved techniques for training gans. NIPS, 29, 2016.
  31. I hear your true colors: Image guided audio generation. In ICASSP, pages 1–5. IEEE, 2023.
  32. Naturalspeech: End-to-end text to speech synthesis with human-level quality. arXiv preprint arXiv:2205.04421, 2022.
  33. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  34. A closer look at spatiotemporal convolutions for action recognition. In CVPR, pages 6450–6459, 2018.
  35. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. arXiv preprint arXiv:2305.11175, 2023a.
  36. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023b.
  37. Diffsound: Discrete diffusion model for text-to-sound generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  38. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023a.
  39. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023b.
  40. Repetitive activity counting by sight and sound. In CVPR, pages 14070–14079, 2021.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Zhifeng Xie (11 papers)
  2. Shengye Yu (1 paper)
  3. Mengtian Li (31 papers)
  4. Qile He (2 papers)
Citations (2)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Yusiissy/SonicVisionLM (76 stars)
  2. SonicVisionLM