FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment (2412.15023v3)

Abstract: Traditional sound design workflows rely on manual alignment of audio events to visual cues, as in Foley sound design, where everyday actions like footsteps or object interactions are recreated to match the on-screen motion. This process is time-consuming, difficult to scale, and lacks automation tools that preserve creative intent. Despite recent advances in vision-to-audio generation, producing temporally coherent and semantically controllable sound effects from video remains a major challenge. To address these limitations, we introduce FolAI, a two-stage generative framework that decouples the when and the what of sound synthesis, i.e., the temporal structure extraction and the semantically guided generation, respectively. In the first stage, we estimate a smooth control signal from the video that captures the motion intensity and rhythmic structure over time, serving as a temporal scaffold for the audio. In the second stage, a diffusion-based generative model produces sound effects conditioned both on this temporal envelope and on high-level semantic embeddings, provided by the user, that define the desired auditory content (e.g., material or action type). This modular design enables precise control over both timing and timbre, streamlining repetitive tasks while preserving creative flexibility in professional Foley workflows. Results on diverse visual contexts, such as footstep generation and action-specific sonorization, demonstrate that our model reliably produces audio that is temporally aligned with visual motion, semantically consistent with user intent, and perceptually realistic. These findings highlight the potential of FolAI as a controllable and modular solution for scalable, high-quality Foley sound synthesis in professional and interactive settings. Supplementary materials are accessible on our dedicated demo page at https://ispamm.github.io/FolAI.

• The paper introduces Stable-V2A, a novel two-stage framework using an RMS-Mapper for temporal guidance and Stable-Foley for diffusion-based audio synthesis conditioned on video and text.
• Evaluations on V2A datasets show Stable-V2A significantly outperforms prior models in temporal alignment and audio realism, improving metrics like E-L1 and FAD.
• Stable-V2A offers sound designers precise temporal and semantic control, automating tasks and opening creative possibilities in film, gaming, and VR.

Stable-V2A: Synthesis of Synchronized Sound Effects with Temporal and Semantic Controls

The paper presents "Stable-V2A," a novel framework for the Video-to-Audio (V2A) synthesis task. It proposes an innovative solution harnessing state-of-the-art audio synthesis techniques, focusing on generating semantically and temporally aligned audio for video inputs, aimed primarily at aiding sound designers and Foley artists. Stable-V2A introduces a two-stage model featuring an RMS-Mapper and Stable-Foley, promising enhancements over existing V2A models in both synchronization and sound quality.

Core Contribution and Methodology

The Stable-V2A model's architecture innovatively splits the task into two stages:

1. RMS-Mapper: This stage is tasked with mapping an envelope from a video, which serves as a temporal guide for audio synthesis. It uses frames and optical flow information, processed through TC-CLIP and RegNet-based modules, to estimate an RMS envelope representative of audio features. This step transforms the challenge of synchronizing video and audio with traditionally disparate resolutions into a more manageable classification problem through amplitude quantization and μ-law encoding.
2. Stable-Foley: Building upon Stable Audio Open, this module generates the final audio output, integrating temporal controls from the RMS-Mapper via a ControlNet—a first for such applications with Diffusion Transformers. Semantic alignment is achieved by conditioning the audio generation on embeddings from the CLAP model alongside frame representations from the CAVP video encoder.

Results and Evaluation

The model was evaluated on the Greatest Hits dataset, a standard in the V2A domain, and a new dataset called Walking The Maps, which was introduced to validate footstep sound synthesis. The results reveal that Stable-V2A significantly surpasses other state-of-the-art models, particularly in metrics assessing temporal alignment and audio realism. For instance, Stable-V2A achieves notable improvements in E-L1 and FAD metrics, underscoring its effectiveness in producing high-fidelity, well-synchronized audio tracks. Importantly, the model offers a high level of control over the temporal characteristics of the sounds, an aspect critical for creative applications in sound design.

Implications and Future Directions

The implications of this research are broad. Practically, Stable-V2A provides a tool not only capable of automating repetitive aspects of sound design but also allowing nuanced control over the output, empowering creatives to focus on artistic aspects. As the model supports both audio and text prompting, it offers extensive flexibility in usage scenarios, spanning film, gaming, and virtual reality applications.

Theoretically, Stable-V2A sets a promising precedent for the use of latent diffusion models in multimodal synthesis tasks, exploiting advancements in neural network architectures that allow for precise cross-modal alignment. The paper highlights the potential of integrating these models with cutting-edge neural encoders and suggests a future where even more intricate controls over sound properties could be integrated.

In conclusion, the proposed Stable-V2A framework not only advances the field of V2A synthesis by addressing key challenges in temporal alignment and audio quality but also opens up new horizons for integrating human-readable and semantically rich controls in automated sound design processes. Future research could explore optimizing semantic controls further or expanding the model's applicability to other aspects of sound synthesis and design.