Video-to-Sound (V2S): Audio Synthesis

• Video-to-Sound (V2S) is a technology that generates semantically consistent audio from silent videos by decoupling 'what' to hear from 'when' it should occur.
• Recent models leverage modular designs that combine semantic conditions—via text, labels, or audio prompts—with temporal controls like energy curves or onset traces.
• Innovative backbones such as diffusion and flow-based models enable fine-grained control over sound synthesis, addressing challenges like synchronization and source-aware composition.

Video-to-Sound (V2S), often used interchangeably with video-to-audio (V2A), denotes the generation of audio for a silent video such that the result is semantically consistent with the visual content and temporally synchronized with visible events (Chen et al., 2024, Zhang et al., 2024). In contemporary work, the label covers several adjacent problem settings: Foley sound synthesis for post-production, general environmental sound generation, joint sound effects and background music generation, video-and-text-to-audio generation, and, in some subliteratures, video-to-speech or unified sound-and-speech generation (Lee et al., 2024, Chen et al., 2024, Cheng et al., 29 Sep 2025). A central theme across recent systems is the separation of what should be heard from when it should occur, typically by combining semantic conditions such as text, labels, audio prompts, or source-aware visual semantics with temporal controls such as energy curves, RMS envelopes, onset traces, or dense video-derived timing features (Jeong et al., 2024, Gramaccioni et al., 2024, Li et al., 2024).

1. Scope, terminology, and task formulations

The most compact formulation in the recent literature treats V2S as a mapping from a silent video to audio aligned with the semantic content of the input video (Chen et al., 2024). One explicit language-mediated factorization is

$v \rightarrow p \rightarrow a,$

where $v$ denotes video frames, $p$ a natural-language scheme generated by a multimodal LLM, and $a$ the generated audio (Chen et al., 2024). This formulation is deliberately modular: the video is interpreted first, then the interpretation is rendered acoustically.

The field, however, is not monolithic. Foley-oriented work emphasizes tightly synchronized sound effects for impacts, footsteps, cloth, doors, or object interactions, often under post-production constraints and creative control requirements (Lee et al., 2024, Gramaccioni et al., 2024). Other systems target broader soundscapes, including background ambience, off-screen events, and background music, as in semantically consistent SFX+BGM generation (Chen et al., 2024, Dai et al., 6 Jan 2026). A narrower usage reserves V2S for non-linguistic audio and separates it from visual speech synthesis, while another line explicitly broadens the acronym to include speech generation from silent talking-face video (Cheng et al., 29 Sep 2025, Choi et al., 2024).

A second shift is from single-task models to unified conditional audio generators. Omni2Sound supports video-to-audio (V2A), text-to-audio (T2A), and joint video-text-to-audio (VT2A) within one diffusion model, while VSSFlow jointly models V2S and VisualTTS in a single flow-matching framework (Dai et al., 6 Jan 2026, Cheng et al., 29 Sep 2025). This broadened scope alters the conceptual status of V2S: it becomes one operating mode of a multimodal audio generator rather than a standalone architecture.

2. Conditioning: semantics, timing, and controllability

A dominant organizing principle in current V2S research is the explicit separation of semantic and temporal control. ReWaS states this directly by using video to estimate the structural information of sound, namely energy, while receiving key content cues from a user prompt (Jeong et al., 2024). FolAI likewise describes a two-stage framework that decouples the when and the what of sound synthesis through a temporal scaffold plus semantically guided diffusion (Gramaccioni et al., 2024). Video-Foley uses Root Mean Square (RMS) as an intuitive temporal-event condition and combines it with semantic timbre prompts supplied as audio or text (Lee et al., 2024).

This conditioning split is implemented in several distinct ways. ReWaS defines an audio energy curve from a mel-spectrogram as

$$e_a = Smoothing \left(\frac{1}{D} \sum_{d=1}^{D} mel_{w,d}\right),$$

and uses that signal as structural control for a text-to-audio model (Jeong et al., 2024). Smooth-Foley introduces semantic guidance from textual labels throughout generation: a frame adapter injects high-resolution frame-wise video features, while a temporal adapter uses conditions derived from similarities of visual frames and textual labels, addressing continuous sounds with moving visual presence (Zhang et al., 2024). Tri-Ergon extends controllability further by combining textual, auditory, and pixel-level visual prompts with Loudness Units relative to Full Scale (LUFS) embeddings for manual control of loudness changes over time and across stereo channels (Li et al., 2024).

A plausible implication is that V2S controllability is no longer reducible to category selection. Recent systems expose timing, intensity, timbre, background environment, and even source-level composition as separate control axes. SSV2A is explicit on this point: it detects local sounding objects, maps them into a Cross-Modal Sound Source manifold, and enables intuitive control by compositing vision, text, and audio conditions (Guo et al., 2024). VarietySound reaches a similar conclusion from a different direction by disentangling temporal information, acoustic information, and background information for timbre-controllable generation (Cui et al., 2022).

3. Generative backbones and cross-modal alignment mechanisms

The field has converged on latent generative models, but not on a single family. Diffusion backbones remain prominent: Smooth-Foley adapts a frozen text-to-audio diffusion model with a frame adapter and a ControlNet-style temporal adapter (Zhang et al., 2024); Stable-Foley conditions Stable Audio Open through a ControlNet branch driven by an RMS envelope (Gramaccioni et al., 2024); Tri-Ergon uses a VAE plus a DiT-based latent diffusion model operating on 44.1 kHz stereo waveforms up to 60 seconds (Li et al., 2024). Flow-based models form a parallel trajectory. VSSFlow learns a continuous velocity field

$$\frac{dx_t}{dt} = v_\theta(x_t, c, t), \qquad x_t = t x_0 + (1-t)x_1,$$

with flow-matching loss

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x_0, x_1}\left\| v_\theta(x_t, c, t) - (x_1 - x_0) \right\|^2,$$

thereby unifying sound and speech generation under one ODE-based model (Cheng et al., 29 Sep 2025).

Architecture design increasingly reflects different inductive biases for different conditions. VSSFlow argues that cross-attention suits ambiguous video conditions, whereas self-attention over concatenated streams better suits deterministic transcript conditions (Cheng et al., 29 Sep 2025). Omni2Sound separates semantics and timing more explicitly: CLIP-derived global visual features and Flan-T5 text tokens enter a semantic cross-attention branch, while Synchformer-derived temporal features modulate the model through AdaLN, giving video one pathway for what and another for when (Dai et al., 6 Jan 2026). YingSound follows a comparable decomposition with an audio DiT, a video DiT, and a learnable Audio-Visual Aggregator inserted between layers, plus later CoT-based refinement for few-shot industrial scenarios (Chen et al., 2024).

A notable recent development is the incorporation of synchronization directly into attention. Flowley proposes Progressive Soft-masked Cross-Attention, which adds a temporally structured mask to audio–video cross-attention in latent space and claims zero additional computational cost compared with standard attention (Tran et al., 7 Jul 2026). This mechanism is significant because it embeds alignment constraints inside the generative backbone rather than outsourcing them to external synchronization modules or to a separate video-to-text stage.

4. Decomposition, source awareness, and long-form control

One influential line of work treats audio not as a unitary target but as a composition of disentangled factors. VarietySound decomposes each target sound into temporal information, acoustic information, and background information, encodes them with separate modules, and reconstructs the result with a mel decoder trained alongside a mel discriminator and a temporal discriminator (Cui et al., 2022). The aim is not only controllability but also specialization: a reference audio may determine timbre, while the silent video determines event timing.

A second line emphasizes explicit loudness or envelope control as a proxy for temporal structure. Tri-Ergon conditions diffusion on a fused multimodal embedding $E_M$, a LUFS sequence $E_L$, and timing information $E_T$, and models the conditional distribution

$v$0

Because LUFS is represented per channel and over time, the method can control loudness trajectories independently of semantic prompts and variable clip duration (Li et al., 2024). Stable-Foley and FolAI adopt a similar principle with RMS or motion-intensity envelopes rather than LUFS, suggesting a broader trend toward envelope-conditioned Foley generation (Gramaccioni et al., 2024).

A third line moves from global scene understanding to source-aware modeling. SSV2A explicitly argues that prior V2A systems rely too heavily on global scene semantics and overlook local sounding objects. Its Sound Source-Aware pipeline detects local sources, learns a Cross-Modal Sound Source manifold, and attentively mixes source-level semantics into a global audio representation before handing control to AudioLDM (Guo et al., 2024). Efficient Video to Audio Mapper with Visual Scene Detection addresses a related problem at the temporal level: multiple scenes within a clip. It detects visual scene boundaries from CLIP feature self-similarity and generates per-scene audio, showing that scene-aware segmentation can improve fidelity and relevance in multi-scene videos (Yi et al., 2024).

A plausible synthesis is that fine-grained V2S control now spans at least three decompositions: source decomposition, semantic–temporal decomposition, and loudness or envelope decomposition. These are not equivalent. Source-aware methods resolve which object is sounding; envelope-based methods resolve when and how strongly; timbre-oriented methods resolve what it sounds like.

5. Datasets, evaluation practice, and representative results

The empirical landscape of V2S is strongly shaped by dataset choice. Some benchmarks emphasize short, tightly aligned impacts; others emphasize general web video; still others are explicitly curated for continuous sounds, unified multimodal conditioning, or timbre control. The table below lists representative datasets that recur in recent work.

Dataset	Role in V2S research	Representative use
VGGSound	General 10-second audio-visual benchmark	Used by Smooth-Foley, VSSFlow, Flowley, ReWaS, and Omni2Sound (Zhang et al., 2024, Cheng et al., 29 Sep 2025, Tran et al., 7 Jul 2026)
Greatest Hits	Visually indicated impact and scratch sounds	Used in Foley-oriented systems and Stable-Foley-style evaluation (Gramaccioni et al., 2024)
VAS	Timbre-controllable V2S benchmark with 8 categories	Used by VarietySound (Cui et al., 2022)
VGGSound-Continuous	Continuous sound scenarios with moving objects	Introduced with 95 challenging test clips for Smooth-Foley (Zhang et al., 2024)
SoundAtlas / VGGSound-Omni	Unified V2A, T2A, VT2A training and evaluation	Introduced by Omni2Sound; SoundAtlas contains 470k pairs (Dai et al., 6 Jan 2026)
MM-V2A / VGGS3	Long-form multimodal V2A and single-source visual-audio training	Used by Tri-Ergon and SSV2A (Li et al., 2024, Guo et al., 2024)

Evaluation is correspondingly heterogeneous. Audio fidelity is typically measured with FAD or FD; semantic relevance with CLAP, CLIP Score, IB-Score, LB-Score, MKL, or KL-based metrics; temporal alignment with AV-Align, Align Acc, Onset Accuracy/AP, DeSync, or envelope error such as E-L1; and perceptual quality with MOS-style human judgments (Zhang et al., 2024, Dai et al., 6 Jan 2026, Li et al., 2024, Cheng et al., 29 Sep 2025). The diversity of metrics reflects the field’s unresolved objective: some systems optimize category fidelity, others source relevance, others synchronization, and others controllable realism.

Several recent systems report strong quantitative gains. Smooth-Foley, on VGGSound-Continuous and VGGSound, reports MKL $v$1 and CLIP Score $v$2 for its frame-wise model, together with subjective scores of $v$3 for semantic alignment, $v$4 for temporal alignment, and $v$5 for audio quality on a 1–10 scale (Zhang et al., 2024). Omni2Sound, on VGGSound-Omni V2A, reports FAD $v$6, FD $v$7, DS $v$8, IB $v$9, PQ $p$0, and IS $p$1, and argues that a single model can reach unified state-of-the-art performance across V2A, T2A, and VT2A (Dai et al., 6 Jan 2026). Tri-Ergon-L, evaluated on VGGSound at 10 seconds, reports FD $p$2, KL $p$3, and AV-Align $p$4 while generating 44.1 kHz stereo audio of varying lengths up to 60 seconds (Li et al., 2024). VSSFlow, on VGGSound, reports FAD $p$5 with VGGish embeddings, KL $p$6, DeSync $p$7, and IB-VA $p$8, while arguing that joint sound–speech learning improves V2S rather than harming it (Cheng et al., 29 Sep 2025). ReWaS, on VGGSound, reports FID $p$9, MKL $a$0, CLAP $a$1, Energy MAE $a$2, and a 204M-parameter training footprint, emphasizing quality, controllability, and training efficiency (Jeong et al., 2024).

6. Open issues, misconceptions, and emerging directions

One recurring misconception is that V2S can be reduced to captioning plus text-to-audio generation. SVA demonstrates that a language-mediated pipeline can produce semantically consistent SFX and BGM with natural language as the interface, but it also states that this remains a coarse-grained method that cannot accurately model the intricate relationship between video and audio and does not attempt temporal synchronization (Chen et al., 2024). Flowley sharpens the criticism from another angle, arguing that converting visual input into text to leverage pretrained text-to-audio models sacrifices fine-grained temporal cues (Tran et al., 7 Jul 2026). This suggests that language is powerful as an interface but insufficient as the sole carrier of temporal structure.

A second misconception is that better semantics automatically imply better synchronization. Smooth-Foley shows that continuous sound scenarios remain difficult when temporal conditions are inaccurate or semantic conditions are low resolution, especially for moving or visually ambiguous objects (Zhang et al., 2024). Source-aware methods and explicit temporal scaffolds arose precisely because global semantic correctness does not guarantee frame-level or event-level alignment.

A third misconception is that unified training necessarily damages video-conditioned performance. Omni2Sound instead introduces a three-stage progressive schedule in which high-quality VT2A data act as a bridge between tasks, and VSSFlow reports that end-to-end joint learning of sound and speech benefits V2S through a shared general audio prior and more stable classifier-free guidance (Dai et al., 6 Jan 2026, Cheng et al., 29 Sep 2025). These findings do not settle the issue, but they shift the debate from whether to unify toward how to control cross-task competition and modality bias.

Current research directions are correspondingly clear. Off-screen and background sound generation is being treated as a first-class problem rather than an error mode (Dai et al., 6 Jan 2026). Caption quality and V-A-T alignment have become data bottlenecks, leading to SoundAtlas and SoundCap-style pipelines for detailed, sound-aware captions (Dai et al., 6 Jan 2026, Tran et al., 7 Jul 2026). Fine-grained controllability is expanding from timbre and prompts to loudness trajectories, stereo dynamics, scene boundaries, and sound-source composition (Li et al., 2024, Yi et al., 2024, Guo et al., 2024). Finally, generative V2S backbones are being repurposed for downstream tasks such as video/text-queried sound separation, as in MMAudioSep, which shows that a fine-tuned separation model can still retain original video-to-audio generation capability (Takahashi et al., 10 Oct 2025).

Taken together, the literature indicates a transition from monolithic video-conditioned sound synthesis toward modular, multimodal, and explicitly controllable systems. The central technical challenge is no longer merely generating plausible audio, but coordinating semantics, timing, source structure, and user intent within a single generative framework.