VSSFlow: Unified Video-Conditioned Audio Generation

• VSSFlow is a unified framework that integrates video-to-sound (V2S) and visual TTS using a single flow-matching model with a shared audio generation prior.
• It employs a DiT backbone with innovative cross-attention for ambiguous video and concatenation with self-attention for deterministic phoneme transcripts.
• Joint, end-to-end training on mixed data enhances convergence, stability, and the quality of both sound and speech generation.

to=arxiv_search.search 天天中彩票中了json {"query":"VSSFlow (Cheng et al., 29 Sep 2025) video-conditioned sound speech generation", "max_results": 5} to=arxiv_search.search 天天中彩票中json {"query":"ti:VSSFlow OR abs:VSSFlow", "max_results": 10} VSSFlow is a unified video-conditioned generative framework that integrates video-to-sound (V2S) and visual text-to-speech (VisualTTS) within a single flow-matching model. Its central premise is that both tasks share an underlying audio generation prior, but require different treatment of their conditioning signals: video is ambiguous and one-to-many, whereas transcripts are comparatively deterministic and tightly constraining. VSSFlow addresses that asymmetry through a task-unified DiT backbone and a condition aggregation mechanism that uses cross-attention for video and concatenation plus self-attention for phoneme transcripts, while relying on end-to-end joint learning rather than multi-stage curricula (Cheng et al., 29 Sep 2025).

1. Problem scope and motivation

VSSFlow targets two tasks that had conventionally been addressed as separate problems. In V2S, the model generates synchronized environmental or non-linguistic sound from silent video. In VisualTTS, it generates speech from a talking video plus transcripts, with emphasis on lip synchronization and speaker consistency. The unification problem is nontrivial because the conditioning modalities are heterogeneous: video can map to multiple acoustically plausible outputs, while transcripts strongly determine lexical content and local timing (Cheng et al., 29 Sep 2025).

The framework is explicitly positioned against two assumptions in prior work. The first is that a single in-context fusion mechanism is too crude to handle distinct condition types. The second is that joint learning across V2S and VisualTTS requires complex training stages or curriculum strategies to avoid task interference. VSSFlow argues that neither assumption is necessary if the architecture respects the different inductive biases of the two condition types and the model is trained end-to-end (Cheng et al., 29 Sep 2025).

A common misconception in this area is that task unification necessarily degrades specialization. VSSFlow reports the opposite finding: joint learning benefits both sound and speech generation, and the explanation advanced by the authors is the emergence of a shared general audio prior that improves convergence, conditional generation, and classifier-free guidance stability (Cheng et al., 29 Sep 2025).

2. Flow-matching formulation

VSSFlow is built on a conditional flow-matching ODE over audio latents:

$$\frac{dx_t}{dt} = v_\theta(x_t, c, t), \quad x_t = x_0 + \int_0^t v_\theta(x_s, c, s)\,ds$$

where $x_0 \sim \mathcal{N}(0,1)$ is source noise, $x_1$ is the target audio latent, $x_t$ is the interpolation state, $t \in [0,1]$, and $c$ denotes the conditioning information. Training uses the linear path

$$x_t = t x_0 + (1-t)x_1, \qquad \dot{x}_t = x_1 - x_0$$

with the flow-matching objective

$$\mathcal{L}_{\text{FM}} = \mathbb{E}_{t, x_0, x_1}\left\|v_\theta(x_t, c, t) - \dot{x}_t\right\|^2.$$

This gives VSSFlow a single generative formulation for both V2S and VisualTTS; the task distinction enters only through the condition $c$ (Cheng et al., 29 Sep 2025).

Inference uses classifier-free guidance (CFG). The model is trained with random condition dropping so that it learns both conditional and unconditional velocities, and the guided velocity is

$$v_\theta^{\mathrm{CFG}}(x_t, c, t) = v_\theta(x_t, \emptyset, t) + \gamma\big(v_\theta(x_t, c, t) - v_\theta(x_t, \emptyset, t)\big),$$

where $x_0 \sim \mathcal{N}(0,1)$0 is the CFG scale. One of the noteworthy analyses in VSSFlow is that joint training makes CFG more stable and more useful, because the unconditional branch learns a broader shared audio prior rather than a narrow task-specific distribution (Cheng et al., 29 Sep 2025).

3. Architecture and latent representations

The generator is a 10-layer cross-attention-based Diffusion Transformer (DiT) in the style of Stable Audio Open. Audio is processed in latent space: the waveform is converted to a mel spectrogram, encoded by an AudioLDM 2 VAE into a latent, denoised by the DiT, decoded back to mel, and finally synthesized into waveform by a vocoder. The audio is resampled to 16 kHz, truncated or padded to 10 seconds, and represented as

$x_0 \sim \mathcal{N}(0,1)$1

with $x_0 \sim \mathcal{N}(0,1)$2 and $x_0 \sim \mathcal{N}(0,1)$3. A timestep token is prepended, so the input latent becomes $x_0 \sim \mathcal{N}(0,1)$4 (Cheng et al., 29 Sep 2025).

The video condition is extracted by CLIP at 10 FPS:

$x_0 \sim \mathcal{N}(0,1)$5

with $x_0 \sim \mathcal{N}(0,1)$6 for 10 seconds and $x_0 \sim \mathcal{N}(0,1)$7. It is then linearly interpolated to match the audio latent length, producing $x_0 \sim \mathcal{N}(0,1)$8. Transcript input is converted to phoneme sequences and embedded as

$x_0 \sim \mathcal{N}(0,1)$9

with $x_1$0, then temporally aligned to $x_1$1. The alignment uses a duration predictor leveraging AV-HuBERT features to derive phoneme-to-frame alignment and repeat or pad phonemes to the audio length (Cheng et al., 29 Sep 2025).

Condition type	Representation	Temporal alignment
Video	CLIP features, $x_1$2 after interpolation	10 FPS features interpolated to audio length
Transcript / phoneme	Phoneme embeddings, $x_1$3 with $x_1$4	Duration predictor with AV-HuBERT-derived alignment
Audio target	VAE latent, $x_1$5	Native latent sequence

This representation design is central to the model’s unification claim: both tasks are solved in the same latent audio space, with modality-specific structure confined to the conditioning pathway rather than the generative backbone (Cheng et al., 29 Sep 2025).

4. Condition aggregation mechanism

The main architectural innovation is the condition aggregation mechanism. VSSFlow explicitly distinguishes two ways of injecting conditions into the DiT: cross-attention, in which the condition acts as keys and values, and concatenation plus self-attention, in which condition tokens are appended to the latent stream. The argument is that these mechanisms encode different inductive biases. Cross-attention is better suited to ambiguous video, because it permits flexible, non-rigid conditioning, while self-attention / concatenation is better suited to deterministic transcript input, where tighter local alignment is desirable (Cheng et al., 29 Sep 2025).

The paper evaluates four condition-fusion variants: CrossV, CrossVS, ConcatV, and ConcatVS. The selected design is CrossV, in which video is introduced by cross-attention and speech by concatenation. Empirically, phoneme concatenation helps speech generation converge faster, while video cross-attention is superior for sound generation. Attention-map analysis further supports the claimed inductive-bias split: self-attention tends to focus on current and nearby positions, which benefits aligned phoneme-to-audio generation, whereas cross-attention is more globally flexible and is better matched to video-conditioned sound synthesis (Cheng et al., 29 Sep 2025).

The significance of this mechanism is methodological rather than merely architectural. VSSFlow does not treat condition fusion as a single universal primitive; it treats fusion choice as part of the problem specification. That is the basis on which the model unifies V2S and VisualTTS without collapsing either task into an unnatural conditioning regime (Cheng et al., 29 Sep 2025).

5. Joint training and the shared general audio prior

VSSFlow is trained end-to-end on mixed task data with three condition patterns: V2S with $x_1$6, VisualTTS with both $x_1$7 and $x_1$8 active, and TTS with $x_1$9. The stated interpretation is that this exposes the model to sound-only, speech-conditioned, and text-conditioned audio generation within one parameterization, allowing it to learn a broader audio distribution than a task-isolated model (Cheng et al., 29 Sep 2025).

The authors describe the resulting representation as a shared general audio prior. In their account, this prior captures low-level and mid-level regularities common to sound and speech, including spectral structure, temporal continuity, waveform realism, and denoising dynamics. The reported practical consequences are better conditional generation, faster convergence, and more stable CFG behavior. In particular, the analysis shows that models trained jointly improve more substantially at higher CFG scales, whereas a V2S-only model may degrade under stronger CFG, suggesting a narrower unconditional prior (Cheng et al., 29 Sep 2025).

Training is conducted for 200 epochs on 4 H100 GPUs with batch size 36 per GPU, learning rate $x_t$0, and 2000 warmup steps. The unconditional drop probability for both $x_t$1 and $x_t$2 is 0.1. At inference, the reported CFG scales are 3.0 for V2S and 1.5 for VisualTTS, and sampling uses the Dopri5 ODE solver (Cheng et al., 29 Sep 2025).

This joint-training result directly challenges the view that multimodal audio generation requires elaborate curriculum design. VSSFlow’s position is that task interference is not inevitable; under an appropriate condition aggregation scheme, sound and speech mutually reinforce one another (Cheng et al., 29 Sep 2025).

6. Experimental evaluation

The V2S benchmark is VGGSound, with approximately 182k training and 15k test examples. VisualTTS evaluation uses Chem, GRID, and LRS2, with approximately 162k training examples in the described setup, while auxiliary TTS data come from LJSpeech and LibriTTS, totaling approximately 160k training examples. The final model is trained on 503k examples across V2S, TTS, and VisualTTS data (Cheng et al., 29 Sep 2025).

For V2S, VSSFlow is compared with autoregressive, mask-based, diffusion, and flow-based baselines, including SpecVQGan, Im2Wav, V-AURA, VAB, Diff-Foley, Seeing and Hearing, V2A-Mapper, FoleyCrafter, TiVA, LoVA, Frieren, and MMAudio. Metrics include FAD, IS, KL, Onset Acc., Onset AP, DeSync, and VA-IB. On VGGSound, VSSFlow reports FAD (vggish) 1.34 and FAD (pann) 11.10, both best; FAD (passt) 187.40, second-best; Onset Acc. 7.16, second-best; DeSync 1.18, competitive; and VA-IB 26.01, among the best and tied with some top baselines. The interpretation given is that VSSFlow improves sound quality and semantic alignment, while remaining slightly weaker on some temporal-alignment metrics than the most specialized V2S systems (Cheng et al., 29 Sep 2025).

For VisualTTS, the baselines are DSU, HPMDubbing, StyleDubber, and EmoDubber, and the metrics are WER, Spk. Sim., UTMOS, MCD, MCD-DTW, MCD-DTW-SL, LSE-C, and LSE-D. On Chem, VSSFlow reports WER 15.1, Spk. Sim. 79.7, UTMOS 3.17, MCD 9.55, MCD-DTW 5.18, MCD-DTW-SL 5.19, LSE-C 6.1, and LSE-D 8.37. On GRID, it reports WER 18.2, Spk. Sim. 51.5, UTMOS 3.31, MCD 8.66, MCD-DTW 5.23, MCD-DTW-SL 5.23, LSE-C 6.37, and LSE-D 8.6. These results are presented as evidence of intelligible speech, strong lip synchronization, and good speaker preservation (Cheng et al., 29 Sep 2025).

The ablation studies are central to the paper’s claims. Condition-mechanism ablations show that video via cross-attention outperforms video concatenation for V2S, while phoneme concatenation accelerates convergence and improves speech quality. Joint-training ablations show that adding speech data improves V2S and that adding sound data does not hurt VisualTTS. CFG analysis shows that the benefit of joint learning becomes more visible at stronger guidance scales (Cheng et al., 29 Sep 2025).

7. Limitations, extensions, and broader significance

The paper identifies three primary limitations: lack of high-quality native joint sound-speech data, a representation bottleneck in compact audio and speech latents, and broader data scarcity for unified video-speech-sound corpora. The mixed sound-speech generation experiments rely on synthesized mixtures rather than native recordings, which the authors describe as useful but likely inferior to real joint data (Cheng et al., 29 Sep 2025).

At the same time, VSSFlow points toward a broader class of unified audio generators. In a continued-training experiment, the model is reported to produce joint sound-speech mixtures aligned with video, including car engine noise, braking sound, and police officer speech. This suggests an extension beyond the canonical V2S and VisualTTS tasks toward more general sound-plus-speech generation (Cheng et al., 29 Sep 2025).

The broader significance of VSSFlow lies less in a single benchmark number than in its reformulation of the design space. It treats heterogeneous conditions not as a nuisance to be normalized away, but as a signal to choose different conditioning operators. It also treats joint learning not as a regularization liability, but as a mechanism for learning a richer audio prior. Within that framing, VSSFlow presents unified video-conditioned sound and speech generation as a single flow-matching problem with modality-specific condition aggregation rather than as two separate model families (Cheng et al., 29 Sep 2025).