ViSAudio: Binaural Audio from Silent Video

• ViSAudio is an end-to-end framework that directly generates immersive spatial audio from silent video using dual-branch conditional flow matching.
• It integrates multimodal features—visual, textual, synchronization, and spatial—via Transformer layers and conditional modulation for precise spatio-temporal alignment.
• Trained on the extensive BiAudio dataset, ViSAudio outperforms previous methods in both objective and subjective evaluations, setting a new state-of-the-art in binaural audio generation.

ViSAudio is an end-to-end framework for generating binaural spatial audio directly from silent video, designed to address the limitations of prior approaches that sequentially generate mono audio and subsequently apply spatialization. ViSAudio introduces a dual-branch conditional flow-matching architecture, tightly integrating multimodal information and spatial cues to achieve precise spatio-temporal alignment between audio and input video. The framework is trained on the BiAudio dataset, a large corpus of video–binaural audio pairs encompassing diverse scenes and camera trajectories. ViSAudio sets state-of-the-art performance in both objective and subjective evaluations for spatial audio generation (Zhang et al., 2 Dec 2025).

1. System Pipeline Overview

ViSAudio processes a silent perspective video $\mathcal{I} = \{I_n\}_{n=1}^N$, with each frame $I_n \in \mathbb{R}^{3\times H \times W}$; it can also optionally incorporate a text caption $\mathcal{T}$. The core pipeline is as follows:

• Initialization: The framework is initialized from MMAudio's pretrained large-scale flow-matching network operating at 44.1 kHz.
• Conditional Flow Matching and Dual-Branch Generation: Two parallel branches predict time-dependent velocity fields $v^l_\theta(t, \mathcal{C}, x_t^l)$ and $v^r_\theta(t, \mathcal{C}, x_t^r)$ for left and right audio latents $x_t^l, x_t^r$. Conditioning $\mathcal{C}$ bundles (1) CLIP-based visual and text features, (2) Synchformer-derived synchronization features, and (3) channel-specific spatial features from a Perception Encoder with stereo embeddings.
• Conditional Spacetime Modulation: A module integrates global and per-frame spatial and temporal context to maintain the balance between channel consistency and spatial distinction.
• Transformer Processing: $N_1$ multimodal joint Transformer layers synchronize left/right latents, followed by $N_2$ single-branch Transformers for channel-specific refinement.
• Latent Integration and Decoding: Starting from two independent Gaussian noises, the framework integrates velocity fields over $t \in [0,1]$, producing final latents $I_n \in \mathbb{R}^{3\times H \times W}$0 that are decoded to mel-spectrograms (via MMAudio VAE) and converted to time-domain waveforms by a neural vocoder (BigVGAN).

This design enables direct generation of spatially immersively binaural audio from video without the need for mono intermediate representations (Zhang et al., 2 Dec 2025).

2. Conditional Flow Matching Formulation

ViSAudio adopts a conditional flow matching (CFM) regime rooted in the displacement interpolation path of optimal transport:

$I_n \in \mathbb{R}^{3\times H \times W}$1

For each audio channel $I_n \in \mathbb{R}^{3\times H \times W}$2:

$I_n \in \mathbb{R}^{3\times H \times W}$3

$I_n \in \mathbb{R}^{3\times H \times W}$4

Here, $I_n \in \mathbb{R}^{3\times H \times W}$5 is a standard Gaussian noise, $I_n \in \mathbb{R}^{3\times H \times W}$6 the ground-truth audio latent for channel $I_n \in \mathbb{R}^{3\times H \times W}$7, $I_n \in \mathbb{R}^{3\times H \times W}$8 is all conditioning, and $I_n \in \mathbb{R}^{3\times H \times W}$9 regresses the velocity field. Inference simulates the reverse ODE:

$\mathcal{T}$0

from $\mathcal{T}$1 to $\mathcal{T}$2, ensuring coherent generation of left/right channel latents.

3. Dual-Branch Audio Generation Architecture

The architecture employs two parallel branches, each responsible for generating one binaural channel:

• Multimodal Feature Extraction: Visual and textual features from CLIP $\mathcal{T}$3, synchronization features from Synchformer $\mathcal{T}$4, and perception encoder spatial features $\mathcal{T}$5 with learnable stereo embeddings $\mathcal{T}$6.
• Frame-Aligned Spatial Features: For each channel,

$\mathcal{T}$7

• Transformer Stack:
  • $\mathcal{T}$8 joint multimodal Transformers keep left/right latents synchronized:

  $\mathcal{T}$9

  $v^l_\theta(t, \mathcal{C}, x_t^l)$0 - $v^l_\theta(t, \mathcal{C}, x_t^l)$1 channel-specific Transformers for spatial and semantic refinement:

  $v^l_\theta(t, \mathcal{C}, x_t^l)$2

• Regression of Velocity Fields: Refined latents are mapped to final velocity vector fields for numerical integration.

This architecture preserves channel synchronization while permitting independent spatial specialization, essential for accurate binaural rendering.

4. Conditional Spacetime Module

The conditional spacetime module injects joint spatial and temporal context into each Transformer branch via feature modulation:

• Feature Fusion: Each channel receives its own frame-aligned $v^l_\theta(t, \mathcal{C}, x_t^l)$3 and global conditioning $v^l_\theta(t, \mathcal{C}, x_t^l)$4 (a concatenation and projection of frame-averaged $v^l_\theta(t, \mathcal{C}, x_t^l)$5).

• Global Channel Features: For each channel,

$v^l_\theta(t, \mathcal{C}, x_t^l)$6

These features are used to modulate (via adaLN or FiLM) the single-branch Transformer blocks, ensuring that each branch captures spatial and temporal dynamics specific to its channel.

The spacetime module is critical for aligning audio with both spatial context and dynamic visual input, supporting robust handling of viewpoint changes, moving sources, and complex scenes.

5. Training Procedure and BiAudio Dataset

ViSAudio training utilizes the BiAudio dataset:

• Dataset Construction: 97,000 video–binaural audio pairs (8 s each, totaling 215 h), curated from Sphere360 FOA and 360° videos.

  • Spherical harmonic analysis to localize dominant directions.
  • Randomized perspective rendering (pitch/yaw/roll).
  • FOA to binaural conversion via HRIR convolution (Omnitone).
  • Filtering to ensure perceptible stereo separation.
  • Dual-stage captioning: Qwen2.5-Omni (free-form) and Qwen3-Instruct-2507 (structured).
• Training Regime:
  • Fine-tuning of MMAudio (large, v2) on BiAudio with over-sampled MUSIC dataset (×3).
  • Batch size 64, learning rate $v^l_\theta(t, \mathcal{C}, x_t^l)$7, weight decay $v^l_\theta(t, \mathcal{C}, x_t^l)$8.
  • Convergence in ≈50,000 iterations (~2 days on 8×A800).
  • Only the CFM loss $v^l_\theta(t, \mathcal{C}, x_t^l)$9 is optimized; no auxiliary losses.
  • Data augmentation through randomized camera trajectories.

This approach enables training on open-domain scenarios with broad spatial and semantic diversity.

6. Evaluation Metrics and Comparative Analysis

ViSAudio's evaluation encompasses objective and subjective measures:

• Objective Metrics (lower is better unless noted):
  • Fused Fréchet Distance scores: FD$v^r_\theta(t, \mathcal{C}, x_t^r)$0, FD$v^r_\theta(t, \mathcal{C}, x_t^r)$1, FD$v^r_\theta(t, \mathcal{C}, x_t^r)$2, FD$v^r_\theta(t, \mathcal{C}, x_t^r)$3
  • KL divergence metrics: KL$v^r_\theta(t, \mathcal{C}, x_t^r)$4, KL$v^r_\theta(t, \mathcal{C}, x_t^r)$5
  • DeSync (audio–video synchrony error)
  • IB-Score (ImageBind vision/audio similarity; higher is better)
• In-Distribution Performance (BiAudio test set, 2.7k clips):

| Method | FD$v^r_\theta(t, \mathcal{C}, x_t^r)$6 | FD$v^r_\theta(t, \mathcal{C}, x_t^r)$7 | KL$v^r_\theta(t, \mathcal{C}, x_t^r)$8 | DeSync | IB-Score | |--------------|--------------------------|---------------------------|---------------|--------|----------| | ThinkSound | 5.125 | 23.801 | 2.462 | 0.903 | 0.191 | | AudioX | 4.224 | 22.240 | 2.167 | 1.157 | 0.235 | | ViSAGe | 14.212 | 62.587 | 3.546 | 1.159 | 0.123 | | See2Sound | 9.573 | 42.853 | 3.410 | 1.244 | 0.088 | | ViSAudio | 2.516 | 13.917 | 1.355 | 0.788 | 0.299 |

• User Study (mean opinion scores; 1–5, 12 experts on 10 clips):

| Method | Spatial Impression | Spatial Consistency | Temporal Align. | Semantic Align. | Audio Realism | |---------------|-------------------|--------------------|-----------------|-----------------|---------------| | ThinkSound | 3.25 ±0.25 | 2.88 ±0.29 | 3.48 ±0.24 | 3.40 ±0.27 | 3.07 ±0.27 | | AudioX | 3.05 ±0.27 | 2.79 ±0.31 | 2.87 ±0.30 | 3.37 ±0.26 | 3.26 ±0.32 | | ViSAGe | 1.66 ±0.23 | 1.52 ±0.21 | 1.70 ±0.31 | 1.73 ±0.31 | 1.49 ±0.21 | | See2Sound | 2.03 ±0.30 | 1.52 ±0.21 | 1.74 ±0.33 | 1.73 ±0.28 | 1.65 ±0.33 | | ViSAudio | 4.13 ±0.29 | 4.10 ±0.29 | 4.28 ±0.27 | 4.29 ±0.24 | 4.16 ±0.28|

• Ablation Analysis: Both the dual-branch and spacetime modules provide significant spatial and semantic gains.
• Generalization: Out-of-distribution evaluation on FAIR-Play confirms the robustness and margin over prior methods.

ViSAudio thus achieves state-of-the-art binaural spatial audio generation from silent video, with significant improvements in spatial realism, consistency, and synchrony over existing techniques (Zhang et al., 2 Dec 2025).