---
title: 'FoleyBench: Benchmark for Video-to-Audio Models'
url: https://www.emergentmind.com/topics/foleybench
type: topic
---

# FoleyBench: Benchmark for Video-to-Audio Models

FoleyBench is a large-scale benchmark specifically designed for the evaluation of video-to-audio (V2A) generation models in Foley-style scenarios, i.e., modeling sound effects causally and temporally synchronized with visible, non-speech, non-music events in video. The dataset and its protocols address longstanding deficiencies in existing V2A benchmarks, which are dominated by speech/music and frequently lack clear audio-visual correspondence relevant to post-production, AR/VR, and sound design applications involving visible Foley effects [2511.13219].

## 1. Construction and Design of the FoleyBench Dataset

FoleyBench leverages a multi-stage, automated pipeline to create a dataset containing 5,000 (video, ground-truth audio, text caption) triplets, each 8–10 seconds in length, along with an extended long-form set ("FoleyBench-Long") of 650 videos each 30 seconds in length.

- **Data collection** uses Creative-Commons licensed videos sourced from YouTube (FineVideo, LVBench) and Vimeo (V3C1).
- **Scene detection** is performed automatically, discarding segments shorter than 8 seconds to ensure sufficient temporal context.
- **Audio filtering with YAMNet** removes clips dominated by speech or music, rejecting frames with a speech/music score ≥ 0.6; this stage eliminates 97.7% of the original candidate pool. Manual spot-checks reveal a post-filtering precision of 47% for true Foley clips.
- **Audio-visual grounding (using Gemini 2.5 Pro)** accepts only clips where sounds are causally and temporally tied to visible on-screen actions, evidenced by a 72% precision on validation data.
- **Yield**: 5,000 clips for the core set (~12 hours), and an additional 650 longer clips for long-form evaluation (~5.4 hours). Each example preserves visible sound sources and strict temporal alignment, e.g., hammer impacts coinciding with visible contact.

In contrast to previous datasets (e.g., VGGSound), FoleyBench ensures robust coverage of Foley-relevant categories; only 13.4% of Universal Category System (UCS) categories have ≤ 3 clips (versus 24.3% in filtered VGGSound), and the Shannon entropy over UCS categories reaches 5.35 (vs. 4.73 in VGGSound-filtered), demonstrating greater diversity relevant to Foley modeling.

| Dataset Segment        | Number of Clips | Duration per Clip | Total Duration |
|-----------------------|-----------------|-------------------|---------------|
| FoleyBench (core)     | 5,000           | 8–10 s            | 12 hours      |
| FoleyBench-Long       | 650             | 30 s              | 5.4 hours     |

## 2. Foley-Specific Taxonomy and Metadata

FoleyBench employs a comprehensive taxonomy and rich metadata schema to enable granular analysis of both dataset content and V2A model performance:

- **Sound category labeling** leverages the UCS and AudioSet taxonomies at the top-class level.
- **Envelope classification** distinguishes between "Discrete" (isolated impacts, e.g., door slam) and "Rest"/continuous sounds (e.g., running water).
- **Source complexity** is annotated as "Single-source" or "Multi-source."
- **Acoustic focus** is labeled as "Background," "Action," or "Combined."
- **Additional metadata** includes a Gemini-generated one-line caption, a YAMNet coarse label for prompting, and detailed JSON-formatted labels containing all above attributes with rationale where applicable.

This approach accommodates both high-level category analysis and fine-grained breakdown of model strengths and failure modes, especially in challenging multi-source or continuous ambience contexts.

## 3. Evaluation Metrics and Protocols

Evaluation of V2A models on FoleyBench is standardized along audio quality and cross-modal alignment axes, utilizing six principal metrics (all PANN-embedding-based unless stated otherwise):

- **Audio Quality:**
  1. Fréchet Audio Distance (FAD): Quantifies the distributional distance between generated and real audio; lower is better.
  2. Inception Score (IS): Assesses the diversity and sharpness of audio events; higher indicates superior generation.
  3. Kullback–Leibler Divergence (KLD): Measures class-distribution match; lower values reflect improved alignment with ground-truth classes.

- **Cross-Modal Alignment:**
  1. ImageBind Score (IB): Cosine similarity between video and audio embeddings; higher reflects superior semantic correspondence.
  2. CLAP Score: Cosine similarity between text caption and audio embeddings; reported for text-conditioned models only.
  3. De-Sync: Average absolute difference in event timing between audio and video; lower indicates tighter temporal synchronization.

In contrast to prior V2A benchmarks, which are largely insensitive to discrete impacts and fine alignment due to speech/music bias, FoleyBench's metrics and category coverage highlight discrete and ambient Foley effects—critical for application realism.

| Metric          | Measures                                 | Optimal Value   |
|-----------------|------------------------------------------|-----------------|
| FAD             | Audio distributional similarity          | Lower           |
| IS              | Audio event diversity/sharpness          | Higher          |
| KLD             | Class-distribution match                 | Lower           |
| IB              | Video-audio semantic alignment           | Higher          |
| CLAP            | Text-audio semantic alignment            | Higher          |
| De-Sync         | Temporal alignment (sec)                 | Lower           |

## 4. Benchmarking and Analysis of V2A Models

Twelve state-of-the-art V2A models spanning autoregressive, diffusion, flow, and adapter-based architectures have been evaluated on the FoleyBench 5,000-clip test split:

- **MMAudio (flow-matching, video–text–audio joint model)** achieves the best FAD (8.76), IS (11.2), KLD (2.43), second-best IB (0.306), and De-Sync (0.447 s).
- **Seeing & Hearing** attains the highest IB (0.371) via masked spectrograms with test-time ImageBind optimization.
- **CAFA (ControlNet adapter)** accomplishes strong CLAP (0.270) and leads among adapters for FAD (15.5).
- **V-AURA (autoregressive)** produces best-in-class temporal synchronization (De-Sync = 0.716 s).

Fine-grained ablations on MMAudio indicate:

- For "Discrete" impacts, De-Sync improves (0.458 → 0.390 s), but FAD worsens (9.02 → 16.35) and IS declines (10.3 → 8.8).
- "Background" ambiences see degraded sync (0.405 → 0.636 s) and IS (11.98 → 4.61), but improved KLD (2.54 → 1.98).
- Multi-source scenes improve semantic IB (0.296 → 0.324), while FAD (9.84 → 11.34) and De-Sync (0.436 → 0.467 s) deteriorate.
- Introduction of text conditioning (SpecMaskFoley ablation) uniformly improves all metrics (e.g., FAD 23.18 → 19.60).

Long-form generation (FoleyBench-Long) is feasible for LOVA, VTA-LDM, and MMAudio. Notably, MMAudio's audio quality measured by FAD degrades (8.76 → 27.5) on long-form, but maintains the best semantic alignment (IB = 0.239, De-Sync = 0.638 s). LOVA yields best long-form FAD (26.2) and IS (5.02), trading off alignment quality [2511.13219].

## 5. Implications and Future Research Directions

Major insights from FoleyBench include:

- Temporal alignment is within reach for state-of-the-art models, yet accurate rendering of "what" the sound should be—especially for discrete impacts and complex ambiences—remains challenging.
- Multi-source scenes often yield audio "mash-ups" that, while semantically plausible, lack auditory fidelity characteristic of true Foley.
- Ambient/background environments and long-form audio coherence constitute open challenges.
- Text conditioning functions as a semantic prior, signaling that purely visual V2A approaches are often under-constrained.
- Standard datasets hide these nuanced failure modes due to speech/music dominance and sparse Foley-relevant coverage.

Suggested directions for advancing the field include:

- Development of visual encoders with explicit modeling of object materials and interaction forces.
- Enhanced ambient synthesis modules leveraging environmental sound libraries.
- Improved disentanglement and mixing strategies for multi-source scenarios.
- Joint pretraining of video, text, and audio modalities at scale to capture semantic context more robustly.
- Architecture innovations targeting long-form consistency, such as memory-augmented diffusion or autoregressive designs.
- Ongoing expansion and refinement of FoleyBench, with more sound categories, extended durations, and validation involving human experts [2511.13219].

By establishing a rigorously curated, taxonomically diverse, and causally-grounded benchmark with precise evaluation protocols, FoleyBench provides a foundation for advancing Foley-style V2A research and for diagnosing failure modes absent from prior datasets.

Source: https://www.emergentmind.com/topics/foleybench