---
title: 'VGGSounder: Multi-label Audio–Visual Benchmark'
url: https://www.emergentmind.com/topics/vggsounder
type: topic
---

# VGGSounder: Multi-label Audio–Visual Benchmark

VGGSounder constitutes a comprehensively re-annotated, multi-label test set for high-fidelity evaluation of audio–visual foundation models. Originating as an extension and correction of the original VGGSound dataset, VGGSounder introduces rigorous per-clip, per-class, and per-modality labeling to address the pitfalls of incomplete annotation, class overlap, and modality misalignment. It is specifically designed to yield reliable benchmarks for contemporary multi-modal recognition and fusion models, providing both nuanced ground truth and the tools for fine-grained analysis of modality-specific performance and model fusion robustness [2508.08237].

## 1. Construction and Annotation Process

VGGSounder is built on top of the publicly available VGGSound test split, comprising 15,446 ten-second YouTube clips spanning 309 classes. The original VGGSound employed an automated labeling pipeline that enabled large-scale annotation but suffered from two critical issues: nearly half (48.4%) of the “ground-truth” labels were either missing or modality-inaccessible, and systematic class overlap was prevalent due to synonyms and hierarchical relationships (e.g., “cow lowing” and “cattle mooing”).

To correct these problems, the VGGSounder annotation pipeline consists of three stages:

1. **Gold-standard reference labeling:** Four in-house experts exhaustively labeled 417 randomly sampled clips (with coverage across all classes), specifying modality presence (audible, visible, or both) for each class. This set functioned both as catch trials and ground truth for vetting external annotators.
2. **Proposal generation:** Label proposals (≈30 per clip) were generated via three channels: (a) top predictions from six pretrained models (CAV-MAE, AV-Siam, Equi-AV, DeepAVFusion, Gemini 1.5 Flash/Pro), (b) consensus labels from overlapping embedding model outputs, and (c) a short list of common ambient classes.
3. **Crowdsourced multi-labeling:** Each clip was labeled by three expert-selected MTurk workers, who indicated modality for each proposal, could introduce missing classes, and flagged meta-classes (background music, voice-over, static images). Annotation batches with less than 25% F1 on gold-standard clips were discarded and repeated.

A post-processing step automatically injected synonymous and hierarchical classes (e.g., mapping “cow lowing” to also include “cattle mooing”). Majority voting merged individual annotations into clip-level gold labels.

### Table: Core Statistics of VGGSounder

| Statistic                | Value                        | Comment                                         |
|--------------------------|------------------------------|-------------------------------------------------|
| Clips                    | 15,446                       | 10s YouTube segments                            |
| Classes                  | 309                          | Multi-modal events                              |
| Labels/clip (avg)        | ≈2.4                         | Substantial event overlap                       |
| Fraction multi-label     | 85%                          | >1 class per clip is the norm                   |
| Modality split (labels)  | A∖V: 26.4%; V∖A: 1.3%; AV: 72.3% | Modalities explicitly distinguished        |
| Meta-classes (any)       | 40%                          | Background music, voice-over, or static images   |

Most VGGSounder clips feature multiple simultaneous classes, explicit modality designations (audible-only, visible-only, or both), and flags for modality-confounding meta-classes.

## 2. Improvements over the Original VGGSound

VGGSound, while foundational, imposes three main limitations for benchmarking modern multi-modal models:

- **Single-label only:** Clips often contain multiple co-occurring events, but only one is credited.
- **Overlapping class definitions:** Coexistence of synonyms or class hierarchies manifests as label ambiguity.
- **Modality misalignment:** A large fraction of clips pair visual labels with inaudible events or audio labels with invisible events.

VGGSounder corrects these by:

- Enabling comprehensive **multi-label classification**—clips can be associated with every event actually present.
- Systematically **merging synonymous and hierarchical labels** in annotation and post-processing.
- Attaching **per-label modality annotations** and **meta-labels** (background music, narration, static visuals), empowering controlled subset and ablation analyses.

*This approach yields a high-fidelity, modality-disambiguated evaluation environment, enabling discrimination between A-only, V-only, AV, and modality-confounded cases.*

## 3. Evaluation Protocol and Metrics

Model evaluation with VGGSounder supports nuanced analyses reflecting the granularity of its annotations. For each input clip, models are assessed using micro-averaged metrics derived from their top-k predictions ($k \in \{1,3,5,10\}$):

- **Subset Accuracy:** Fraction of clips for which the predicted set exactly matches the true label set:
  $$
  \mathrm{Acc}_{\mathrm{subset}} = \frac1N\sum_{x=1}^N\mathbf{1}\bigl[\hat y(x)=y(x)\bigr]
  $$
- **F₁-score:** Micro-averaged harmonic mean of per-clip precision and recall:
  $$
  \mathrm{Precision} = \frac{|\hat y\cap y|}{|\hat y|},\quad
  \mathrm{Recall} = \frac{|\hat y\cap y|}{|y|},\quad
  F_1 = \frac{2}{1/\mathrm{Precision}+1/\mathrm{Recall}}
  $$
- **Hit Rate:** Fraction of clips for which the predicted and ground-truth sets overlap:
  $$
  \mathrm{Hit} = \frac1N\sum_{x=1}^N \mathbf{1}\bigl[\hat y(x)\cap y(x)\neq\emptyset\bigr]
  $$
- **Modality Confusion ($\mu$):** Percentage of clips where model accuracy *drops* when provided with both modalities (relative to best unimodal performance) for a specific subset $M$ (audible, visible, or both):
  $$
  \mu_M = 100 \frac{1}{N}\sum_{x=1}^N \mathbf{1}\bigl[m(x)\text{-correct} \wedge av(x)\text{-wrong}\bigr] \qquad (x\in M)
  $$

Evaluation is conducted in audio-only, visual-only, and audio–visual input modes; results are stratified by modality subset and by meta-classes.

## 4. Benchmark Models and Testing Regimes

Two major model categories are benchmarked on VGGSounder:

- **Embedding models:** (e.g., CAV-MAE, AV-Siam, DeepAVFusion, Equi-AV) are fine-tuned on the VGGSounder training set and evaluated via top-k classification.
- **Foundation models:** (e.g., Gemini 1.5 Flash/Pro, Gemini 2.0 Flash, VideoLLaMA-2, Unified-IO-2, PandaGPT, Ola) are assessed zero-shot (via class-list prompts) or with LLM-assisted scoring (open-source via Qwen-3).

Testing regimes are explicitly designed to analyze both overall performance and fusion-specific effects, leveraging VGGSounder’s per-clip, modality-aware labeling structure.

## 5. Experimental Findings and Analytical Insights

VGGSounder-driven experiments yield several key findings:

- **Superiority of AV fusion:** Across all metrics, models achieve best performance when both modalities are available. However, specialized embedding models retain superiority on pure audio tasks.
- **Prevalent visual bias:** Most foundation models display a “visual bias,” outperforming on visual streams by ≈10–20 percentage points compared to audio, when evaluated unimodally.
- **Fusion brittleness (modality confusion):** Every model exhibits nonzero modality confusion; 4–11% of clips are classified correctly by one modality but misclassified with both. Embedding models more often suffer when adding audio to visual input; foundation models often lose audio accuracy when provided visuals, indicating imperfect fusion strategies.
- **Meta-class effects:** 
  - Presence of background music reduces audio F₁ by 3–5 points across all models.
  - Voice-over narration severely impairs embedding model audio F₁ (≈9 point drop) yet has minor or even beneficial effects on foundation models such as VideoLLaMA-2 and Unified-IO-2, suggesting learned speech-ignoring mechanisms.
  - Static images degrade visual F₁ by 6–8 points for all models, while sometimes boosting audio-only accuracy, indicating model adaptation when vision is uninformative.

**Benchmarking summary:** Specialized embedding models excel on audio; foundation models match or surpass at audio–visual classification but resort to visual shortcuts. Fusion remains fragile: adding a second modality negatively impacts 5–10% of evaluated cases, captured by $\mu$.

## 6. Impact and Applications

By transforming a noisy, single-label, and modality-ambiguous dataset into a modality-aware, multi-label suite, VGGSounder sets a high standard for audio–visual foundation model evaluation. Its annotation rigor and inclusion of modality confusion metrics allow researchers to diagnose fusion failures, modality bias, and performance on confounded or multi-event scenes—scenarios prevalent in real-world media.

A plausible implication is that future benchmarks and models should not only aim for higher mean accuracy, but must also exhibit robust modality fusion and resilience to meta-class phenomena such as background noise or visual channel corruption.

## 7. Conclusion

VGGSounder reshapes the landscape of audio–visual model benchmarking by introducing structured, multi-label, modality-precise annotation, robust evaluation metrics, and a principled framework for revealing model fusion weaknesses [2508.08237]. Its adoption enables precise, reproducible, and context-aware assessment of the next generation of multi-modal foundation models, driving both model and dataset advancement in the domain.

Source: https://www.emergentmind.com/topics/vggsounder