VGG-Sound+ Audio-Visual Dataset

• VGG-Sound+ is a large-scale audio-visual dataset comprised of 199,467 ten-second clips from YouTube, annotated with 309 distinct and visually grounded sound classes.
• The dataset employs an iterative pipeline combining visual verification, audio filtering, and classifier-driven deduplication to ensure high-quality, noise-reduced samples.
• Baseline evaluations using CNN architectures demonstrate robust performance metrics, and the dataset supports advanced extensions like binaural audio spatialization and multi-label reannotation.

VGG-Sound+ is a large-scale audio-visual dataset curated for training and evaluating deep learning models, particularly in audio recognition and multi-modal modeling. It expands upon the original VGGSound resource by incorporating rigorous noise filtering, refined annotation pipelines, and explicit provisions for future-proof experimental protocols, making it foundational for research into unconstrained, real-world sound recognition and audio-visual understanding (Chen et al., 2020, Zverev et al., 11 Aug 2025, Wang et al., 21 Jan 2026).

1. Dataset Structure and Taxonomy

VGG-Sound+ comprises 199,467 ten-second video/audio clips, totaling approximately 550 hours. Each clip is sourced from public YouTube content and annotated with one of 309 mutually exclusive “leaf-node” classes. The label set is deliberately flat: no explicit hierarchy, taxonomic structure, or superclasses are encoded in the released annotations.

Class design follows principles of visual-groundedness and exclusivity: each class refers to a visually identifiable source or event (e.g., “playing cello,” “dog barking,” “airplane flyby”), with no abstract or synthetic categories included. For convenience, classes can be informally regrouped into semantic families such as people sounds, musical instruments, animal sounds, vehicles, sports, nature and ambience, home and tool sounds, and miscellaneous events.

Per-class distribution is long-tailed, with between 200 and 1,050 clips per class (mean ≈645). A minimum threshold of 200 clips per class is imposed; more frequent categories can exceed 1,000. Partitioning yields 177,837 training samples and 21,630 held-out clips (validation and test) distributed across classes (Chen et al., 2020).

2. Data Collection Pipeline

The construction of VGG-Sound+ follows a four-stage, scalable pipeline:

1. Source Video Acquisition: Approximately 1 million YouTube videos (≥10s duration) are downloaded using search queries derived from ≈600 candidate sound class names (drawn from prior ontologies such as AudioSet and expanded via manual curation in multiple languages).
2. Visual Verification: Each candidate video undergoes frame-wise classification with a pre-trained OpenImages CNN. Sound classes are mapped to visual classifier categories either by name-matching or via semantic embedding: word2vec vectors $S\in\mathbb{R}^{600\times512}$ (sound classes) and $O\in\mathbb{R}^{5000\times512}$ (OpenImages), with an affinity matrix $A=SO^{\top}\in\mathbb{R}^{600\times5000}$. Top-10 frames passing a confidence threshold (score >0.2 on any visual signature) are used to extract centered 10 s segments.
3. Audio Verification: Clips are filtered by a fine-tuned 3-way VGGish model (classes: {speech, music, others}). For each candidate, if the non-target component's probability exceeds 0.5, the clip is rejected (e.g., for “dog barking,” reject if speech or music >0.5).
4. Iterative Noise Reduction and Deduplication: Manual spot-checks (20 clips/class) remove low-purity classes (<50% correct). Classifier-based pruning uses ResNet18 to select “easy positives” (top-3 predicted), with visual similarity retrieval used to recover “hard positives.” Final rounds retrain on these unions, and further deduplication removes YouTube re-uploads via visual fingerprint comparison.

The following table summarizes the pruning cascade:

Stage	Number of Classes	Number of Clips
Initial download	~600	~1,000,000
After visual verification	470	550,000
After audio verification	390	260,000
Final clean-up and dedup	309	199,467

3. Annotation Principles and Noise Management

VGG-Sound+ adopts a strict inclusion policy for both sound classes and individual clips:

• Visual and Auditory Correspondence: Only instances where the labelled event/object is visually present are retained (disambiguating, e.g., “dog barking” with the dog on-screen, not off-camera).
• Broad Noise Filtering: Class-dependent application of VGGish model ensures that non-relevant speech or music is excluded, tuned to prefer coverage over perfection.
• Classifier and Human Filtering: An iterative process involving ResNet18 audio models and visual similarity scores recovers false negatives and reduces residual false positives.
• No Explicit SNR Metrics: The pipeline does not report per-clip signal-to-noise ratio, beyond empirical thresholds for accepted class probability and manual spot-checking.

This suggests that while the dataset maintains strong practical cleanliness, certain sound mixtures or partial event overlaps may persist, as identified in subsequent benchmark extensions (Zverev et al., 11 Aug 2025).

4. Benchmarking Protocols and Baselines

Baseline audio classification experiments are provided for CNN architectures (ResNet18/34/50). The standard training protocol consists of 10 s audio (16 kHz) transformed to log-mel spectrograms, with cross-entropy loss and global average pooling. The optimizer is Adam with initial learning rate $10^{-3}$, reduced on plateau. All models are evaluated on the VGG-Sound test set (all 309 classes) and an AStest (AudioSet intersection).

Key metrics reported:

• mean Average Precision (mAP): up to 0.532 (ResNet50, full VGG-Sound test)
• Area Under the ROC Curve (AUC): up to 0.973
• $d'$ (discriminability index): up to 2.735
• Top-1 Accuracy: up to 0.510
• Top-5 Accuracy: up to 0.764

Deeper models consistently improve performance, with reduced mAP on the more challenging multi-source AudioSet benchmark. No multi-modal fusion or temporal localization architectures are baseline-evaluated, although a plausible implication is that visual information could yield additional gains (Chen et al., 2020).

5. BinauralVGGSound: Spatial Audio Expansion

BinauralVGGSound (also known as SpatialV2A) is a derived resource wherein each mono track from VGGSound is replaced by a generated binaural audio pair. This transformation leverages spherical harmonic domain expansion and HRIR-based convolution according to:

• $s'_m(t) = (D^\top D)^{-1} D^\top \Psi(t)$ (spherical harmonics expansion)
• $\hat{l}(t) = \sum_{m=1}^M h_l(\theta'_m) * s'_m(t)$ and $\hat{r}(t) = \sum_{m=1}^M h_r(\theta'_m) * s'_m(t)$ (binaural rendering)

A U-Net-based spatialization network, conditioned on ResNet–18 video features, supervises the transformation for all 187,195 clips (totaling 519 hours, same 309 classes and captions). Each file includes spatial metadata: horizontal centroid, area fraction, spatial variance, left-right bias, and shape anisotropy derived from sound-source heatmaps. Metrics such as interaural level and time difference and human MOS-S (mean opinion score for spatialization, 3.90/5) demonstrate strong spatial consistency and perceptual quality (Wang et al., 21 Jan 2026).

6. VGGSounder: Multi-Label and Modality-Aware Test Set

VGGSounder (referred to as VGG-Sound⁺ in some contexts) is a comprehensively re-annotated test set for VGGSound. It addresses three major issues: incomplete ground truth, class overlap, and modality misalignment:

• Multi-label Reannotation: Every 10s clip is relabelled so that all co-occurring events are annotated (mean ≈2.4 labels per clip; >60% multi-label).
• Synonym and Superclass Linking: Subclass (e.g., “snare drum”) or synonym (e.g., “timpani”/“tympani”) annotations link to superclasses and equivalents for taxonomic coverage.
• Per-label Modality Tags: Each class is marked audible-only (A$\setminus$V), visible-only (V$\setminus$A), or audio-visual (AV).
• Meta-labels: Binary markers identify confounders—background music, voice-over, and static imagery—present in ~40% of clips.

A crowd-sourced annotation pipeline, seeded by model predictions and verified by expert gold sets, achieves macro-averaged F₁ ≈ 0.68 on a gold-standard set.

VGGSounder supports detailed evaluation protocols: accuracy, F₁, Hit, and a modality confusion metric ($O\in\mathbb{R}^{5000\times512}$0), which quantifies degradation when multi-modal models are presented with additional modalities that distract from their unimodal strengths. For instance, $O\in\mathbb{R}^{5000\times512}$1 denotes the fraction of clips correctly solved by audio only but missed when both audio and video are provided. This suggests practical value in diagnosing and benchmarking multi-modal model fusion (Zverev et al., 11 Aug 2025).

7. Limitations and Prospective Enhancements

Identified constraints include the absence of a hierarchical label structure, long-tail class imbalance, no SNR or temporal event localization, and limited modality fusion at the baseline. Recommended future directions encompass:

• Introducing hierarchical label sets for taxonomy alignment (e.g., with AudioSet).
• Implementing sampling or augmentation for rare classes and class-aware loss functions like focal loss.
• Explicit SNR and event-activity fraction annotation.
• Temporal event localization using combined visual and auditory cues.
• Stronger baselines leveraging transformer or fused CNN-Transformer architectures.
• Expansion to additional data sources and modalities.
• Multi-label and modality-aware benchmarks, as operationalized in VGGSounder.

A plausible implication is that, with these extensions, VGG-Sound+ will continue to serve as a primary benchmark for complex real-world audio-visual tasks, providing rigorous evaluation for foundation models, spatial audio generation, and multi-modal fusion methods (Chen et al., 2020, Zverev et al., 11 Aug 2025, Wang et al., 21 Jan 2026).