---
title: Audio-Based Grounding Dataset
url: https://www.emergentmind.com/topics/audio-based-grounding-dataset
type: topic
---

# Audio-Based Grounding Dataset

An audio-based grounding dataset is a curated or constructed resource that explicitly links raw or processed audio signals to spatial, temporal, or semantic elements in another modality, most commonly vision (images, video, or 3D point clouds). These datasets are central to the development, evaluation, and benchmarking of models that localize, match, or segment visual elements using sound—especially when the grounding cue is given as speech or non-speech audio rather than canonical text. The field now encompasses tasks including temporal localization in video from spoken utterances, segmentation of 2D/3D visual regions from environmental sounds, spatial grounding of impacts or events given audio, and even affordance segmentation cued by action sounds.

## 1. Key Dataset Types and Scope

Audio-based grounding datasets are heterogeneous in their technical design, modalities, and target tasks. The major axes of differentiation include:

- **Modality pairing**: Audio–video (spoken video grounding, temporal alignment), audio–image (segmentation, object localization), audio–3D (audio-conditioned 3D point cloud grounding), audio–robotics (robotic manipulation via multimodal input), and audio–question-answer (audio-visual instruction and QA).
- **Annotation granularity**: Pixel-level (segmentation masks), bounding box, temporal interval, or class-level labels.
- **Audio type**: Raw environmental sounds, impact sounds, speech (human or TTS), or synthetic mixtures (speech plus sound).
- **Dataset scale**: Varies from thousands to hundreds of thousands of examples.

For large-scale spoken video grounding, the ActivityNet Speech dataset exemplifies direct audio–video query alignment, offering over 70,000 spoken queries temporally aligned to unconstrained ActivityNet videos with synthetic yet challenging noise augmentation [2209.00277]. For spatial segmentation grounded by single-word speech, the single-word audio-guided image segmentation dataset includes 66,202 image–audio pairs paired with 35 object-class keywords spanning 35 accents, supporting robust, cross-accent grounding [2511.22025]. In 3D, Audio-3DVG generated 172,995 synthetic audio descriptions for 2,714 3D indoor scenes, with referential utterances mapped to ScanNet-derived real-world environments [2507.00669]. Task coverage further includes pixel-level affordance mask prediction from action sounds (AV-AG/AVAGD, 12,768 images/masks, 5,203 sounds, 97 objects, 55 affordances) [2512.02005], grounding of real pointwise impact sounds to object mesh locations (RealImpact, 150,000 multichannel samples, 50 objects, 600 microphone poses each) [2306.09944], and simultaneous speech/non-speech grounding in images with controlled mixtures (Extended-IS3, 6,840 mixtures) [2503.18880].

## 2. Annotation Protocols and Technical Construction

Annotation schemes and design methodology are highly task-specific but share key rigor:

- **Temporal alignment**: For activity grounding from speech, segment times are inherited or mapped from existing text-captioned datasets, e.g., ActivityNet or movie audio description corpora (e.g., MAD, 384,000 temporally anchored sentences in 1,207 h video) [2112.00431].
- **Spatial supervision**: Segmentation- and detection-oriented datasets propagate or manually annotate per-pixel or bounding-box regions (e.g., IS3/Extended-IS3, AV-AG).
- **Sound-event labeling**: In text-to-audio grounding, sound events are identified via automated phrase chunking (NP/VP extraction), then manually merged and temporally segmented in the audio [2102.11474].
- **Audio manuscript and diversity**: Audio sources may be crowdsourced (spontaneous speech or actors/readers, e.g., ActivityNet Speech), synthetic via TTS (Audio-3DVG, single-word), or curated from environmental recordings (RealImpact, ASPED v.b [2509.19295]).
- **Multimodal QA**: Benchmarks such as audio-visual QA datasets involve human-authored audio-attentive questions with dual annotator passes and inter-annotator agreement metrics (κ=0.82 for AudioVisQA [2407.15046]).

Realistic background noise is often injected (ActivityNet Speech mixes ESC-50 noise at α ∈ [0.5, 0.7]), and close attention is paid to class balance, coverage, and split protocols, with many resources providing stratified “seen/unseen” splits (AV-AG zero-shot setting, single-word dataset with held-out images/utterances).

## 3. Data Formats, Splits, and Statistical Properties

Audio-based grounding datasets typically standardize input modalities and splits:

- **Audio**: 16 kHz PCM WAV for speech and environmental sounds, occasionally 48 kHz for multichannel impact capture (RealImpact).
- **Visual**: Untrimmed videos, still images, or 3D point clouds (ScanNet-derived PCDs in Audio-3DVG), all mapped to their audio referents.
- **Masks/Labels**: Pixel-accurate masks (PNG), bounding boxes (xyxy), temporal intervals, or class indicators.
- **Splits**: Standard train/validation/test, stratified by utterance, scene, or object-action pair, or a single held-out evaluation set for benchmarking (Extended-IS3).
- **Examples**:

  | Dataset                   | #Audio    | #Visuals    | Split       | Annotation            |
  |---------------------------|----------:|------------:|:------------|:---------------------|
  | ActivityNet Speech        | 70,000    | 20,000      | 37k/18k/17k | Temporal intervals    |
  | AV-AG (AVAGD)             | 5,203     | 12,768      | seen/unseen | Pixel masks (dual)    |
  | Audio-3DVG                | 173,000   | 2,714       | per dataset | 3D bbox, class label  |
  | RealImpact                | 150,000   | 50          | per object  | Impact position, RGBD |
  | Single-word Segmentation  | ~2,300    | 66,202      | 50k/8k/8k   | Pixel mask            |
  | Extended-IS3              | 6,840     | 3,420       | eval only   | Pixel masks, transcript|
  | ASPED v.b                 | N/A       | N/A         | 1,056/132/132h | Presence flag, 1 fps |

Common notational conventions include $V = \{v_i\}_{i=1}^{N_v}$ for visual objects, $Q = \{q_j\}$ for queries, $A$ or $F$ for audio-derived features, and $[τ_s^{(j)}, τ_e^{(j)}]$ for temporal localization.

## 4. Supported Tasks and Benchmarking Protocols

Audio-based grounding datasets are usually designed to support one or more of the following tasks:

- **Temporal video grounding**: Localization of a video fragment corresponding to a spoken or audio query, with evaluation via $R@K$ (recall at K proposals) at IoU thresholds $m \in \{0.3,0.5,0.7\}$, and mean IoU (mIoU) [2209.00277, 2112.00431].
- **Spatial segmentation or detection**: Pixel-level or region-level mask prediction given an audio cue, evaluated via mIoU, F-score, or object-level mAP [2511.22025, 2512.02005].
- **Simultaneous mixed-audio grounding**: Two or more overlapping sources (e.g., speech plus sound), with separate segmentation heads and disentanglement loss (Extended-IS3) [2503.18880].
- **3D grounding/localization**: Audio-conditioned retrieval of 3D bounding boxes or proposals, with Acc@0.25/0.5 and top-1 accuracy [2507.00669].
- **Object recognition and weakly supervised grounding**: Classification of the object mentioned in paired speech and image, and multimodal retrieval [1906.06147].
- **Audio-visual question answering**: Open-ended QA with audio-aware questions, measured by accuracy, exact match, and F1 metrics [2407.15046].
- **Pedestrian or event detection**: Audio-based presence prediction aligned with visual frames, with balanced accuracy in noise-dominant environments [2509.19295].
- **Affordance segmentation**: Given an action sound, segment out the functional and dependency regions in the corresponding object image; performance is measured by mIoU and F-score [2512.02005].

Loss functions include binary cross-entropy (for masks or framewise activity), $L_{IoU}$, cross-entropy for classification, ranking or alignment losses, and specific disentanglement objectives for mixed-audio domains.

## 5. Data Access, Licensing, and Limitations

Access protocols and licensing regimes vary:

- **Open access**: Most datasets are available via GitHub or Hugging Face, with links explicitly provided (e.g., ActivityNet Speech: https://github.com/ActivitySpeech/ActivitySpeech, RealImpact: https://github.com/marmot-xy/Spoken-Video-Grounding, AVAGD: https://jscslld.github.io/AVAGFormer/, Audio-3DVG: Spark-TTS scripts and code released, Extended-IS3: https://github.com/kaist-avelab/seeing-speech-and-sound, etc.).
- **Licensing**: Datasets generally inherit source licenses (e.g., ActivityNet’s research license) or are released with specific academic/non-commercial clauses (e.g., AudioVisQA: CC-BY-NC 4.0 [2407.15046]).
- **Limitations**:
  - Coverage: Some datasets are relatively small, domain-specific, or English-only (e.g., AudioVisQA: 120 clips).
  - Annotation: Not all include human freeform annotation or inter-annotator agreement; some rely on templates or synthetic generation.
  - Noisy or synthetic speech: Synthetic speech (Audio-3DVG, single-word segmentation) may not fully capture natural acoustic variability.
  - Task coverage: Future directions include expanding to multi-sound scenes, multilingual QA, higher event density, and improved simulation-to-real transfer.

## 6. Significance, Research Impact, and Open Challenges

Audio-based grounding datasets have catalyzed advances in:

- **End-to-end learning**: They enabled direct training from raw audio without textual intermediates, increasing the robustness of multimodal systems to linguistic and acoustic variability [2511.22025].
- **Sim-to-real evaluation**: Standardized impact/affordance datasets (RealImpact, AV-AG) provide calibration testbeds bridging simulation and real-world sound spatialization [2306.09944].
- **Zero-shot and generalization analysis**: Explicit “unseen” splits facilitate probing of model extrapolation to novel objects or affordances, which is rarely covered in canonical AVS or 3DVG settings [2512.02005].
- **New tasks**: Datasets such as Extended-IS3 support joint and disentangled visual grounding from mixed audio, and Audio-VisQA introduces multi-turn, audio-attentive, open-ended question answering [2503.18880, 2407.15046].

Outstanding challenges include scaling multimodal coverage, refining annotation for real-world and noisy conditions, and extending beyond segmentation or bounding box formats to richer scene graphs or narrative-based grounding. The convergence of large-scale datasets, audio-visual foundation models, and nuanced benchmarking will continue to drive the development of robust, generalizable audio-based grounding systems.

Source: https://www.emergentmind.com/topics/audio-based-grounding-dataset