AudioCoT: Multimodal Audio & CoT Dataset

• AudioCoT is a multimodal dataset featuring paired audio segments, videos, natural-language captions, and stepwise chain-of-thought (CoT) annotations designed for integrated audio understanding and synthesis.
• It comprises a classic audio captioning subset with 5,000 non-speech audio clips and an expanded multimodal resource for video-to-audio generation and editing tasks.
• The dataset employs rigorous crowdsourcing and hybrid automated protocols to ensure high-quality, diverse, and interpretable annotations for benchmarking and model training.

AudioCoT is a freely-available, large-scale multimodal dataset designed to support research in audio captioning and Chain-of-Thought (CoT) reasoning for audio generation and editing. It provides paired audio segments, videos, natural-language captions, and stepwise structured reasoning annotations, with coverage designed to advance both audio understanding and reasoning-guided synthesis tasks in multimodal LLMs. AudioCoT exists in two main forms: (1) the original human-annotated audio captioning dataset for training and evaluating natural language descriptions of non-speech audio (Lipping et al., 2019), and (2) the expanded multimodal reasoning resource supporting video-to-audio generation and editing, with fine-grained CoT annotations (Liu et al., 26 Jun 2025).

1. Dataset Composition and Structure

AudioCoT comprises distinct data modalities and annotation layers for modern audio and multimodal research requirements:

• Classic Audio Captioning Subset (Lipping et al., 2019):
  • Clips: 5,000 distinct non-speech audio files (15–30 seconds), sourced from Freesound.org, normalized to unity peak amplitude. Speech, music, and sound effects are excluded.
  • Captions: Each clip is associated with five final captions, human written, grammatically reviewed, and accuracy rated. The aggregate corpus contains 25,000 captions.
  • File Formats: Mono WAV audio (16 kHz or original sampling rate), and JSON files mapping audio clip IDs to final captions, AMT HIT IDs, and scoring metadata.
• Multimodal CoT Resource (Liu et al., 26 Jun 2025):
  • Audio–Video Pairs: 741.1 hours spanning VGGSound and non-speech AudioSet.
  • Audio–Text Pairs: 1,790.7 hours from AudioSet-SL, Freesound, AudioCaps, and BBC Sound Effects.
  • Temporal Segmentation: All clips are uniformly segmented to 9.1 seconds, resulting in ≈1,000,000 samples.
  • Content: Roughly balanced between music and sound effects, with extensive coverage of urban scenes, wildlife, mechanical actions, domestic Foley, musical instruments, and ambient nature.
  • Annotations: JSONL files store for each segment: sample ID, file paths, free-text caption, and a structured CoT chain encoding step-by-step reasoning.
  • CoT Chains: For each sample, structured sequences describe visual cues, semantic audio events, and detailed sound properties (volume, timbre, temporal duration), optionally linked to object region-of-interest (ROI) data.

2. Crowdsourcing and Automated Annotation Protocols

AudioCoT’s annotation protocols are designed for comprehensive coverage and quality:

• Three-Stage Crowdsourcing Pipeline (Lipping et al., 2019):
  • Step 1: Five independent workers per audio clip provide single-sentence descriptions (≥8 words), restricted to observable content.
  • Step 2: Separate annotators (excluding original authors) perform grammatical correction/rephrasing.
  • Step 3: Three independent raters evaluate all 10 captions (5 initial, 5 edited) per clip for accuracy and fluency (scale 1–4); the five best by sum accuracy (then fluency) are selected.
• Automated and Hybrid Annotation Process (Liu et al., 26 Jun 2025):
  • Visual Cues Extraction: VideoLLaMA2 extracts temporal and semantic video features.
  • Audio Description: Qwen2-Audio generates audio descriptions.
  • Chain Synthesis: GPT-4.1-nano assembles structured CoT chains (stepwise reasoning) for each modality and task.
  • ROI Annotation: Grounded SAM2 localizes objects and associates corresponding reasoning segments.
  • Editing Chains: Existing chains are reorganized for instruction-based audio editing scenarios.

3. Quality Control and Diversity Metrics

Rigorous quality control is integral to AudioCoT:

• Spell Check and Typographical Error Rates (Lipping et al., 2019):
  • All captions processed with CyHunspell (US and UK dictionaries).
  • Step 2 reduces typographical errors by ~45% relative to initial captions.
  • Selected final captions show up to 18× fewer typos in high-fluency items compared to low-fluency ones.
• Lexical Diversity and Jaccard Similarity (Lipping et al., 2019):
  • Words are stemmed with NLTK Snowball stemmer; Jaccard similarity between two captions $$J(a,b)=|\mathbb{W}_a\cap\mathbb{W}_b|/|\mathbb{W}_a\cup\mathbb{W}_b|$$.
  • Mean Jaccard for initial/edited: 0.62; among final 5 captions, 0.24—indicative of significant diversity yet shared semantic core.
• Alignment and Filtering (Liu et al., 26 Jun 2025):
  • CLAP score < 0.2 triggers automated re-prompting/discarding for misaligned pairs.
  • 5% of samples in each stage are manually reviewed; thresholds enforced if failure rates exceed 5%.
• CoT Chain Statistics (Liu et al., 26 Jun 2025):
  • Average chain length: mean ≈ 5.2 steps, median = 5.
  • Step-count distribution: 2 steps (8%), 3 (15%), 4 (20%), 5 (25%), 6 (17%), 7+ (15%).

4. Annotation Schema and Data Formats

AudioCoT employs explicit and structured annotation schemas:

Split	Filetype	Content/Annotation Type
audio/	.wav	Audio segments (9.1 s)
video/	.mp4	Video segments (9.1 s)
metadata/	.jsonl	For train/val/test; includes CoT and caption data
rois/	.json	Region-of-interest bounding boxes and metadata

• Classic Captioning: Each audio file maps to five final captions, with granular accuracy and fluency scores.
• CoT Reasoning Chains: Each entry includes sample and file IDs, caption, and a series of stepwise annotations (step index, visual cue, semantic event, synthesis parameters). Object-centric tasks add ROI IDs and bounding boxes.

5. Benchmarking, Applications, and Evaluation Metrics

AudioCoT is designed to serve as a benchmark and training resource for several research tasks:

• Primary Applications:
  • MLLMs for audio/V2A generation: Training/fine-tuning multimodal LLMs with explicit reasoning chains.
  • Object-centric audio design and editing: ROI-based reasoning supports interactive and instruction-based Foley workflows.
  • Evaluation and Benchmarking: In-distribution (e.g., VGGSound test split) and out-of-distribution (MovieGen Audio Bench) evaluation sets.
  • Causal and temporal reasoning: Supporting modeling of temporally and spatially grounded audio phenomena.
• Dataset Metrics (Liu et al., 26 Jun 2025):
  • Fréchet Distance (FD) in OpenL3 space to compare distributional alignment between generated and reference audio.
  • KL-Divergence for class distribution similarity.
  • CoT Chain Length and Distribution for annotation completeness.

A plausible implication is that the explicit CoT annotation structure enables interpretability and error analysis in reasoning-driven audio generation pipelines.

6. Limitations and Release Details

AudioCoT’s scope and design constraints impose several caveats:

• Domain Coverage: Restriction to non-speech audio; no speech, dialogue, or multilingual content included.
• Temporal/Spatial Resolution: Reasoning annotations generally omit sub-100 ms timing cues and fine-grained spatialization.
• Bias and Coverage: Certain rare, culturally specific or indigenous sounds are under-represented.
• Annotation Consistency: Some automated CoT chains may contain errors, hallucinations, or over-generalizations, though filtering steps and strict length/token constraints mitigate the issue.
• Licensing and Tools: Distributed under CC-BY; accompanied by ingestion scripts, data loaders (PyTorch/TensorFlow), and utilities for metric computation.

AudioCoT is accessible via the Tampere University Audio Research Group’s website, GitHub, and the ThinkSound project page, with structured metadata and APIs enabling immediate integration into contemporary audio or multimodal machine learning pipelines (Lipping et al., 2019, Liu et al., 26 Jun 2025).