Enriched Prompt Pipeline for Audio-Text Synthesis

• Enriched Prompt Pipeline is a multi-stage process that combines audio extraction, language modeling, and contrastive alignment to generate robust audio captions.
• It employs prompt chaining and structured prompts to integrate fine-grained details, improving retrieval performance and zero-shot classification.
• The pipeline demonstrates scalable synthetic caption generation while highlighting trade-offs like inherent biases and computational cost.

AudioSetCaps is a large-scale synthetic audio-caption corpus built from AudioSet through a fully automatic audio-to-caption pipeline that combines a large audio-LLM, a LLM, and contrastive audio-text scoring for refinement. It was introduced to address the scarcity and cost of paired audio-language data for audio-language learning, and it comprises 1.9 million audio-caption pairs, which the authors describe as the largest audio-caption dataset at the time of writing. The same pipeline was also applied to YouTube-8M and VGGSound, producing a broader release of 6,117,099 captions and 18,414,789 audio question-answer pairs, for a total of 24,531,888 audio-language pairs (Bai et al., 2024).

1. Scope, motivation, and dataset definition

AudioSetCaps was proposed in the context of rapidly expanding audio-language modeling, where model quality increasingly depends on the availability of large paired corpora but manual captioning remains time-intensive and labour-heavy. The central premise is that synthetic caption generation can be scaled if the pipeline first extracts fine-grained audio content, then generates captions under explicit constraints, and finally filters or regenerates outputs that appear weakly aligned with the source audio (Bai et al., 2024).

The dataset is derived from AudioSet and consists of 1.9 million audio-caption pairs. Its reported average caption length is 28 words and its vocabulary size is 21k. The same generation pipeline was extended to YouTube-8M and VGGSound, producing 4,023,990 captions and 12,086,037 audio question-answer pairs for YouTube-8M, and 182,189 captions and 592,680 question-answer pairs for VGGSound. Across AudioSet, YouTube-8M, and VGGSound, the released resources total 6,117,099 captions and 24,531,888 audio-language pairs when question-answer data are included (Bai et al., 2024).

A notable design choice is that AudioSetCaps is not only a caption dataset but also an experiment in structured synthetic data generation. The paper treats the corpus as evidence that prompt engineering for audio-language data benefits from decomposition, controlled vocabularies, and automatic alignment checks rather than a single unrestricted generative pass (Bai et al., 2024).

2. Three-stage automated generation pipeline

The AudioSetCaps pipeline has three stages: audio content extraction, caption generation, and caption refinement. In the first stage, Qwen-Audio-Chat is used as the audio-LLM responsible for extracting structured descriptors from an input clip. In the second stage, Mistral-7B converts those descriptors, together with AudioSet labels, into a natural-language caption. In the third stage, LAION CLAP with an HTSAT audio encoder and RoBERTa text encoder scores the audio-caption alignment and triggers regeneration when the generated caption appears less aligned than the corresponding label phrase (Bai et al., 2024).

The extracted information is explicitly partitioned into three channels: an overall audio content description, a speech features description, and a music features description. These texts are lightly sanitized before being inserted into a structured prompt for Mistral-7B. The prompt also includes the AudioSet ground-truth labels, but the instructions explicitly forbid outputting those labels verbatim and forbid mentioning the specific content of speech. Captions are also constrained to be within 50 words (Bai et al., 2024).

The refinement step uses CLAP similarity as an external quality signal. Conceptually, the paper describes a score of the form

$$s(a,c)=\mathrm{sim}\big(f_{\text{audio}}(a), f_{\text{text}}(c)\big),$$

where the generated caption is compared against the audio and then against a simple label-based reference. If the generated caption scores worse than the label phrase, it is treated as suspicious or hallucinated and regeneration is triggered, with a maximum number of attempts to control cost (Bai et al., 2024).

This architecture makes the dataset generation process explicitly modular. The audio model is used for perceptual extraction, the LLM for fluency and compression into caption form, and the contrastive model for post hoc alignment control. The paper presents this division of labour as a core reason the resulting captions remain both detailed and sufficiently grounded (Bai et al., 2024).

3. Enriched prompting, prompt chaining, and structured caption synthesis

A defining feature of AudioSetCaps is prompt chaining in the extraction stage. Rather than asking a single broad question such as describing the audio in detail, the pipeline breaks extraction into sequential subtasks. Qwen-Audio-Chat first produces an overall scene description, then receives follow-up questions about speech only if speech is present, and then receives follow-up questions about music only if music is present. The speech stage targets language, emotion, and gender; the music stage targets genre and instruments (Bai et al., 2024).

The fine-grained categories are fixed in advance. Spoken language is restricted to English, French, Chinese, Japanese, Arabic, Spanish, Hindi, Russian, German, and Portuguese. Speech emotion is restricted to Neutral, Calm, Angry, Excited, Sad, Happy, Fearful, Surprised, Frustrated, and Nervous. Instrument classes are Guitar, Bass, Piano, Drums, Violin, Flute, Trumpet, Saxophone, Clarinet, and Harp. Genre classes are Electronic, Pop, Folk, Rock, Classical, Country, Jazz, Blues, Hip hop, and Reggae. The paper describes this as a classification-style prompting regime that constrains Qwen’s outputs into a structured schema rather than unconstrained prose (Bai et al., 2024).

The caption-generation prompt is likewise strongly structured. It uses a role-setting frame in which the model is asked to act as a professional crowd-sourced worker, then provides three content fields—Description, Speech, and Music—followed by the ground-truth labels and two explicit constraints: do not mention the specific content of speech, and do not output the ground-truth labels in the caption. This arrangement is intended to force semantic integration of trusted labels and extracted audio metadata while preventing trivial copying (Bai et al., 2024).

The paper’s ablations show that this enriched prompting is not a superficial stylistic choice. When fine-grained speech and music content is removed from prompts, retrieval performance drops from 39.7 to 38.4 in text-to-audio R@1 and from 50.9 to 48.3 in audio-to-text R@1 on the AudioCaps-based ablation setting. Replacing prompt chaining with a single complex Qwen prompt reduces audio-to-text R@1 from 50.9 to 49.2, and removing caption refinement reduces it to 48.9. This suggests that decomposition, structured metadata, and post-generation scoring each contribute measurably to downstream utility (Bai et al., 2024).

4. Scale, caption properties, and fine-grained annotation coverage

AudioSetCaps is characterized not only by scale but also by unusually high fine-grained attribute density. The paper contrasts it with earlier synthetic audio-caption resources by reporting both caption length and the frequency of explicit mentions of language, emotion, instrument, and genre (Bai et al., 2024).

Dataset	Avg. caption length	Vocabulary / fine-grained note
LAION-Audio-630K	7 words	Short captions
WavCaps	8 words	Limited fine-grained mentions
Auto-ACD	18 words	Moderate fine-grained coverage
Sound-VECaps	40 words	Longer captions
AudioSetCaps	28 words	21k vocabulary

The paper also quantifies fine-grained attribute mentions in thousands. AudioSetCaps contains 13.8K language mentions, 336.0K emotion mentions, 690.0K instrument mentions, and 682.0K genre mentions. These counts exceed those reported for WavCaps, Auto-ACD, and Sound-VECaps, and the authors directly attribute this increase to prompt-chained extraction and structured conditioning (Bai et al., 2024).

Dataset	Language (K)	Emotion (K)	Instrument (K)	Genre (K)
WavCaps	0.01	1.4	11.4	3.8
Auto-ACD	3.5	16.6	299.8	214.3
Sound-VECaps	6.7	64.3	207.7	134.1
AudioSetCaps	13.8	336.0	690.0	682.0

The distribution of these attributes is not uniform. In AudioSetCaps, 86% of speech annotations are English and 80% of emotion annotations are Neutral; guitar and bass dominate instruments, and electronic dominates genres. The paper presents this as both evidence of richness and evidence of bias inherited from the underlying source distribution (Bai et al., 2024).

5. Training roles and empirical performance

AudioSetCaps is used for both automated audio captioning and audio-text retrieval. For automated audio captioning, the paper trains an HTSAT encoder with a BART decoder using a standard autoregressive cross-entropy objective,

$$\mathcal{L}_{AAC}=-\frac{1}{N}\sum_{n=1}^{N}\log p(c_n \mid c_{1:n-1}, x).$$

When trained on AudioSetCaps alone and evaluated on AudioCaps, the model reaches BLEU$_1$ 73.9, BLEU$_4$ 30.8, ROUGE$_L$ 52.7, METEOR 26.2, CIDEr 83.9, SPICE 18.6, and SPIDEr 51.3. Training on AudioSetCaps together with AudioCaps and Clotho raises CIDEr to 84.8 and SPIDEr to 51.6, which the paper reports as the best setting in its captioning experiments (Bai et al., 2024).

For audio-text retrieval, the downstream model uses HTSAT and RoBERTa with a CLIP-style symmetric InfoNCE formulation:

$$\mathcal{L}_{i}^{a}=-\log \frac{\exp \left(e_{i}^{a} \cdot e_{i}^{t} / \tau\right)}{\sum_{j=1}^{B} \exp \left(e_{i}^{a} \cdot e_{j}^{t} / \tau\right)},$$

$$\mathcal{L}_{i}^{t}=-\log \frac{\exp \left(e_{i}^{t} \cdot e_{i}^{a} / \tau\right)}{\sum_{j=1}^{B} \exp \left(e_{i}^{t} \cdot e_{j}^{a} / \tau\right)},$$

$$\mathcal{L}_{ATR}=\frac{1}{2B} \sum_{i=1}^{B}(\mathcal{L}_{i}^{a} + \mathcal{L}_{i}^{t}).$$

Pre-training solely on AudioSetCaps yields text-to-audio retrieval of R@1 46.3, R@5 79.7, and R@10 90.0, and audio-to-text retrieval of R@1 59.7, R@5 86.0, and R@10 93.2. The paper describes these results as state of the art among monolingual pipelines and close to a multilingual SONAR-based system (Bai et al., 2024).

The same pre-training also improves zero-shot classification on tasks aligned with the extracted metadata. Reported results include 32.6% on CommonLanguage speech language recognition, 28.5% on CREMA-D emotion classification, 25.7% on RAVDESS emotion classification, 70.5% on GTZAN music genre recognition, and 59.7% on OpenMIC instrument recognition. The paper directly connects these gains to the explicit inclusion of language, emotion, genre, and instrument information in generated captions (Bai et al., 2024).

6. Limitations, biases, and methodological significance

The paper identifies several limitations. First, the fine-grained annotations inherit the source distribution’s biases, most visibly the dominance of English language and Neutral emotion. Second, residual hallucinations remain even after CLAP-guided regeneration. Third, scaling the three-model inference pipeline to tens of millions of pairs incurs substantial computational cost, even though the implementation is described as runnable on consumer GPUs (Bai et al., 2024).

These limitations also clarify the methodological stakes of AudioSetCaps. The dataset is not presented merely as a static resource but as a case study in automatic multimodal corpus construction. Its core design principles are task decomposition through prompt chaining, controlled vocabularies for perceptual attributes, separation of extraction and generation roles across models, structured language-model prompts with trusted but non-copyable labels, and contrastive-model scoring as a hallucination check. The paper argues, and the ablations support, that the gains come from this orchestration rather than from synthetic generation alone (Bai et al., 2024).

A plausible implication is that AudioSetCaps marks a shift in audio-language data construction from template-centric caption synthesis toward pipeline-centric synthesis. In this view, the dataset’s empirical importance lies not only in its scale but in demonstrating that caption quality, retrieval effectiveness, and zero-shot transfer can all benefit when caption generation is treated as a controlled, multi-stage alignment problem rather than a one-shot generative task (Bai et al., 2024).