Audio Question Answering (AQA)

• Audio Question Answering (AQA) is a multimodal task that interprets acoustic scenes to answer questions on event detection, temporal relations, and compositional nuances.
• State-of-the-art methods leverage transformer-based architectures with spectro-temporal processing and cross-modal fusion to enhance reasoning and accuracy.
• Applications include environmental monitoring, speech or animal call analysis, with curriculum learning and RL-based fine-tuning significantly boosting performance.

Audio Question Answering (AQA) is a multimodal reasoning task in which a model must understand the content of a complex audio scene and answer a natural-language question about that scene (Wijngaard et al., 9 Jul 2025). The field is motivated by the broader ambition of auditory intelligence: developing systems that not only perceive and label acoustic events, but also reason about temporal, spectral, and compositional relations, often under real-world conditions of imbalance, ambiguity, and noise. Typical applications include environmental sound understanding, speech or animal call analysis, and explorations of general audio–language reasoning. Research in AQA spans dataset development, model architectures, learning paradigms, evaluation metrics, and the design of benchmarks for both general and task-specialized settings.

1. Task Definition, Objectives, and Challenges

AQA is formally defined as mapping an audio clip and accompanying question to an answer, which may be in natural language or as a multiple-choice selection. Inputs are acoustic recordings (e.g., environmental, speech, animal calls) and corresponding questions that query various properties: event detection (“What is the main sound?”), temporal relations (“Which animal called after the horn?”), attribute identification (“How many bird calls?”), or compositional inferences (“Which animal call overlaps with background music?”) (Wijngaard et al., 9 Jul 2025, Behera et al., 2023).

Principal objectives are:

• Acoustic understanding: extracting event attributes (onset, duration, count, type)
• Temporal reasoning: ordering, duration, causal/relational queries
• General reasoning: composition, integration of world/background knowledge

Key challenges include:

• Data imbalance: Certain classes (e.g., speech, music) dominate, causing poor minority-class generalization (Wijngaard et al., 9 Jul 2025).
• Question variation: Difficulty ranges from trivial to highly abstract.
• Training instability: RL-based fine-tuning can be noisy, especially on challenging or ambiguous queries.

2. Datasets, Data Generation, and Curation

AQA research utilizes a spectrum of datasets, from synthetic (CLEAR, DAQA) to crowdsourced (ClothoAQA) and procedurally generated large-scale corpora (Abdelnour et al., 2018, Fayek et al., 2019, Lipping et al., 2022, Behera et al., 2023). Recent advances have included automated pipelines for question and answer generation using LLMs (Behera et al., 2023), leading to the construction of data resources with greater diversity and scale:

Dataset	#Clips	#Unique Questions	#Unique Answers
ClothoAQA	1,991	9,153	830
AQUALLM-Clotho	5,929	438,600	25,235
AQUALLM-AudioCaps	51,308	728,310	35,008
CLEAR (synthetic)	50,000	130,957	47

Synthetic datasets often rely on functional-program templates for compositional reasoning (Abdelnour et al., 2018), while large-scale automated pipelines (e.g., AQUALLM) employ answer candidate extraction, LLM-driven question generation and verification, and paraphrasing to maximize question/answer diversity and mitigate overfitting (Behera et al., 2023).

Benchmarking initiatives like DCASE 2025's AQA task integrate domain diversity with three subsets—bioacoustics, temporal soundscapes, and complex reasoning—posing additional stress on both data balance and generalization (Yang et al., 12 May 2025).

3. Model Architectures and Learning Paradigms

AQA model architectures have evolved from visual-QA adaptations, such as FiLM or MAC networks (Abdelnour et al., 2019, Abdelnour et al., 2021, Fayek et al., 2019), to attention-based and transformer-derived cross-modal models. Distinctive inductive biases for audio have proven crucial:

• Spectro-temporal front ends: 1D convolutions along time/frequency (Abdelnour et al., 2021), spectrogram transformers (e.g., AST) (Sridhar et al., 2024).
• Multimodal fusion: Cross-attention or joint encoding of audio-text (Sudarsanam et al., 2023, Li et al., 2023).
• Temporal reasoning modules: Multi-scale attention to encode both short- and long-duration events (Li et al., 2023).
• Auxiliary conditioning: Feature-wise linear modulation (FiLM) and multi-controller extensions (MALiMo) for adaptive fusion (Fayek et al., 2019).
• Large Audio-LLMs (LALMs): Recent systems fuse pretrained audio encoders with LLM backbones (Qwen2-Audio, AudioFlamingo2, Gemini), sometimes using adapters or projection modules for alignment (Yang et al., 12 May 2025, Sridhar et al., 2024).

Curriculum learning and statistical data balancing have been shown to substantially improve training dynamics and final accuracy. Difficulty labels generated by LLMs enable staging of easy-to-hard sample presentation, and statistical filtering reweights underrepresented categories for more uniform learning (Wijngaard et al., 9 Jul 2025). Hybrid strategies combine supervised pretraining with RL-based policy optimization, notably Group Relative Policy Optimization (GRPO), to leverage small, curated datasets efficiently while avoiding overfitting (Li et al., 14 Mar 2025, Gibier et al., 18 Nov 2025).

4. Evaluation Methodologies and Metrics

Standard AQA evaluation employs top-1 accuracy (for multiple-choice) or exact-match accuracy, with domain-averaged reporting in mixed benchmarks (Yang et al., 12 May 2025, Wijngaard et al., 9 Jul 2025). For open-form responses, traditional metrics (BLEU, METEOR, ROUGE, BERTScore) have been adapted from NLP and captioning, but typically fail to capture question context, reasoning, or partial correctness, leading to weak correlation with human judgments (Dixit et al., 6 Oct 2025). The AURA Score was introduced to address these deficiencies: combining LLM-based question-aware scoring (via chain-of-thought rationales), a partial credit system, and an audio-text entailment module to achieve state-of-the-art human correlation (Dixit et al., 6 Oct 2025).

Further, benchmarks such as AQEval systematically sample binary, single-word, and long-form QAs for human annotation, enabling robust metric development and validation.

5. Main Experimental Results and Findings

Recent studies have demonstrated that careful data curation and curriculum-driven training lead to significant empirical gains. On the DCASE 2025 AQA benchmark, pure data curation (statistical filtering and staged curriculum) yields an 11.7 point increase in accuracy versus strong LALM baselines (Qwen2-Audio-7B, AudioFlamingo2, Gemini-2.0-Flash), reaching 64.2% overall accuracy (Wijngaard et al., 9 Jul 2025). Statistically filtered data alone provides 1.1 pp gain, curriculum-only matches this, and both together are additive. Guided decoding for multiple-choice formats further reduces output errors with negligible computational overhead.

Across public datasets, RL with GRPO outperforms supervised fine-tuning—even when using only modestly sized post-training samples—and closes substantial portions of the human-model performance gap (Li et al., 14 Mar 2025). Systems trained with AQUALLM-generated synthetic QA pairs reach test accuracy of ≥95.6%, well above those trained only on crowdsourced gold-standard QAs (Behera et al., 2023). Hybrid symbolic-neural pipelines that extract segment-level acoustic events and encode them as structured prompts enable LALMs to surpass unconstrained QA models by directly injecting event reasoning into the answer space (Gibier et al., 18 Nov 2025).

6. Specialized Extensions: Temporal, Spatial, and Knowledge-Intensive AQA

Temporal Reasoning: Temporal QA remains a bottleneck. Data-augmented temporal QA pipelines synthesize event/timestamp-labeled samples via LLM prompting, leveraging curriculum learning to preserve performance on core skills while enhancing temporal competence (Sridhar et al., 2024). Frame-level temporal resolution via spectrogram transformers, explicit metadata, and architectural modifications significantly improve temporal QA accuracy.

Spatial AQA: Spatial audio question answering has been advanced via motion-centric data augmentation, explicit spatial reasoning architectures, and source separation modules. Parametric trajectory synthesis generates diverse spatial scenes, while "thinking mode" LLMs enable explicit reasoning chains, especially when paired with query-conditioned source separation (AGM), yielding substantial accuracy boosts when reasoning about dynamic source movements (Sridhar et al., 18 Feb 2026).

Knowledge-Intensive AQA: Recent attention to knowledge-enriched AQA seeks to answer queries that transcend direct audio perception by integrating KB-derived facts. Audio entity linking (AEL) modules pair audio with ASR-transcribed text and retrieve KB entries, with downstream LALMs leveraging these snippets to achieve order-of-magnitude accuracy gains on synthetic knowledge-intensive QA benchmarks (Penamakuri et al., 2024).

7. Implications, Open Problems, and Future Directions

Empirical evidence indicates that data quality and balance are more impactful than architectural novelty for current AQA benchmarks (Wijngaard et al., 9 Jul 2025). RL-driven fine-tuning outperforms traditional SFT in resource-constrained regimes (Li et al., 14 Mar 2025, Zhang et al., 7 Oct 2025). Temporal and spatial structure remain unresolved core challenges, with denser supervision, specialized modules, and richer synthetic data offering promising directions (Sridhar et al., 2024, Sridhar et al., 18 Feb 2026).

Key open areas include:

• Generalization beyond overrepresented classes and rare or long-duration events
• Multi-phase curriculum learning with adaptive pacing and chain-of-thought annotations
• Integration of explicit causal and goal-driven reasoning (ASPIRE, AUX, AUGMENT paradigms) for explainability and human alignment (Nam, 11 Aug 2025)
• Evaluation beyond surface similarity, towards holistic correctness and partial credit (Dixit et al., 6 Oct 2025)
• Knowledge integration using entity linking, down-stream KB access, and multi-modal retrieval (Penamakuri et al., 2024)
• Resource-efficient adaptation, including test-time RL and label-free self-improvement (Zhang et al., 7 Oct 2025)

The methodological toolkit developed for AQA—combining statistical data curation, staged curriculum learning, hybrid RL pipelines, and question-aware evaluation—offers a robust foundation for the design of the next generation of audio–language intelligence systems. These techniques generalize to related tasks including long-form audio QA, multimodal and spatially-aware QA, and knowledge-intensive audio reasoning.