---
title: Audio Hallucination QA Dataset
url: https://www.emergentmind.com/topics/audio-hallucination-qa-dataset
type: topic
---

# Audio Hallucination QA Dataset

The Audio Hallucination QA Dataset is a specialized benchmark designed to systematically evaluate and analyze hallucinations arising in large audio-language and multimodal models when answering questions about audio content. Hallucinations, in this context, refer to model-generated outputs that are not grounded in the actual audio signal but instead reflect spurious, invented, or visually biased information. The dataset enables rigorous measurement of hallucination prevalence, type categorization, and assessment of mitigation methods across diverse model architectures and tasks.

## 1. Dataset Construction and Annotation Protocol

The canonical Audio Hallucination QA Dataset, as established in "On the Audio Hallucinations in Large Audio-Video Language Models" [2401.09774], comprises 1,000 QA instances derived by prompting Video LLAMA on videos from the FAVDBench test split. Each instance employs the standardized query “What do you hear?”, focusing the model on audio perception over visual cues. Output sentences are annotated by human experts with binary labels: hallucinated (audio description generated without support from actual audio cues) or non-hallucinated. For hallucinated responses, secondary annotation specifies one of three taxonomy-driven hallucination types: Type (A) – both object and action hallucinated, Type (B) – object correct, action hallucinated, Type (C) – action correct, object hallucinated.

| Split | Total Instances | Hallucinated | Hallucination Categorization   |
|-------|----------------|--------------|-------------------------------|
| FAVDBench QA | 1,000         | 323–332      | Types (A), (B), (C)           |

The annotation protocol includes noun and verb extraction (excluding formulaic phrases), facilitating statistical analysis of specific error signatures for each hallucination type.

## 2. Hallucination Taxonomies and Benchmark Extensions

Taxonomical organization of hallucination types, validated across several works [2405.09589][2410.16130][2410.18325], underpins benchmark design and cross-study comparisons. The three-type schema has become canonical for audio hallucinations in multimodal settings, supported by both classification experiments and error analyses:

| Type | Error Manifestation                       | Example                 | Key Characteristics                |
|------|-------------------------------------------|-------------------------|------------------------------------|
| A    | Objects and actions hallucinated          | "a baby crying"         | Generic/ambient hallucinations     |
| B    | Object correct, action hallucinated       | "woman playing a harp"  | Action misattribution              |
| C    | Action correct, object hallucinated       | "trumpet" vs "tuba"     | Instrument/source substitution     |

Further extensions (e.g., AVHBench [2410.18325]) introduce multiple judgment tasks: audio-driven video hallucination, video-driven audio hallucination, audio-visual matching, and audio-visual captioning. These tasks enable disambiguation of cross-modal confounds and compositional reasoning errors, particularly relevant for multi-sensory environments.

## 3. Model Evaluation Protocols and Classification Methodologies

The dataset enables evaluation via both zero-shot embedding-based classifiers and fine-tuning pipelines. In embedding-based approaches, pre-trained encoders (MS-CLAP, LAION-CLAP) project audio ($E_a(A)$) and textual ($E_t(T)$) features into a joint space; cosine similarity ($\cos(h_a, h_t)$) below a threshold $\alpha$ implies hallucination. Fine-tuned models employ MLP layers atop frozen encoders, with element-wise combination and sigmoid-based prediction:

\[
\hat{h}_a = F_a(h_a), \quad \hat{h}_t = F_t(h_t), \quad \hat{h}_{at} = \hat{h}_a \odot \hat{h}_t, \quad \hat{y} = F_{at}(\hat{h}_{at}), \quad L = \text{BCE}(\hat{y}, y)
\]

Experimental results indicate fine-tuned MS-CLAP achieves $87.9\%$ F$_1$, decisively surpassing zero-shot (approx. $52.2$–$52.9\%$ F$_1$) and random baseline ($40.3\%$ F$_1$).

Benchmarks such as AVHBench utilize accuracy, precision, recall, and F$_1$ for binary tasks, and METEOR/CIDEr/GAVIE-A for captioning. Paired and before-after question design (as in MATCH [2410.16130]) enables nuanced analysis of response discrimination and consistency (C-C, C-I metrics).

## 4. Hallucination Mitigation Strategies and Empirical Safeguards

Recent mitigation approaches include:
- **Audio-Aware Decoding (AAD)** [2506.07233]: Contrastive decoding compares token logits with and without audio input, promoting tokens with audio-grounded evidence. The method is formally defined as 
  \[
  p_\text{AAD}^{(t)} = \text{softmax}\left((1+\alpha) \cdot \text{logit}_\text{with audio}^{(t)} - \alpha \cdot \text{logit}_\text{without audio}^{(t)}\right)
  \]
  Empirically, AAD achieves F$_1$ increases from $0.046$ to $0.428$ over baseline on object hallucination datasets.
  
- **Adaptive Vector Steering (AVS)** [2510.12851]: Layer-wise steering of model activations using contrastive representations (audio vs. silent input) with adaptive intervention strength. Later layers receive higher steering weights, calibrated by effect size analysis. AVS yields F$_1$ improvements (Gemma: $0.550 \to 0.619$, Qwen: $0.626 \to 0.632$) and $8\%$ accuracy gain on MMAU.

- **Post-processing in ASR** [2501.11378]: The Bag of Hallucinations (BoH) captures frequent spurious outputs; combined with delooping and string search (Aho–Corasick), post-processing robustly suppresses recurring ASR hallucinations, reducing WER by up to $6.5–9.4\%$.

- **Attention-Guided Explainable Scores** [2505.15386]: RePPL attributes token-wise uncertainty to semantic propagation and generation. The uncertainty scores detect chaotic patterns typical of hallucination and can be mapped to audio QA by aligning scores with spectrogram regions or transcribed tokens.

## 5. Cross-Modal and Acoustic Condition Benchmarks

Benchmarks such as RePOPE-Spk [2510.08581] extend evaluation to spoken queries under variable acoustic conditions. Hallucination error rates escalate as queries shift from text to speech (by $3\%$ under clean speech, up to $20\%$ with environmental noise). Experiments demonstrate that input order and query duration affect robustness, with longer spoken queries helping mitigate but not eliminate performance drops.

AVHBench [2410.18325] systematically addresses cross-modal hallucinations in audio-visual LLMs. It shows models often perform near chance on multimodal settings, but unimodal or text-converted inputs enhance reliability. Ablations reveal that improved audio-to-LLM alignment and LoRA fine-tuning dramatically boost F$_1$ and captioning metrics.

## 6. Broader Implications and Future Directions

The Audio Hallucination QA Dataset exposes specific weaknesses in current model architectures, notably the over-reliance on visual cues, susceptibility to compositional confusion, and vulnerability to acoustic noise and input ordering in voice-driven interfaces. Improvements in model design—closer integration of audio features, refined multi-turn and chain-of-thought prompting, and informed post-processing—are advancing reliability but not fully solving the issue.

Future work is anticipated to extend benchmarks with more challenging, diverse scenarios and to investigate methods that enhance modality-specific attention, noise robustness, and dynamic input handling. The taxonomy-driven evaluation and systematic mitigation approaches established in these datasets will inform both diagnostic toolchain development and the training of more grounded, trustworthy audio and multimodal language models.

Source: https://www.emergentmind.com/topics/audio-hallucination-qa-dataset