AuditoryBench++: Can Language Models Understand Auditory Knowledge without Hearing? (2509.17641v1)

Abstract: Even without directly hearing sounds, humans can effortlessly reason about auditory properties, such as pitch, loudness, or sound-source associations, drawing on auditory commonsense. In contrast, LLMs often lack this capability, limiting their effectiveness in multimodal interactions. As an initial step to address this gap, we present AuditoryBench++, a comprehensive benchmark for evaluating auditory knowledge and reasoning in text-only settings. The benchmark encompasses tasks that range from basic auditory comparisons to contextually grounded reasoning, enabling fine-grained analysis of how models process and integrate auditory concepts. In addition, we introduce AIR-CoT, a novel auditory imagination reasoning method that generates and integrates auditory information during inference through span detection with special tokens and knowledge injection. Extensive experiments with recent LLMs and Multimodal LLMs demonstrate that AIR-CoT generally outperforms both the off-the-shelf models and those augmented with auditory knowledge. The project page is available at https://auditorybenchpp.github.io.

• The paper introduces AuditoryBench++ to evaluate auditory commonsense in LLMs using text-only tasks and special auditory reasoning tokens.
• The methodology employs a novel two-stage AIR-CoT process that combines span detection and auditory knowledge injection to boost performance.
• Experimental results show substantial improvements, notably 83.89% accuracy in pitch comparison and 82.67% in auditory context reasoning.

AuditoryBench++: Evaluating Auditory Knowledge in LLMs Without Audio Input

Introduction

AuditoryBench++ addresses a critical gap in the evaluation of LLMs and multimodal LLMs (MLLMs): their ability to reason about auditory concepts in the absence of direct audio input. While humans can effortlessly imagine and reason about sounds based on textual descriptions, LLMs typically lack this form of auditory commonsense. AuditoryBench++ introduces a comprehensive benchmark for assessing auditory knowledge and reasoning in text-only settings, spanning basic auditory comparisons to contextually grounded reasoning. The paper also presents AIR-CoT, a novel auditory imagination reasoning method that enables LLMs to dynamically generate and integrate auditory information during inference.

Figure 1: Overview of AuditoryBench++, which assesses auditory knowledge of LLMs without audio input.

AuditoryBench++ Benchmark Design

AuditoryBench++ comprises five distinct tasks, each probing a different aspect of auditory knowledge:

• Pitch Comparison: Binary decision on which of two described sounds has a higher pitch.
• Duration Comparison: Binary decision on which sound lasts longer.
• Loudness Comparison: Binary decision on which sound is louder.
• Animal Sound Recognition: Multiple-choice mapping of onomatopoeic expressions to animal sources.
• Auditory Context Reasoning: Multiple-choice questions requiring contextual interpretation of auditory cues.

The benchmark construction pipeline integrates resources such as AuditoryBench, AudioTime, and MMAU, applying rigorous filtering, statistical estimation, and human verification to ensure reliability and objectivity. For comparison tasks, only instrument-based or segment-level annotated data are used, with statistical significance ($p < 0.01$) enforced for pairwise differences. For context reasoning, audio clips are captioned using Qwen2-Audio and rewritten into text-only problems via GPT-4o, followed by human refinement.

AIR-CoT: Auditory Imagination Reasoning Chain-of-Thought

AIR-CoT is a two-stage training paradigm designed to endow LLMs with auditory imagination capabilities:

1. Span Detection via Special Tokens: The model is fine-tuned to detect spans in the input that require auditory reasoning, emitting special tokens ({\tt[imagine]} and {\tt[/imagine]}) during decoding. The loss is computed only on these tokens, excluding downstream answer tokens to avoid biasing final predictions.
2. Knowledge Injection via Imagination: Upon encountering the {\tt[/imagine]} token, the model pauses generation and invokes an imagination process. Audio embeddings are generated using CLAP and injected into the model via a 2-layer MLP projector, aligning the embedding dimension with the model's hidden size. Only the projector is trained in this stage, with loss computed on answer tokens.

   Figure 2: Pipeline of the proposed AIR-CoT. (a) Data Preparation. Training data is augmented with {\tt[imagine]} tokens to mark spans requiring auditory reasoning. (b) Stage 1: Span Detection. The model is fine-tuned to detect the spans by generating the special tokens during decoding. (c) Stage 2: Knowledge Injection. When encountering the {\tt[/imagine]} token, the model pauses to generate the embedding using CLAP and injects it for auditory reasoning.

Experimental Results and Analysis

Experiments on AuditoryBench++ demonstrate that off-the-shelf LLMs and MLLMs perform close to random guessing on most auditory comparison tasks, highlighting the lack of inherent auditory commonsense. Augmented models such as AudioBERT and Imagine to Hear show modest improvements, but remain limited in scope.

AIR-CoT achieves substantial gains, particularly in pitch comparison (+8.25%), animal sound recognition (+9.34%), and auditory context reasoning (+11.88%) over the best baselines. Notably, AIR-CoT attains 83.89% accuracy on pitch comparison and 82.67% on auditory context reasoning, indicating effective integration of imagined auditory knowledge. However, improvements in duration and loudness comparison are marginal, attributed to the limitations of current audio representations, which are primarily semantic and lack explicit encoding of temporal and amplitude cues.

Implementation Details

• Base Model: Qwen2.5-7B for both AIR-CoT stages.
• Imagination Module: CLAP text encoder + 2-layer MLP projector.
• Training: Stage 1 uses SFT with special tokens, 10 epochs, batch size 4, learning rate $1 \times 10^{-5}$, AdamW optimizer. Stage 2 trains only the projector for 10 epochs, batch size 4, learning rate $1 \times 10^{-4}$, weight decay 0.01, AdamW optimizer.
• Evaluation Metric: Accuracy across all tasks.

Implications and Future Directions

AuditoryBench++ and AIR-CoT establish a new paradigm for evaluating and enhancing auditory reasoning in LLMs without perceptual input. The results indicate that explicit imagination mechanisms, coupled with knowledge injection, can significantly improve model performance on tasks requiring auditory commonsense. However, the limited gains in duration and loudness tasks underscore the need for audio representations that encode quantitative properties such as time and amplitude more directly. Future research should focus on developing temporally and dynamically enhanced audio-text representations, potentially leveraging advances in contrastive pretraining and compositional reasoning.

The benchmark and methodology have practical implications for multimodal systems deployed in environments where only textual descriptions are available, such as assistive technologies, conversational agents, and educational tools. Theoretically, the work advances the understanding of how imagination and commonsense reasoning can be operationalized in neural models, bridging the gap between human and machine cognition in multimodal contexts.

Conclusion

AuditoryBench++ provides a rigorous framework for assessing auditory knowledge in LLMs under text-only constraints. AIR-CoT demonstrates that auditory imagination, operationalized via span detection and knowledge injection, enables LLMs to reason about sounds without direct audio input. The benchmark and method lay the groundwork for future research on multimodal reasoning and the development of models with more human-like imagination capabilities.