AudioLLMs: Multimodal Auditory Language Models
- AudioLLMs are multimodal systems that integrate auditory encoding with language modeling to process speech, music, and complex sounds.
- Architecturally, AudioLLMs combine audio encoders, alignment modules, and pretrained LLMs to support tasks from transcription to paralinguistic analysis.
- Benchmarks like AudioBench and MAE evaluate AudioLLMs on transcription, multi-audio reasoning, and selective auditory attention under challenging conditions.
Searching arXiv for the cited AudioLLM papers to ground the article and verify metadata.
arxiv_search(query="Audio LLMs benchmark AudioBench universal benchmark (Wang et al., 2024)", max_results=5)
arxiv_search(query="SALMONN generic hearing abilities (Tang et al., 2023)", max_results=5)
arxiv_search(query="DeSTA2.5-Audio (Lu et al., 3 Jul 2025)", max_results=5)
Auditory LLMs (AudioLLMs), also referred to in the literature as large audio-LLMs (LALMs) and audio-LLMs (ALLMs), are multimodal systems that extend LLMs with auditory capabilities so that an (audio, instruction) pair can be mapped to a free-form textual response (Wang et al., 2024). In the strongest formulations, they are intended to support “generic hearing abilities” over speech, audio events, and music rather than only automatic speech recognition, while also retaining the instruction-following and reasoning behavior of the underlying language backbone (Tang et al., 2023). The field has consequently expanded from audio-conditioned transcription and question answering to broader concerns including paralinguistic perception, multi-audio reasoning, speech quality evaluation, selective auditory attention, interpretability, and cognitively grounded assessment (Lu et al., 3 Jul 2025).
1. Conceptual scope and problem setting
AudioLLMs are defined operationally by a simple interface: audio is encoded, aligned to a language-model token space, and answered in natural language under an explicit textual query (Wang et al., 2024). That formulation distinguishes them from both narrow audio classifiers and cascaded pipelines in which audio is first transcribed and only then passed to a text LLM. The distinction matters because several benchmarks report that cascades can be strong on speech-heavy tasks, while direct AudioLLMs are preferable when non-speech audio and paralinguistic information are essential (Wang et al., 2024).
A second defining feature is breadth of auditory coverage. SALMONN formalizes this breadth as a unified treatment of speech, audio events, and music, arguing that “hearing” for embodied or real-world AI agents must cover general auditory information rather than only lexical content (Tang et al., 2023). Later work generalizes the same premise to task-agnostic instruction following, where robust auditory perception and language preservation are treated as joint design targets rather than separate optimization problems (Lu et al., 3 Jul 2025).
The literature also increasingly separates auditory knowledge from auditory grounding. A text-only LLM may encode substantial knowledge about sound categories, music, pronunciation, or audio quality through text pre-training, and that backbone knowledge is strongly correlated with later audio-grounded performance after multimodal alignment (Lu et al., 19 Mar 2026). At the same time, probing work on text-based LLMs shows that code-mediated audio generation remains rudimentary: simple musical notes are feasible, some environmental sounds are partially recoverable, and speech generation fails completely in the reported setup (Anbazhagan et al., 4 May 2025). This suggests that AudioLLMs are not merely text LLMs with auditory vocabulary, but systems whose performance depends on both backbone priors and the quality of audio-language alignment.
2. Architectural patterns and alignment strategies
The canonical AudioLLM architecture combines an audio encoder, an alignment or connection module, and a pretrained LLM. SALMONN is a representative early design: it uses the encoder from Whisper-Large-v2 for speech, BEATs for non-speech audio, a window-level Q-Former as the connection module, Vicuna-13B as the backbone LLM, and LoRA on Vicuna’s self-attention layers (Tang et al., 2023). This dual-encoder pattern reappears in later systems whenever broad semantic content and fine-grained affective or acoustic cues must be disentangled rather than collapsed into a single representation.
One line of work augments a strong base encoder with lightweight specialists. MoWE-Audio adds a “Mixture of Weak Encoders” to a standard AudioLLM, using a data-independent router and a data-dependent router with top-1 routing to select small auxiliary encoders per sample (Zhang et al., 2024). The full training objective is reported as
with the router loss combining entropy and diversity terms so that routing is both confident and non-collapsed (Zhang et al., 2024). Empirically, this architecture is presented as a way to increase multitask capacity without making the base encoder itself much larger.
Another line of work focuses on supervision design rather than expert routing. DeSTA2.5-Audio argues that many prior LALMs suffer catastrophic forgetting because audio-alignment targets are written by humans or by an external teacher with a different response style than the backbone LLM (Lu et al., 3 Jul 2025). Its remedy is self-generated cross-modal alignment: the same Llama3.1-8B-Instruct backbone that will later serve as the language core first generates responses from a structured textual surrogate of the audio, and those responses become the supervision targets for learning the audio-text interface. The resulting DeSTA-AQA5M corpus contains 5 million training samples derived from 7,000 hours of audio spanning 50 datasets, with Whisper-large-v3 as the audio encoder and a 6-layer Q-Former with 64 queries as the modality adapter (Lu et al., 3 Jul 2025). The central claim is that distributional compatibility between teacher and backbone matters more than simply using a stronger external annotator.
ALARM extends this self-generation idea to reasoning backbones, where naive self-generated targets can expose the textual surrogate in the chain-of-thought itself (Grinberg et al., 10 Mar 2026). Its “self-rephrasing” stage rewrites such traces into audio-native variants while preserving distributional alignment with the frozen reasoning model. ALARM also replaces single-encoder reliance with a four-encoder stack—Whisper, W2V-BERT 2.0, MuQ, and SSLAM—together with learned fusion and compression modules, and trains on 6.08M instances spanning 18.89K hours (Grinberg et al., 10 Mar 2026). In parallel, the UAS framework argues that ASR-centric supervision itself is the source of a persistent perception deficit, and replaces flat transcript targets with a structured JSON schema containing transcription, paralinguistics, and nonLinguisticEvents (Zhang et al., 14 Apr 2026). This moves supervision from “what was said” toward a decomposition of “what was said,” “how it was said,” and “what else occurred acoustically.”
3. Benchmarking and evaluation regimes
AudioLLM evaluation has moved from isolated task reports toward larger benchmark ecosystems. AudioBench is presented as a universal benchmark with 8 tasks and 26 datasets, covering speech understanding, audio scene understanding, and voice understanding over more than 400 hours of audio and 100k+ samples (Wang et al., 2024). Its evaluation protocol is explicitly mixed: ASR uses WER, audio captioning uses METEOR in addition to model-as-judge, and most other tasks are scored with a judge model because outputs are free-form (Wang et al., 2024). The paper reports that Llama-3-70B-Instruct shows Spearman correlation above 0.85 with GPT-4 on three representative datasets, motivating its use as the open-source judge (Wang et al., 2024).
Several later benchmarks specialize this general framework toward missing capabilities.
| Benchmark | Focus | Reported scope |
|---|---|---|
| AudioBench | Universal audio instruction following | 8 tasks, 26 datasets |
| MAE | Multi-audio evaluation | 20 datasets, 11 tasks |
| FinAudio | Financial-domain audio | 3 tasks, 5 datasets |
| MUSA | Selective auditory attention | 200 items, 4 domains |
| RAIL | CHC-grounded auditory cognition | 5,306 samples, 32 tasks |
MAE is the first dedicated multi-audio benchmark for ALLMs, built from 20 datasets and 11 multi-audio tasks spanning speech and sound, with both open-ended and closed-form settings (Chen et al., 2024). Its motivation is that single-audio competence does not guarantee correct reasoning over relations between clips. The reported experiments on 15 ALLMs show that open-source models are often near chance on balanced binary tasks, sometimes collapsing to a fixed answer, and that Qwen-Audio reaches only 39.6% average accuracy on MAE-Speech (Chen et al., 2024). FinAudio introduces a domain-specific benchmark for short financial ASR, long financial ASR, and long financial audio summarization, over five datasets and 400+ hours of financial audio, and reports sharp degradation on long recordings for all evaluated models (Cao et al., 26 Mar 2025).
MUSA addresses a different failure mode: cocktail-party selective auditory attention under semantically plausible multilingual interference (Koo, 17 May 2026). Each item contains an English target dialogue, a distractor in English, Spanish, Korean, or Chinese, a target-grounded question, and four answer options, evaluated in single-stream, source-separation, and end-to-end cocktail-party conditions across controlled SNRs from dB to dB (Koo, 17 May 2026). RAIL, by contrast, argues that task-centric benchmarks are not enough because they do not identify the underlying cognitive ability being tested. Grounded in the CHC framework, it decomposes auditory intelligence into Auditory Processing, Fluid Reasoning, Memory, Processing Efficiency, and Acquired Knowledge, and instantiates these as 32 fine-grained subcapabilities over 5,306 samples and 30.6 hours (Jin et al., 9 Jun 2026).
Taken together, these benchmarks reframe evaluation from raw end-task accuracy toward capability-specific diagnosis. A plausible implication is that “AudioLLM performance” is no longer a single quantity but a bundle of partially independent competencies: transcription, scene understanding, paralinguistics, memory, source grounding, efficiency, and domain knowledge.
4. Beyond transcription: paralinguistics, perception, and quality awareness
A persistent theme in the literature is that ASR-level semantic understanding is not enough. The emotion-reasoning work on multitask AudioLLMs argues that current systems remain limited in modeling paralinguistic cues such as emotion, and that classification-centric emotion recognition offers little interpretability (Zhang et al., 7 Jun 2025). Its proposed dual-encoder system combines a fixed Whisper-Large-v3 speech encoder with an emotion-centric encoder selected from Whisper, HuBERT, or Emotion2Vec, followed by task-alternating training and reasoning-augmented supervision in which each emotion label is paired with a natural-language explanation (Zhang et al., 7 Jun 2025). On IEMOCAP, MELD-ER, and MELD-SR, label-only supervision is reported to underperform strongly relative to interpretive or evidence-grounded reasoning targets, and the final “AudioLLM-Reasoning” model reaches 63.8, 53.0, and 61.1 on those three tasks respectively (Zhang et al., 7 Jun 2025). Groundedness analysis with Llama-3-70B-Instruct reports that 49.2%–73.8% of predictions contain extractable quotations, with average groundedness 82.8/100 and relevance 65.3/100 (Zhang et al., 7 Jun 2025).
UAS addresses the same gap from the supervision side. It attributes the field’s “performance inversion” to ASR-centric training that teaches models to suppress paralinguistic cues and non-speech events as nuisance factors (Zhang et al., 14 Apr 2026). By enforcing a structured output with explicit paralinguistic subfields—Age, Gender, Emotion, Accent, Prosody, and Timbre—and explicit non-linguistic event fields, UAS-Audio improves fine-grained perception on MMSU by 10.9 points over same-size state-of-the-art models while keeping reasoning essentially unchanged at 77.4% versus 77.6% for Qwen2.5-Omni (Zhang et al., 14 Apr 2026). This is one of the clearest demonstrations that perception and reasoning are separable optimization targets.
Speech quality assessment introduces a further extension of perception-aware AudioLLMs. “Audio LLMs Can Be Descriptive Speech Quality Evaluators” constructs the first natural language-based speech evaluation corpus from authentic human ratings and proposes ALLD, an alignment approach with LLM distillation (Chen et al., 27 Jan 2025). The best reported Qwen2-Audio with ALLD (2×) achieves LCC 0.93, SRCC 0.93, MSE 0.17, and BLEU 25.84 for MOS prediction, while joint A/B evaluation reaches BLEU 30.17 and accuracy 98.6% (Chen et al., 27 Jan 2025). A related study shows that instruction-tuned AudioLLMs can predict MOS, SIM, A/B preference, and generate natural-language descriptions, with SALMONN competitive on MOS and SIM and finetuned open-source models reaching a GPT-4o-mini-based description correlation score of 0.64 on the NISQA description task (Wang et al., 2024).
These results directly challenge the common misconception that AudioLLMs are primarily “speech recognizers with prompts.” The strongest systems are increasingly asked to describe evidence, track affect, assess signal quality, and separate lexical from non-lexical information.
5. Reasoning over complex auditory scenes
The field’s next major challenge is reasoning under complexity rather than on isolated clips. MAE shows that existing ALLMs are powerful in comprehending primary audio elements in individual inputs but struggle when the question depends on relations across multiple clips (Chen et al., 2024). The proposed MALLM addresses this using discriminative learning on synthetic paired speech and sound data without human annotation, and is reported to achieve 73.8% average accuracy on the speech benchmark and 74.3% on the sound benchmark while remaining competitive on single-audio tasks (Chen et al., 2024). On the sound side, the headline comparison is 57.0% for Qwen-Audio versus 74.1% for MALLM (Chen et al., 2024).
Auditory cognition under realistic interference is studied more directly in work on test-time compute. A self-collected database with 10 participants defines three increasingly difficult tasks—audio event recognition, speech comprehension with mild noise, and overlapping speech with noise—and evaluates Qwen2-Audio, Audio-Flamingo 2, Gemini-2.0-Flash, Gemini-1.5-Pro, and GPT-4o (Dang et al., 30 Mar 2025). Performance drops sharply on overlapping speech for most models, while GPT-4o is reported to surpass average human performance on one difficult overlapping-speech condition (Dang et al., 30 Mar 2025). Five test-time compute methods are introduced, including CoT, temperature-based majority vote, beam search, and verifier-based reranking, with reported gains ranging from roughly 9% to 150% depending on model and task (Dang et al., 30 Mar 2025). The implication is that inference-time search and verification can compensate, at least partially, for limited auditory-scene robustness without retraining.
MUSA shows that single-stream excellence does not transfer automatically to selective auditory attention (Koo, 17 May 2026). At 0 dB, Gemini-2.0-Flash is reported to fall from 0.955 in the single setting to 0.242 in the cocktail-party setting, while source separation recovers performance to 0.952 (Koo, 17 May 2026). The paper’s error taxonomy shows that failures are dominated not by arbitrary hallucination but by distractor-grounded source confusion, with interference rates such as 0.918 for Gemini-2.0-Flash under cocktail-party conditions and very low ungrounded-error rates (Koo, 17 May 2026). Separation therefore reduces acoustic overlap but does not solve source attribution, a point that reinforces the broader claim that source grounding is a distinct capability from either transcription or separation.
RAIL broadens this perspective by showing that state-of-the-art LALMs are highly uneven across cognitive abilities: Knowledge is the strongest broad capability on average, Memory is also strong, and Auditory Processing is the weakest (Jin et al., 9 Jun 2026). Humans rank 7th overall among the 26 evaluated models, yet remain best on Auditory Processing and Processing Efficiency (Jin et al., 9 Jun 2026). This suggests that language-heavy reasoning and retrieval can look strong even when low-level auditory cognition and rapid response remain comparatively underdeveloped.
6. Internal representations, model editing, and frontier problems
As AudioLLMs have become more capable, interpretability work has shifted from output inspection to internal mechanism analysis. AudioLens applies Logit Lens–style vocabulary projection to DeSTA2, Qwen-Audio-Chat, and Qwen2-Audio-Instruct, and finds that attribute information at the prediction point often degrades with layer depth when recognition fails, whereas earlier effective resolution correlates with higher accuracy (Yang et al., 5 Jun 2025). The reported Pearson correlations between accuracy and critical-layer depth are strongly negative for several settings, and a simple layer-enrichment intervention yields a 16.3% relative accuracy improvement on the difficult DeSTA2 animal-recognition case without additional training (Yang et al., 5 Jun 2025). One central conclusion is that many LALMs query the auditory input directly at the final prediction position rather than consolidating enough attribute evidence into earlier hidden states.
AR&D moves from probing to mechanistic interpretation using TopK sparse autoencoders on Qwen2-Audio-7B-Instruct residual-stream activations (Chowdhury et al., 24 Feb 2026). It retrieves representative audio clips for latent features, names them through automatic captioning and summarization, and validates them with human evaluation and steering. On 50 random concepts, the automatically generated names achieve expert-rated score versus for raw polysemantic features, and steering experiments show large gains in sensitivity such as 0.75 versus 0.13 for Neutral Happy and 0.61 versus 0.09 for Female Male (Chowdhury et al., 24 Feb 2026). The main significance is that AudioLLM internal features can be both interpretable and causally actionable.
Knowledge editing exposes another layer of difficulty. SAKE is the first benchmark for editing auditory attribute knowledge in LALMs, covering speaker gender, speaker emotion, spoken language, and animal sound, and evaluating reliability, generality, locality, and portability on DeSTA2.5-Audio and Qwen2-Audio-Instruct (Yang et al., 19 Oct 2025). The benchmark shows that methods can often make the edited instance correct while still damaging unrelated intra-attribute knowledge, generalizing poorly across alternative audio realizations, or failing to propagate the edit through related reasoning chains (Yang et al., 19 Oct 2025). Sequential editing is especially unstable for some methods, with explicit reports of degeneration and repetitive outputs under continued edits (Yang et al., 19 Oct 2025). This suggests that auditory attribute knowledge is more entangled than a simple symbolic fact store.
A final frontier concerns the choice of language backbone itself. The holistic evaluation of auditory knowledge in LLM backbones introduces AKB-2000, a 2,000-question multiple-choice benchmark over Sound, Paralinguistic, Phonetic, Music, Audio Quality, and Technical Knowledge, and shows that text-only auditory knowledge correlates strongly with later audio-grounded performance (Lu et al., 19 Mar 2026). Pairwise correlations within the text-only evaluations are about 0.94, and correlations between text-only and audio-grounded metrics range from 0.71 to 0.82 (Lu et al., 19 Mar 2026). The same study identifies phonetics, stress, rhyme, and music theory as particularly hard subcategories for text-pretrained backbones (Lu et al., 19 Mar 2026). A plausible implication is that AudioLLM progress will continue to depend not only on larger audio corpora and better encoders, but on backbone selection, pronunciation-sensitive knowledge, multilingual robustness, long-audio handling, multi-speaker perception, and evaluation regimes that isolate cognition rather than only end-task success.
