Large Audio-Language Models (LAMs)

• Large Audio-Language Models (LAMs) are multimodal systems that combine pre-trained audio encoders and large language models to enable comprehensive audio understanding and reasoning.
• They employ modular architectures—end-to-end, cascaded, and agentic frameworks—to perform tasks such as ASR, translation, and conversation across various domains.
• LAMs use diverse training paradigms and tool-augmented methods to boost performance, safety, and multilingual capabilities, while addressing challenges like adversarial robustness and hallucination.

Large Audio-LLMs (LAMs) are a class of multimodal neural systems that couple audio encoders with LLMs, enabling universal audio understanding and reasoning across diverse linguistic, paralinguistic, and cognitive tasks. By bridging the traditionally separate domains of speech processing, environmental sound understanding, and text-based reasoning, LAMs fundamentally expand the interface and potential of foundation models, providing the infrastructure for open-domain, voice-first, and speech-driven AI applications.

1. Architectural Foundations and Model Taxonomy

Modern LAMs are modular architectures comprising (i) a pre-trained audio encoder (e.g., Whisper, AudioLM, CLAP), (ii) a modality-adapter (typically a projection or cross-attention module), and (iii) a LLM backbone (often a transformer decoder, such as PaLM-2, LLaMA, or Qwen2.5) (Rubenstein et al., 2023, Wang et al., 19 Feb 2025, Liu et al., 3 Nov 2025, Gao et al., 6 Dec 2024). Input audio is transformed into high-level embeddings, which are then injected into the LLM either as a soft prompt (prepended embeddings) or cross-attended within the transformer block. LAMs may support bidirectional modality—processing both speech-to-text and text-to-speech—by sharing or expanding vocabulary embeddings and positional encodings for both text and audio tokens (Rubenstein et al., 2023). Architectures span:

• End-to-end models: unified transformers (e.g., AudioPaLM) that operate jointly over mixed audio-text token streams, performing tasks such as ASR, speech translation, and TTS from a single parameter set.
• Cascaded/adapter-based models: separately trained audio encoders joined to frozen or partially-finetuned LLMs via lightweight adapters (linear projections or LoRA blocks) (Fathullah et al., 2023, Wang et al., 19 Feb 2025, Cappellazzo et al., 18 Sep 2024).
• Multimodal agentic frameworks: LLMs operating as central agents, invoking modular audio-language tools (AudioToolAgent, Audio-Maestro) through explicit tool-selection and result integration, thus supporting multi-step reasoning and tool-augmented verification (Lee et al., 13 Oct 2025, Wijngaard et al., 3 Oct 2025).

Key advances include modular toolkits for extensibility (AudioToolAgent), efficient batch inference and prompt standardization (AU-Harness (Surapaneni et al., 9 Sep 2025)), agentic multi-step tool-calling, and language-region or dialect-specific adaptation (SeaLLMs-Audio (Liu et al., 3 Nov 2025)).

2. Training Paradigms and Supervision Regimes

LAMs are trained with objectives reflecting the interaction of language and audio. The canonical loss is token-level cross-entropy over mixed audio-text streams, with task tags signaling function (ASR, AST, S2ST, etc.) (Rubenstein et al., 2023, Liu et al., 3 Nov 2025). Pretraining occurs on massive mixtures of paired speech–text (LibriLight, GigaSpeech, AudioCaps), text-only (for language transfer), and sometimes synthetic audio–text pairs in under-resourced languages.

Alternative supervision paradigms include:

• Text-only supervision: leveraging frozen audio-language alignment models (e.g., CLAP) and aligning LLMs to audio-conditioned latent spaces without any audio during training (MATS (Wang et al., 19 Feb 2025)). The Santa mechanism augments during inference by memory-based fusion of CLAP-aligned noisy text vectors and actual audio embeddings.
• Curriculum and mixed-modality fine-tuning: curriculum learning, where paired speech-label data are introduced only in the late training phase, strengthening language grounding before incorporating limited acoustic grounding. Empirically, as little as 2–5% of speech data yields substantial Spoken Language Understanding (SLU) gains (up to 27 F1 points over text-only baselines), with curriculum schedules outperforming direct mixing when data are scarce (Choi et al., 18 Sep 2025).
• Joint instruction tuning across modalities: integrating multi-task instruction and dialogue data to unify open-domain question answering, summarization, and reasoning over both speech and environmental audio (Liu et al., 3 Nov 2025, Gao et al., 6 Dec 2024).

Pretraining objectives are augmented with task tags and, for speech synthesis or S2ST, voice prompts and hierarchical codec tokens to preserve speaker identity and prosody (Rubenstein et al., 2023).

3. Capabilities, Evaluation, and Limitations

LAMs demonstrate state-of-the-art results across a variety of tasks:

• Automatic Speech Recognition (ASR)/Speech-to-Text: Conformer or Whisper-based LAMs achieve 0.8–1.1% WER on LRS3 and ∼18% relative WER reduction over monolingual baselines on Multilingual Librispeech, even with frozen LLMs and large strides (up to 960 ms), enabling efficient long-form audio processing (Fathullah et al., 2023, Cappellazzo et al., 18 Sep 2024).
• Audio-Visual Speech Recognition (AVSR): Models such as Llama-AVSR leverage paired video encoders and achieve 0.77% WER on clean LRS3 (Cappellazzo et al., 18 Sep 2024), with LoRA adaptation and modality-aware compression governing the computation–accuracy tradeoff.
• Audio Reasoning/QA and Multi-Task Understanding: In zero-shot audio captioning, question answering, and environmental sound classification, models trained under text-only supervision (MATS) surpass prior text-only systems and approach fully audio-supervised ones (Wang et al., 19 Feb 2025).
• Temporal Reasoning and Dialogue: Curriculum-augmented LALMs substantially outperform vanilla LAMs on open and synthetic temporal reasoning, achieving up to 0.70 SPIDER and 0.73 FENSE on temporal QA (Sridhar et al., 10 Sep 2024). However, diarization and temporal event comprehension remain open weaknesses, with word-diarization error rates ∼35% for all LALMs evaluated in AU-Harness (Surapaneni et al., 9 Sep 2025).
• Open-ended Audio Dialogue: Benchmarks such as ADU-Bench (Gao et al., 6 Dec 2024) reveal that current LALMs still underperform cascaded ASR+LLM pipelines in open-ended dialogue, especially in mathematical reasoning, roleplay behaviors, and phonetic ambiguity (intonation, pauses, homophones).
• Multilingual and Region-Specific Performance: SeaLLMs-Audio demonstrates strong performance for Indonesian, Thai, and Vietnamese (human-rated quality >4/5), outperforming general-purpose European LAMs on Southeast Asian audio tasks (Liu et al., 3 Nov 2025).

A systematic evaluation regime encompasses static benchmarks (ASR, QA, paralinguistics), interactive user preference studies (revealing only weak correlation to static metrics, τ≤0.33, R²=0.30 (Li et al., 21 Feb 2025)), and fair, standardized prompting protocols (AU-Harness) to mitigate instruction-modality artifacts (up to 9.5 absolute points performance difference on audio vs. text prompts).

4. Safety, Reliability, and Robustness

The expanded capabilities of LAMs introduce new failure modes and attack surfaces:

• Reliability and Knowledge Boundaries: Standard LAMs over-commit, failing to admit lack of knowledge or uncertainty. Training-free multi-modal chain-of-thought (MCoT) prompting or supervised fine-tuning to inject explicit "IDK" (I don't know) outputs quantitatively raises reliability by ∼9–10 points on MMAU, with the Reliability Gain Index (RGI) formalizing the tradeoff between humbleness and conservativeness. This "meta ability" to reject confidently transfers across modalities (sound, music, speech) (Ma et al., 25 May 2025).
• Hallucination: LAMs are prone to hallucinating absent audio content due to model priors. Audio-Aware Decoding (AAD) mitigates this with contrastive reweighting of logits, comparing predictions with and without audio context and promoting audio-grounded token selection. AAD reduces hallucination F1 by up to +0.43 and increases general audio QA accuracy by up to 10.3% (Hsu et al., 8 Jun 2025).
• Jailbreak Attacks and Alignment Vulnerabilities: LAMs are susceptible to adversarial spoken prompts. StyleBreak introduces a two-stage style-aware transformation pipeline, jointly perturbing textual and audio style attributes (emotion, age, gender) and uses a learned policy network to find strongly adversarial variants. This approach increases attack success rates up to 4× for Qwen2-Audio, MERaLiON, and other state-of-the-art models, with attack success rate gains of 20–40 percentage points in controlled experiments (Li et al., 12 Nov 2025).
• Audio Adversarial Robustness: Benchmarks such as AJailBench-Base and AJailBench-APT (with the Audio Perturbation Toolkit) demonstrate significant safety degradation under small, semantic-preserving perturbations, with attack success rates increasing from 0.14–0.95 (baseline) to 0.43–0.95 (adversarial), underscoring the necessity of semantically-aware defense mechanisms. No current model is robust across all categories (Song et al., 21 May 2025).

Defense strategies emphasize adversarial fine-tuning, consistency regularization, front-end filtering, and context- or uncertainty-aware refusal calibration.

5. Tool-Augmented and Agentic Reasoning

Several recent frameworks extend LAMs beyond end-to-end inference:

• Modular Tool Use: AudioToolAgent orchestrates a central LLM agent that coordinates pretrained audio–LLMs (tools) through HTTP adapters, selecting and sequencing tool calls for audio QA, verification, and multi-step reasoning. State-of-the-art performance is achieved on MMAU (74.10%), MMAR (68.80%), and MMAU-Pro (57.96%), without any new data collection or fine-tuning. Shapley value analysis quantifies individual tool contributions (Wijngaard et al., 3 Oct 2025).
• Tool-Augmented Reasoning: Audio-Maestro enables LALMs to call external, timestamped analysis tools (diarization, chord recognition, emotion detection). Empirically, tool augmentation consistently improves reasoning accuracy by 2.9–4.7 percentage points (e.g., Gemini-2.5-flash: 67.4%→72.1%), especially for tasks requiring fine-grained, structured audio analysis (Lee et al., 13 Oct 2025).
• Prompt Standardization and Evaluation: AU-Harness provides a unified evaluation harness enabling high-throughput, standardized, and modality-aware LAM benchmarking, and exposes corpus- and prompt-dependent weaknesses in instruction following, diarization, and oral reasoning (Surapaneni et al., 9 Sep 2025).

The agentic paradigm enables plug-and-play extension with no retraining, but raises implications for inference latency, error propagation from third-party tools, and the necessity for robust cross-tool consensus mechanisms.

6. Specialized Applications and Emerging Directions

LAMs are rapidly expanding into domain-specialized and human-centric applications:

• Disfluent and Child Speech Processing: Models evaluated under child stuttering and disfluency (single-channel separation + child-only summarization) demonstrate that state-of-the-art audio-first LALMs (Audio Flamingo-3, Kimi Audio) can produce clinically relevant summaries from mixed audio, though purity and faithfulness are not fully captured by standard metrics (BERTScore, ROUGE) and require expert or LLM-judge assessment (Okocha et al., 21 Oct 2025).
• Region- and Language-Specific Models: SeaLLMs-Audio demonstrates the feasibility of scaling LAMs for underrepresented Southeast Asian languages, achieving state-of-the-art task success and language quality on Indonesian, Thai, and Vietnamese benchmarks (Liu et al., 3 Nov 2025).
• Open-ended Dialogue and Multilingual Interaction: Audio dialogue benchmarking reveals broad deficiencies in math-heavy dialogue, ambiguous intent interpretation (intonation, homophone discrimination), roleplay, and non-Latin script handling, even for large-scale models (Gao et al., 6 Dec 2024).
• Audio-Visual Fusion and Long-Context Processing: Compression strategies (e.g., modality-aware striding up to 960 ms) enable low-overhead, long-context audio-visual speech recognition (Cappellazzo et al., 18 Sep 2024, Fathullah et al., 2023).

Future research directions target prosody-sensitive encoders, symbolic/audio fusion (math OCR), persona and pragmatics alignment, privacy-preserving and on-device inference, reinforcement learning from spoken feedback, and scalable, open-source evaluation blends of static and live user interaction (Li et al., 21 Feb 2025, Sridhar et al., 10 Sep 2024, Liu et al., 3 Nov 2025). Robust, semantically aware alignment remains a central challenge, as evidenced by the persistent vulnerabilities exposed in adversarial, style-oriented, and tool-augmented settings.