A Survey of Large Audio Language Models: Generalization, Trustworthiness, and Outlook

Abstract: The foundational capabilities established by LLMs have paved the way for Multimodal LLMs (MLLMs), within which Large Audio LLMs (LALMs) are essential for realizing universal auditory intelligence. Despite their remarkable performance, the escalation of LALMs' capabilities has significantly outpaced the development of systemic frameworks to ensure their trustworthiness. This survey provides a comprehensive investigation into the endogenous mechanisms of LALMs, detailing the architectural innovations and alignment algorithms that facilitate emergent reasoning. Specifically, we analyze how the transition to unified end-to-end frameworks and the integration of continuous acoustic signals inherently expand the attack surface. To rigorously evaluate the risks within these paradigms, we establish a comprehensive taxonomy of trustworthiness, categorizing critical vulnerabilities such as cross-modal jailbreaking, latent acoustic backdoors, and biometric privacy leakage. We review the state-of-the-art through six analytical pillars: hallucination, robustness, safety, privacy, fairness, and authentication. The profound imbalance between a mature offensive landscape and underdeveloped defenses further validates the critical trustworthiness gaps and multidimensional risks facing audio-centric intelligence. Finally, we propose a strategic roadmap advocating for "Defense-in-Depth" architectures, causal auditory world modeling, and intrinsic representation engineering to bridge the gap between empirical performance and intrinsically trustworthy audio intelligence. Our project has been uploaded to GitHub https://github.com/Kwwwww74/Awesome-Trustworthy-AudioLLMs.

• The paper analyzes the evolution of LALMs from cascaded ASR systems to integrated end-to-end architectures with audio-specific Chain-of-Thought prompting.
• The paper quantitatively evaluates trustworthiness issues including hallucination, robustness, privacy, fairness, and adversarial vulnerabilities using dedicated benchmarks.
• The paper outlines future trajectories by proposing input sanitization, privacy-preserving representation, and dynamic agent-based benchmarking for improved LALM safety.

Survey of Large Audio LLMs: Generalization, Trustworthiness, and Outlook

Evolution and Generalization Architectures of LALMs

Large Audio LLMs (LALMs), as surveyed in "A Survey of Large Audio LLMs: Generalization, Trustworthiness, and Outlook" (2605.20266), have emerged as core primitives for universal auditory intelligence in the era of multimodal LLMs. This transition from simplistic, task-specific cascaded systems—such as automatic speech recognition (ASR) or sound event detection—to highly integrated end-to-end neural architectures is a defining characteristic of recent advances in the field.

Figure 1: The Evolutionary Roadmap of LALMs from cascaded ASR/sound event systems toward end-to-end causal cognition frameworks, 2022–2026.

The survey provides an exhaustive dissection of these endogenous mechanisms, especially focusing on the co-design of acoustic encoders, alignment projectors, and LLM backbones within modern LALMs. Sequence representation diverges into discrete token-based and continuous manifold-based designs, with recent encoder architectures leveraging factorized audio tokenization, segmentwise pruning, and multimodal cross-sensory alignment. Specialized techniques such as Mixture-of-Experts adapters address gradient conflicts and promote cross-domain representational disentanglement, while knowledge distillation and contrastive pre-training enable robust cross-modal reasoning. Notably, emergent reasoning mechanisms are shifting toward audio-specific "Chain-of-Thought" (Audio-CoT) prompting, where intermediate reasoning steps can be traced prior to final model outputs.

Figure 3: The Audio-CoT architecture prompts LALMs to expose hidden reasoning steps, in contrast to direct-response pathways, highlighting superior cognitive alignment and interpretability.

The survey identifies a marked movement from rigid, modular pipelines to architectures focusing on causal auditory world modeling, cross-modal knowledge transfer, agentic frameworks with synchronous duplexing, and intrinsic representation engineering. These design choices not only drive improvements in auditory reasoning but significantly expand the risk surface for adversarial or unaligned behaviors.

Taxonomy and Quantitative Growth of Trustworthiness Problems

The integration of audio as a native modality presents unique challenges in trustworthiness that substantially diverge from text-centric LLMs. Notably, the attack surface is qualitatively and quantitatively larger due to the continuous, high-dimensional nature of acoustic data. The paper formulates a six-pillar taxonomy: hallucination, robustness, safety, privacy, fairness, and authentication.

Figure 2: Visualization of LALM trustworthiness pillars, identifying specific failure scenarios in hallucination, robustness, fairness, privacy, authentication, and safety.

The survey documents an unprecedented acceleration in both offensive research and corresponding benchmarking initiatives from 2024–2026.

Figure 4: Timeline demonstrating quantitative growth in scholarly publications and benchmarks relating to LALM trustworthiness.

The synthesis of benchmarks provided covers dedicated evaluations for each pillar. Hallucination in LALMs is frequently tied to the acoustic-semantic gap, where models hallucinate plausible but unfounded events, fail to ground outputs in actual audio evidence, or exhibit strong text modality dominance—even in explicitly auditory tasks. Modality neglect is empirically confirmed: replacing audio signals with silence often produces near-identical LALM outputs on emotional reasoning and other paralinguistic tasks. Alignment-enhancing mitigations, such as audio-contribution-aware post-training and cross-modal attentive reweighting, are quantitatively validated to improve acoustic grounding but are not sufficient to eliminate failures.

Robustness failures emerge from both environmental perturbations (realistic noise, speaker or context variation) and sophisticated adversarial acoustic attacks. The introduction of backdoor vulnerabilities via imperceptible pattern triggers during pre-training or alignment fine-tuning is especially severe in continuous modalities, even under minimal poisoning regimes. The "reasoning tax"—the empirically observed trade-off between adversarial robustness and authentic reasoning—remains unresolved.

With deepfake generation, LALMs have enabled both advanced speaker authentication (framed as AQA tasks) and robust anti-spoofing via interpretable component analysis. Partially manipulated, segment-level deepfakes and fine-grained localization benchmarks expose the brittleness of current detector facilities, especially under adversarial transfer.

Privacy concerns are magnified due to the biometric richness of voice: LALMs leak not only speaker identity information but also bystander and environmental context, substantiated by metrics on the HearSay and SH-Bench datasets.

Demographic, clinical, and positional bias penetrate the reasoning pipelines—LALMs propagate accent, timbre, and ordering biases into downstream decision-making, with measurable negative impact in high-stakes domains such as clinical recommendation systems.

Safety alignment is not inherited from text-only RLHF training; it is actively circumvented through style-based, emotional, or accent-based jailbreaking attacks. The effectiveness of exogenous guardrails and endogenous representation-shaping is limited, with defense mechanisms frequently resulting in increased "over-rejection" of benign queries.

Evaluation Paradigms and Holistic Safety Alignment

The survey advances a hierarchical framework for LALM evaluation along three pillars: fidelity (grounding and hallucination detection), stability/robustness (behavioral consistency), and safety/alignment (adversarial risk, privacy, bias, and authentication). This structure is necessary to expose mechanistic failures that are often invisible in aggregate performance metrics.

Figure 5: Conceptual taxonomy of LALM evaluation, partitioning approaches into fidelity/grounding, stability/robustness, and safety/alignment domains.

Noteworthy findings include:

• Audio hallucination (perceptual hallucination) is fundamentally a cross-modal failure, not simply a language generation error. Large-scale benchmarks such as HalluAudio show persistent rates of ungrounded generation even in top models.
• Behavioral instability in long-context settings is substantial: information loss, attention dilution, and reasoning decay are profound as audio duration increases, with up to 90% performance degradation in some tasks when moving from short to long-form.
• Instruction following is highly brittle—a change in prompt phrasing, speaker, or format reliably induces model divergence, revealing a severe lack of interactional stability.
• Safety and authentication evaluation demonstrate that jailbreaks exploiting non-semantic acoustic features (e.g., intonation, accent) bypass text-based safety filters, and new adversarially optimized attacks (e.g., over-the-air, stealth, emotional hijacking) remain effective against leading defenses.

Future Trajectories and Structural Research Gaps

The survey concludes by exposing a fundamental developmental asymmetry: the adversarial (offensive) landscape of LALMs is far more mature, nuanced, and diverse than the currently available defense mechanisms. Fundamental obstacles arise from the continuous/discrete dichotomy, lack of unified, dynamic safety benchmarks, and insufficient multimodal alignment.

Figure 6: The Outlook of LALM. Projected research trajectories emphasize intrinsic mechanistic advances, multi-layered safety, and rigorous, agent-based evaluation protocols.

The authors advocate three interlocking pillars for next-generation LALM security:

1. Input-Level Audio Sanitization: Pre-encoder purification employing diffusion and smoothing to neutralize adversarial triggers.
2. Intrinsic Privacy-Preserving Representation: Architectural anonymization decoupling speaker and environmental factors from semantic content.
3. Dynamic, Agent-Based Benchmarking: Automated evaluation agents capable of real-time probing of behavioral and latent state safety through multimodal red teaming.

Crucially, the future of trustworthy audio intelligence requires shifting safety and privacy from post-hoc detection to intrinsic model property—enabling causal world modeling, physically grounded reasoning, and structural disentanglement.

Conclusion

This survey articulates the current frontiers, architectural advances, and multidimensional trustworthiness challenges of LALMs as native auditory-computational agents. Although auditory generalization, emergent reasoning, and duplex conversational architectures are rapidly scaling, the associated trustworthiness landscape is dominated by sophisticated attack vectors and substantially lagging defensive maturity. Substantial progress will require bridging mechanistic alignment gaps, developing privacy- and safety-centric model architectures, and institutionalizing rigorous evaluation ecosystems that stress-test LALMs across real-world, adversarial, and ethically fraught operational regimes. The survey provides the theoretical and practical groundwork for these advances but highlights that addressing intrinsic trustworthiness is now the central challenge in deploying LALMs at scale.