DeSTA2.5-Audio: Modular Audio-LLM

• DeSTA2.5-Audio is a general-purpose Large Audio Language Model (LALM) that integrates a frozen audio encoder with an instruction-tuned LLM using a Q-Former adapter.
• It employs a self-generated cross-modal alignment strategy to fuse rich acoustic features with natural language, enabling versatile audio understanding.
• Evaluated on diverse benchmarks, the model demonstrates competitive performance in tasks like voice assistance, real-time audio analysis, and multimodal retrieval.

DeSTA2.5-Audio is a general-purpose Large Audio LLM (LALM) characterized by a modular architecture that integrates state-of-the-art audio processing with robust LLMing. Designed to facilitate instruction-following and multimodal auditory perception without requiring task-specific audio instruction-tuning, DeSTA2.5-Audio introduces a distinctive self-generated cross-modal alignment strategy that preserves the original language proficiency of the backbone LLM while supporting a broad array of audio understanding tasks (2507.02768).

1. Architectural Overview

DeSTA2.5-Audio adopts a fusion-based architectural paradigm combining a frozen, pre-trained audio encoder and a frozen, instruction-tuned LLM, interfaced through a Q-Former-based modality adapter. The default instantiation utilizes Whisper-large-v3 as the audio encoder and Llama3.1-8B-Instruct as the LLM. The architectural flow is as follows:

• Audio Encoder: Processes raw audio waveforms into multi-scale acoustic features via stacked transformer layers.
• Q-Former Modality Adapter: For each selected layer $\ell$ of the encoder, learnable query embeddings $Q^{(\ell)} \in \mathbb{R}^{N \times d}$ interact with hidden states $h^{(\ell)} \in \mathbb{R}^{T \times d}$ to yield query-oriented features: $$f^{(\ell)} = \mathrm{Q}\text{-}\mathrm{Former}\left(Q^{(\ell)}, h^{(\ell)}\right)$$.
• Feature Aggregation and Projection: Outputs $f^{(\ell)}$ are linearly aggregated with learnable weights $\alpha^{(\ell)}$ ($\sum \alpha^{(\ell)} = 1$), then projected to an LLM-compatible embedding:

$$F = \mathrm{Linear}\left(\sum_{\ell} \alpha^{(\ell)} f^{(\ell)}\right).$$

• Transcribed Audio Embedding (Optional): If available, transcribed textual tokens $E$ are embedded and optionally concatenated, yielding the final audio representation $A = [F; E]$.
• Prompt Conditioning and Generation: Given prompt embeddings $P$, the LLM receives the audio embedding $A$ and autoregressively generates response tokens:

$y_i = \mathrm{LLM}(P, A, y_{<i}).$

This architecture enables effective integration of temporally rich audio signals with natural language, ensuring efficient audio-text alignment and downstream response generation.

2. Self-Generated Cross-Modal Alignment Strategy

DeSTA2.5-Audio circumvents catastrophic forgetting—a common problem when LLMs are augmented via large-scale, cross-modal, instruction-tuned datasets—by introducing a self-generation mechanism for constructing training supervision. The process involves:

• Metadata Extraction: For each audio sample, structured metadata (e.g., time range, gender, emotion) is formatted as textual descriptors (e.g., “[00:00-00:05] Hello world (Gender:Female, Emotion:Happy, …)”).
• Prompt Sampling: A prompt $p$ is randomly selected from a large, diverse pool of instructions.
• Self-Target Generation: The same LLM that forms the model’s backbone generates the training target $y$ in response to $(x^{(audio)}, x^{(text)}, p)$.
• Training Tuple: The pipeline thus produces training quadruples $(x^{(audio)}, x^{(text)}, p, y)$.

This strategy ensures that the generated targets preserve the linguistic characteristics and knowledge distribution of the backbone LLM, as opposed to responses from alternate (potentially misaligned) models or human annotators. Empirical comparisons indicate that this reduces perplexity, preserves native language behavior, and substantially improves instruction-following generalization.

3. Data Construction: DeSTA-AQA5M

Training is performed on the DeSTA-AQA5M dataset, comprising 5 million triplets derived from approximately 7,000 hours of audio aggregated over 50 diverse public datasets. The distribution is diversified across:

• Speech: Including paralinguistic cues, speaker attributes, and various linguistic content.
• Environmental Sounds: Non-verbal audio from everyday environments.
• Music: Capturing genre, timbre, instrumentation, and musical structure.

Structured metadata extracted or inferred from source datasets is consistently translated into textual descriptors, ensuring the self-generation procedure is well grounded for any supported audio modality.

Only parameters in the Q-Former modality adapter (and, optionally, lightweight Low-Rank Adaptation (LoRA) modules) are updated during training; both the audio encoder and the backbone LLM weights remain frozen. This selective adaptation fosters robust cross-modal alignment without diminishing the general language proficiency of the LLM.

4. Performance Across Audio-Language Benchmarks

DeSTA2.5-Audio is evaluated using a comprehensive suite of benchmarks designed to probe a wide spectrum of audio-language competence:

• Dynamic-SUPERB: Multi-dimensional evaluation across content, semantics, paralinguistics, degradation, and speaker-sensitive tasks.
• MMAU (Multi-modal Audio Understanding): Multiple-choice tasks spanning speech, general sound, and music.
• SAKURA: Assessment of single-hop and multi-hop reasoning capabilities.
• Speech-IFEval: Diagnostics for instruction following and forgetting (measured by IFrate and $\Delta$).
• VoiceBench: Focused on voice assistant and conversational interaction performance.

Despite a modest training data scale (7,000 hours, compared to some baselines exceeding 100,000 hours), DeSTA2.5-Audio attains state-of-the-art or competitive results across all categories. In multi-hop reasoning and instruction following, it outperforms cascaded baselines (ASR+LLM) and prior LALMs such as Qwen2-Audio-Instruct and earlier DeSTA versions.

5. Comparative Analysis and Insights

A key finding is the performance advantage conferred by self-generated targets over alternatives generated by disjoint LLMs. When supervision targets are sourced from models other than the backbone LLM, distributional mismatches arise, manifesting in elevated perplexity and degraded evaluation metrics. The self-generation paradigm yields alignment both at the level of factual content and at the stylistic/linguistic distribution, maintaining continuity with the pre-trained LLM’s capabilities. This demonstrates the centrality of data construction, especially distributional properties of linguistic targets, in LALM engineering.

6. Applications and Future Directions

DeSTA2.5-Audio’s generalization and robustness render it suitable for a broad range of real-world applications:

• Voice assistants and interactive dialogue systems: Enhanced instruction following and contextual response in dynamic, real-world settings.
• Real-time audio analysis: For applications such as environmental monitoring and music information retrieval.
• Multimodal retrieval and recommendation: Integrating audio cues with text or visual modalities for complex multimedia tasks.
• Accessibility systems: Conversational support across varied dialects, accents, and noisy input conditions.

Future research aims to relax the dependence on textual intermediaries for audio, thereby better capturing non-textual and paralinguistic acoustic cues. Other directions include extending to more languages, refining the modality adapter, and developing advanced prompt engineering and multi-hop reasoning capabilities. Enhancing model sensitivity to subtle acoustic phenomena not readily expressible textually remains an important open challenge.

7. Significance and Impact

DeSTA2.5-Audio exemplifies a scalable paradigm for LALMs in which structural modularity, minimal task-specific fine-tuning, and rigorous data construction jointly deliver robust audio-language alignment. The empirical results affirm that preserving the backbone LLM’s native abilities during cross-modal alignment is critical for zero-shot generalization and instruction following. The model’s strong performance across diverse benchmarks, with relatively moderate compute and data requirements, has practical implications across academic research and industry deployments. Future progress in this domain is likely to emphasize further cross-modal grounding, architectural innovations that capture richer acoustic nuances, and expanded deployment in multilingual and specialized audio domains.