SALMONN: Multimodal Audio-Language Model
- SALMONN is a multimodal large language model that integrates speech, audio, and music processing through dual auditory encoders, a window-level Q-Former, and a frozen Vicuna LLM.
- The architecture employs parameter-efficient training methods, updating only 0.24% of its parameters to achieve robust cross-modal alignment and emergent reasoning capabilities.
- Variants like video-SALMONN and SALMONN-omni extend the original design to audio-visual integration and full-duplex speech interaction, demonstrating significant performance improvements on diverse benchmarks.
SALMONN, short for Speech Audio Language Music Open Neural Network, is a multimodal LLM that integrates a pre-trained text-based LLM with speech and audio encoders into a single multimodal model, enabling the model to directly process and understand general audio inputs spanning speech, audio events, and music. In the literature, the name also denotes a broader line of systems built on the same research program, including video-SALMONN for speech-enhanced audio-visual understanding and SALMONN-omni for codec-free full-duplex speech understanding and generation (Tang et al., 2023, Sun et al., 2024, Yu et al., 2024).
1. Foundational formulation and research objective
SALMONN was introduced as a response to the claim that hearing is an essential ability of AI agents in the physical world, where auditory perception must cover at least three classes of sounds: speech, audio events, and music. The original model was described as the first multimodal LLM specifically designed to endow AI with generic hearing abilities, and as the first model of its type among systems that jointly address speech, general audio, and music in a single end-to-end framework (Tang et al., 2023).
The model’s stated objective was not limited to supervised auditory tasks. Alongside competitive performance on tasks used in training, the original work emphasized cross-modal emergent abilities unseen in training, including speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning. This combination of trained and emergent behavior positioned SALMONN as a step toward open-ended auditory-language intelligence rather than a narrowly optimized task model (Tang et al., 2023).
A plausible implication is that SALMONN occupies an intermediate point between classic task-specific audio models and later omni or full-duplex conversational systems: it retains the generality of an instruction-following LLM while grounding that capability in continuous auditory perception.
2. Core architecture and representation pipeline
The original SALMONN architecture is built from dual auditory encoders, a window-level Q-Former connection module, and a frozen Vicuna-13B backbone with LoRA adaptation. The speech encoder is Whisper-Large-v2, while the non-speech audio encoder is BEATs with fine-tuned AS2M weights. Both encoders produce frame-wise outputs at 50 Hz, and their features are concatenated along the feature dimension for each time frame before being passed to the Q-Former (Tang et al., 2023).
The Q-Former is modified from BLIP-2 and applied at the window level rather than to the whole sequence. In the original configuration, it uses trainable query per window of frames, corresponding to approximately 0.33 s per window; for a 30 s audio input, the system produces about 88 tokens. This design preserves temporal granularity for speech tasks while keeping the token budget compatible with the text backbone (Tang et al., 2023).
Only a small subset of parameters is updated during training. The original report states that only the LoRA modules and the Q-Former are trainable, accounting for about 33M parameters, or 0.24% of the model. Subsequent task-specific finetuning studies preserved the same parameter-efficient pattern: both audio encoders and the Vicuna LLM remain frozen, while the connection module and LoRA layers are trained (Tang et al., 2023, Wang et al., 2024).
A later analysis of spoken LLMs characterized SALMONN’s modality adapter more precisely. In that study, SALMONN was described as using a Whisper Large v2 speech encoder together with a BEATs general audio encoder and a window-level Q-Former modality adapter. Every 30 s of input yields 88 tokens, so each adapter output represents about 340 ms of audio, roughly 3–4 phonemes (Ògúnrèmà et al., 2 Oct 2025).
3. Training procedure and emergent abilities
The original SALMONN training recipe is explicitly three-stage. The first stage is pre-training for cross-modal alignment using large-scale speech recognition and audio captioning data, including LibriSpeech, GigaSpeech, WavCaps, AudioCaps, and Clotho. The second stage is instruction tuning on task-specific datasets covering speech recognition, speech translation, audio captioning, phone recognition, emotion recognition, music captioning, overlapped speech recognition, speaker verification, gender recognition, and speech/audio/music question answering. The third stage is few-shot activation tuning, introduced to re-awaken emergent reasoning and generation abilities that were reported to be suppressed by task overfitting after instruction tuning (Tang et al., 2023).
Activation tuning is central to SALMONN’s original identity. The reported mechanism generates a handful of challenging cross-modal samples—such as stories or question answering outputs—using a discounted LoRA scaling factor that increases diversity, and then applies teacher-forcing for a very small number of updates. The paper states that even 12 stories with one gradient step per sample were sufficient to activate emergent abilities while avoiding catastrophic forgetting (Tang et al., 2023).
The quantitative profile reported in the original work separates trained tasks from emergent tasks. On Level 1 tasks, SALMONN reported 2.1 WER on LibriSpeech clean ASR, 34.4 BLEU4 on En2Zh, 25.6 METEOR on AudioCaps, and .93 accuracy on speaker verification without activation tuning. On emergent tasks, activation tuning substantially raised instruction following and response quality: spoken-query QA increased from .19 (.29 FR) to .41 (.98 FR), slot filling from .33 (.77 FR) to .41 (.99 FR), speech audio co-reasoning from .02 (.04 FR) to .50 (.73 FR), and story diversity from 7.77 (0.0 FR) to 82.57 (1.0 FR) (Tang et al., 2023).
This training design matters conceptually because SALMONN does not treat auditory-language competence as a fixed property inherited from pre-trained encoders and a frozen LLM. Instead, it treats emergent auditory reasoning as a capability that can be suppressed by deterministic supervision and later reactivated by minimal but targeted cross-modal conditioning.
4. Family variants and system-level extensions
The SALMONN line subsequently expanded in two main directions: audio-visual grounding and full-duplex spoken interaction.
| Variant | Characterization | Distinguishing mechanism |
|---|---|---|
| SALMONN | Generic hearing over speech, audio, and music | Whisper-Large-v2, BEATs, window-level Q-Former, Vicuna backbone |
| video-SALMONN | Speech-enhanced audio-visual LLM | Multi-Resolution Causal Q-Former, diversity loss, unpaired audio-visual mixed training |
| SALMONN-omni | Codec-free, full-duplex speech understanding and generation | Streaming speech encoder, LLM core, streaming speech synthesizer, thinking mechanism |
video-SALMONN extends the auditory formulation to video processing. It combines a visual encoder based on InstructBLIP, a speech encoder based on Whisper-large-v2, and BEATs for non-speech audio, with all encoders kept frozen. Its main architectural novelty is the Multi-Resolution Causal Q-Former (MRC Q-Former), which aligns multimodal features and projects them into the LLM input space at multiple temporal resolutions. The work also introduces diversity loss and an unpaired audio-visual mixed training scheme to avoid frame or modality dominance. On the introduced SAVE benchmark, video-SALMONN reported more than 25% absolute accuracy improvements on the video-QA task and over 30% absolute accuracy improvements on audio-visual QA tasks with human speech (Sun et al., 2024).
SALMONN-omni moves from passive understanding to full-duplex interaction. It is described as a codec-free, full-duplex speech understanding and generation model whose architecture consists of a streaming speech encoder, an LLM core, and a streaming speech synthesizer, all communicating through continuous embeddings rather than text tokens or quantized speech codecs. The model introduces a duplex spoken dialogue framework with time blocks of size , explicit state tokens <start_speak> and <end_speak>, and a > token that acts as a placeholder for internal, non-spoken cognitive activity during non-speaking blocks (Yu et al., 2024).
The training objective for the thinking mechanism is described as
with to avoid output collapse on the thinking token. The later standalone report on SALMONN-omni makes the architecture more concrete by specifying a streaming speech encoder based on Mamba LM blocks, Llama-3-8B-Instruct as the backbone, and a streaming speech synthesizer built on CosyVoice2. That report describes fixed-length time blocks of 80 ms, an explicit thinking strategy that outperformed implicit thinking, and Direct Preference Optimization (DPO) for context-dependent barge-in and backchanneling. It further states that SALMONN-omni achieves at least 30% relative performance improvement over existing open-source full-duplex models and a 35.9% relative average accuracy improvement over previous open full-duplex models under predicted turn-taking (Yu et al., 2024, Yu et al., 17 May 2025).
5. Empirical profile across benchmarks and downstream tasks
Large benchmark studies present SALMONN as strong but highly non-universal. In Dynamic-SUPERB Phase-2, a benchmark with 180 tasks across speech, music, and environmental audio, SALMONN-13B recorded the best reported English ASR performance with 2.79% WER on Superb-ASR (LibriSpeech-TestClean) and also excelled in speaker verification with 93.5% accuracy on SuperbSV. At the same time, it underperformed on several other task families, including emotion recognition at 30.9% on Superb ER, music classification, sound event detection, and parts of spoken language understanding. The benchmark authors concluded that no model performed well universally (Huang et al., 2024).
Task-specific finetuning studies show that SALMONN can adapt effectively when supervision is aligned with the target objective. In automatic speech quality assessment, auditory LLMs including SALMONN were finetuned to predict MOS, SIM, A/B testing outcomes, and natural language descriptions. The reported SALMONN (vic1.5) results were .861/.859/.347 on NISQA, .826/.833/.282 on BVCC, .644/.636/.196 at utterance level and .894/.891/.034 at system level on SOMOS, and .796/.809/1.374 on VoxSim for speaker similarity. The same study reported a 0.64 correlation score for finetuned natural language descriptions, emphasizing interpretability alongside numeric evaluation (Wang et al., 2024).
Audio deepfake detection exposes a different empirical pattern. In a study framing the problem as Audio Question-Answering, SALMONN combined audio inputs with prompt variants such as a direct binary prompt, a yes/no prompt, a context-rich descriptive prompt, and a combined multi-prompt setting. In zero-shot evaluation on ASVspoof 2019 LA, SALMONN with Prompt#1 achieved ACC = 0.46 and mF1 = 0.46, described as close to random guessing. After LoRA finetuning, the same work reported ACC = 0.98 and mF1 = 0.98 for the Multi prompt on in-domain data, but only ACC = 0.66 and mF1 = 0.62 on In-the-Wild out-of-domain evaluation, indicating limited generalization (Chuchra et al., 2 Jan 2026).
A separate zero-shot spoofing study reached an even more pessimistic conclusion. Across ASVspoof2019, In-the-Wild, and WaveFake, the authors reported severe predictive biases for all evaluated large audio-LLMs, including SALMONN, and argued that their practical performance was effectively equivalent to random classification when judged by balanced metrics. For SALMONN specifically, FP16 quantization caused negligible degradation relative to FP32, reducing memory from 51.47 GB to 27.34 GB and inference time from 1020.38 s to 960.38 s per 100 samples, whereas INT8 intensified bias and slowed inference (Dutta et al., 7 Jun 2025).
6. Representation analysis, multilingual extensions, and current research directions
Interpretability studies of SALMONN’s internal representations have focused on the modality adapter. An investigation of intermediate representations in spoken LLMs found that for Whisper-based systems such as SALMONN, the modality adapter output is predominantly an English-based semantic representation rather than a phonetic transliteration. For every input language tested, over 90% of nearest-token analyses labeled SALMONN’s adapter output as English, and word-level analyses indicated that the outputs often correspond to translations or semantic equivalents in English. The authors hypothesized that this behavior arises because Whisper was trained for both ASR and speech translation to English. This suggests a hidden interlingua that may explain generalization to languages unseen in instruction tuning (Ògúnrèmà et al., 2 Oct 2025).
Multilingual instruction-following work reinforced that interpretation. The XS-CoT framework was applied to SALMONN and other speech LLMs to improve non-core language performance by inserting a cross-lingual reasoning chain comprising target-language instruction tokens, core-language instruction tokens, core-language response tokens, and target-language response tokens. The paper reports up to 45% in GPT-4 score improvement over direct supervised finetuning on SALMONN and states that semi-implicit XS-CoT reduces token delay by more than 50% with only a slight drop in GPT-4 scores. In the SALMONN-specific training recipe, only the Q-former is tuned during modal alignment, while only the LLM is tuned with LoRA during XS-CoT and semi-implicit training (Xue et al., 29 Apr 2025).
Long-context audio understanding is another active extension point. A context-extension study introduced Partial YaRN, which modifies only audio token positions while leaving text positions intact, and evaluated it directly on SALMONN. On YODAS2-MCQA, vanilla SALMONN dropped from 49.01% at 1 min to 23.47% at 10 min, whereas Partial YaRN achieved 57.35% at 1 min, 59.60% at 2 min, 38.53% at 5 min, and 32.93% at 10 min. The same work proposed Virtual Longform Audio Training (VLAT) as a training-time positional augmentation strategy for extending Partial YaRN to unseen audio lengths (Chaichana et al., 17 Oct 2025).
Safety and hallucination research has also intersected with the SALMONN line. For video-SALMONN, the training-free AVCD decoding framework improved AVHBench accuracy by 11% over base decoding, from 60.00 to 66.83, by dynamically identifying less dominant modalities and applying attentive masking during trimodal contrastive decoding (Jung et al., 27 May 2025). A separate safety-focused system named SALMONN-Guard, built on Qwen2.5-Omni-7B rather than the original SALMONN backbone, was introduced together with SACRED-Bench to address speech-audio compositional attacks. That work reported a reduction in overall attack success rate from 66.75% for Gemini 2.5 Pro to 11.32% for SALMONN-Guard, alongside 100% accuracy on benign samples (Yang et al., 13 Nov 2025).
Taken together, these studies depict SALMONN as a technically influential auditory-language architecture whose strengths are clearest in English ASR, speaker-related tasks, and adaptable instruction-following settings, but whose limitations remain substantial in universal zero-shot coverage, out-of-domain forensic robustness, and several paralinguistic, music, and environmental audio tasks. The literature consistently treats SALMONN not as a closed solution, but as a platform from which audio-visual reasoning, full-duplex dialogue, safety controls, multilingual transfer, and long-context audio understanding continue to be developed (Huang et al., 2024, Chuchra et al., 2 Jan 2026).
