---
title: 'SALMONN: Multimodal Audio-Language Model'
url: https://www.emergentmind.com/topics/salmonn
type: topic
---

# SALMONN: Multimodal Audio-Language Model

SALMONN, short for **Speech Audio Language Music Open Neural Network**, is a multimodal large language model that integrates a pre-trained text-based large language model with speech and audio encoders into a single multimodal model, enabling the model to directly process and understand general audio inputs spanning speech, audio events, and music. In the literature, the name also denotes a broader line of systems built on the same research program, including **video-SALMONN** for speech-enhanced audio-visual understanding and **SALMONN-omni** for codec-free full-duplex speech understanding and generation [2310.13289] [2406.15704] [2411.18138].

## 1. Foundational formulation and research objective

SALMONN was introduced as a response to the claim that hearing is an essential ability of AI agents in the physical world, where auditory perception must cover at least three classes of sounds: speech, audio events, and music. The original model was described as the first multimodal large language model specifically designed to endow AI with generic hearing abilities, and as the first model of its type among systems that jointly address speech, general audio, and music in a single end-to-end framework [2310.13289].

The model’s stated objective was not limited to supervised auditory tasks. Alongside competitive performance on tasks used in training, the original work emphasized **cross-modal emergent abilities** unseen in training, including speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning. This combination of trained and emergent behavior positioned SALMONN as a step toward open-ended auditory-language intelligence rather than a narrowly optimized task model [2310.13289].

A plausible implication is that SALMONN occupies an intermediate point between classic task-specific audio models and later omni or full-duplex conversational systems: it retains the generality of an instruction-following LLM while grounding that capability in continuous auditory perception.

## 2. Core architecture and representation pipeline

The original SALMONN architecture is built from **dual auditory encoders**, a **window-level Q-Former** connection module, and a frozen **Vicuna-13B** backbone with **LoRA** adaptation. The speech encoder is **Whisper-Large-v2**, while the non-speech audio encoder is **BEATs** with fine-tuned AS2M weights. Both encoders produce frame-wise outputs at **50 Hz**, and their features are concatenated along the feature dimension for each time frame before being passed to the Q-Former [2310.13289].

The Q-Former is modified from BLIP-2 and applied at the **window level** rather than to the whole sequence. In the original configuration, it uses **\(N=1\)** trainable query per window of **\(L=17\)** frames, corresponding to approximately **0.33 s** per window; for a **30 s** audio input, the system produces about **88 tokens**. This design preserves temporal granularity for speech tasks while keeping the token budget compatible with the text backbone [2310.13289].

Only a small subset of parameters is updated during training. The original report states that only the LoRA modules and the Q-Former are trainable, accounting for about **33M parameters**, or **0.24%** of the model. Subsequent task-specific finetuning studies preserved the same parameter-efficient pattern: both audio encoders and the Vicuna LLM remain frozen, while the connection module and LoRA layers are trained [2310.13289] [2409.16644].

A later analysis of spoken language models characterized SALMONN’s modality adapter more precisely. In that study, SALMONN was described as using a **Whisper Large v2** speech encoder together with a **BEATs** general audio encoder and a **window-level Q-Former modality adapter**. Every **30 s** of input yields **88 tokens**, so each adapter output represents about **340 ms** of audio, roughly **3–4 phonemes** [2510.02569].

## 3. Training procedure and emergent abilities

The original SALMONN training recipe is explicitly **three-stage**. The first stage is **pre-training** for cross-modal alignment using large-scale speech recognition and audio captioning data, including **LibriSpeech**, **GigaSpeech**, **WavCaps**, **AudioCaps**, and **Clotho**. The second stage is **instruction tuning** on task-specific datasets covering speech recognition, speech translation, audio captioning, phone recognition, emotion recognition, music captioning, overlapped speech recognition, speaker verification, gender recognition, and speech/audio/music question answering. The third stage is **few-shot activation tuning**, introduced to re-awaken emergent reasoning and generation abilities that were reported to be suppressed by task overfitting after instruction tuning [2310.13289].

Activation tuning is central to SALMONN’s original identity. The reported mechanism generates a handful of challenging cross-modal samples—such as stories or question answering outputs—using a discounted LoRA scaling factor that increases diversity, and then applies teacher-forcing for a very small number of updates. The paper states that even **12 stories** with **one gradient step per sample** were sufficient to activate emergent abilities while avoiding catastrophic forgetting [2310.13289].

The quantitative profile reported in the original work separates trained tasks from emergent tasks. On Level 1 tasks, SALMONN reported **2.1 WER** on **LibriSpeech clean** ASR, **34.4 BLEU4** on **En2Zh**, **25.6 METEOR** on **AudioCaps**, and **.93** accuracy on **speaker verification** without activation tuning. On emergent tasks, activation tuning substantially raised instruction following and response quality: **spoken-query QA** increased from **.19 (.29 FR)** to **.41 (.98 FR)**, **slot filling** from **.33 (.77 FR)** to **.41 (.99 FR)**, **speech audio co-reasoning** from **.02 (.04 FR)** to **.50 (.73 FR)**, and **story** diversity from **7.77 (0.0 FR)** to **82.57 (1.0 FR)** [2310.13289].

This training design matters conceptually because SALMONN does not treat auditory-language competence as a fixed property inherited from pre-trained encoders and a frozen LLM. Instead, it treats emergent auditory reasoning as a capability that can be suppressed by deterministic supervision and later reactivated by minimal but targeted cross-modal conditioning.

## 4. Family variants and system-level extensions

The SALMONN line subsequently expanded in two main directions: **audio-visual grounding** and **full-duplex spoken interaction**.

| Variant | Characterization | Distinguishing mechanism |
|---|---|---|
| **SALMONN** | Generic hearing over speech, audio, and music | Whisper-Large-v2, BEATs, window-level Q-Former, Vicuna backbone |
| **video-SALMONN** | Speech-enhanced audio-visual large language model | Multi-Resolution Causal Q-Former, diversity loss, unpaired audio-visual mixed training |
| **SALMONN-omni** | Codec-free, full-duplex speech understanding and generation | Streaming speech encoder, LLM core, streaming speech synthesizer, thinking mechanism |

**video-SALMONN** extends the auditory formulation to video processing. It combines a visual encoder based on **InstructBLIP**, a speech encoder based on **Whisper-large-v2**, and **BEATs** for non-speech audio, with all encoders kept frozen. Its main architectural novelty is the **Multi-Resolution Causal Q-Former (MRC Q-Former)**, which aligns multimodal features and projects them into the LLM input space at multiple temporal resolutions. The work also introduces **diversity loss** and an **unpaired audio-visual mixed training scheme** to avoid frame or modality dominance. On the introduced **SAVE** benchmark, video-SALMONN reported **more than 25% absolute accuracy improvements on the video-QA task** and **over 30% absolute accuracy improvements on audio-visual QA tasks with human speech** [2406.15704].

**SALMONN-omni** moves from passive understanding to **full-duplex** interaction. It is described as a **codec-free, full-duplex speech understanding and generation model** whose architecture consists of a **streaming speech encoder**, an **LLM core**, and a **streaming speech synthesizer**, all communicating through **continuous embeddings** rather than text tokens or quantized speech codecs. The model introduces a duplex spoken dialogue framework with **time blocks** of size **\(\Delta t\)**, explicit state tokens **`<start_speak>`** and **`<end_speak>`**, and a **`<think>`** token that acts as a placeholder for internal, non-spoken cognitive activity during non-speaking blocks [2411.18138].

The training objective for the thinking mechanism is described as
$$
\mathcal{L} = \lambda_\text{text}\mathcal{L}_\text{text}
+ \lambda_\text{speech}\mathcal{L}_\text{speech}
+ \lambda_\text{think}\mathcal{L}_\text{think},
$$
with **\(\lambda_\text{think} < 0\)** to avoid output collapse on the thinking token. The later standalone report on SALMONN-omni makes the architecture more concrete by specifying a **streaming speech encoder based on Mamba LM blocks**, **Llama-3-8B-Instruct** as the backbone, and a **streaming speech synthesizer built on CosyVoice2**. That report describes **fixed-length time blocks** of **80 ms**, an **explicit thinking** strategy that outperformed implicit thinking, and **Direct Preference Optimization (DPO)** for context-dependent barge-in and backchanneling. It further states that SALMONN-omni achieves **at least 30% relative performance improvement over existing open-source full-duplex models** and a **35.9% relative average accuracy improvement** over previous open full-duplex models under predicted turn-taking [2411.18138] [2505.17060].

## 5. Empirical profile across benchmarks and downstream tasks

Large benchmark studies present SALMONN as strong but highly non-universal. In **Dynamic-SUPERB Phase-2**, a benchmark with **180 tasks** across speech, music, and environmental audio, **SALMONN-13B** recorded the best reported **English ASR** performance with **2.79% WER** on **Superb-ASR (LibriSpeech-TestClean)** and also excelled in **speaker verification** with **93.5%** accuracy on **SuperbSV**. At the same time, it underperformed on several other task families, including **emotion recognition** at **30.9%** on **Superb ER**, music classification, sound event detection, and parts of spoken language understanding. The benchmark authors concluded that no model performed well universally [2411.05361].

Task-specific finetuning studies show that SALMONN can adapt effectively when supervision is aligned with the target objective. In **automatic speech quality assessment**, auditory LLMs including SALMONN were finetuned to predict **MOS**, **SIM**, **A/B testing** outcomes, and **natural language descriptions**. The reported **SALMONN (vic1.5)** results were **.861/.859/.347** on **NISQA**, **.826/.833/.282** on **BVCC**, **.644/.636/.196** at utterance level and **.894/.891/.034** at system level on **SOMOS**, and **.796/.809/1.374** on **VoxSim** for speaker similarity. The same study reported a **0.64** correlation score for finetuned natural language descriptions, emphasizing interpretability alongside numeric evaluation [2409.16644].

Audio deepfake detection exposes a different empirical pattern. In a study framing the problem as **Audio Question-Answering**, SALMONN combined audio inputs with prompt variants such as a direct binary prompt, a yes/no prompt, a context-rich descriptive prompt, and a combined multi-prompt setting. In **zero-shot** evaluation on **ASVspoof 2019 LA**, SALMONN with **Prompt#1** achieved **ACC = 0.46** and **mF1 = 0.46**, described as close to random guessing. After **LoRA** finetuning, the same work reported **ACC = 0.98** and **mF1 = 0.98** for the **Multi** prompt on in-domain data, but only **ACC = 0.66** and **mF1 = 0.62** on **In-the-Wild** out-of-domain evaluation, indicating limited generalization [2601.00777].

A separate zero-shot spoofing study reached an even more pessimistic conclusion. Across **ASVspoof2019**, **In-the-Wild**, and **WaveFake**, the authors reported severe predictive biases for all evaluated large audio-language models, including SALMONN, and argued that their practical performance was effectively equivalent to random classification when judged by balanced metrics. For SALMONN specifically, **FP16** quantization caused negligible degradation relative to **FP32**, reducing memory from **51.47 GB** to **27.34 GB** and inference time from **1020.38 s** to **960.38 s per 100 samples**, whereas **INT8** intensified bias and slowed inference [2506.06756].

## 6. Representation analysis, multilingual extensions, and current research directions

Interpretability studies of SALMONN’s internal representations have focused on the **modality adapter**. An investigation of intermediate representations in spoken language models found that for **Whisper-based** systems such as SALMONN, the modality adapter output is predominantly an **English-based semantic representation** rather than a phonetic transliteration. For **every input language tested**, **over 90%** of nearest-token analyses labeled SALMONN’s adapter output as **English**, and word-level analyses indicated that the outputs often correspond to **translations or semantic equivalents in English**. The authors hypothesized that this behavior arises because Whisper was trained for both **ASR** and **speech translation to English**. This suggests a **hidden interlingua** that may explain generalization to languages unseen in instruction tuning [2510.02569].

Multilingual instruction-following work reinforced that interpretation. The **XS-CoT** framework was applied to SALMONN and other speech large language models to improve non-core language performance by inserting a cross-lingual reasoning chain comprising target-language instruction tokens, core-language instruction tokens, core-language response tokens, and target-language response tokens. The paper reports **up to 45% in GPT-4 score** improvement over direct supervised finetuning on SALMONN and states that **semi-implicit XS-CoT** reduces token delay by **more than 50%** with only a slight drop in GPT-4 scores. In the SALMONN-specific training recipe, only the **Q-former** is tuned during modal alignment, while only the **LLM** is tuned with **LoRA** during XS-CoT and semi-implicit training [2504.20835].

Long-context audio understanding is another active extension point. A context-extension study introduced **Partial YaRN**, which modifies only audio token positions while leaving text positions intact, and evaluated it directly on SALMONN. On **YODAS2-MCQA**, vanilla SALMONN dropped from **49.01%** at **1 min** to **23.47%** at **10 min**, whereas **Partial YaRN** achieved **57.35%** at **1 min**, **59.60%** at **2 min**, **38.53%** at **5 min**, and **32.93%** at **10 min**. The same work proposed **Virtual Longform Audio Training (VLAT)** as a training-time positional augmentation strategy for extending Partial YaRN to unseen audio lengths [2510.15231].

Safety and hallucination research has also intersected with the SALMONN line. For **video-SALMONN**, the training-free **AVCD** decoding framework improved **AVHBench** accuracy by **11%** over base decoding, from **60.00** to **66.83**, by dynamically identifying less dominant modalities and applying attentive masking during trimodal contrastive decoding [2505.20862]. A separate safety-focused system named **SALMONN-Guard**, built on **Qwen2.5-Omni-7B** rather than the original SALMONN backbone, was introduced together with **SACRED-Bench** to address speech-audio compositional attacks. That work reported a reduction in overall attack success rate from **66.75%** for **Gemini 2.5 Pro** to **11.32%** for **SALMONN-Guard**, alongside **100% accuracy on benign samples** [2511.10222].

Taken together, these studies depict SALMONN as a technically influential auditory-language architecture whose strengths are clearest in **English ASR**, **speaker-related tasks**, and adaptable instruction-following settings, but whose limitations remain substantial in **universal zero-shot coverage**, **out-of-domain forensic robustness**, and several **paralinguistic**, **music**, and **environmental audio** tasks. The literature consistently treats SALMONN not as a closed solution, but as a platform from which audio-visual reasoning, full-duplex dialogue, safety controls, multilingual transfer, and long-context audio understanding continue to be developed [2411.05361] [2601.00777].

Source: https://www.emergentmind.com/topics/salmonn