SAM Audio: Audio Separation Framework

• SAM Audio is a foundation model framework for prompt-driven, open-domain audio separation using text, visual, and span prompts.
• It employs a diffusion-transformer with cross-attention and dedicated prompt fusion to achieve high-fidelity separation and segmentation.
• The framework extends to audio-visual segmentation and incremental learning, consistently setting new benchmarks in performance and perceptual evaluation.

SAM Audio is a foundation model framework for prompt-driven, open-domain audio source separation and multimodal segmentation. It defines a new paradigm in audio processing by supporting text, visual, and temporal prompts for highly flexible, interactive, and controllable audio extraction tasks. The term encompasses both the SAM Audio model itself (Shi et al., 19 Dec 2025), multimodal architectures for audio-visual segmentation leveraging the Segment Anything Model (SAM) (Nguyen et al., 2024, Liu et al., 1 Jun 2025, Lee et al., 22 Feb 2025, Huang et al., 2024), and metrics for reference-free perceptual evaluation (SAM Audio Judge) (Wang et al., 27 Jan 2026). These advances extend the flexibility and generalization of large-scale visual foundation models to the broader audio and audio-visual domains.

1. Model Architecture and Prompting Modalities

At the core of the SAM Audio framework is a generative diffusion-transformer model operating in a continuous latent space provided by a pretrained DAC-VAE (frames at 25 Hz, 128-dimensional latent vectors) (Shi et al., 19 Dec 2025). Given a mixture, SAM Audio jointly produces target and residual stems by concatenating their latent representations. Its training is governed by a rectified flow-matching objective, where the Diffusion Transformer predicts an instantaneous velocity field $u(x_t, c, t; \theta)$ transporting sampled latent noise $x_0$ toward the data manifold, conditioned on multimodal prompts $c$.

The model supports three major prompting interfaces:

• Text prompts: NP/VP-style descriptions (e.g., "dog barking"), encoded by a frozen T5-base model and fused via cross-attention in the DiT.
• Visual prompts: Pixel-level binary masks (e.g., from SAM 2) coupled with video frames processed by a Perception Encoder, projected and frame-aligned with the audio features.
• Span prompts: Explicit or predicted time intervals, encoded as token sequences and concatenated with the latent mixture trajectory.

Prompt information is modulated into the separation process via dedicated DiT blocks through cross-attention (for text) and concatenation (for visual/spans), enabling arbitrary control over "what" (semantics) and "when/where" (temporal/spatial extent) to extract.

Auxiliary loss terms align internal DiT hidden states with embeddings from a pretrained audio event detection (AED) network, improving the model's focus on both content and localization.

2. Training Data, Objective Functions, and Evaluation Protocols

SAM Audio is pretrained and fine-tuned on an extensive, multi-regime training protocol:

• Fully real mixtures: Professionally recorded multi-track music (~10k pieces, 536h) and 21k hours of conversational speech.
• Synthetic mixtures: Mixes of clean music, in-the-wild sound events, and speech–noise blends at a range of SNRs.
• Pseudo-labeled mixtures: An intermediate SAM Audio checkpoint is applied to 1 million hours of unlabeled video/audio, producing separation hypotheses via text prompts generated by an audio LLM (PLM-Audio). Multi-modal filtering (CLAP similarity, aesthetic scores, ImageBind for visuals) ensures only high-quality pseudo-labels are retained.

The separation process is trained via a mean-squared error (MSE) flow-matching loss:

$$\mathcal{L}_\mathrm{FM} = \mathbb{E}_{x_0, t} \| u(x_t, c, t; \theta) - [x_1 - (1-\sigma_\mathrm{min})x_0] \|^2$$

with auxiliary representation alignment:

$$\mathcal{L}_\mathrm{aux} = \mathbb{E}_t [ 1 - \mathrm{sim}(\phi_\mathrm{aux}(h_t), \mathrm{AED}(x_\mathrm{tgt})) ]$$

and a final loss $$\mathcal{L} = \mathcal{L}_\mathrm{FM} + \lambda \mathcal{L}_\mathrm{aux}$$.

Evaluation is performed with both ground-truth references (e.g., standard subjective and objective source separation metrics) and the reference-free, perceptually aligned SAM Audio Judge (SAJ). SAJ is a neural model predicting recall, precision, faithfulness, and overall perceptual scores, with high human correlation (PCC up to 0.88 for speech) (Wang et al., 27 Jan 2026).

3. Audio-Visual Segmentation: Adaptations of SAM for Audio

Extending SAM for audio-visual segmentation has spawned multiple model designs:

• Adapter-based fusion (SAVE): Adapters are inserted in SAM's ViT blocks to integrate audio-specific cues, using residual audio encoder adapters for deep fusion. Audio features (e.g., from VGGish) are transformed to produce sparse prompts that serve as input to the SAM mask decoder (Nguyen et al., 2024).
• Feature-pyramid fusion (AuralSAM2): AuralFuser modules attach externally to SAM2's feature pyramid, providing dense and sparse audio-visual prompts to the decoder at multiple scales. Audio-guided contrastive learning aligns cross-modal features and suppresses visually dominant false matches (Liu et al., 1 Jun 2025).
• Semantic alignment via text (AV2T-SAM): Audio (CLAP) and visual (CLIP) embeddings are projected into a shared semantic space and element-wise fused to form cross-modal features, which are subsequently mapped to SAM's text prompt space (Lee et al., 22 Feb 2025).

These audio-visual SAM variants consistently achieve state-of-the-art mIoU and F-score on AVSBench S4/MS3 and Ref-AVS, even at quarter-resolution and with all base weights frozen, via efficient fine-tuning of adapters only. Fully prompt-driven ("zero-shot") pipelines using multimodal guidance, GPT-based reasoning (AL-Ref-SAM2), and language-unified references can match or surpass fully supervised benchmarks (Huang et al., 2024).

4. Applications in Separation, Segmentation, and Incremental Learning

The modular prompt-driven design of SAM Audio and its derivatives enables broad application:

• General sound, speech, and music separation: Outperforms open-domain and specialized baselines (AudioSep, FlowSep, Demucs, Spleeter, etc.), attaining SAJ subjective-judge scores exceeding 4.5 in most domains (Shi et al., 19 Dec 2025).
• Multimodal segmentation: Provides high-fidelity temporal and spatial segmentation of sources in audio-visual contexts (e.g., localizing a "sounding drum set" or "dog barking in video"), addressing data scarcity and annotation bottlenecks (Liu et al., 1 Jun 2025, Lee et al., 22 Feb 2025).
• Class-incremental learning: Frozen SAM Audio encoders provide stable multimodal representations, while dual-level (feature/logit) distillation objectives and audio-guided attention dramatically reduce forgetting, with mean accuracy gains >10 percentage points over prior CIL methods on AVE-CI and VS100-CI (Gupta et al., 9 Jun 2026).
• Perceptual evaluation and data curation: SAJ can rerank model outputs, filter pseudo-labels by perceptual quality, and stratify samples by robustness/difficulty, greatly reducing manual evaluation costs (Wang et al., 27 Jan 2026).
• Plug-and-play pipelines: Fully training-free segmentation using audio/language references and GPT-based pivot selection enables strong open-vocabulary and open-modality performance, mitigating the need for retraining as new data or referencing schemes arise (Huang et al., 2024).

5. Benchmarking, Performance, and Ablation Insights

On audio and audio-visual separation and segmentation tasks:

• Separation: For 3B-parameter SAM Audio-Large, speech extraction yields SAJ=4.67, general SFX SAJ=4.35, and music/instrument separation SAJ up to 4.82 (MUSDB). SAM Audio achieves net win rates of 20–40% over both open-domain and specialized baselines (Shi et al., 19 Dec 2025).
• Segmentation: AuralSAM2 reports mIoU up to 86.62 on AVSBench V1m (multi-source), exceeding GAVS and prior SAM-based variants by 1–3 points (Liu et al., 1 Jun 2025). AV2T-SAM delivers mIoU=86.67, F-score=0.924 (S4 subset), up to 1.56 points over previous SOTA (Lee et al., 22 Feb 2025). SAVE, with a frozen backbone and only trained adapters, attains mIoU=85.11/0.912 (S4) and generalizes well even with synthetic-only pretraining (Nguyen et al., 2024).
• Evaluation: SAJ reference-free perceptual metrics show PCC/SRCC up to 0.88/0.82 for overall quality, recall, precision, and faithfulness, outperforming CLAP, SDR Estimators, and commercial LLMs (Wang et al., 27 Jan 2026).
• Ablation studies: Larger model scales mostly benefit instrument and highly specialized tasks. Joint text+span prompting outperforms text-only prompting by large margins (up to +39% net win rate), and combining text with predicted spans improves controllability and quality further. Auxiliary task pretraining and additional pseudo-labels consistently improve downstream performance.

6. Limitations, Failure Cases, and Recommendations

• Mismatch with downstream models: Aggressive audio "cleaning" with SAM Audio can degrade the performance of zero-shot automatic speech recognition (e.g., Whisper), despite objective signal improvements (PSNR +3.7 dB), due to departures from the pretrained input manifold, especially for larger ASR models. Even imperceptible artifacts alter fine-grained time–frequency structures critical for end-to-end ASR, yielding systematic increases in WER/CER across >90% of samples (Islam et al., 5 Mar 2026).
• Superficial fusion pitfalls: Injecting audio features into vision encoders or relying solely on adapter-based fusion can be susceptible to biases from dominant visual patterns or insufficient cross-modal alignment if not properly regularized (e.g., via contrastive audio-visual loss or pyramid-level fusion) (Liu et al., 1 Jun 2025).
• Prompt modality efficacy: Span-only prompts may underperform for continuous or ambiguous events, while joint prompt modalities deliver consistently higher extraction and localization accuracy.
• Prompt engineering: Quality of segmentation and separation is highly sensitive to prompt construction; pipelines like AL-Ref-SAM2 leverage GPT-based Chain-of-Thought for pivot frame and region selection, substantially narrowing the performance gap with manually fine-tuned systems (Huang et al., 2024).

A plausible implication is that end-to-end joint training of denoising and downstream recognition or segmentation, as well as prompt-aware data augmentation, will be essential for future end-user applications.

7. Perspectives and Open Directions

SAM Audio establishes a flexible and generalizable blueprint for multimodal source separation and segmentation, but several directions remain open:

• Unified semantic and mask prediction: Current pipelines may depend on external "stepping-stones" to align semantic class tokens and mask outputs (e.g., in AuralSAM2); end-to-end architectures integrating class semantic awareness in the mask decoder remain to be established (Liu et al., 1 Jun 2025).
• Task-aware enhancement: Coupling audio separation with recognition (ASR-aware enhancement), e.g., through multi-task losses or adaptation of the denoising front-end, is advocated to avoid degradation in downstream task performance (Islam et al., 5 Mar 2026).
• Robustness to domain shift: Pretraining on synthetic or out-of-domain mixtures delivers robust zero-shot performance, but fine-tuning and prompt adaptation for new domains are required to unlock the full potential of foundation audio models (Nguyen et al., 2024).
• Reference-free perceptual metrics: Tools like SAM Audio Judge demonstrate strong alignment with human ratings and facilitate scalable evaluation, data curation, and model selection across text, visual, or span prompted tasks, serving as foundational infrastructure for prompt-driven audio AI research (Wang et al., 27 Jan 2026).

In conclusion, SAM Audio and its associated ecosystem (SAM Audio Judge, audio-visual prompt-driven SAM variants) constitute the current state-of-the-art in controllable, scalable, and high-fidelity audio separation and segmentation. These models enable interactive, multimodal pipelines for both scientific research and real-world deployment, spanning open-domain sound separation, AV segmentation, incremental learning, and perceptual evaluation (Shi et al., 19 Dec 2025, Wang et al., 27 Jan 2026, Liu et al., 1 Jun 2025, Lee et al., 22 Feb 2025, Nguyen et al., 2024, Huang et al., 2024, Islam et al., 5 Mar 2026, Gupta et al., 9 Jun 2026).