Audio Mamba (AuM): Efficient Audio SSM Models

• Audio Mamba (AuM) is a family of neural sequence models based on structured state-space models that deliver linear-time complexity and efficient long-range dependency capture.
• They combine dynamic input-adaptive SSM recurrences with selective gating and convolutional preprocessing to achieve competitive accuracy on tasks like ASR, enhancement, and speaker verification.
• The architecture integrates Mamba blocks with attention-like modules, ensuring scalability and real-time performance across diverse audio processing applications.

Audio Mamba (AuM) is a family of neural sequence models for audio and speech processing built upon the Mamba paradigm of structured state-space models (SSMs), offering linear-time complexity in sequence length and competitive or superior accuracy compared to Transformer baselines in a wide variety of tasks. By leveraging input-adaptive SSM recurrences and selective gating, AuM architectures efficiently capture long-range temporal dependencies while maintaining scalability for long audio sequences and high-resolution inputs. The AuM models have been successfully instantiated for streaming automatic speech recognition (ASR), speech enhancement, source separation, self-supervised audio representation learning, speaker verification, audio captioning, bioacoustic analysis, time-frequency spatial modeling, and trajectory estimation.

1. Structured State-Space Model Foundations

At the core of AuM is the Mamba state-space block, a discrete-time approximation of continuous SSMs with an input-dependent, selective mechanism. For a single feature or channel, a Mamba layer evolves a hidden state $h_t\in\mathbb{R}^N$ via

$$h_t = A_t\,h_{t-1} + B_t\,x_t,\qquad y_t = C_t^{\mathsf T} h_t ,$$

where $A_t$, $B_t$, $C_t$ are typically diagonal or low-rank and are modulated by the current input $x_t$. Key to the Mamba paradigm is the "selective" input-dependent parameterization—at each time step, a small feed-forward or gating network computes the effective SSM parameters as a function of input features, providing dynamic context-awareness absent in stationary SSMs. Discretization utilizes methods such as zero-order hold, bilinear transform, or exponentiation to map continuous SSMs to stable, efficient recurrences.

By replacing the quadratic-cost self-attention of Transformers, this SSM core enables true linear time and memory complexity in sequence length, making it especially suited for long-form audio, streaming, and real-time tasks. Empirically derived model architectures use stacks of such SSM blocks, interleaved with convolutional preprocessing, gating non-linearities, and, in some variants, attention or cross-modal fusion modules (Fang et al., 30 Sep 2024, Erol et al., 5 Jun 2024, Kühne et al., 1 Jul 2025, Dang et al., 24 Dec 2024, Yadav et al., 4 Jun 2024).

2. Key Architectural Features and Variations

2.1. Encoder and Block Design

The canonical Audio Mamba block comprises:

• Causal or bidirectional SSM (Mamba) layer: dynamic, input-gated state evolution.
• Pre-convolutional expansion: 1D or 2D convolution expands channel dimension, facilitating expressive spatial mixing.
• Gate or residual branch: enables skip connections and gated information flow.
• Optional addition of cross-attention for text alignment (e.g., in TTS/voice editing) or time-frequency multi-head attention for robust generalization (MambAttention).
• Output projection and normalization (e.g., LayerNorm).

For sequence-to-sequence or masked patch modeling, inputs are typically tokenized as spectrogram patches, embedded, position-augmented, and stacked through multiple Mamba blocks. In convolutional hybrid architectures (e.g., U-Mamba-Net for source separation), Mamba serves as a global temporal filter atop a U-Net local feature extractor, providing multi-resolution abstraction with global context (Dang et al., 24 Dec 2024).

2.2. Bidirectionality and Gating

Bidirectional Mamba employs forward and backward SSM scans (with shared or separate parameters), outputting either their sum or a gated fusion. This mechanism enhances context-capturing ability and brings SSM performance closer to self-attention models on non-causal tasks (Erol et al., 5 Jun 2024, Shams et al., 20 May 2024, Xiao et al., 8 Sep 2024).

2.3. Spectrum and Spatial Modeling

Specialized applications, such as source localization, deploy two-dimensional arrangements where SSMs operate both along the time and frequency axes, with time–frequency fusion and residual exchanges. The bidirectional temporal-frequency SSM in TF-Mamba achieves state-of-the-art spatial localization using parameter-sharing, dual-axis modeling (Xiao et al., 8 Sep 2024).

3. Applications Across Audio Domains

The flexibility of the AuM framework allows instantiations across a diverse set of audio processing tasks:

• Streaming ASR: AuM encoder + convolutional lookahead + Unimodal Aggregation (UMA) + early termination. Achieves 5.55% CER and ∼200–500 ms latency on AISHELL-1, outperforming causal Transformer and matching or surpassing chunked Conformer on accuracy–latency trade-offs (Fang et al., 30 Sep 2024).
• Speech and Audio Enhancement: SEMamba and MambAttention integrate causal/non-causal SSMs with (optionally shared) multi-head attention for single- and multi-channel enhancement, attaining state-of-the-art PESQ (3.69 with PCS on VoiceBank-DEMAND) and strong out-of-domain generalization on DNS 2020 and EARS-WHAM_v2 (Chao et al., 10 May 2024, Kühne et al., 1 Jul 2025).
• Speaker Verification: Integration of local-context bidirectional Mamba (LCB-Mamba) and Tri-Mamba blocks into ECAPA-TDNN yields MASV, reducing EER to 0.795% on private large-scale datasets, significantly surpassing CNN and Transformer alternatives in efficiency and accuracy (Liu et al., 14 Dec 2024).
• Source Separation: U-Mamba-Net alternates U-Net blocks and Mamba filters, selectively passing long-range structure while suppressing reverberation and noise, reaching 8.50 dB SI-SNR on Libri2Mix with just 4.4 M params and 2.5 GMACs, much more efficient than RNN/Transformer models (Dang et al., 24 Dec 2024).
• Self-Supervised Representation Learning: Audio Mamba (SSAM/SSAMBA) and BioMamba employ masked spectrogram patch modeling and/or HuBERT-style SSL tasks, providing significant gains (+20 points in [0,100]-scaled aggregate metrics over SSAST) while reducing inference memory footprint by 90–95% (Yadav et al., 4 Jun 2024, Shams et al., 20 May 2024, Tang et al., 3 Dec 2025).
• Audio Captioning: Mamba-2 audio captioners with LoRA adaptation approach or match the captioning scores of 7B+ param transformer LLMs (SPICE 16.8/11.9 on AudioCaps/Clotho) with just 2.7B parameters, using efficient SSM-LLoRA integration and simple connector compression (Lee et al., 19 Sep 2025).
• Trajectory and Bioacoustic Analysis: TAME leverages twin (temporal, spectral) SSMs with cross-attention for 3D drone localization (APE=0.55 m, Acc=98%) and BioMamba matches transformer LLMs at 40% lower VRAM in bioacoustic detection/classification (Xiao et al., 17 Dec 2024, Tang et al., 3 Dec 2025).

4. Empirical Performance, Scalability, and Efficiency

The linear complexity of Mamba-based architectures provides substantial scalability advantages over Transformer encoders. On benchmarks such as AudioSet, VGGSound, and various speech/audio command tasks, AuM variants match or exceed Transformer accuracy, often with a fraction of the parameter count and order-of-magnitude reductions in VRAM usage and inference latency:

Model	Task/Benchmark	Metric	Params	Relative Efficiency	Reference
AuM (ASR)	AISHELL-1/2	CER, Latency	6–30 M	Linear O(T)	(Fang et al., 30 Sep 2024)
U-Mamba-Net	Libri2Mix	SI-SNR	4.4 M	10× less compute	(Dang et al., 24 Dec 2024)
SEMamba	VoiceBank-DEMAND	PESQ, STOI	3–6 M	⅓ Transformer FLOPs	(Chao et al., 10 May 2024)
SSM-Audio LLMs	AudioCaps/Clotho	SPICE	2.7B	Linear in sequence	(Lee et al., 19 Sep 2025)

In self-supervised and audio tagging tasks, AuM achieves 90%+ inference speedup and 95%+ memory reduction compared to SSAST-like transformers at equivalent or higher downstream performance (Yadav et al., 4 Jun 2024, Shams et al., 20 May 2024, Tang et al., 3 Dec 2025). In real-time streaming and low-latency deployments, the convolutional lookahead and bidirectional variants ensure bounded delay with minimal computational overhead (Fang et al., 30 Sep 2024).

5. Advantages, Limitations, and Analysis

5.1. Advantages

• Linear-Complexity: O(T) training and inference unlocks efficient long-context modeling, very long sequence support, and deployment on resource-limited hardware.
• Parameter Efficiency: Smaller models maintain or exceed transformer performance (e.g., 12M-param Audio Mamba-Nano matches HT-SAT with 31M).
• Data and Resolution Scaling: AuM models scale favorably with dataset size and patch (token) resolution.
• Global and Local Context: Through bidirectionality, cross-modal fusion, and hybrid attention-integration, SSMs can unify causal (streaming/real-time) and non-causal (global context) requirements.

5.2. Limitations and Stability Considerations

• Numerical Stability: Strict parameterization (e.g., $|a(i)|<1$ for all SSM channels) is necessary for gradient stability at large D, challenging at scale without specialized normalization or parameter clamping (Xiong et al., 2 Sep 2025).
• Flexibility: SSMs are less flexible than full-attention for arbitrary cross-token dependencies, notably in raw multimodal or highly structured tasks (Lin et al., 22 May 2024, Lee et al., 19 Sep 2025).
• Generalization: While pure Mamba-based models excel in-domain, out-of-domain generalization for speech enhancement is maximized by integrating shared multi-head attention (MambAttention) and placing attention before each Mamba block (Kühne et al., 1 Jul 2025).

5.3. Hybrid and Specialized Variants

Hybrid approaches, such as MambAttention (shared T/F MHA + Mamba blocks) or U-Mamba-Net (U-Net × Mamba alternation), enhance generalization and exploit inductive priors, further closing residual gaps with Transformer-based models in some edge cases (Dang et al., 24 Dec 2024, Kühne et al., 1 Jul 2025). In temporal-spectral tasks, cross-modal attention or parallel SSMs (TAME) ensure competitive performance for multidimensional sequence inference (Xiao et al., 17 Dec 2024, Xiao et al., 8 Sep 2024).

6. Research Impact and Outlook

Audio Mamba has established the SSM paradigm as a drop-in, superior-efficiency alternative to Transformer architectures in audio understanding and generation. It is now the foundation for state-of-the-art models in streaming ASR, enhancement, speaker verification, self-supervised pretraining, and scalable audio-language modeling (Fang et al., 30 Sep 2024, Dang et al., 24 Dec 2024, Yadav et al., 4 Jun 2024, Tang et al., 3 Dec 2025, Lee et al., 19 Sep 2025).

The framework supports efficient deployment for on-device, real-time processing, extremely long input durations, and high-resolution spectrogram analysis, making it practical for edge, embedded, and low-resource applications (Yadav et al., 4 Jun 2024, Tang et al., 3 Dec 2025, Shams et al., 20 May 2024). The success of cross-modal and hybrid approaches (e.g., MAVE, TAME, MambAttention) demonstrates the viability of combining SSMs with attention for tasks requiring both precise global synchronization and fine-grained local modeling (Mohammad et al., 6 Oct 2025, Kühne et al., 1 Jul 2025, Xiao et al., 17 Dec 2024).

Ongoing research aims to address open challenges in large-scale pretraining stability, long-context generalization, and flexible inductive bias integration. Augmenting SSMs with lightweight attention, hierarchical SSM stacks, or application-specific priors represents a promising direction for further performance gains and broader applicability (Xiong et al., 2 Sep 2025, Lee et al., 19 Sep 2025, Lin et al., 22 May 2024).