---
title: 'XLSR-Mamba: Efficient Speech Modeling'
url: https://www.emergentmind.com/topics/xlsr-mamba-model
type: topic
---

# XLSR-Mamba: Efficient Speech Modeling

XLSR-Mamba model refers to a family of speech sequence models that integrate a multilingual self-supervised encoder (XLSR, typically wav2vec 2.0 XLS-R) with state-space and Mamba-based backbones to efficiently capture long-range temporal dependencies for tasks such as spoofing attack and audio deepfake detection. The main instantiations include XLSR-Mamba—featuring a Dual-Column Bidirectional Mamba (BiMamba) state-space network—and the more recent XLSR-MamBo, a modular framework that hybridizes Mamba-style SSMs and attention mechanisms. Both serve as high-performance, computationally efficient alternatives to traditional transformer architectures in challenging speech-processing scenarios [2411.10027][2601.02944][2406.16808].

## 1. Architectural Foundation

The canonical XLSR-Mamba pipeline processes input speech in the following sequence:

1. **Input:** Raw waveform audio, typically sampled at 16 kHz.
2. **XLSR (wav2vec 2.0 XLS-R) Feature Extraction:** A convolutional front end maps the input to a $\tfrac{N}{320} \times C$ feature map ($C=1024$), followed by 24 transformer encoder layers, yielding contextualized embeddings of size $T \times C$.
3. **Linear Projection:** A fully connected projection reduces the embedding size from $\mathbb{R}^C$ to $\mathbb{R}^D$ (typical $D=144$), resulting in the sequence $X = [x_1, \dots, x_T]$, $x_t \in \mathbb{R}^D$.
4. **BiMamba Backbone:** The projected sequence is processed through $L=12$ stacked Bidirectional Mamba (BiMamba) layers, each implementing a dual-column structure to capture both forward and backward temporal dependencies.
5. **Pooling & Classification:** Mean pooling across temporal dimension, then a linear classifier predicts class labels (bonafide or spoof).
6. **Data Augmentation:** RawBoost is applied to input waveforms during training to enhance generalization [2411.10027].

Recent variants such as XLSR-MamBo generalize this backbone by interleaving variants of SSMs (Mamba, Mamba2, Gated DeltaNet, and Hydra) with self-attention layers in multi-layered topologies [2601.02944].

## 2. Mathematical Formulation of Mamba and BiMamba

At its core, a Mamba block implements a time-varying, input-dependent linear state-space model (SSM) over the feature sequence:

\[
h_t = A_t h_{t-1} + B_t x_t,\quad y_t = C_t h_t + D_t x_t
\]

where $A_t$, $B_t$, $C_t$, $D_t$ are dynamically generated (input-conditioned) matrices for each $t$. The output for the full sequence can be computed as a time-varying convolution:

\[
y = x * \mathcal{K},\quad \mathcal{K} = [C_1B_1,\;C_2A_2B_2,\;\dots,\;C_TA_T\cdots A_2B_1]
\]

Bidirectionality is achieved in XLSR-Mamba using a dual-column structure:

- The forward column processes $\{x_t\}$ sequentially.
- The backward column processes time-reversed $\{x_{T-t+1}\}$, then its outputs are reversed to match original time indices.
- Outputs from both columns are concatenated at each time step ($z_t = [h^f_t; h^b_t]$).
- A projection brings $z_t$ back to $\mathbb{R}^D$ with residual connection [2411.10027].

Hydra, as introduced in XLSR-MamBo, generalizes this to native bidirectional mixing within a single SSM block. The SSM is parameterized as a quasiseparable matrix yielding bidirectional context in $O(Td)$ time, circumventing the need for separate forward/backward passes [2601.02944].

## 3. Integration with Self-Supervised Learning and Multilingual Pre-Training

XLSR-Mamba leverages pre-trained wav2vec 2.0 XLS-R models (0.3B–0.3B parameters, multilingual) as its front end. During fine-tuning for spoofing detection, all network parameters (convolutional feature encoder, transformer layers, projection, Mamba or hybrid backbone, classifier) are updated end-to-end.

A general framework for extending Mamba with cross-lingual self-supervised objectives comprises:

- **Masked Frame Prediction:** Mask a proportion of acoustic frames and predict original features using the output of Mamba blocks, with L1 or L2 loss.
- **Contrastive Loss:** A projection head after specific Mamba layers predicts masked positions, with negatives sampled from other frames/tokens; minimized by InfoNCE:
  \[
  \mathcal{L}_\text{NCE} = -\sum_t \log\frac{\exp(h_t^\top\,c_t/\tau)}{\sum_{t'}\exp(h_t^\top\,c_{t'}/\tau)}
  \]
- **Multilingual Parametrization:** Prepend each utterance with a language embedding, which conditions the SSM selection layers ($W_B, W_C, W_\Delta$). Relative positional bias can be added when necessary [2406.16808].

XLSR-Mamba and its variants are typically pre-trained with large multilingual speech corpora (e.g., MLS, CommonVoice), and fine-tuned for downstream tasks without the self-supervised heads.

## 4. Training Regimes, Objective Functions, and Implementation

Key components of XLSR-Mamba empirical training setups include:

- **Objective:** Weighted binary cross-entropy for binary spoofing classification. For instance,
  \[
  \mathcal{L}_{\mathrm{CE}} = -\frac{1}{N}\sum_{i=1}^N w_{y_i}\,\bigl[y_i\log p_i + (1-y_i)\log(1-p_i)\bigr]
  \]
  with $w_{0},w_{1}$ balancing class priors [2411.10027].
- **Optimization:** Adam (learning rate $1\times10^{-6}$, weight decay $10^{-4}$), or AdamW for XLSR-MamBo [2601.02944]. Early stopping is used (patience=7).
- **Augmentation:** RawBoost includes linear/nonlinear convolutive, impulsive additive, and stationary additive noise.
- **Hyperparameters:** Typical settings are $L=12$ BiMamba layers ($d=144$), batch size 20 utterances, segment length 4 s (64600 samples). XLSR-MamBo reduces D ($\mathbb{R}^{1024} \to \mathbb{R}^{128}$), with SSM state dim 64 and varying backbone depth $L \in \{5,7\}$.
- **Evaluation metrics:** Equal Error Rate (EER%) and minimum t-DCF are standard.

## 5. Comparative Empirical Performance

Empirical evaluation demonstrates XLSR-Mamba’s superiority in both accuracy and efficiency relative to transformer-based approaches:

| System                          | Params | 21LA EER% | 21DF EER% | ITW EER% | min t-DCF      |
|----------------------------------|--------|-----------|-----------|----------|---------------|
| XLSR-Mamba [2411.10027]         | 319 M  | 0.93      | 1.88      | 6.71     | 0.208         |
| XLSR-Conformer+TCM              | 320 M  | 1.03      | 2.06      | 7.79     | —             |
| Fake-Mamba                      | 320 M  | 0.97      | 1.74      | 5.85     | —             |
| MamBo-3-Hydra-N3 [2601.02944]   | 319 M  | 0.81      | 1.70      | 4.97     | —             |

On In-the-Wild data, XLSR-Mamba outperforms XLSR+SLS and XLSR-Conformer+TCM baselines. XLSR-MamBo further reduces EER, especially in more diverse or unseen conditions (e.g., DFADD dataset) owing to stronger bidirectional context modeling via Hydra. Runtime analysis shows 30–40% faster inference (RTF ≈ 0.15 for XLSR-Mamba vs 0.25 for XLSR-Conformer) [2411.10027][2601.02944].

## 6. Scaling Properties, Variants, and Generalization

XLSR-MamBo systematically explores SSM-attention hybridization and scaling effects:

- **Topologies:** Interleaving SSM and attention layers (MamBo-3) with deeper stacking (N steps per Mamba block) improves both mean EER and variance across checkpoints.
- **SSM Variants:** Hydra’s native bidirectional mixing outperforms dual-branch directional Mamba in both accuracy and computational efficiency.
- **Depth and Stacking:** Higher backbone depth ($L$) and greater SSM recurrence ($N$) compress the spread in top-n checkpoint EERs, yielding more robust models under distribution shift.
- **Ablation:** MamBo-1/2 (denser intra-block hybridization) and MamBo-4 (maximum alternation) show smaller or less stable improvements. Deeper models offset checkpoint instability.

In challenging scenarios (unseen synthetic attacks, such as diffusion or flow-matching synthesis), all MamBo variants generalize well, but bidirectional SSMs (notably Hydra) at higher depth achieve near-zero EERs on most subsets.

## 7. Broader Applications and Context

The Mamba architecture, when evaluated across tasks such as ASR, text-to-speech, spoken language understanding, and summarization, matches or outperforms state-of-the-art transformers. Its main strengths include linear-time complexity ($O(NL)$ per block), the ability to process very long contexts, and scalability to long-form speech (e.g., 600 s video summarization out-of-memory for transformers) [2406.16808]. In speech applications where global context and efficient handling of long sequences are critical, the XLSR-Mamba line constitutes a class of highly efficient, robust, and generalizable neural architectures.

*This suggests that XLSR-Mamba and its hybrids are positioned as competitive, computationally efficient backbones for multilingual and robust speech sequence modeling, especially in adversarial or out-of-distribution settings, and scale favorably with backbone depth and hybridization.*

Source: https://www.emergentmind.com/topics/xlsr-mamba-model