Mamba-SEUNet: Efficient Speech Enhancement

• The paper introduces Mamba-SEUNet, combining a U-Net hierarchy with linear-time state-space Mamba blocks for efficient, bidirectional long-range speech enhancement.
• It employs TS-Mamba blocks to fuse multi-scale features and process parallel magnitude-phase enhancements in the STFT domain using skip connections.
• Empirical results on the VCTK+DEMAND benchmark show competitive WB-PESQ scores with significantly lower computational cost compared to Transformer and Conformer models.

Searching arXiv for the primary paper and closely related Mamba-U-Net work. Mamba-SEUNet is a time-frequency monaural speech enhancement model that combines a U-Net encoder–decoder hierarchy with Mamba, a linear-time selective state-space model, in order to improve long-range dependency modeling while keeping computation low. The architecture operates in the STFT domain, uses bidirectional Mamba blocks to model forward and backward dependencies of speech signals at different resolutions, and employs skip connections for multi-scale feature fusion. On the VCTK+DEMAND benchmark, the reported best standard configuration attains a WB-PESQ score of 3.59, and the reported score rises to 3.73 when combined with Perceptual Contrast Stretching (PCS) (Wang et al., 2024).

1. Position within monaural speech enhancement

Mamba-SEUNet is situated in a line of speech enhancement systems in which Transformer and Conformer variants have become predominant, but are constrained by the quadratic complexity of self-attention with respect to sequence length. The model is introduced specifically to address three stated requirements: capturing global long-range temporal and frequency dependencies, preserving fine local spectral details, and supporting efficient inference on constrained hardware (Wang et al., 2024).

The design premise is to combine two complementary mechanisms. U-Net provides hierarchical coarse-to-fine feature extraction and reconstruction with skip-connected multi-resolution fusion, while Mamba provides linear-time sequence modeling through selective state-space dynamics. This yields a model that is not merely a flat sequence processor applied to speech spectra, but a multi-resolution enhancement network in which sequence modeling is embedded inside an encoder–decoder structure.

A common misunderstanding is to treat Mamba-SEUNet as a generic attention replacement. The reported formulation is narrower and more specific: Mamba is inserted into a U-Net-like speech enhancement backbone through TS-Mamba blocks, and the resulting system performs parallel magnitude and phase enhancement in the STFT domain rather than end-to-end waveform enhancement (Wang et al., 2024).

2. State-space formulation and computational rationale

Mamba is presented through a continuous-time state-space model mapping input $x(t)$ through hidden state $h(t)$ to output $y(t)$:

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$

For discrete speech signals, zero-order hold discretization with time scale $\Delta$ is used:

$$\overline{A}=e^{(\Delta A)}, \qquad \overline{B}=(\Delta A)^{-1}(e^{(\Delta A)}-I)\cdot(\Delta B)$$

which yields the recurrent discrete form

$$h(t)=\overline{A}h(t-1)+\overline{B}x(t), \qquad y(t)=\overline{C}h(t)$$

and, when unrolled, a global convolution interpretation

$$\overline{K}=(C\overline{B},\, C\overline{A}\overline{B},\, \ldots,\, C\overline{A}^{L-1}\overline{B})$$

$y=x * \overline{K}$

where $L$ is sequence length (Wang et al., 2024).

The computational argument follows directly from this formulation. Mamba’s hardware-aware scan is described as keeping most operations in fast SRAM and reducing HBM access, thereby avoiding the $h(t)$0 behavior of self-attention and retaining linear complexity in sequence length. In the context of speech enhancement, this is used to justify replacing or augmenting Transformer-style sequence modeling with state-space modeling that is more deployment-friendly.

This suggests that the architectural contribution is not only representational but also systems-oriented. The model is explicitly framed as a response to the gap between strong enhancement quality and practical deployment cost, especially when long-context modeling is required.

3. STFT-domain pipeline and encoder–decoder organization

The network processes noisy speech $h(t)$1 in the STFT domain. The input waveform is transformed into magnitude $h(t)$2 and phase $h(t)$3. A feature encoder maps these to intermediate features

$h(t)$4

which are then processed by a stack of TS-Mamba blocks inside a U-Net-like encoder–decoder with skip connections. The decoder predicts enhanced magnitude and phase, and ISTFT reconstructs the enhanced speech waveform (Wang et al., 2024).

The encoder is inspired by MP-SENet and consists of two convolutional layers, a Dilated DenseNet, and frequency downsampling. The first convolution raises the channel dimension to $h(t)$5, and the second convolution halves the frequency resolution to reduce computation. The Dilated DenseNet has depth $h(t)$6 and dilation factor $h(t)$7. This encoder is described as capturing foundational spectral structure while preserving efficiency.

The decoders for magnitude and phase mirror the encoder, each using a Dilated DenseNet, a 2D transposed convolution, and a $h(t)$8 convolution. Their output nonlinearities differ: the magnitude decoder uses learnable sigmoid (L-Sigmoid), whereas the phase decoder uses Arctan2. This implements a parallel magnitude-phase enhancement strategy in the spirit of MP-SENet (Wang et al., 2024).

Skip connections link encoder and decoder stages at matching resolutions. Their role is defined in standard U-Net terms but with task-specific emphasis: preserving fine-grained spectral details, recovering information lost through downsampling, and enabling the decoder to combine high-level semantic context with local detail. In this sense, Mamba-SEUNet is a multi-scale feature fusion model as much as it is a sequence model.

4. TS-Mamba blocks and bidirectional context modeling

The central enhancement module is the TS-Mamba block, where “TS” denotes time-stream and frequency-stream processing. Mamba blocks are applied sequentially along time and frequency so that dependencies are learned along both axes of the time-frequency representation. These blocks appear throughout the U-Net hierarchy at multiple resolutions, with downsampling on the encoder side and upsampling on the decoder side (Wang et al., 2024).

At each resolution, TS-Mamba is intended to learn coarse global structure at low resolution and fine local patterns at higher resolution. This multi-resolution arrangement is presented as a primary reason the architecture outperforms simpler flat sequence models. A plausible implication is that the U-Net scaffold supplies the scale hierarchy within which state-space modeling becomes more effective for enhancement than if it were applied at only one resolution.

To model both past and future context, the network uses bidirectional Mamba. For input $h(t)$9,

$y(t)$0

$y(t)$1

$y(t)$2

where $y(t)$3 denotes forward Mamba, $y(t)$4 denotes backward Mamba, $y(t)$5 reverses the sequence, $y(t)$6 is root mean square normalization, and $y(t)$7 concatenates along the channel dimension (Wang et al., 2024).

Each directional Mamba block uses a linear projection from $y(t)$8 to $y(t)$9, followed by a 1-D convolution with kernel size $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$0, SiLU activation, an SSM branch,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$1

and a parallel gated branch without SSM,

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$2

The branches are then concatenated and projected:

$h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$3

The paper states that this dual-branch design helps compensate for information loss associated with sequential SSM constraints. In architectural terms, the bidirectional wrapper addresses contextual symmetry, while the dual-branch directional block addresses expressivity within each scan direction.

5. Model variants, optimization, and benchmark protocol

Four model variants are defined. XS uses $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$4 and $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$5 with 0.99M parameters; S uses $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$6 and $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$7 with 1.88M parameters; M uses $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$8 and $h'(t)=Ah(t)+Bx(t), \qquad y(t)=Ch(t)$9 with 3.78M parameters; and L uses $\Delta$0 and $\Delta$1 with 6.28M parameters. Additional settings are $\Delta$2 and $\Delta$3 (Wang et al., 2024).

Training uses segments of 30600 samples. The STFT configuration is FFT length $\Delta$4, window length $\Delta$5, and hop size $\Delta$6. Optimization uses AdamW with initial learning rate 0.0005, learning-rate decay by $\Delta$7 per epoch, and 200 training epochs (Wang et al., 2024).

Evaluation is conducted on the VCTK+DEMAND benchmark. Clean speech is drawn from VoiceBank/VCTK and noise from DEMAND. The dataset contains 12,396 utterances from 30 speakers, with 28 speakers for training and 2 for testing. Training uses 10 noise types at SNRs from 0 to 15 dB, while testing uses 5 noise types at SNRs from 2.5 to 17.5 dB. All audio is downsampled to 16 kHz (Wang et al., 2024).

The reported evaluation metrics are WB-PESQ, STOI, CSIG, CBAK, COVL, and FLOPs. FLOPs are computed on a 2-second, 16 kHz sample on GPU. This benchmark choice is significant because the paper states that it is the same benchmark widely used for monaural enhancement, which makes comparisons directly meaningful.

6. Empirical results, ablations, and adjacent Mamba-U-Net designs

The reported baseline comparison includes MetricGAN+, TSTNN, CMGAN, DPCFCS-Net, MP-SENet, MUSE, S4ND U-Net, and SEMamba. The best standard result is obtained by Mamba-SEUNet (L), which reports WB-PESQ = 3.59, STOI = 0.96, CSIG = 4.80, CBAK = 4.02, COVL = 4.32, 18.17G FLOPs, and 6.28M parameters (Wang et al., 2024).

The efficiency argument is sharpened by the smaller S configuration. Mamba-SEUNet (S) reports 1.88M parameters, 4.62G FLOPs, and WB-PESQ 3.54, and this is contrasted with CMGAN at 63.15G FLOPs, MP-SENet at 74.29G FLOPs, DPCFCS-Net at 130.65G FLOPs, a Transformer baseline at 51.69G FLOPs, and a Conformer baseline at 34.07G FLOPs. The explicit claim is that Mamba-SEUNet delivers strong enhancement with dramatically reduced compute relative to many Transformer and Conformer baselines.

With Perceptual Contrast Stretching, performance improves further. SEMamba + PCS reports WB-PESQ 3.69, STOI 0.96, CSIG 4.79, CBAK 3.63, and COVL 4.37. Mamba-SEUNet (S) + PCS reports WB-PESQ 3.70, STOI 0.96, CSIG 4.79, CBAK 3.64, and COVL 4.37. Mamba-SEUNet (L) + PCS reports the paper’s highlighted best PESQ result, WB-PESQ 3.73, with STOI 0.96, CSIG 4.82, CBAK 3.67, and COVL 4.40 (Wang et al., 2024).

The ablation on the number of TS-Mamba blocks $\Delta$8 in Mamba-SEUNet (M) reports a monotonic increase in WB-PESQ from 3.46 at $\Delta$9 to 3.51 at $$\overline{A}=e^{(\Delta A)}, \qquad \overline{B}=(\Delta A)^{-1}(e^{(\Delta A)}-I)\cdot(\Delta B)$$0, 3.56 at $$\overline{A}=e^{(\Delta A)}, \qquad \overline{B}=(\Delta A)^{-1}(e^{(\Delta A)}-I)\cdot(\Delta B)$$1, and 3.57 at $$\overline{A}=e^{(\Delta A)}, \qquad \overline{B}=(\Delta A)^{-1}(e^{(\Delta A)}-I)\cdot(\Delta B)$$2, while parameters increase from 1.11M to 2.00M, 2.89M, and 3.78M, respectively. The stated conclusion is that more TS-Mamba blocks consistently improve quality, with gains beginning to plateau by $$\overline{A}=e^{(\Delta A)}, \qquad \overline{B}=(\Delta A)^{-1}(e^{(\Delta A)}-I)\cdot(\Delta B)$$3.

A controlled comparison using the same U-Net architecture reports that the Mamba version achieves WB-PESQ 3.57 with 3.78M parameters and 10.28G FLOPs, compared with Conformer at WB-PESQ 3.45, 2.94M parameters, and 34.07G FLOPs, and Transformer at WB-PESQ 3.52, 4.55M parameters, and 51.69G FLOPs. This is used to support the paper’s central claim that Mamba is more effective and much cheaper computationally in this setting (Wang et al., 2024).

Within the broader family of Mamba-enhanced U-shaped architectures, MUCM-Net provides a related but distinct example in medical image segmentation rather than speech enhancement. It is described as a lightweight Mamba-enhanced U-Net-style segmentation network that integrates Mamba State-Space Models into a UCM-Net architecture to improve long-range feature modeling while keeping computation very low. The comparison is conceptually useful because it underscores that Mamba-U-Net hybrids are not specific to speech; however, MUCM-Net is not Mamba-SEUNet, and its hybrid block structure, segmentation objective, and dermoscopic application domain are different (Yuan et al., 2024).

Taken together, the reported evidence supports a concise characterization of Mamba-SEUNet as a bidirectional, multi-resolution Mamba-U-Net for monaural speech enhancement in the STFT domain. Its reported performance is tied to three interacting components: the U-Net hierarchy for coarse and fine spectral structures, bidirectional Mamba for preceding and succeeding context, and TS-Mamba blocks for multi-scale processing. This suggests that the model’s main significance lies in demonstrating that state-space sequence modeling can be integrated into a speech enhancement U-Net in a way that preserves competitive quality while substantially reducing computational cost relative to attention-centric alternatives.