SEMamba++: Advanced Speech Restoration

• SEMamba++ is a family of advanced speech restoration architectures that extend SEMamba with selective state-space modeling and speech-specific spectral priors.
• The design integrates global, local, and periodic spectral patterns using a time-frequency dual-processing framework, optimized via GAN-based training and multiple reconstruction losses.
• Variants include multi-channel, audio-visual, and EMG-enhanced systems, each achieving notable performance gains in metrics like PESQ, UTMOS, and SI-SDR.

SEMamba++ designates a set of SEMamba-derived architectures built around Mamba-style selective state-space modeling for speech restoration, but the label is not used uniformly across the literature. In the narrowest and most formal sense, it refers to the model introduced in "SEMamba++: A General Speech Restoration Framework Leveraging Global, Local, and Periodic Spectral Patterns," a 2026 discriminative time–frequency speech restoration system that adds speech-specific inductive biases to the SEMamba lineage (Lee et al., 12 Mar 2026). In a broader and earlier sense, related papers use the label or its functional equivalent to describe enhanced SEMamba variants for single-channel perceptual refinement, multi-channel speech enhancement, audio-visual target extraction, and AC+EMG fusion, while the original SEMamba paper itself does not use the term "SEMamba++" (Chao et al., 2024, Ting et al., 2024, Chao et al., 19 Aug 2025, Feng et al., 11 Jan 2025).

1. Terminology and scope

The literature associates "SEMamba++" with several distinct extensions of SEMamba rather than a single canonical model family. This suggests that the name functions both as an exact model title and as an informal label for "enhanced" SEMamba systems.

Paper	Relation to “SEMamba++”	Primary extension
(Chao et al., 2024)	Term not used	SEMamba-advanced with CL and PCS is the closest enhanced configuration
(Ting et al., 2024)	MC-SEMamba is described as “SEMamba++”	Multi-channel front end with minimal parameter increase
(Chao et al., 19 Aug 2025)	AVSEMamba is presented as the improved/extended SEMamba implied by the label	Full-face audio-visual fusion for multi-speaker enhancement
(Feng et al., 11 Jan 2025)	Explicitly described as SEMamba++	AC+EMG multi-modal enhancement
(Lee et al., 12 Mar 2026)	Exact model title	General speech restoration with GLP, MR-parallel TFDP, and learnable mapping

The most historically prior anchor is SEMamba, introduced for single-channel speech enhancement as a Mamba-based alternative to Transformer or Conformer sequence modeling (Chao et al., 2024). The most formal current anchor is the 2026 general speech restoration framework, which explicitly names itself SEMamba++ and redefines the architecture around global, local, and periodic spectral priors (Lee et al., 12 Mar 2026). Between these endpoints, the label is used to denote several domain-specific upgrades, including multi-microphone, audio-visual, and EMG-assisted systems (Ting et al., 2024, Chao et al., 19 Aug 2025, Feng et al., 11 Jan 2025).

2. Foundations in SEMamba

SEMamba originated as a speech enhancement system that replaces attention layers with Mamba, a selective state-space model that scales linearly in sequence length (Chao et al., 2024). The paper studies Mamba in both a basic causal pipeline and an advanced non-causal magnitude-and-phase pipeline adapted from MP-SENet. The basic model operates in STFT magnitude, whereas the advanced model operates jointly on magnitude and phase and decodes both a magnitude mask and complex components. The best reported configuration, SEMamba-advanced with consistency loss and perceptual contrast stretching, achieves a PESQ of 3.69 on VoiceBank-DEMAND; the authors describe this as the closest faithful mapping to what one might informally call "SEMamba++" in that paper.

The underlying state-space formulation is the standard discrete-time SSM used in the Mamba lineage:

$$h_n = \bar{A} h_{n-1} + \bar{B} x_n,\qquad y_n = C h_n.$$

With discretization over step $\Delta$,

$$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$

For a 1-D input sequence, the SSM induces a convolution kernel

$K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$

Mamba adds input-dependent selection, conceptually modulating $\bar{B}(x_n)$, $C(x_n)$, and potentially $\Delta(x_n)$, and accelerates the resulting recurrence with a hardware-aware scan algorithm.

In SEMamba-advanced, the core processing is a time-frequency Mamba block applied $N=4$ times. Bidirectional Mamba is implemented by parallel forward and flipped passes:

$$y = \mathrm{Conv1D}\big(M_{\mathrm{uni}}(x)\oplus M_{\mathrm{uni}}(\mathrm{flip}(x))\big).$$

The paper also adds a consistency loss

$$L_{\mathrm{consistency}} = \|\hat{S} - \mathrm{STFT}(\mathrm{iSTFT}(\hat{S}))\|_1,$$

and reports that consistency loss improves PESQ from 3.52 to 3.55, while perceptual contrast stretching further increases PESQ to 3.69 and COVL to 4.37, with CBAK decreasing from 3.95 to 3.63. Relative to Transformer and Conformer baselines, the paper’s central claim is that Mamba matches or slightly exceeds their performance with fewer FLOPs or parameters in the compared settings. That result establishes the technical baseline from which later "++" variants depart.

3. The canonical 2026 SEMamba++ architecture

In its strictest usage, SEMamba++ is the 2026 general speech restoration framework that hard-codes three speech-specific inductive biases—global, local, and periodic spectral structure—into an efficient Mamba-based time–frequency dual-processing architecture (Lee et al., 12 Mar 2026). The model is discriminative and operates in the TF domain. Its defining additions over prior SEMamba variants are a Frequency GLP block, a multi-resolution parallel TFDP block with frequency-only downsampling, and a learnable frequency-aware mapping-based magnitude decoder.

The front end uses 16 kHz audio and an STFT with Hann window length 400 and hop 100. Let $\Delta$0 denote the complex spectrum. Magnitude is power-law compressed before the encoder. A dilated DenseNet encoder expands the channel dimension to $\Delta$1 and reduces frequency bins by $\Delta$2 to $\Delta$3. The bottleneck contains $\Delta$4 blocks, each built from a multi-resolution parallel TFDP module. Two decoders, one for magnitude and one for phase, reconstruct the spectrum; transposed convolutions restore frequency resolution, and iSTFT yields the waveform.

The Frequency GLP block is the paper’s principal frequency model. It processes $\Delta$5 through parallel global-periodic and local branches, then fuses them selectively. For a fixed time $\Delta$6 and channel $\Delta$7, the global-periodic branch applies a Fourier Analysis Network along the frequency axis:

$\Delta$8

with

$\Delta$9

This branch is intended to capture harmonic and periodic dependencies across frequency bins. In parallel, the local branch applies frequency convolutions,

$$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$0

to model short-range spectral structure such as formants and local smoothness. The two branches are concatenated and projected,

$$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$1

and then passed through a channel-wise FAN feed-forward stage before residual fusion. The paper’s motivation is that generic SSMs and local convolutions do not inherently encode spectral periodicity or full-band frequency structure.

The TFDP block processes three frequency resolutions in parallel: $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$2, $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$3, and $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$4. Each branch first applies frequency-only downsampling via 2D convolution with stride $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$5 and kernel $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$6. Time modeling is then performed with Time-Mamba along the temporal axis for each frequency bin independently. The selective SSM dynamics are written as

$$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$7

Frequency GLP is then applied to the temporally processed representation, and the three branches are fused by upsampling lower-frequency branches back toward the top branch. The model deliberately downsamples only in frequency, not time; the paper argues that this preserves temporal fidelity while still yielding quadratic savings in the GP branch cost because GLP complexity scales as $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$8 when the FAN widths are proportional to $$A_d = e^{\Delta A},\qquad B_d = \int_{0}^{\Delta} e^{\tau A}\, d\tau\, B.$$9.

The magnitude decoder departs from masking and instead uses a learnable per-frequency softplus mapping:

$K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$0

where $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$1 is learnable for each frequency bin. The paper’s rationale is that masking is poorly suited to bandwidth extension, since upper bands may contain near-zero energy and therefore require generation rather than attenuation or reweighting. A dedicated phase decoder runs in parallel, and the final waveform is reconstructed from the mapped magnitude and predicted phase.

4. Objective functions, training setup, and empirical results

SEMamba++ uses vocoder-style LSGAN training together with multiple reconstruction losses (Lee et al., 12 Mar 2026). For generator $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$2 and discriminators $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$3,

$K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$4

$K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$5

The generator also optimizes magnitude $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$6, anti-wrapping phase loss, consistency loss, complex RI loss, multi-scale mel loss, and discriminator feature matching:

$K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$7

Hyperparameters follow prior SEMamba work for $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$8, $K_t = C \bar{A}^{t-1}\bar{B},\qquad y = x * K.$9, $\bar{B}(x_n)$0, and $\bar{B}(x_n)$1, with $\bar{B}(x_n)$2 and $\bar{B}(x_n)$3.

Training uses VCTK 0.92 excluding p280 and p315, with WHAM! and DNS 2020 noises and Arni and DNS5 RIRs. Degradations are simulated by random combinations of noise at SNR $\bar{B}(x_n)$4 to $\bar{B}(x_n)$5 dB, reverberation, bandwidth limitation with cutoffs 2–7 kHz, and clipping. Optimization uses 1.5 s segments, AdamW with learning rate $\bar{B}(x_n)$6, $\bar{B}(x_n)$7, $\bar{B}(x_n)$8, batch size 8, 100 epochs, and exponential decay $\bar{B}(x_n)$9.

The reported quantitative behavior emphasizes perceptual quality and out-of-domain robustness. On VCTK-GSR test, SEMamba++ achieves UTMOS 3.55 and OVRL 3.14 with PESQ 1.77, while SEMamba attains UTMOS 2.34, OVRL 2.88, and PESQ 2.13. On URGENT 2025 validation, SEMamba++ reports UTMOS 2.82 and OVRL 3.20; on the URGENT 2025 blind test, it reports UTMOS 2.61 and OVRL 3.13. On DNS 2020 test, the model achieves DNSMOS SIG 3.487, BAK 4.02, and OVRL 3.206. On CCF-AATC 2025, with 2.7M parameters and RTF 0.021, it reports DNSMOS SIG 3.45, BAK 4.01, and OVRL 3.18. The paper explicitly notes a tradeoff: switching from MetricGAN to vocoder-style GAN objectives improves SCOREQ, UTMOS, and OVRL, but incurs a cost to PESQ because MetricGAN directly optimizes PESQ.

The ablation study attributes gains to each of the three main components. For frequency mixing, GLP is compared against Mamba, Transformer, Conformer, and SpatialNet, and is reported as giving the best and most consistent performance, especially out of domain. Removing the GP branch or replacing FAN with a linear layer degrades results. For scale handling, MR-parallel outperforms MR-sequential and a single-resolution wider model. The paper further reports that gradient attribution shows bottom, middle, and top branches specialize differently—respectively to noise patterns, speech content, and harmonics—and that IoU of influential gradients across resolutions is lower for MR-parallel than MR-sequential. Frequency-only downsampling outperforms both no downsampling and time-downsampling. The learnable mapping decoder improves OOD robustness relative to learnable masking, particularly in restoration settings that require bandwidth generation.

5. Multi-channel, audio-visual, and EMG-conditioned interpretations

A major source of ambiguity around the term "SEMamba++" is that several papers use it, or map to it, for modality-specific extensions rather than for the 2026 general speech restoration framework.

In multi-channel speech enhancement, MC-SEMamba is described as a minimal, effective multi-channel extension of SEMamba that can be viewed as "SEMamba++" because it keeps SEMamba’s generator architecture and training framework while upgrading the front end to ingest $C(x_n)$0-channel recordings and learn spatial information jointly with time–frequency dynamics (Ting et al., 2024). The mechanism is pure channel stacking at the input: each microphone contributes two channels, compressed magnitude and phase, and adding a microphone increases the first 2D convolution’s input channels by 2. With default output channels $C(x_n)$1 and kernel size $C(x_n)$2, this adds only $C(x_n)$3 parameters per microphone. On CHiME3, MC-SEMamba-6 reports PESQ(N-1)=3.48, PESQ(N-2)=3.50, PESQ(W)=3.07, STOI=98.5%, SI-SDR=21.3 dB, and SDR=21.9 dB with about 2.3M generator parameters. The paper further reports that increasing the number of microphones from 1 to 6 improves PESQ(W) by 0.64, STOI by 3.2 points, SI-SDR by 5.2 dB, and SDR by 5.3 dB.

In audio-visual speech enhancement, AVSEMamba is presented as the improved or extended SEMamba that a label such as "SEMamba++" implies (Chao et al., 19 Aug 2025). The model adds a full-face visual stream encoded by a pretrained and frozen 3D ResNet-18 operating at 25 fps, followed by Conv1D temporal alignment to the audio frame rate. Audio magnitude features, audio phase features, and visual embeddings are concatenated and passed through a shared feature encoder before stacked Mamba-based temporal-frequency blocks. Separate decoders reconstruct enhanced magnitude and phase. On the AVSEC-4 development set, the model reports MBSTOI 0.8037, PESQ 2.97, and UTMOS 2.32, compared with the noisy input’s MBSTOI 0.4161, PESQ 1.30, and UTMOS 1.32. On the blind test set, it reports PESQ 2.31, STOI 0.77, and UTMOS 2.24, compared with the noisy input’s PESQ 1.31, STOI 0.55, and UTMOS 1.37. The paper states that AVSEMamba ranked first on the monaural leaderboard.

In the AC+EMG setting, "SEMamba++" is used explicitly for a two-stage multi-modal extension with limited-channel electromyography (Feng et al., 11 Jan 2025). Stage 1 converts 8-channel EMG sampled at 1000 Hz to an acoustical speech signal via speech-unit prediction and a unit-to-Mel plus HiFi-GAN pipeline. Stage 2 jointly enhances the noisy AC speech and the EMG-predicted speech using a modified SEMamba with per-branch DenseEncoders, a fully connected cross module, a TF-Mamba core with bidirectional Mamba along both time and frequency axes, and dual decoders for magnitude mask and complex reconstruction. On matched low SNR at $C(x_n)$4 dB, the proposed SEMamba++ with TF-Mamba$C(x_n)$5 reports PESQ 2.876 versus 2.641 for AC-only SEMamba, a gain of 0.235, with STOI 0.925 versus 0.898. On mismatched low SNR at $C(x_n)$6 dB, it reports PESQ 2.081 versus 1.554, a gain of 0.527, with STOI 0.862 versus 0.756. The paper argues that the EMG-derived speech prior stabilizes enhancement in extremely low-SNR and non-stationary noise conditions while using only 8 EMG channels rather than the 35-channel setups common in earlier EMG-based work.

Taken together, these variants show that "SEMamba++" has come to denote a recurring design pattern: retain the Mamba-based SEMamba backbone, then inject a new inductive bias or modality—spatial channels, full-face video, EMG-derived speech, or explicit global–local–periodic frequency structure.

6. Limitations, misconceptions, and open directions

A common misconception is that SEMamba++ refers to a single universally agreed architecture. The literature instead shows several incompatible but related uses of the name or its implied meaning. The exact titled model is the 2026 general speech restoration framework (Lee et al., 12 Mar 2026), but earlier and parallel works use the label informally for perceptually enhanced SEMamba, MC-SEMamba, AVSEMamba, and AC+EMG fusion (Chao et al., 2024, Ting et al., 2024, Chao et al., 19 Aug 2025, Feng et al., 11 Jan 2025). Any technical discussion therefore has to specify which formulation is intended.

The 2026 framework’s own limitations are explicit. The GP branch applies linear mixing directly on frequency bins, so deployment in sampling-rate-agnostic settings may require weight sharing tied to physical frequency or learned continuous kernels. Extreme bandwidth truncation remains difficult for a purely discriminative mapping decoder; the paper notes that very large generative models may hallucinate plausible content more aggressively, though at higher cost and with possible content drift. Although phase is modeled with a dedicated decoder and anti-wrapping loss, the authors identify joint end-to-end waveform modeling and explicit pitch conditioning as plausible next steps. Loss design is also open, because the vocoder-style GAN improves perceptual metrics but lowers PESQ relative to MetricGAN-oriented training.

The modality-specific extensions have parallel limitations. MC-SEMamba learns spatial cues implicitly from stacked multi-channel inputs and shared convolutions, without explicit IPD or ILD features and without beamforming; the paper suggests that explicit spatial feature heads or MVDR/GEV-style integration could push SI-SDR and SDR higher. AVSEMamba requires synchronized, quality full-face video and may degrade under occlusion, poor lighting, motion blur, missing frames, or large audio-visual asynchrony. The EMG-based SEMamba++ depends on electrode placement quality, synchronization, and a single 8-channel EMG corpus, and its fusion module remains a stack of fully connected layers rather than a more adaptive cross-modal mechanism.

These constraints point to a coherent research trajectory. One direction is causal or streaming Mamba processing, which is already structurally compatible with selective scanning but not yet standard in the strongest non-causal SEMamba-derived systems. Another is richer cross-channel or cross-modal fusion, whether for microphone arrays, visual guidance, or biosignals. A third is more explicit speech inductive bias, exemplified by the 2026 model’s claim that global, local, and periodic spectral patterns should be encoded directly rather than left entirely to generic sequence modeling. In that sense, the evolution of SEMamba++ is less a single architecture than a research program: progressively augmenting SEMamba with task-specific structure while trying to preserve the linear-time efficiency that motivated Mamba in speech restoration in the first place.