MossFormer2: Hybrid Speech Separation

• The paper demonstrates that integrating Transformer self-attention with an FSMN-based recurrent branch effectively captures both global context and fine-scale speech patterns to achieve superior SI-SDR improvements.
• MossFormer2 employs a parallel encoder–separator–decoder framework with gated convolution units and dilated FSMN blocks to model long-range dependencies and localized phonetic details.
• The model sets state-of-the-art benchmarks across major datasets while enabling efficient, real-time processing and integration into multimodal and neuro-speech applications.

MossFormer2 is a hybrid neural architecture developed for enhanced time-domain monaural speech separation, integrating Transformer-based self-attention with an RNN-free recurrent module. Leveraging feedforward sequential memory networks (FSMNs) and gated convolutional units, MossFormer2 effectively models both long-range dependencies and fine-scale recurrent patterns in speech signals. Its computational design permits efficient, parallelized processing while maintaining the ability to capture sequential details typically missed by attention-only models. MossFormer2 sets a state-of-the-art benchmark across major speech separation datasets and has been incorporated in broader toolkits and multimodal architectures for various speech processing tasks.

1. Architectural Principles and Motivation

The foundational MossFormer architecture employs joint self-attention over non-overlapping segments of encoded speech to capture global context and long-range dependencies. However, self-attention in isolation inadequately learns localized recurrent patterns such as phonetic repetition and prosodic variation. MossFormer2 addresses this limitation by integrating a recurrent module based on FSMN—a design that eschews traditional recurrent connections for pure feedforward operation. This hybridization targets two essential requirements:

• Global Modeling: Transformer-based MossFormer module captures coarse-scale dependencies.
• Local Recurrency: The added recurrent module enables modeling of fine-scale, localized sequential patterns through parallel feedforward memory blocks.

This dual-module approach is specifically motivated by the necessity to capture speech features manifesting across diverse temporal scales—a capability that prior attention-centric models lacked (Zhao et al., 2023).

2. Detailed Model Structure

MossFormer2 retains the standard encoder–separator–decoder layout, operating entirely in the time domain:

• Encoder: 1-D convolution followed by a ReLU activation, mapping the input mixture waveform to a non-negative embedding sequence.
• Separator: Integrates both MossFormer (self-attention) and FSMN-based recurrent modules.
• Decoder: Transposed 1-D convolution reconstructs separated speaker waveforms.

Within the separator, the two modules operate in parallel. The recurrent branch employs a bottleneck layer (1×1 convolution + PReLU + LayerNorm) to downsample embeddings and facilitate memory-efficient processing. The core operation within the Gated Convolutional Unit (GCU) of the recurrent module is:

$$U = \text{Conv}_U(X) \ V = \text{Conv}_U(X) \ Y = \text{Dilated\_FSMN}(V) \ O = X + (U \odot Y)$$

where $X \in \mathbb{R}^{N'\times S}$ is the bottlenecked embedding, $U$ and $V$ are pointwise convolutions, $Y$ results from the dilated FSMN block, and $\odot$ is element-wise multiplication. The skip connection ($O$) is critical for gradient stability and fusing recurrent modeling with the original features.

3. FSMN-Based Recurrent Module

The "RNN-free" FSMN module in MossFormer2 comprises several specialized components:

• Bottleneck Control: Dimensionality reduction via $1\times1$ convolution with PReLU and LayerNorm regulates input complexity.
• Gated Convolutional Unit: Implements gating and integrates the dilated FSMN block for expanded receptive field.
• Dilated FSMN Block: Stacked two-dimensional convolutions with exponentially increasing dilation factors $(1, 2, …, 2^{L-1})$ and dense skip connections allow aggregation of information at multiple resolutions. The $l$-th memory layer input is formalized as:

$X_l = H_l([X_0, X_1, ..., X_{l-1}])$

where $H_l$ is a convolutional transformation and $[\,\cdot\,]$ denotes concatenation of all prior layers.

• Conv-U Block: Combines normalization, linear projection, SiLU activation, depthwise convolution, and skip connections to reinforce position-wise dependencies.

This structure enables parallel sequence modeling, eliminating the computational bottleneck of sequential RNN unrolling and facilitating large-context aggregation.

4. Performance Evaluation and Comparative Benchmarks

MossFormer2 demonstrates consistent improvements in speech separation efficacy, as measured by SI-SDR improvement (SI-SDRi), across major benchmarks:

Dataset	SI-SDRi (MossFormer)	SI-SDRi (MossFormer2)	Parameter Count	Comparative Methods
WSJ0-2mix	22.8 dB	24.1 dB	55.7 M	SepFormer, QDPN, DPTNet
WSJ0-3mix	—	Improved over SOTA	—	SepFormer, QDPN
Libri2Mix	—	Improved over SOTA	—	DPTNet, QDPN
WHAM!/WHAMR!	—	Superior Accuracy	—	MossFormer, Recent Methods

Notably, MossFormer2 surpasses models such as SepFormer and QDPN in SI-SDRi while maintaining lower parameter counts (55.7 M vs. QDPN's 200 M), indicating improved efficiency. In ablation studies, scaled-down MossFormer2 variants still outperform base MossFormer, demonstrating that the gains stem primarily from architectural advancements rather than sheer capacity. The real-time factor (RTF) remains low, confirming suitability for practical deployment (Zhao et al., 2023).

5. Integration in Speech Processing Toolkits and Multimodal Systems

Within the ClearerVoice-Studio toolkit (Zhao et al., 24 Jun 2025), MossFormer2 serves as the principal feature mapping engine for interconnected tasks:

• Speech Enhancement: Phase-sensitive mask prediction from mel-spectrograms (MossFormer2_SE_48K); joint mask-spectrum prediction via a dual-decoder setup (MossFormerGAN_SE_16K).
• Speech Separation: Time-domain mask prediction and source waveform reconstruction (MossFormer2_SS_16K).
• Speech Super-Resolution: HiFiGAN-supported restoration of high-resolution signals post spectrogram refinement (MossFormer2_SR_48K).
• Multimodal Speaker Extraction: Early fusion of visual features augments speaker extraction capabilities in AV_MossFormer2_TSE_16K.

Benchmarks from the ClearerVoice-Studio paper show MossFormer2-based models achieving PESQ scores up to 3.57 (DNS-2020), NB_PESQ of 3.88, and STOI of 98.05 in enhancement. SI-SNR improvements and high restoration fidelity are also reported in corresponding tasks, confirming state-of-the-art status and versatility.

6. MossFormer2 in Multimodal Neuro-Speech Applications

In TFGA-Net (Si et al., 14 Oct 2025), MossFormer2 is integrated as the separator for brain-controlled speaker extraction, combining auditory and EEG-derived features:

• Feature Fusion: Mixed audio and EEG features concatenated, merged by 1-D convolution, and processed by MossFormer2.
• Separator Details: Local full-attention and linearized global attention capture context; output refined by gated convolutional units (Equation 8). The RNN-free recurrent module refines temporal details (Equations 9/10).
• Benchmark Results: TFGA-Net with MossFormer2 achieves SI-SDR of 15.91 dB on Cocktail Party dataset and 16.9 dB on KUL dataset, outperforming prior methods (UBESD, BASEN, M3ANet, NeuroHeed) by significant margins (7.37 dB, 4.35 dB, and 3.02 dB, respectively).

A plausible implication is that MossFormer2's ability to capture rhythmic and prosodic patterns, in tandem with EEG-driven attention cues, enhances separation efficacy in complex scenarios.

7. Prospects and Evolution

The hybrid architecture of MossFormer2 offers a template for future models aimed at time-domain audio analysis and multimodal fusion. Potential onward trajectories include:

• Optimizing the recurrent module with alternative gating/dilation strategies;
• Extending to other sequence modeling domains (e.g., noise suppression, real-time enhancement, multi-speaker extraction);
• Incorporating advanced training objectives and loss functions for improved perceptual quality;
• Expanding to multimodal settings (e.g., combining with visual or additional neural signals).

This suggests that MossFormer2, beyond setting benchmarks in classical separation tasks, may form the backbone of next-generation models for contextual and robust auditory processing applications.