MIMO-TasNet: Multi-Channel Speech Separation

• MIMO-TasNet is a multi-channel extension of TasNet that directly reconstructs spatially-consistent waveforms for separated speech sources.
• It integrates time-domain convolutional encoders, causal TCNs, and iterative beamforming to enhance spatial fidelity with preserved ILD and ITD cues.
• Experimental results show significant SNR improvements and reduced spatial errors, enabling low-latency performance for binaural and microphone array applications.

MIMO-TasNet denotes a class of multi-input-multi-output (MIMO) extensions of TasNet, a time-domain neural architecture for speech separation, specifically designed to process multi-channel mixture signals and directly produce multi-channel output waveforms for each separated source. In contrast to single-channel TasNet or earlier multi-channel variants producing single-channel outputs, MIMO-TasNet architectures maintain and reconstruct spatial cues such as interaural level difference (ILD) and interaural time difference (ITD), enabling spatially consistent source separation suitable for binaural, microphone array, and hearing-assistive applications. Representative instantiations include real-time systems for binaural spatial-cue preservation (Han et al., 2020) and frameworks integrating iterative beamforming with data-driven separation (Chen et al., 2021).

1. MIMO-TasNet Model Architectures

MIMO-TasNet architectures extend time-domain audio separation networks (TasNet/Conv-TasNet) by supporting multiple input and output channels. Two principal instantiations are canonical: (a) real-time binaural MIMO-TasNet for spatial-cue preservation (Han et al., 2020), and (b) the Beam-Guided TasNet framework integrating MC-Conv-TasNet with iterative MVDR beamforming (Chen et al., 2021).

1.1 Binaural MIMO-TasNet

Given left and right mixture waveforms $x^{(l)}(t), x^{(r)}(t) \in \mathbb{R}^T$, parallel 1-D convolutional encoders (64 filters, 2 ms/16-sample kernels at 8 kHz) yield encoded representations: $$E^{p}_{n,\tau} = \sum_{\ell=0}^{L-1} w_{n,\ell}\;x^{(p)}[\tau R + \ell]$$ for $p\in\{l,r\}$, $n=1,\ldots,N$. The encoder outputs ($E^l$, $E^r$) are concatenated and submitted to a causal Temporal Convolutional Network (TCN)—comprising 4 stacks of 8 dilated conv blocks (with layer normalization, PReLU, causal depthwise and pointwise convolutions, and residual/skip connections).

The TCN estimates $2C$ non-negative masks $M_i^p(n, \tau)$ ($i=1,\ldots,C$, $p\in\{l,r\}$), one per source and per input channel. For each source $i$, the mask-and-sum operator combines the encoded streams: $$R_{i,n,\tau} = M_i^{l}(n,\tau)\,E^{l}_{n,\tau} + M_i^{r}(n,\tau)\,E^{r}_{n,\tau}$$ Each separated source $\hat s_i^{(p)}[t]$ in each channel $p$ is then reconstructed by the corresponding linear decoder (transpose convolution): $$\hat{s}_i^{(p)}[t]=\sum_{n=1}^N \sum_\tau d_{n,t-\tau R}\,S_{i,n,\tau}$$

1.2 Beam-Guided TasNet Framework

The Beam-Guided TasNet (Chen et al., 2021) generalizes MIMO-TasNet to $C$-channel inputs and outputs, connecting a multi-channel Conv-TasNet (MC-Conv-TasNet) with MVDR beamforming in an iterative, cyclic manner. The mixture $\{y_c(t)\}_{c=1}^C$ is processed as:

1. Stage 1: MC-Conv-TasNet estimates source images, followed by MVDR beamforming for each separated source.
2. Stage 2 (iterative): The original mixture, along with previous MVDR outputs, is input to MC-Conv-TasNet, triggering refined mask estimation and beamforming; this cycle repeats for $N$ iterations with shared network parameters across iterations.

Key notations: $$R = \text{ParEnc}(\{y_c(t)\}) \in \mathbb{R}^{N\times L}$$

$\hat{z}_{s,c}(t) = \text{ParDec}(M_{s,c} \odot R)$

Spatial covariance matrices estimated from $\hat{z}_{s,c}(t)$ inform the closed-form MVDR filter design, directly integrating neural and beamforming pipelines.

2. Spatial Cue Preservation Mechanisms

MIMO-TasNet maintains binaural/spatial cues critical for downstream spatial audio applications. By producing separate output signals for each input channel and source, the architecture preserves both ILD and ITD.

In binaural MIMO-TasNet, channel-specific masks $M_i^l$, $M_i^r$ ensure that each channel's contribution is learned directly, preserving the true-source ILD: $$\text{ILD}_i = 10\log_{10} \frac{\|\hat{s}_i^{l}\|_2^2}{\|\hat{s}_i^{r}\|_2^2}$$ No explicit IPD features are required, as the end-to-end time-domain design (encoder/decoder) faithfully reconstructs HRIR-induced time disparities.

Spatial-cue consistency is enforced implicitly: the network is trained to reconstruct binaural reference signals filtered by individual HRIRs, obviating the need for explicit spatial losses (Han et al., 2020). In Beam-Guided TasNet, iterative beamforming refines spatial separation by leveraging improved speaker spatial statistics via the SCMs.

3. Training Objectives and Loss Functions

Optimizing MIMO-TasNet for robust signal separation and spatial cue fidelity involves a targeted loss design.

3.1 Signal Quality Metrics

While scale-invariant SDR (SI-SDR) is commonly used, it is insensitive to level cues: $$\mathrm{SI\!-\!SDR}(s,\hat s)=10\log_{10}\frac{\|\alpha s\|_2^2}{\|\hat s-\alpha s\|_2^2}$$ with $\alpha = \frac{\hat s^\top s}{s^\top s}$.

Instead, real-time binaural MIMO-TasNet employs the non‐scale‐invariant SNR objective: $$\mathrm{SNR}(s,\hat s)=10\log_{10}\frac{\|s\|_2^2}{\|s-\hat s\|_2^2}$$ The aggregate loss is the negative average SNR across all sources and channels.

In Beam-Guided TasNet, an end-to-end permutation-invariant training (PIT) strategy with unfolding is adopted. The loss sums the negative SNR over each beamforming/refinement iteration: $$L = -\sum_n \mathrm{SNR}( \hat{z}_{s,c}^{(n)}, x_{s,c} )$$

3.2 Spatial Fidelity Evaluation

Spatial error metrics include ITD and ILD deviations: $$\Delta_{ITD} = |\mathrm{ITD}(s^l, s^r) - \mathrm{ITD}(\hat{s}^l, \hat{s}^r)|$$

$$\Delta_{ILD} = |10\log_{10}\frac{\|s^l\|^2}{\|s^r\|^2} - 10\log_{10}\frac{\|\hat{s}^l\|^2}{\|\hat{s}^r\|^2}|$$

4. Real-Time and Computational Considerations

MIMO-TasNet architectures target real-time, low-latency contexts. The real-time binaural system uses strictly causal convolutional processing throughout (encoder, TCN, decoder), filter duration $L=2$ ms, and avoids STFT or any large windowing. Total algorithmic latency is $\lesssim 2$ ms; including all computational overhead, “well below 5 ms” live separation is consistently achieved, enabling deployment in hearing aids and AR platforms (Han et al., 2020).

In the Beam-Guided TasNet case, iterative refinement trades off separation accuracy against computational load and inference latency: each iteration improves separation/beamforming but increases overall complexity (Chen et al., 2021).

5. Experimental Performance

5.1 Binaural MIMO-TasNet Results

On the anechoic WSJ0-2mix two-talker task, MIMO-TasNet outperforms previous models in both signal and spatial metrics:

Method	SNRᵢ (dB)	ΔITD (μs)	ΔILD (dB)
Single-channel TasNet	10.2	5.8	0.46
+ sin(IPD),cos(IPD),ILD features	14.4	1.8	0.20
+ parallel-encoder	15.0	2.0	0.20
+ parallel encoder + mask–sum (MIMO)	15.6	1.8	0.19

Robustness is demonstrated on three-speaker, noisy, and reverberant mixtures, with MIMO-TasNet consistently delivering the highest SNR improvements and the lowest spatial cue errors across conditions. Pearson’s correlations between SNR improvement and spatial error are $-0.77$ (ITD) and $-0.85$ (ILD), indicating that improved signal separation is tightly coupled with spatial cue accuracy (Han et al., 2020).

5.2 Beam-Guided TasNet (Iterative MIMO)

On 4-mic reverberant WSJ0-2mix, iterative Beam-Guided TasNet achieves the following on reference channel $c=1$:

Model	Iterations	SDR (dB)	WER (%)
Beam-TasNet (baseline)	—	17.4	13.4
Beam-Guided TasNet	1	19.1	12.3
Beam-Guided TasNet	2	20.0	12.1
Beam-Guided TasNet	4	21.5	12.1
Oracle IRM-MVDR	—	17.6	12.8
Oracle signal-based MVDR	—	23.5	11.9

After four iterations, Beam-Guided TasNet closes the gap to the oracle MVDR to $2.0$ dB, highlighting the impact of cyclically integrating beamforming with neural separation (Chen et al., 2021).

6. Algorithmic Mechanisms and Iterative Refinement

MIMO-TasNet’s mask-and-sum paradigm in the time domain is central to both architectures' effectiveness. In the Beam-Guided TasNet, iterative connection of neural separation and classical MVDR beamforming forms a directed cyclic flow. Each refinement cycle allows updated speaker spatial covariance matrices, estimated from improved neural masks, to yield progressively more powerful beamforming filters. This “cyclic MIMMO interaction” accelerates convergence towards the oracle spatial filter performance.

Weight sharing across iterations (deep unfolding) maintains parameter efficiency, while the PIT with unfolding incentivizes consistent improvement across steps.

7. Open Challenges and Future Prospects

Outstanding issues include designing lighter-weight, low-latency iterative schemes, broadening scalability to more than two sources or to dynamically varying speaker configurations, and directly optimizing for ASR end goals or alternative beamformers (e.g., GEV, Wiener). The statistical robustness of SCM estimation—particularly regarding phase information in the neural outputs—remains critical for further progress. Expanding the operational envelope to challenging, real-world acoustic conditions and more generalized microphone array configurations is a prominent research direction (Han et al., 2020, Chen et al., 2021).