Direction-Preserving MIMO Speech Enhancement

• The paper introduces algorithms that preserve inter-channel phase and level cues through MVDR, DP-MWF, and hybrid deep learning techniques.
• It leverages spatial covariance estimation, spherical harmonics encoding, and neural covariance prediction to maintain accurate direction-of-arrival and spatial fidelity.
• Empirical results demonstrate improved PESQ, STOI, and SI-SDR metrics, enabling effective real-time beamforming and localization in multi-microphone systems.

Direction-preserving multiple-input multiple-output (MIMO) speech enhancement refers to the class of algorithms and neural architectures that enhance speech signals in multi-microphone environments while explicitly preserving the spatial (directional) characteristics of the sources in the output. Unlike single-output (MISO) enhancement that collapses spatial information, direction-preserving MIMO techniques aim to retain or reconstruct output signal fields whose inter-channel phase and level relationships are consistent with an acoustically plausible, spatially localized source—enabling downstream applications such as beamforming, binaural rendering, and localization. Recent advances leverage spatial covariance modeling, deep-learning-driven spatial feature extraction (e.g., spherical harmonics, directional embeddings), and model-based constraints (e.g., MVDR) to ensure the preservation of the target directionality and spatial cues.

1. Theoretical Foundations: Signal Models and Directional Constraints

The standard observation model for multi-channel arrays is

$$\mathbf{x}(t, f) = \mathbf{h}_f s_{t,f} + \mathbf{n}_{t,f}$$

where $\mathbf{x}(t,f)\in\mathbb{C}^D$ is the microphone array STFT vector, $s_{t,f}$ is the target speech, $\mathbf{h}_f$ is the frequency-dependent steering vector (transfer function), and $\mathbf{n}_{t,f}$ is noise plus interference. The spatial structure is described by covariance matrices: $$\mathbf{R}_{ss}(f) = \sigma_s^2(f) \mathbf{h}_f \mathbf{h}_f^H,\quad \mathbf{R}_{nn}(f) = \mathbb{E}[\mathbf{n}_{t,f}\mathbf{n}_{t,f}^H]$$ The principal eigenvector of $\mathbf{R}_{ss}(f)$ encodes the target direction, while the noise subspace is embedded in $\mathbf{R}_{nn}(f)$. The objective of direction-preserving enhancement is to build a filter $\mathbf{W}(f)$ such that the enhanced output retains the target’s directional cues, typically enforced via distortionless constraints (e.g., $\mathbf{w}^H\mathbf{h}=1$) as in MVDR (Haeb-Umbach et al., 13 Jan 2025). MIMO formulations generalize this to full-rank $\mathbf{x}(t,f)\in\mathbb{C}^D$0, enabling multi-output spatial rendering (Deppisch, 13 Apr 2026).

2. Model-Based and Hybrid Methods

Classical direction-preserving enhancement methods are rooted in spatial filtering frameworks, most notably:

• MVDR Beamformer: Minimizes output noise power under the distortionless constraint, with solution

$\mathbf{x}(t,f)\in\mathbb{C}^D$1

ensuring unity gain for the target direction (Haeb-Umbach et al., 13 Jan 2025, Bai et al., 2024).

• Direction-preserving MIMO Wiener Filter (DP-MWF): For MIMO enhancement, the filter is

$\mathbf{x}(t,f)\in\mathbb{C}^D$2

where $\mathbf{x}(t,f)\in\mathbb{C}^D$3 is derived to trade off speech distortion and noise suppression, and $\mathbf{x}(t,f)\in\mathbb{C}^D$4 ensures preservation of noise and target directionality (Deppisch, 13 Apr 2026).

• Multi-norm Beamforming: Optimization-based approaches impose an $\mathbf{x}(t,f)\in\mathbb{C}^D$5–$\mathbf{x}(t,f)\in\mathbb{C}^D$6 cost, e.g.,

$\mathbf{x}(t,f)\in\mathbb{C}^D$7

where $\mathbf{x}(t,f)\in\mathbb{C}^D$8 is the steering vector, ensuring that target direction is preserved (Qin et al., 24 Jul 2025).

These methods require accurate estimation of spatial covariance matrices, a challenge addressed by data-driven and hybrid approaches.

3. Deep Learning and Spatial Feature Encoding

Several recent architectures integrate spatial feature representations into deep neural networks to achieve direction-preserving MIMO enhancement:

• Spherical Harmonics Encoding: Injects spherical harmonics transform (SHT) coefficients, $\mathbf{x}(t,f)\in\mathbb{C}^D$9, as auxiliary inputs. The SHT provides a spatially complete, orientation-invariant basis, allowing the network to jointly process STFT and spatially-encoded features. The architecture merges separate spectro-temporal and SHT encoders, fusing their representations to predict the enhanced STFT, which preserves directional cues (Pan et al., 2023).
• Neural Covariance Estimation: OnlineSpatialNet is a lightweight network predicting a scale-normalized Cholesky factor of the noise covariance. The predicted covariance is used in a DP-MWF, ensuring the output's spatial eigenstructure matches the target/noise subspaces and thus preserves directionality (Deppisch, 13 Apr 2026).
• Triple-Steering Neural Enhancement: The CDUNet uses three steering vectors—centered on the target direction and two flanks—to condition the network on directional selectivity. This configuration enables dynamic adjustment of the spatial passband and maintains enhancement focused on the correct angular region (Wen et al., 2024).
• DOA-Aware Directional Embeddings: MIMO-DBnet learns per-frame, per-source embeddings that encode direction-of-arrival (DOA), mitigating phase-wrapping ambiguity and guiding the beamformer to preserve high-frequency spatial cues (Fu et al., 2022).
• Wavelet–Conformer Hybrid Architectures: WTFormer leverages wavelet transforms for multi-resolution decomposition and multi-dimensional collaborative attention (MCA) blocks that jointly attend to channel, time, and frequency, further reinforced by a MUSIC-based spatial loss to explicitly enforce spatial-cue preservation (Han et al., 27 Jun 2025).

4. Spatial Covariance Estimation and Directionality Preservation

Fully data-driven enhancement (end-to-end DNNs) can compromise spatial cues if the network focuses solely on spectro-temporal features (Haeb-Umbach et al., 13 Jan 2025). To counteract this, contemporary approaches use neural networks to estimate soft masks or directly predict spatial covariance matrices:

• Mask-Driven Covariance Estimation: Deep models output time–frequency masks $s_{t,f}$0 used as weights in SCM computation, which are then fed into spatial filtering (e.g., MVDR) (Haeb-Umbach et al., 13 Jan 2025, Bai et al., 2024).
• Neural Cholesky Predictors: Instead of time–frequency masks, a neural network (e.g., OnlineSpatialNet) predicts the Cholesky factor for the noise covariance, ensuring accurate multi-channel spatial modeling (Deppisch, 13 Apr 2026).
• Self-Attentive RNN Beamforming: Temporal and spatial self-attention modules absorb per-frame or cross-channel dependencies in covariance matrices to learn beamformer weights that are both distortionless and sensitive to spatial structure (Li et al., 2021).

The combination of explicit spatial modeling and model-based beamforming ensures the preservation of DOA information, inter-channel phase (IPD), inter-channel level difference (ILD), and other essential spatial cues.

5. Loss Functions, Metrics, and Empirical Validation

• Loss Functions: Time-domain mean squared error (MSE), scale-invariant SNR (SI-SDR), and spatial-cue preservation terms such as MUSIC-based spectrum MSE are prevalent. For example, WTFormer uses

$s_{t,f}$1

where $s_{t,f}$2 penalizes deviations in the output MUSIC spectrum (Han et al., 27 Jun 2025).

• Directionality Metrics: Direction preservation is assessed using beam pattern plots, inter-channel cue errors ($s_{t,f}$3IPD, $s_{t,f}$4ILD, $s_{t,f}$5ITD), Target Speaker Over-Suppression (TSOS), covariance alignment (cosine similarity), and downstream ASR/DOA tasks (Bai et al., 2024, Deppisch, 13 Apr 2026, Han et al., 27 Jun 2025).
• Empirical Results: Direction-preserving approaches demonstrate significant PESQ/STOI/SI-SDR gains over non-directional architectures, with improved main-lobe angular accuracy and reduced WER/TSOS. SHT-injected and neural-covariance systems achieve PESQ gains of up to +0.16, STOI of +5%, and SI-SDR increases exceeding 1 dB compared to baseline mask-based or MISO systems (Pan et al., 2023, Deppisch, 13 Apr 2026, Wen et al., 2024, Fu et al., 2022).

6. Computational Aspects and Real-Time Feasibility

Advances in architecture efficiency enable direction-preserving MIMO enhancement to be deployable in real-time and resource-constrained settings:

• Compact architectures (e.g., OnlineSpatialNet with 0.82M parameters, CDUNet with 74.4k parameters, WTFormer with 0.98M parameters) support real-time inference on CPUs/MCUs with typical factors of $s_{t,f}$60.5$s_{t,f}$7 (Wen et al., 2024, Han et al., 27 Jun 2025, Deppisch, 13 Apr 2026).
• Efficient covariance estimation (Cholesky parameterization, narrow/wide-band mixing, attention mechanisms) reduces required FLOPs by $s_{t,f}$8 versus mask-driven baselines (Deppisch, 13 Apr 2026, Bai et al., 2024).
• MIMO structures that avoid output-channel collapse (i.e., not mapping many channels to one) and maintain multi-output generation are key to preserving directionality (Li et al., 2022, Deppisch, 13 Apr 2026).

7. Practical Applications and Limitations

Direction-preserving MIMO speech enhancement is foundational for microphone-array front ends in beamforming, scene-aware ASR, binaural rendering, and robust speaker localization. High spatial fidelity in output signals is critical for such applications, particularly under reverberant, noisy, or multi-speaker conditions. The integration of explicit spatial encoding, model-based filtering, and efficient DNNs advances the state-of-the-art in both separation quality and spatial cue retention.

Limitations include the need for sufficiently dense and well-calibrated arrays (especially for SHT-based schemes), the open challenge of reliable covariance estimation in non-stationary or mismatched acoustic environments, and robustness to reverberation and non-far-field propagation. Ongoing research investigates generalized spatial encoding, uncertainty-aware learning, and real-time constraints.

---

Key Methods and Corresponding arXiv Papers

Approach	Spatial Modeling	Directionality Mechanism
SHT Dual Encoder (Pan et al., 2023)	Spherical Harmonics	Auxiliary SHT input
DP-MWF + Neural Cov. (Deppisch, 13 Apr 2026)	Cholesky Covariance	Full-rank MIMO filter
MVDR (Hybrid, masks) (Haeb-Umbach et al., 13 Jan 2025)	Mask-driven SCM	Distortionless constraint
DOA Embeddings (MIMO-DBnet) (Fu et al., 2022)	DOA Neural Embedding	High-freq direction coding
WTFormer (Han et al., 27 Jun 2025)	Wavelet+Conformer+MCA	MUSIC spatial loss
Triple-Steering CDUNet (Wen et al., 2024)	Parametric spatial input	Input-flank steering vectors
PCG-AIID (Li et al., 2022)	Dense MIMO masking	Per-channel output

These approaches collectively advance the theory and practice of direction-preserving MIMO speech enhancement by combining model-based signal processing principles, learned spatial feature representations, and computationally efficient architectures.