Multi-Mic Complex Spectral Mapping

Updated 2 May 2026

Multi-microphone complex spectral mapping is a technique that estimates clean speech spectra directly from multichannel STFT observations by leveraging both spatial and spectral cues.
It employs deep neural architectures, including RI-based representations, mask-based filtering, and state-space models, to enhance performance in noisy and reverberant settings.
Hybrid pipelines integrate DNN-based estimation with classical beamforming and adaptive filtering, achieving significant improvements in metrics like SI-SDR, PESQ, and STOI.

Multi-microphone complex spectral mapping is a class of signal processing and deep learning techniques that aim to directly estimate time-frequency domain complex spectral representations of clean or enhanced speech from raw or preprocessed multi-microphone observations. These approaches leverage both spatial and spectral cues present in the multichannel complex short-time Fourier transform (STFT) signals, learning nonlinear mappings that generalize and, in several methodologies, subsume classical beamforming, spatial filtering, and post-filtering techniques.

1. Foundations and Problem Formulation

Multi-microphone complex spectral mapping is formulated in discrete STFT space, where the observed signals at $M$ microphones are denoted $X_m(\tau, f)$ for channel $m$ , frame $\tau$ , and frequency $f$ . In reverberant or noisy environments, these observations can be modeled as

$X_m(\tau,f) = D_m(\tau,f) + N_m(\tau,f),$

where $D_m(\tau, f)$ models the (possibly reverberant) direct-path speech component and $N_m(\tau, f)$ represents additive noise. The canonical goal is to recover, at each frame and frequency, either the clean source signal at a reference microphone or, in MIMO or separation tasks, channel- or source-specific estimates, by learning mappings from the multichannel (real/imaginary or magnitude/phase) STFT domain directly to the target complex spectra (Halimeh et al., 2021, Wang et al., 2020, Wang et al., 2020, Pan et al., 2023, Ren et al., 2024, Zhang et al., 2022).

2. Core Neural Architectures and Spectral Mapping Strategies

Deep learning-based multi-microphone complex spectral mapping frameworks universally leverage convolutional, recurrent, and state-space models with STFT-domain features:

RI/Complex Input Representation: The real-imaginary (RI) parts of each microphone’s STFT coefficients are stacked, creating $2M$-dimensional feature vectors per T-F bin. This formulation underpins both MISO and MIMO architectures, as well as more general spatial encodings (Wang et al., 2020, Wang et al., 2020).
Mask-based Filter-and-Sum: Complex-valued masks are estimated per channel or per-reference by deep networks (such as COSPA), effecting channel-wise filtering prior to summation:

$\hat S(\tau, f) = \sum_{m=1}^M M_m(\tau, f) X_m(\tau, f)$

where $X_m(\tau, f)$ 0 are learned masks (Halimeh et al., 2021).

Spectral Mapping in Beamspace/SH Domain: Frameworks such as hierarchical SHT modeling project raw multichannel observations into spatial basis representations (e.g. spherical harmonics), allowing DNNs to estimate clean “spatial coefficients” hierarchically for improved spatial selectivity (Pan et al., 2023).
Temporal Modeling: Pipelines frequently insert GRU/BLSTM/LSTM or state-space (e.g. Mamba) blocks to capture temporal non-stationarity, both in the spectral and spatial cues (Halimeh et al., 2021, Ren et al., 2024, Zhang et al., 2022).
Hybrid Architectures: Recent models (e.g., MCMamba) extract both multi-scale spatial and spectral features in parallel streams—full/narrow-band spatial, sub/full-band spectral—feeding them into a structured ensemble of state-space or recurrent modules, followed by late fusion to produce complex ratio masks (Ren et al., 2024).

Approach	Spatial Handling	Output Domain
COSPA (Halimeh et al., 2021)	Per-mic complex masks, sum	Complex STFT
MISO/MIMO (Wang et al., 2020)	RI stack DNNs, beamforming	Complex STFT
Hier. SH (Pan et al., 2023)	Spherical harmonics, explicit	SH + STFT
MCMamba (Ren et al., 2024)	Multi-scale streams, SSM	Complex mask
LCSM (Zhang et al., 2022)	MIMO, channelwise LSTM	Complex STFT

3. Exploiting Spatial and Spectral Cues

Direct learning from complex spectra enables joint modeling of spatial and spectral structure:

Preservation and Manipulation of Inter-channel Phase: Applying initial per-channel or global complex masks preserves natural spatial cues (such as phase differences) critical for spatial selectivity. Later per-channel decoding enables adaptive, time-frequency-specific beam pattern formation (Halimeh et al., 2021).
Hierarchical Spatial Modeling: Spherical harmonic transforms convert microphone signals into spatial order coefficients, which can be sequenced through DNN stages such that coarse spatial features guide the estimation of finer spatial details (Pan et al., 2023).
Spatial-Spectral Feature Fusion: Advanced frameworks (such as MCMamba) compute inter-channel phase differences (IPD), magnitude features, and sub/full-band spectral contexts, feeding these jointly to dynamic sequence models to leverage the full spatiospectral context for mask estimation (Ren et al., 2024).
Time-varying Adaptive Filtering: Time-adaptive masks, as opposed to static beamformers, enable exploitation of nonstationary speech activity, responding to speech onset/offset and dynamic scene changes (Halimeh et al., 2021, Wang et al., 2020).

4. Training Objectives and Losses

Learning-based complex spectral mapping frameworks generally minimize direct time-frequency or time-domain error measures between estimated and target spectral components:

Complex-Domain MSE/L1 Loss: Both real-imaginary and magnitude losses are used to penalize amplitude and phase errors in the estimated spectra or SH coefficients (Wang et al., 2020, Pan et al., 2023, Ren et al., 2024).
MVDR-Targeted Losses: Some models are trained to mimic oracle beamformer outputs (e.g., MVDR), enhancing their spatial selectivity even in the absence of explicit spatial parameter supervision (Halimeh et al., 2021).
Permutation Invariant Losses for Separation: For multi-source separation, permutation-invariant training (uPIT) is applied to resolve speaker-output ambiguities (Wang et al., 2020).
Cross-domain Objectives: Hybrid time-frequency losses (e.g., combining SDR with spectral L1/MAE) are employed to enforce perceptual and signal-level fidelity (Zhang et al., 2022).

5. Integration with Classical Beamforming and Post-filtering

Several approaches hybridize deep spectral mapping with spatial filtering:

DNN + MVDR Pipeline: Direct estimation of clean spectral components by a DNN is followed by MVDR beamforming using DNN-provided spatial statistics, and optionally a second DNN post-filter. This integration allows learned nonlinear front-ends and physically grounded beamforming to be co-optimized (Wang et al., 2020, Wang et al., 2020).
DNN as Adaptive Beamformer: Models such as COSPA re-interpret the DNN-derived complex masks as generalized, data-driven adaptive spatial filters, where the shape and selectivity can potentially surpass classical spatial-only methods (Halimeh et al., 2021).
Post-filter Stacking: Multiple stages of complex spectral mapping (e.g., initial mapping, beamforming, post-filter mapping) yield improvements in SI-SDR, PESQ, and word error rate (WER) over both single-channel and conventional multichannel pipelines (Wang et al., 2020, Wang et al., 2020).

6. Empirical Benchmarks, Metrics, and Observations

Multi-microphone complex spectral mapping methods are benchmarked using standard speech enhancement, separation, and dereverberation metrics, often on open-source synthetic and real-world corpora:

Speech Enhancement: COSPA achieves $X_m(\tau, f)$ 1SINR = 7.5 dB (close to single-channel 7.7 dB and far exceeding DNN-MVDR 5.3 dB); also, achieves PESQ gain of 0.23 versus 0.16 for single-channel baselines. The measured SDR and $X_m(\tau, f)$ 2STOI further corroborate competitive performance relative to oracle MVDR and GMVDR (Halimeh et al., 2021).
Speech Dereverberation: DNN MISO models produce SI-SDR = 8.6 dB and PESQ = 3.24 (2-mic, MISO1-BF-MISO2), outperforming WPE and BeamformIt baselines, and yielding significantly lower WER on the REVERB Challenge dataset (Wang et al., 2020).
Speech Separation: Full DNN+MVDR+post-filter chains stabilize performance at SI-SDR = 15.6 dB and WER = 8.28% (oracle: 6.40%) on SMS-WSJ. Robust generalization to real arrays with geometrical perturbations ( $X_m(\tau, f)$ 30.4 dB SI-SDR decrease at 5 mm perturbation) is observed (Wang et al., 2020).
Hierarchical SH Mapping: Relative improvements with explicit spatial modeling yield PESQ gain of +0.12 and STOI gain of +2.16% over the DPCRN baseline at high reverberation, with optimal performance at spherical harmonic order $X_m(\tau, f)$ 4 (Pan et al., 2023).
MCMamba CHiME-3: SOTA results reported: WB-PESQ = 2.98, NB-PESQ = 3.49, STOI = 98.2%, SDR = 20.8 dB, surpassing McNet and earlier causal baselines (Ren et al., 2024).
Echo Cancellation - LCSM: LCSM achieves up to +15 dB higher ERLE and +0.3 PESQ points over magnitude-masking and CRN baselines, while maintaining a footprint of 0.55 M parameters for embedded deployment (Zhang et al., 2022).

7. Limitations and Prospective Directions

Most models assume a fixed microphone array geometry; geometry mismatch sensitivity is minimal under small perturbations but remains a challenge for unknown configurations (Wang et al., 2020, Wang et al., 2020).
Some approaches (e.g., COSPA) are trained only to match reverberant MVDR outputs, not dry dereverberation targets, and thus do not perform dereverberation per se (Halimeh et al., 2021).
Explicit modeling of interfering speakers within the same network is largely unexplored; separation tasks generally require multi-stream architectures and label permutation handling (Wang et al., 2020).
Real-time and causal deployment is now achievable in highly parameter- and compute-efficient frameworks (e.g., LCSM) (Zhang et al., 2022).
Future research may extend these techniques to explicitly dereverberate, extract multiple simultaneous sources, generalize to unknown arrays, incorporate explicit side-information (e.g., DOA), and leverage spatial basis decompositions for further complexity/performance trade-offs (Pan et al., 2023, Halimeh et al., 2021).