NeuralPMWF: Hybrid Neural Speech Enhancement

• NeuralPMWF is a hybrid speech enhancement system that combines a classical PMWF with a neural controller to balance noise suppression and speech distortion.
• It employs a compact MaskDNN to dynamically estimate time-frequency masks, covariance smoothing factors, and distortion parameters (β) in real time.
• Designed for low-power devices like smart glasses, it achieves a 16 ms latency while preserving the interpretability of classical multi-channel filtering.

NeuralPMWF is a hybrid multi-channel speech enhancement system that combines a classical parameterized multi-channel Wiener filter (PMWF) with a tiny neural network that learns to control the filter in real time. Its design target is low-power, low-latency hardware such as smart glasses, where multi-channel speech enhancement must balance two competing objectives: strong noise suppression and low speech distortion. In NeuralPMWF, the neural network does not replace the beamformer; instead, it estimates the masks, statistics, and control parameters required by a classical PMWF, including a complex multi-channel mask, frequency-dependent covariance smoothing factors, and a time-frequency-dependent distortion parameter $\beta[t,w]$ (Grinstein et al., 18 Jul 2025).

1. Problem formulation and design rationale

The system is formulated in the short-time Fourier transform (STFT) domain for $T$ time frames, $F$ frequency bins, and $M$ microphones. The observed multi-channel mixture is

$$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$

where $\mathbf{S}$ denotes the target speech contribution and $\mathbf{N}$ denotes noise, possibly including interfering speakers. The target of estimation is the target speech at reference channel $0$,

$$\mathbf{S}_0 = \mathbf{S}[:, :, 0] \in \mathbb{C}^{T \times F}.$$

A beamformer applies a complex vector filter $\mathbf{h}[t,w] \in \mathbb{C}^{M}$ to the multi-channel mixture to produce the enhanced estimate

$T$0

The central design problem is the suppression-distortion trade-off. Aggressive enhancement can reduce residual noise substantially but often introduces artifacts, temporal smearing, or other distortions. Conversely, conservative linear beamforming can preserve speech but leave considerable residual noise. NeuralPMWF addresses this by keeping the actual filtering within a classical, interpretable PMWF and assigning only the control function to a lightweight neural model (Grinstein et al., 18 Jul 2025).

This design explicitly contrasts with end-to-end neural speech enhancement systems based on deep nonlinear mappings such as U-Nets, CRNs, and Conv-RNNs. In the NeuralPMWF formulation, the network is a controller rather than the enhancement operator itself. A common misunderstanding is therefore to classify it as a purely neural enhancer; in fact, its defining feature is that the PMWF remains the mechanism performing the multi-channel filtering.

The hardware-oriented constraints are equally central. The system uses a 16 ms STFT window and causal temporal modeling, so the algorithmic latency is 16 ms. The architecture is also constrained by compute and parameter budget, with deployment scenarios including smart glasses and hearing-aid-like devices where CPU/MIPS, memory, and strict real-time operation are limiting factors.

2. PMWF formulation and the role of $T$1

NeuralPMWF adopts the PMWF formulation from Souden et al. (2010). For a multi-channel signal $T$2, where $T$3, the spatial covariance matrix is

$T$4

These yield the speech covariance $T$5 and noise covariance $T$6. NeuralPMWF then defines

$T$7

and uses the first column of $T$8, corresponding to reference channel $T$9, to compute the PMWF filter

$F$0

The scalar $F$1 determines the distortion-suppression balance. Small $F$2 moves the system toward more distortionless behavior, described in the paper as toward MVDR; $F$3 corresponds to the standard MWF in the formulation used; and larger $F$4 yields increasingly aggressive suppression at the cost of greater distortion. PMWF is therefore treated as a unifying framework in which MVDR-like, MWF, and aggressive MWF-like operating points arise from the choice of $F$5 (Grinstein et al., 18 Jul 2025).

This structure is the core inductive bias of the method. Rather than learning arbitrary enhancement transformations, NeuralPMWF learns how to operate a beamformer whose behavior remains inspectable through $F$6, $F$7, and $F$8. The interpretability of the classical filter is preserved, while the neural network supplies data-driven control over the most consequential degrees of freedom.

3. Neural control architecture

The full pipeline begins with complex multi-channel STFT input and ends with inverse STFT waveform reconstruction. The neural component, termed MaskDNN, receives the multi-channel STFT decomposed into real and imaginary parts and represented as a real-valued tensor of size $F$9. From this input it produces a multi-channel complex mask $M$0 as well as parameters used to compute speech presence, covariance smoothing, and distortion control (Grinstein et al., 18 Jul 2025).

Once $M$1 is obtained, masked speech is estimated channelwise as

$M$2

and noise is estimated residually as

$M$3

These multi-channel target and noise estimates are then used for covariance estimation.

MaskDNN is divided into spatial and temporal blocks. The spatial block consists of a sequence of four convolutional layers applied per frequency, with no weight sharing across frequency bins, and is described as inspired by frequency-domain Filter-and-Sum beamforming. The activations are Parametric ReLU (PReLU). The first three layers preserve a 10-channel feature dimension, and the final spatial layer produces an additional channel that is passed to the temporal stage.

The temporal block first projects that channel through a linear layer into a feature space of size $M$4 per time-frequency location. It then applies three causal SplitGRU layers. In SplitGRU, the feature dimension is split into $M$5 chunks, each chunk is processed by a separate GRU, and the outputs are remixed before the next layer. This reduces compute by roughly a factor of $M$6 relative to a full GRU of size $M$7. A final linear layer maps the temporal output to size $M$8 per frame, producing one scalar per frequency. The temporal output is treated as an additional real mask that modulates the spatial block output, yielding the final complex multi-channel mask $M$9.

The architecture is deliberately small: four spatial convolutional layers, three SplitGRU layers with 96 hidden units split into two groups per layer, about 165k parameters in the descriptive summary, and a reported complexity of 164.9k parameters and 24.95 MMACs/s at 16 kHz in the final complexity table. Because all recurrent processing is causal, no future context is used and the only algorithmic delay is the STFT window length.

4. Covariance adaptation and dynamic distortion control

NeuralPMWF uses the neural mask not only to separate speech and noise proxies but also to derive the PMWF control variables. The paper defines

$$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$0

where $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$1 is the sigmoid and $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$2, $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$3, $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$4, $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$5, and $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$6 are learned frequency-dependent real scalars. Here $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$7 functions as a speech presence probability (SPP), estimated from the magnitude of the reference-channel mask (Grinstein et al., 18 Jul 2025).

This parameterization has a direct operational interpretation. When $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$8 is high, speech is likely present and $$\mathbf{Y} = \mathbf{S} + \mathbf{N} \in \mathbb{C}^{T \times F \times M},$$9 becomes small, reducing distortion. When $\mathbf{S}$0 is low, speech is likely absent and $\mathbf{S}$1 becomes large, allowing more aggressive noise suppression. The covariance smoothing parameters $\mathbf{S}$2 and $\mathbf{S}$3 are frequency-dependent but time-invariant at inference time: they are learned during training and then fixed.

Spatial covariances are estimated by exponential smoothing:

$\mathbf{S}$4

Larger $\mathbf{S}$5 implies faster adaptation and more weight on the current frame; smaller $\mathbf{S}$6 implies slower adaptation and stronger temporal smoothing. The learned patterns reported in the paper show $\mathbf{S}$7, indicating that speech statistics are treated as changing faster than noise statistics. The paper also reports that the learned baseline $\mathbf{S}$8 is smaller at low frequencies, where speech energy is high, and larger at high frequencies. In the final SPP-driven formulation, $\mathbf{S}$9 can become very large when speech is absent, exceeding 30 in the visualization described for Fig. 4c.

A key consequence is that NeuralPMWF does not use a single global suppression setting. It performs fine-grained, time-frequency-dependent control of distortion and suppression through $\mathbf{N}$0, while separately learning the temporal adaptation rates used to estimate the beamformer statistics.

5. Training regime and simulated data generation

The system is trained end-to-end from STFT input to time-domain output using a weighted combination of two losses. The first is a time-domain SNR loss, following Roux et al. (2019), described as equivalent to a scale-invariant SNR or SI-SDR-type metric. The second is the frequency-domain phase-constrained magnitude (PCM) loss from Pandey and Wang (2021), which encourages accurate magnitude estimation while respecting mixture phase constraints. Training is supervised with parallel clean-noisy multichannel pairs, and no ground-truth masks, covariances, or $\mathbf{N}$1 values are required; these are learned implicitly through speech reconstruction objectives (Grinstein et al., 18 Jul 2025).

The training corpus is generated from the Interspeech 2020 Deep Noise Suppression (DNS) Challenge corpus. Clean speech and noise are drawn from DNS and split into train, validation, and test sets with an 85/5/10 ratio. All signals are resampled to 16 kHz. Room simulation uses random room dimensions with length and width sampled from $\mathbf{N}$2 m and height from $\mathbf{N}$3 m. Microphone array location and orientation are randomized within the room. Target speech is placed at 0.5–2.5 m from the array within field-of-view azimuth $\mathbf{N}$4 and elevation $\mathbf{N}$5. The simulation also includes 1–10 noise sources at more than 0.5 m, 0–10 interfering talkers at more than 3 m, SNR in $\mathbf{N}$6 dB, SIR in $\mathbf{N}$7 dB, and room impulse responses generated by the image method of order 6 using Pyroomacoustics, with wall absorption in $\mathbf{N}$8.

The microphone configuration is a five-microphone arrangement resembling Ray-Ban Meta smart glasses, and measured anechoic acoustic transfer functions are used to model head-related effects including directivity, diffraction, and absorption. The final dataset size is 320k training utterances, 600 validation utterances, and 3.2k test utterances.

Training uses PyTorch for 100 epochs on 10-second utterances with batch size 128. Optimization uses Adam with AMSGrad, gradient norm clipping at 1, a learning rate of 0.001 for the first 70 epochs, and then a 10% reduction every 10 epochs. Multi-GPU training is performed on Nvidia H100 hardware.

6. Experimental results and ablation structure

Evaluation is reported on the test set using STOI, NB-PESQ, SNR with respect to the reverberant target speech, and SI-SDR via a time-domain SNR-like metric. The STFT front end uses a window size of 256 samples, frame shift of 128 samples, sampling rate of 16 kHz, and 129 frequency bins. This yields an algorithmic latency of 256 samples, approximately 16 ms (Grinstein et al., 18 Jul 2025).

The main baseline systems are all hybrid methods that combine a neural front end with MWF-type beamforming. TinyGRU+MWF, attributed to Pandey (2024), uses a single-channel complex mask estimated by a small GRU network and operates at a similar compute budget. GTCRN+MWF, attributed to Rong et al. (2024), adapts a very low-compute CRN to multichannel processing. MC-CRN+MWF, denoted MCCRN and attributed to Xu et al. (2024), uses a multi-channel CRN with 20 maxDI beamformers plus MWF.

Method	STOI / SI-SDR / SNR / NB-PESQ	Compute / Params
Input	58.1 / -2.83 / -2.84 / 1.47	– / –
MCCRN+MWF	69.9 / 3.88 / 5.78 / 1.79	35 MMACs / 117k
GTCRN+MWF	72.7 / 4.66 / 6.23 / 2.01	91 MMACs / 26.5k
TinyGRU+MWF	73.1 / 5.02 / 6.56 / 2.08	24.37 MMACs / 152.8k
NeuralPMWF	74.3 / 5.5 / 6.91 / 2.12	24.95 MMACs / 164.9k

Relative to the input condition, NeuralPMWF improves STOI by 16.2 points and SI-SDR by about 8.3 dB, from -2.83 to 5.5 dB. Relative to TinyGRU+MWF, which is the most directly comparable baseline in compute budget, the gains are consistent across all reported metrics: STOI improves from 73.1 to 74.3, SI-SDR from 5.02 to 5.5 dB, SNR from 6.56 to 6.91 dB, and NB-PESQ from 2.08 to 2.12.

The ablations separate covariance smoothing strategy from distortion-parameter strategy. For covariance smoothing, cumulative mean performs worst at STOI 69.8, SI-SDR 3.62, SNR 5.57, and PESQ 1.93. Fixed exponential smoothing improves to STOI 71.8, SI-SDR 4.83, SNR 6.40, and PESQ 2.04. Frequency-dependent learned exponential smoothing gives the best reported $\mathbf{N}$9-variant performance at STOI 72.3, SI-SDR 4.90, SNR 6.46, and PESQ 2.06. SPP-driven exponential smoothing is close, with STOI 71.9 and SI-SDR 4.90, but does not clearly surpass the simpler learned frequency-dependent constants. This is why the final system uses frequency-dependent learned $0$0 and $0$1.

The distortion-parameter ablation is more decisive. Static $0$2 and $0$3 both yield essentially identical performance, around STOI 69.8–69.9, SI-SDR 3.62, SNR 5.57, and PESQ 1.93. Aggressive static $0$4 degrades performance further to STOI 67.2, SI-SDR 1.71, SNR 4.42, and PESQ 1.87. A learned but time-invariant frequency-dependent $0$5 gives only slight improvement, with STOI 70.0, SI-SDR 3.70, SNR 5.62, and PESQ 1.93. By contrast, the final SPP-driven formulation,

$0$6

achieves the best overall results: STOI 74.3, SI-SDR 5.5, SNR 6.91, and PESQ 2.12. The principal empirical conclusion is therefore that static choices of $0$7, even when frequency-dependent, do not fully exploit PMWF, whereas time-varying SPP-modulated control yields substantial gains in intelligibility and suppression quality.

7. Applications, limitations, and relation to hybrid beamforming

The reported application profile is explicitly aligned with smart glasses and augmented-reality headsets. The system is trained and evaluated on a five-microphone geometry resembling Ray-Ban Meta smart glasses, with field-of-view constraints built into the room simulation. The same operating characteristics also make it relevant to hearing aids, hearables, teleconferencing systems, mobile devices, and other far-field multi-microphone speech-capture platforms where latency and power budgets are stringent (Grinstein et al., 18 Jul 2025).

Its limitations follow directly from the experimental design. The method is trained and evaluated on a specific five-microphone configuration, so transfer to other array geometries may require retraining or adaptation. Covariance estimation depends on stable multi-channel STFT statistics, and highly reverberant or strongly nonstationary conditions may challenge that estimation process. The training data are simulated DNS mixtures with simulated rooms and measured acoustic transfer functions; the paper notes no study of strongly mismatched real environments, so degradation under such mismatch remains a plausible implication rather than a quantified claim.

Within the broader literature, NeuralPMWF belongs to a family of hybrid neural-plus-signal-processing systems. The paper situates it relative to mask-based MVDR and MWF methods such as those associated with Heymann et al. (2016) and Ochiai et al. (2023), where a DNN predicts time-frequency masks and a classical beamformer computes the filter coefficients. It also aligns conceptually with neural Wiener-filter approaches such as the one attributed to Hsieh et al. (2024), and with systems that learn covariance or PSD estimates to drive classical filters, including NICE-Beam, ADL-MVDR, and DeepMMSE-style methods. What distinguishes NeuralPMWF within that space is the explicit learning of both distortion control and covariance smoothing, rather than mask estimation alone.

The future directions proposed in the paper extend this hybrid-control viewpoint. These include exploring alternative PMWF parameterizations, such as speech-distortion-weighted MWF and other regularizations; further optimizing the spatial-temporal network for other array geometries and scenarios; investigating robustness and transfer to microphone layouts beyond the Ray-Ban Meta-like configuration; and combining the learning of masks, SPP, and $0$8 with explicit multi-task objectives, including ASR-related or intelligibility-oriented losses. The broader implication is that resource-efficient multi-channel enhancement need not choose between rigid classical filtering and heavy end-to-end neural models; NeuralPMWF instead treats learned control of a classical filter as the primary design principle.