Binaural Speech Enhancement

• Binaural speech enhancement is a set of algorithms that improve speech quality captured by two spatially separated microphones while preserving interaural level and time differences.
• Techniques range from model-based filtering and mask-based spectral suppression to deep learning methods that jointly optimize noise reduction and spatial cue fidelity.
• Real-time implementations with low latency and efficient processing are crucial for applications in hearing aids, AR/VR, telecommunications, and consumer audio.

Binaural speech enhancement refers to the set of algorithms and systems designed to improve the intelligibility and quality of speech signals received at two spatially separated microphones (typically situated at the ears) in the presence of noise, while preserving—or explicitly controlling—the binaural spatial cues essential for source localization and spatial awareness. The field encompasses model-based, signal-processing, and learning-based methods with applications in hearing aids, augmented and virtual reality, telecommunication, and consumer audio. Distinct from monaural and generic multi-channel enhancement, binaural enhancement methods must not only reduce noise but also maintain or manipulate interaural level and time differences (ILD and ITD), ensuring that spatial information critical to perceptual realism and user orientation is preserved.

1. Signal Modeling and Spatial Cue Formalism

Binaural speech enhancement builds upon a dual-channel (left/right) signal model: $$\begin{align*} y_L(n) &= x_L(n) + n_L(n) \ y_R(n) &= x_R(n) + n_R(n) \end{align*}$$ where $x_{L,R}(n) = h_{L,R} \ast s(n)$ represent the head-related impulse response (HRIR)-filtered speech at each ear, and $n_{L,R}(n)$ are noise components, which may be directional or diffuse. In the STFT domain, the spatial cues exploited or preserved are:

• Interaural Level Difference (ILD): $$20 \log_{10} \left( \frac{|X_L(f,t)|}{|X_R(f,t)|} \right)$$, primarily at higher frequencies.
• Interaural Phase Difference (IPD)/Interaural Time Difference (ITD): $\arg(X_L(f,t)) - \arg(X_R(f,t))$, most salient at lower frequencies.

Preserving these cues in the enhancement process maintains the externalization and localization of sources, directly supporting natural binaural listening (Tokala et al., 2024, Tokala et al., 26 Jul 2025).

2. Architecture Classes and Methodological Approaches

2.1 Model-Based and Signal Processing Methods

Classical approaches include Kalman filtering exploiting autoregressive speech/voiced-unvoiced models with binaural codebook-based parameter estimation (Kavalekalam et al., 2018), mask-based spectral suppression using spatially informed cues such as coherence-to-diffuse ratio (CDR) (Ghanavi et al., 2022), and OM-LSA gain control (Tokala et al., 2022). Notably, the better-ear listening strategy, which applies channel-selective masking via $m_b(t,f)=\max\{m_L(t,f), m_R(t,f)\}$, is shown to preserve both ILD and ITD by enforcing identical gain in both channels (Tokala et al., 2022).

2.2 Deep Complex Networks and End-to-End Models

Modern systems predominantly use deep architectures:

• Complex-valued encoder-decoder CRN/Transformer hybrids: These process STFTs from both ears via complex convolutional and attention layers, estimating complex ratio masks (CRMs) per channel (Tokala et al., 2024, Tokala et al., 26 Jul 2025). Architectures commonly apply explicit spatial-cue losses (see Sec. 3).
• Lightweight, real-time convolutional networks: For embedded hearing aids and mobile devices, low-parameter networks with explicit RATF (relative acoustic transfer function) modeling or lightweight backbone modules demonstrate significant computational gains without sacrificing MBSTOI or spatial cue preservation (Wang et al., 2024, Lu et al., 17 Sep 2025).
• Mixture-of-experts (MoE) beamforming: Binaural rendering via an MoE bank enables real-time source tracking for moving talkers, adapting spatial filtering weights without explicit DoA estimation (Mittal et al., 16 Sep 2025).
• Time-domain MIMO-TasNet: Real-time, causal convolutional architectures performing mask estimation in learned filterbanks demonstrate sub-5 ms latency, robust ILD/ITD preservation, and strong separation—even for multiple sources (Han et al., 2020).

3. Spatial-Cue Preservation: Loss Functions and Mask Strategies

State-of-the-art binaural speech enhancement jointly optimizes noise reduction, intelligibility (MBSTOI, STOI), and spatial cue fidelity. Loss functions typically combine: $$\mathcal L = \alpha\,\mathcal L_{\mathrm{SNR}} + \beta\,\mathcal L_{\mathrm{STOI}} + \gamma\,\mathcal L_{\mathrm{ILD}} + \kappa\,\mathcal L_{\mathrm{IPD}}$$ where:

• $\mathcal L_{\mathrm{SNR}}$ is the per-channel (or average) scale-sensitive SNR loss.
• $\mathcal L_{\mathrm{STOI}}$ is the (modified) STOI of enhanced vs. clean reference.
• $\mathcal L_{\mathrm{ILD}}$ penalizes deviation in level differences, typically above 1500 Hz.
• $x_{L,R}(n) = h_{L,R} \ast s(n)$0 penalizes interaural phase/ITD distortion, usually below 1500 Hz (Tokala et al., 2024, Tokala et al., 26 Jul 2025).

Direct preservation of ITD/ILD is also achieved by restricting gains to be identical across channels or by employing physically-constrained RATF/BRIR estimation as in LBCCN (Wang et al., 2024). Mask-based methods (both binary and soft) are often trained to maximize MBSTOI or to follow oracle-better-ear strategies that emulate human spatial unmasking (Tokala et al., 2022).

4. Real-Time Constraints and Embedded Implementation

Hearing aid and wearable applications demand ultra-low latency (≤2–10 ms) and low computational complexity. Strategies include:

• Group-Communication and Quantization-Aware Training (QAT): Neural architectures leveraging grouped weight sharing, integer-only arithmetic, and minimal parameter footprints (e.g., 164 kB for GCFSnet) allow processing on embedded SoCs with 2 ms latency (Westhausen et al., 2023).
• Binaural linkage under low bit-rate constraints: Communication of quantized IPD/log-magnitude features across ears is shown to add negligible algorithmic latency and achieves marked performance gains with as little as 4–8 bits and ≤12 ms cross-ear delays (Westhausen et al., 2023).
• Block-causal, streaming implementations: Deep MIMO-TasNet, complex-(recurrent) convolutional, and lightweight transformer networks are implemented with STFT frame shifts of 1–6 ms and total algorithmic delays in the <20 ms range, enabling real-time on-device enhancement (Han et al., 2020, Tokala et al., 2024, Tokala et al., 26 Jul 2025, Wang et al., 2024, Lu et al., 17 Sep 2025).

5. Objective Performance and Evaluation Metrics

Objective metrics assess multiple facets:

• Noise reduction: Segmental SNR improvement (ΔSegSNR, fwSegSNR, SI-SDR) (Tokala et al., 2024, Han et al., 2020).
• Intelligibility: MBSTOI (modified binaural STOI) gain of 0.15–0.25 over baselines; PESQ improvements up to 1.0; and in some user studies, +0.61 MOS over raw signals (Chatterjee et al., 2022, Tokala et al., 26 Jul 2025, Lu et al., 17 Sep 2025).
• Spatial-cue errors: Mean ILD errors <1 dB and IPD errors <10° are consistently reported for state-of-the-art methods (Tokala et al., 2024, Tokala et al., 26 Jul 2025, Lu et al., 17 Sep 2025, Wang et al., 2024).
• Robustness: Performance is maintained across anechoic and reverberant conditions, noise types, and for variable HRIRs (Tokala et al., 26 Jul 2025, Tokala et al., 2024).
• Latency and computational efficiency: Measured in terms of MACs and real-time factor (RTF), with recent models achieving RTFs <0.2 and memory footprints <130 kB (Wang et al., 2024, Lu et al., 17 Sep 2025).

6. Advances, Challenges, and Future Directions

Modern binaural speech enhancement demonstrates:

• Joint optimization without spatial cue sacrifice: Simultaneous gains in SNR/MBSTOI and minimal ILD/IPD distortion through complex-valued deep architectures and explicit spatial loss regularization (Tokala et al., 26 Jul 2025, Tokala et al., 2024).
• Flexible spatial processing: MoE beamforming enables real-time, geometry-agnostic, and direction-adaptive enhancement for dynamic scenes, supporting applications beyond static localization (e.g., continuous talker motion in AR/VR) (Mittal et al., 16 Sep 2025).
• Trade-offs in computation: Low-complexity designs such as LBCCN and GAF-Net mediate the tension between spatial-cue fidelity and reduced parameter/MAC budgets, supporting on-chip deployment (Wang et al., 2024, Lu et al., 17 Sep 2025).
• Generalization and robustness: DNN-based masking and end-to-end approaches may still show vulnerabilities to non-stationary noise and unseen environments (Tokala et al., 2022). Research focuses on normalization to canonical perceptual spaces, self-supervised pretraining, and data augmentation to address these gaps (Xu et al., 2023).
• Integration of multimodal inputs: Inclusion of audio-visual cues (e.g., lip landmarks) can significantly boost intelligibility and generalization (Gogate et al., 2019).

Research continues towards greater robustness under reverberation and dynamic scenes, incorporation of perceptual loss criteria, and even lighter-weight real-time architectures for embedded and consumer devices (Lu et al., 17 Sep 2025, Westhausen et al., 2023, Wang et al., 2024). Multi-microphone and distributed-microphone settings, as well as scalable frameworks for AR/VR spatial audio, comprise active directions.