Directional Speech Enhancement

• Directional speech enhancement is a multichannel technique that uses directional cues like DOA to isolate target speakers and suppress noise.
• It integrates classical beamforming with neural and hybrid methods to boost speech intelligibility and ASR accuracy in far-field and multi-talker settings.
• Recent advances leverage online adaptation, feedback control, and multimodal inputs to dynamically optimize speech extraction in complex acoustic scenarios.

Directional speech enhancement (DSE) is the multichannel front-end stage in a spatial speech perception pipeline that exploits spatial information, such as a direction-of-arrival (DOA) estimate, to extract or enhance a target speaker while suppressing interferers and noise. In a typical SSL $\rightarrow$ DSE $\rightarrow$ ASR pipeline, DSE bridges localization and recognition by converting a DOA estimate into a spatial filter or spectral mask, thereby improving speech intelligibility and downstream ASR accuracy in far-field, noisy, or multi-speaker conditions. The field spans classical beamforming, neural enhancement, and hybrid systems that couple localization, enhancement, and recognition under real-time and computational constraints (Shao et al., 2 Jul 2026).

1. Scope, operating assumptions, and pipeline role

DSE is situated between sound source localization and speech recognition, but it is not reducible to either. Its defining property is target selection by direction: the enhancement objective is conditioned on spatial information, not solely on spectral denoising. In microphone-array settings, this spatial information is supplied either explicitly, through steering vectors or DOA embeddings, or implicitly, through learned multichannel representations that preserve directional structure (Shao et al., 2 Jul 2026).

A recurrent design assumption is that DOA or equivalent spatial cues are available frame by frame or block by block. The assumption is operational rather than incidental. In the AR-headset system of Kinoshita et al., target-direction identification is provided by the headset’s cameras plus user input: face tracking on the RGB camera finds candidate speaker positions in 3-D, the user “points” to one face via eye-gaze and pinching hand gesture, and the resulting 3-D unit vector $\theta$ is fed into both front- and back-ends (Sekiguchi et al., 2022). In the all-neural directional recurrent network (DRN), one-hot azimuth and elevation vectors are supplied at each frame and fused into the extraction model through channel-wise and frame-wise embeddings, enabling frame-rate directional control (Pandey et al., 2024).

The dependence on upstream localization also creates a central failure mode. DOA errors directly affect filter steer and mask quality, and several recent systems address this issue explicitly. The Demucs-based real-time correction loop of Valin et al. formalizes the problem as

$\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$

where $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$ is the online enhanced speech produced by steering toward $\theta$ and scoring the result with a non-intrusive speech-quality metric (Rascon, 2024). This formulation makes directional accuracy part of the enhancement loop itself rather than a fixed upstream prerequisite.

The application domain is correspondingly broad. The survey literature identifies robot audition, hearing assistance, smart speakers, teleconferencing systems, and voice-controlled assistants as representative settings in which DSE must operate under background noise, reverberation, competing speakers, and dynamic acoustic conditions (Shao et al., 2 Jul 2026).

2. Classical directional filtering and spatial covariance formulations

Classical DSE is dominated by beamforming and spatial covariance modeling. Delay-and-sum (DS) beamforming aligns microphone signals to a hypothesized direction $\theta$ by delays $\{\tau_m(\theta)\}_{m=1}^M$ and averages:

$$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$

with frequency-domain weights

$$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$

Its advantages are simplicity, interpretability, and low complexity; its limitation is restricted interference rejection (Shao et al., 2 Jul 2026).

Minimum Variance Distortionless Response (MVDR) beamforming replaces fixed averaging with constrained optimization. Given spatial covariance $\rightarrow$0 and steering vector $\rightarrow$1, the objective is to minimize output power subject to unit gain in the target direction. Assuming noise covariance $\rightarrow$2 is known, the closed-form solution is

$\rightarrow$3

GEV or max-SNR beamforming instead maximizes the Rayleigh quotient $\rightarrow$4, while the generalized sidelobe canceller (GSC) decomposes the constrained problem into a fixed DS branch and an adaptive blocked branch updated by LMS or RLS (Shao et al., 2 Jul 2026).

A more structured spatial model appears in rank-constrained spatial covariance matrix estimation (RCSCME). In the STFT domain, the observed mixture is written as

$\rightarrow$5

with directional point-source speech $\rightarrow$6 and diffuse full-rank noise $\rightarrow$7. The instantaneous covariance is parameterized as

$\rightarrow$8

where $\rightarrow$9 and $\theta$0 are time-frequency source variances. The rank-constrained noise model starts from a partial rank-$\theta$1 estimate

$\theta$2

and restores a full-rank noise covariance through

$\theta$3

The final estimate is obtained by a multichannel Wiener filter,

$\theta$4

which couples directional speech modeling with explicit diffuse-noise structure (Misawa et al., 2021).

This classical lineage remains relevant because many neural systems either reproduce these operators in differentiable form or use them as back-end modules. A common misconception is that DSE became purely end-to-end once deep learning entered the field. The literature instead shows sustained reuse of steering vectors, spatial covariance matrices, Wiener filtering, and beamformer initialization inside contemporary systems (Shao et al., 2 Jul 2026).

3. Neural directional conditioning and learned spatial representations

Modern neural DSE systems differ primarily in how they encode directionality. A baseline family operates in the STFT domain and estimates masks $\theta$5, using losses such as MSE, SI-SDR, or permutation-invariant training. A second family learns filter-and-sum operations directly in the time domain. A third family conditions explicitly on DOA through embeddings, steering vectors, or auxiliary spatial features (Shao et al., 2 Jul 2026).

A compact example of explicit spatial encoding is spherical harmonics injection. Zhang et al. compute spherical harmonics coefficients (SHCs) via a spherical harmonics transform (SHT) and use them as auxiliary model inputs. For microphones at known $\theta$6 on radius $\theta$7, the discrete approximation is

$\theta$8

With truncation order $\theta$9, this yields $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$0 complex coefficients per time-frequency bin. Their model uses two encoders, one for multi-channel STFT inputs and one for SHCs, each implemented with six cascaded inplace GLUs, and fuses the resulting embeddings in a decoder with a channel-wise bi-LSTM (Pan et al., 2023). The stated rationale is that the STFT domain poorly disentangles spatial structure, whereas SHCs concisely represent spatial distributions and give the network a direct handle on directionality.

The DRN of Gu et al. replaces hand-crafted directional features with learned DOA embeddings from a predefined spatial grid. For each frame $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$1, channel-wise azimuth and elevation embeddings are produced by learned linear layers followed by LayerNorm and PReLU, summed into $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$2, projected, and fused multiplicatively with the spatial latent tensor. Frame-wise embeddings are similarly injected after each causal LSTM layer. The time-domain version operates with $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$3 ms frames and hop $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$4, so algorithmic latency is 2 ms end-to-end (Pandey et al., 2024). Because the DOA is supplied every frame, abrupt target switching is handled by feeding the new one-hot DOA at the next frame; no state reset is required.

Directional conditioning also appears in systems that tie enhancement to downstream transcription. In Directional-SpeechLlama, a 7-mic smart-glasses array is paired with a bank of 12 fixed NLCMV beamformers, each steered to one of 12 azimuth bins at $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$5 resolution. On each beamformed signal, the system computes magnitude spectrograms, inter-channel phase differences (IPD), and inter-channel level differences (ILD), flattens the resulting tensor, and projects it into the frozen audio encoder’s 1,536-dimensional space. Direction tokens $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$6 are added to the LLM vocabulary, and Serialized Directional Output Training (S-DOT) forces the model to emit a direction token before each speaker turn (Xie et al., 17 Jun 2025). The same system adds Contrastive Direction Data Augmentation (CDDA), which explicitly pushes embeddings for distractor directions away from target-direction embeddings.

A plausible implication is that “directionality” in neural DSE is no longer a single mechanism. It can be represented as steering-vector channels, SHCs, beam indices, IPD/ILD tensors, one-hot DOA grids, or discrete direction tokens, depending on whether the downstream objective is waveform quality, source separation, or recognition.

4. Online adaptation, feedback control, and dynamic-scene operation

Real-time DSE increasingly treats directional control as an adaptive process rather than a fixed pre-processing step. The most explicit example is DOA correction through speech-quality feedback. Valin et al. define $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$7 and minimize it with Adam, using an instantaneous finite difference of quality with respect to $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$8 as the gradient approximation. In the published implementation, bias correction is omitted; the hyper-parameters are $\theta^*=\arg\max_\theta SQ(\hat{x}(\theta)),$9, $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$0, $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$1, and step-size $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$2 (Rascon, 2024). The loop publishes $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$3 to the beamformer + Demucs path, reads a new smoothed quality score from Squim every $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$4 s, estimates the gradient, updates the Adam moments, and steps the DOA accordingly.

The feedback variable is deliberately non-intrusive. Squim, described as a state-of-the-art, non-intrusive MOS predictor available in torchaudio, processes windows of length $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$5 s with VAD pre-filtering and emits a quality score every $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$6 s. Raw Squim outputs fluctuate heavily, with $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$7–$\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$8 MOS points; exponential smoothing with $\hat{x}(\theta)=\mathrm{Enhance}(X;\theta)$9 reduces high-frequency jitter at the cost of slight lag (Rascon, 2024). Residual variability remains the main source of noise in the Adam objective.

A second adaptive pattern is teacher–student online adaptation. In the AR-headset system, a DNN-based MVDR front end runs continuously with low latency, while FastMNMF runs in the back end on blocks of approximately 9 s. The noisy and enhanced speech pairs produced by FastMNMF are buffered and used every 3 min to fine-tune the front-end DNN with the same SI-SDR loss used in pre-training, mixed 1:1 with original data (Sekiguchi et al., 2022). Weighted Prediction Error (WPE) dereverberation is applied both before beamforming in the front end and before or inside FastMNMF in the back end.

Dynamic-scene handling also appears in low-latency neural extraction. The DRN operates at a 500 Hz frame rate in its 2 ms setting, which the authors use to support abrupt DOA changes without resetting recurrent state (Pandey et al., 2024). The survey literature frames this more generally as the requirement that DSE models track moving DOAs and adapt masks or filters on the fly (Shao et al., 2 Jul 2026).

These systems indicate that DSE is increasingly coupled to control, adaptation, and tracking. This suggests a shift from “estimate DOA once, then enhance” toward continuous optimization in which localization uncertainty and enhancement quality are jointly managed.

5. Evaluation protocols and representative empirical results

DSE is evaluated with both signal-level and downstream metrics. The survey literature lists $\theta$0SNR, SI-SDR, SIR, SAR, PESQ, STOI, and WER reduction as standard criteria, and notes representative survey-level improvements that vary by scene and method: MVDR with $\theta$1SNR $\theta$2–$\theta$3 dB and PESQ $\theta$4–$\theta$5; mask-based systems with SI-SDR $\theta$6–$\theta$7 dB, PESQ $\theta$8, STOI $\theta$9; end-to-end FaSNet with SI-SDR $\theta$0–$\theta$1 dB and real-time latency $\theta$2 ms; and DOA-conditioned systems with $\theta$3SNR $\theta$4–$\theta$5 dB and downstream WER reduction $\theta$6–$\theta$7 (Shao et al., 2 Jul 2026).

Several papers provide more specialized numbers under concrete array geometries and task settings.

Method	Setup	Reported result
Proposed-parallel SH injection	TIMIT, SNR $\theta$8 dB	PESQ $\theta$9; STOI $\{\tau_m(\theta)\}_{m=1}^M$0 (Pan et al., 2023)
Time-domain DRN, $\{\tau_m(\theta)\}_{m=1}^M$1, 2 ms latency	8-mic circular array, dynamic scenes	STOI $\{\tau_m(\theta)\}_{m=1}^M$2; WB-PESQ $\{\tau_m(\theta)\}_{m=1}^M$3; SI-SDR $\{\tau_m(\theta)\}_{m=1}^M$4 dB (Pandey et al., 2024)
DOA correction via quality feedback	Initial error $\{\tau_m(\theta)\}_{m=1}^M$5	convergence in $\{\tau_m(\theta)\}_{m=1}^M$6–$\{\tau_m(\theta)\}_{m=1}^M$7 s; $\{\tau_m(\theta)\}_{m=1}^M$8 good runs at $\{\tau_m(\theta)\}_{m=1}^M$9 (Rascon, 2024)
Online adaptive AR headset	12 min teacher data	WER $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$0 offline fine-tune; WER $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$1 online adaptation (Sekiguchi et al., 2022)

For SH injection, the best model outperforms IGCRN at all three reported SNRs while also reducing parameter count from 1.89 M to 1.82 M and FLOPs from 21.01 G to 19.52 G (Pan et al., 2023). For the DRN, the frequency-domain 16 ms variant reports STOI $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$2, WB-PESQ $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$3, and SI-SDR $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$4 dB, while the 32 ms variant achieves 86.4% STOI and 1.79 PESQ at 9.0 GMAC/s versus MCNet’s 85.8% and 1.72 at 30.2 GMAC/s (Pandey et al., 2024). The same paper reports that under DOA mismatch, $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$5SNR remains within 0.7 dB of peak for $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$6 and is flat within $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$7.

Results on smart glasses illustrate a recurrent asymmetry between target channels. In directional source separation for Meta’s Project Aria glasses, Neural BF-13 improves the wearer stream from PESQ 2.89 and SI-SDR 18.17 dB to PESQ 3.11 and SI-SDR 20.44 dB, while the partner stream improves from PESQ 1.80 and SI-SDR 8.50 dB to PESQ 1.89 and SI-SDR 9.51 dB (Feng et al., 2023). For ASR, the best wearer WER drops from 7.99% to 6.51%, but partner WER can worsen unless beam and separation signals are fused or jointly trained. The joint system with $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$8 reaches 13.25% overall WER (Feng et al., 2023).

The RCSCME literature shows a different pattern: gains from coupling deep spectral priors with explicit spatial models. Under simulated mixtures at 0 dB overall SNR with four microphones and four noise types, unsupervised ILRMA gives SDR improvements of approximately 7–9 dB, ILRMA+RCSCME reaches approximately 14–15 dB, proposed IDLMA+RCSCME reaches approximately 15–16 dB, and the noise self-supervised variant reaches approximately 16–17 dB (Misawa et al., 2021).

These results clarify a frequent misconception. Higher separation quality does not automatically translate into uniformly better recognition outcomes. The smart-glasses studies report that directional source separation benefits ASR for the wearer but not for the conversation partner, and the survey explicitly notes that enhancement artifacts can distort phonetic cues and harm WER if not jointly optimized (Feng et al., 2023, Shao et al., 2 Jul 2026).

6. Integration with recognition, latency constraints, and open problems

DSE is increasingly evaluated not only as a signal-processing module but as part of integrated perception systems. Joint models such as DBNet, JointNet, and D-ASR use differentiable beamforming, shared features, bidirectional feedback, or CTC-based ASR losses to align enhancement with recognition (Shao et al., 2 Jul 2026). Directional-SpeechLlama extends this tendency into speech LLMs by combining beamformed multichannel front ends, spatial features, direction tokens, and a joint loss $$Y_\mathrm{DS}(t)=\sum_{m=1}^M x_m\bigl(t+\tau_m(\theta)\bigr),$$9 (Xie et al., 17 Jun 2025). In multi-talker target-direction ASR, the final system reports sWER $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$0 and SR $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$1; in single-talker far-field recognition, adding S-DOT yields WER avg $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$2, Dir-Acc $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$3, and L/R Acc $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$4 (Xie et al., 17 Jun 2025).

Latency and compute remain decisive design constraints. Filtering-based DS, MVDR, and GSC have low, deterministic latency below 1 ms per frame and require simple linear algebra on small $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$5 matrices. Masking-based neural models incur STFT buffer delays and network compute, leading to algorithmic latency of 30–100 ms, with models ranging from 1–20 M parameters. End-to-end time-domain systems can achieve sub-frame latency below 10 ms but require efficient convolution and careful causal design (Shao et al., 2 Jul 2026). The DRN reports 2.3 GMAC/s with 1.8 M parameters for the 2 ms, $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$6 model and 7.8 GMAC/s with 6.7 M parameters for $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$7 (Pandey et al., 2024). The AR-headset system reports front-end runtime of approximately 0.25 s per 3 s block, back-end runtime of approximately 0.7 s per block, and 42 s fine-tuning cost per 12 min buffer (Sekiguchi et al., 2022).

Open problems are consistent across the surveyed and task-specific literature. The survey highlights mismatch between signal-level and task-level objectives, robustness under dynamic or moving targets, propagation of localization uncertainty, joint SSL–DSE–ASR optimization, streaming and causal architectures, benchmarking and reproducibility, and multimodal integration with visual or inertial cues (Shao et al., 2 Jul 2026). More specialized papers make these abstractions concrete. The DOA-correction work recommends replacing naïve Adam with a control-theoretic observer or estimator, introducing adaptive step-size, dynamically re-injecting updated $$\mathbf{w}_\mathrm{DS}(\omega,\theta)=\tfrac{1}{M}\bigl[e^{j\omega\tau_1(\theta)},\dots,e^{j\omega\tau_M(\theta)}\bigr]^\mathsf{T}.$$8 from the localization front end, and reducing speech-quality estimation variability through metric ensembling, larger windows, higher-order smoothing, or silence-robust VAD and outlier rejection (Rascon, 2024). The smart-glasses source-separation study flags “fairness and efficiency” as next steps, while the DRN identifies the predefined DOA grid and the dependence on accurate per-frame DOA input as current limitations (Feng et al., 2023, Pandey et al., 2024).

Taken together, these lines of work define DSE as a family of spatially conditioned enhancement methods whose core problem is not merely denoising, but target-aware extraction under uncertainty, latency constraints, and task coupling. Classical beamformers provide low-latency structure and interpretability; neural systems provide flexible spatial conditioning and adaptation; hybrid systems increasingly treat localization, enhancement, and recognition as a single coupled problem.