Ultra-Low Latency Speech Enhancement

Updated 22 June 2026

Ultra-low-latency speech enhancement is a set of techniques that reduce processing delays to 1–3 ms using optimized buffer management and efficient model architectures.
Key methodologies include asymmetric STFT windowing, time-domain causal models, and minimum-phase FIR filtering to achieve high-quality, real-time audio processing.
Trade-offs between latency, computational load, and speech quality are managed through advanced pruning, quantization, and hybrid spatial-temporal processing strategies.

Ultra-low-latency speech enhancement refers to algorithmic pipelines and model architectures engineered to process noisy speech signals and return an intelligible, high-quality enhanced version with an end-to-end processing delay constrained to a few milliseconds or less. Such stringent latency requirements are dictated by immersive, interactive domains including hearing assistive devices, true wireless stereo (TWS) earbuds, telepresence, and AR/VR, wherein total processing delays above 5–10 ms begin to degrade usability, speech intelligibility, and user comfort. The design of these systems is shaped by a multifaceted optimization problem spanning algorithmic latency, model footprint, computational complexity, and enhancement fidelity, under power and memory constraints typical of embedded hardware.

1. Fundamental Latency Constraints and Measurement

Ultra-low-latency regimes are typically defined as algorithmic delays below 5 ms, with recent literature targeting 1–3 ms or even sub-millisecond (sample-level) operation (Dementyev et al., 2024, Bae et al., 2024, Cheng et al., 2024). The precise measurement of end-to-end latency ( $L_\text{total}$ ) must account for:

Analysis and Synthesis Windows: In transform-domain methods, algorithmic latency is dictated by the synthesis (output) window length $L_s$ , not the analysis (input) window:

$L_\text{algo} = \frac{L_s - P}{f_s}$

where $P$ is hop size and $f_s$ is sampling rate (Wu et al., 2024).

Internal Model Buffering/Lookahead: Any explicit future-frame context or model-induced delay.
Hardware Processing Time: The time required for model inference, typically depending on MACs/s and the available FLOPs.

Many studies report both the algorithmic and total (hardware + algorithmic) latency, with a hard upper bound (often 4 or 5 ms) imposed by application domain requirements (Bae et al., 2024 Westhausen et al., 2023).

2. Signal Representation and Architectural Strategies

A central design axis in ultra-low-latency speech enhancement is the choice of forward/inverse transforms and model topology. The most prominent approaches include:

Short-Time Fourier Transform (STFT) with Asymmetric Windows: Asymmetric analysis-synthesis windowing enables high-frequency-resolution analysis with ultra-short, low-latency synthesis (e.g., 32 ms analysis, 4–8 ms synthesis). This reduces algorithmic latency to the synthesis window length while retaining mask learning performance (Wang et al., 2021, Wang et al., 2022, Wu et al., 2024, Wood et al., 2019).
Learnable or Adaptive Filterbanks: Deep filter-bank equalizers and adaptive trainable filterbanks provide shorter fixed-latency synthesis and allow the network to approximate frequency-domain manipulation in the time domain (Zheng et al., 2022, Wu et al., 2024).
Time-domain End-to-end Models: Causal U-Nets with LSTM or SSM bottlenecks operate on raw waveform frames (1–3 ms), avoiding transform buffer delays entirely (Bae et al., 2024, Cheng et al., 2024).
Token-based Approaches: Autoregressive or non-autoregressive speech generation via discrete audio tokens or vector quantized codes provides direct mapping from noisy to clean speech codes with minimal lookahead (Li et al., 10 Mar 2026, Xue et al., 2023).
Minimum-phase FIR Filtering: In sample-by-sample enhancement, minimum-phase FIR filters generated by LSTM models are updated every millisecond or less, achieving mean algorithmic latency as low as 0.32 ms in hardware (Dementyev et al., 2024).

A summary comparison of representation strategies and minimum achievable algorithmic latency is given below:

Approach	Min. Algo. Latency	Exemplary Systems
Asym. STFT Dual Windows	2–8 ms	(Wang et al., 2021, Wang et al., 2022, Wood et al., 2019)
Deep/time-domain FBE	1–4 ms	(Zheng et al., 2022, Wu et al., 2024)
Token/quantized methods	1–5 ms	(Li et al., 10 Mar 2026, Xue et al., 2023)
Causal time-domain DNNs	1–3 ms	(Bae et al., 2024, Cheng et al., 2024)
Min-phase FIR (samplewise)	0.3–1.2 ms	(Dementyev et al., 2024)

3. Model Architectures and Complexity Management

Achieving sub-5 ms latency under resource constraints requires both minimal buffering and highly efficient model architectures:

Depthwise Separable and Pointwise Convolutions: Used to reduce compute (GMACs/s) while maintaining receptive field coverage (Romaniuk et al., 2020, Bae et al., 2024, Larraza et al., 21 Jan 2026).
Fast and Quantized Recurrence: Efficient gated RNN variants (e.g., FastGRNN) with state-drift correction mechanisms (Larraza et al., 21 Jan 2026).
Group-communication and Weight Sharing: Extensive weight sharing and group-communication (GCFSnet) for low-complexity, low-latency multichannel enhancement (Westhausen et al., 2023).
Separation of Spatial and Temporal Processing: Decoupled blocks for spatial filtering and LSTM-based temporal processing achieve both low complexity and 2 ms latency (Pandey et al., 2024).
State-Space Model Modulation: SlowFast frameworks modulate lightweight fast-branch SSMs with episodic summaries from a slow branch, allowing compute reductions of up to 70% at <2 ms latency (Cheng et al., 2024).
Model Pruning and Quantization: Structured pruning (e.g., SPDY+OBC) and per-layer quantization bring time-domain U-Nets with LSTM bottleneck to <0.3 GMAC and 3 ms latency on-die (Bae et al., 2024), while neural Wiener filters and recurrent DNNs achieve similar complexity/latency on multi-channel inputs (Hsieh et al., 2024).
Tokenization/Detokenization and Non-autoregressive Parallelism: Non-autoregressive, parallelized token-based systems enable latencies below 10 ms on commodity hardware (Li et al., 10 Mar 2026).

A prevalent pattern is the reduction of buffer-induced delay via architectural decisions, and the use of highly parallelizable, quantizable, and/or prunable network blocks aligned with targeted hardware.

4. Performance and Trade-offs: Latency, Complexity, and Enhancement Quality

The trade-off between latency, computational load, and enhancement quality is empirically characterized in all major ultra-low-latency studies:

Model Size vs. Window/Frame Size: Shrinking the frame/hop/window size raises computational demands per second and degrades enhancement unless compensated by scaling up model size (parameters) (Wu et al., 2024, Wang et al., 2023).
Complexity-Latency Tuning: Systems such as MASnet (Romaniuk et al., 2020) and D-LL-RNN (Pandey et al., 2024) demonstrate linear scaling of GMAC/s with decreasing latency; performance plateaus when further reductions create bottlenecks in either spatial or temporal context modeling.
Objective Metrics: PESQ, STOI, SI-SDR, and DNSMOS OVRL are most commonly reported. In all scenarios, meticulous balancing of model size and frame settings can recover enhancement performance at 2–4 ms latency to within ≲0.05–0.1 MOS or ≲0.3 dB SI-SDR of 20 ms baselines (Wu et al., 2024, Wang et al., 2023, Pandey et al., 2024).
Subjective and Real-data Testing: Ultra-low-latency models, when trained and evaluated on large open-domain datasets, generalize to unseen noise and reverberation conditions with only minor degradation compared to larger-latency systems (Wu et al., 2024, Hsieh et al., 2024, Dementyev et al., 2024).

System	Latency	SI-SDR (dB)	PESQ	DNSMOS OVRL	Compute
Deep FIR (Dementyev et al., 2024)	0.32–1.25 ms	4.1 (mean SI-SDRi)	–	2.01	388 MIPS
Wave-U-Net+LSTM pruned (Bae et al., 2024)	3 ms	–	3.01	3.90 (UTMOS)	0.21 GMAC/s
GCFSnet (uni/lowB/bin) (Westhausen et al., 2023)	2 ms	0–1.5 (SI-SDR)	1.18–1.23	–	0.36 GMAC/s
FSB-LSTM 6ch (Wang et al., 2023)	4 ms	7.8	2.61	–	3.4 GMAC/s
D-LL-RNN-128-8-8 (Pandey et al., 2024)	2 ms	5.8	2.60	–	3.67 GFLOP/s

5. Applications, Hardware Constraints, and Implementation

Targeted applications for ultra-low-latency speech enhancement demand strict power, memory, and compute budgets:

Hearing Aids / TWS Hearables: End-to-end latency must be below 5 ms (preferred 2–3 ms); compute must fit into 0.2–0.5 GMAC/s, model size ≪ 1 MB, with real-time fixed-point or SIMD-accelerated pipelines and quantization-aware training (Bae et al., 2024, Westhausen et al., 2023, Dementyev et al., 2024).
Telephony / Conferencing / AR/VR: Margins can be more relaxed, but end-to-end latency must remain sub-10 ms; model scaling and cross-device deployment highlighted in transmission+enhancement pipelines (Bokaei et al., 2024).
Embedded Implementation: Block-structured pruning, depthwise convolution, group-communication, and modular SSM/LSTM core blocks are proven strategies for matching MACs/s and RAM budget on commercial DSPs and ASICs (Bae et al., 2024, Cheng et al., 2024).

Characteristic implementation strategies include structured pruning with per-layer MAC budget optimization (SPDY+OBC (Bae et al., 2024)), fixed-point quantization, and delayed-feature links (for binaural hearing aids (Westhausen et al., 2023)).

6. Comparative Analysis and Research Frontiers

Cross-system evaluation shows:

Frequency-domain vs. Time-domain: Both can attain <5 ms latencies when using short synthesis windows or direct waveform modeling, with neither dominating across all hardware and quality metrics (Wu et al., 2024, Wang et al., 2022).
Non-autoregressive Tokenization vs. Masking: Token-based methods achieve similar MOS and SI-SDR at reduced compute and latency, especially when combined with parallelizable quantization steps (Li et al., 10 Mar 2026, Xue et al., 2023).
Resource-efficient Spatial-Temporal Decoupling: Multi-channel systems benefit from decoupling spatial and temporal modeling; dense group communication and element-wise fusion are key enablers (Pandey et al., 2024).

No single architecture is optimal for all constraints; design choices are dictated by the precise mix of latency, compute, memory, enhancement targets, and application-specific acoustic scenarios. A major research trend is the further reduction of latency toward the sample-level, matched to ever-stricter constraints from next-generation hearables and ultra-low-power processors (Cheng et al., 2024, Dementyev et al., 2024).

7. Key Open Problems and Directions

While practical ultra-low-latency enhancement is now achievable, the literature notes several ongoing challenges:

Maintaining Quality Under Extreme Constraints: Preserving enhancement performance and intelligibility at <1 ms latency and ≪1 MB model size, especially in non-stationary or unseen noise environments (Dementyev et al., 2024).
Long-term State Stability: Drift in RNN state over long sequences is an issue in efficient architectures; methods such as trainable complementary filters mitigate this (Larraza et al., 21 Jan 2026).
Generalization and Adaptivity: Ensuring robustness across users, speakers, and acoustic contexts under small-buffer and low-resourced conditions.
Evaluation Standardization: Past work is often confounded by inconsistent benchmarks, noise types, and model reporting. Unified large-scale real-world evaluation is necessary (Wu et al., 2024).

A plausible implication is that further progress will hinge on co-optimization of transforms, model sparsity/quantization, and on-chip/hardware-aware inference, tightly coupled to the constraints and opportunities of next-generation personal audio platforms.

References

(Dementyev et al., 2024) Dementyev, A., et al., "Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables," 2024.
(Bae et al., 2024) Kim, J., et al., "Speech Boosting: Low-Latency Live Speech Enhancement for TWS Earbuds," 2024.
(Cheng et al., 2024) Takamichi, S., et al., "Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement," 2024.
(Wu et al., 2024) Martin, R., et al., "Ultra-Low Latency Speech Enhancement - A Comprehensive Study," 2024.
(Larraza et al., 21 Jan 2026) Srivastava, A., et al., "Fast-ULCNet: A fast and ultra low complexity network for single-channel speech enhancement," 2026.
(Li et al., 10 Mar 2026) Shi, F., et al., "LL-SDR: Low-Latency Speech enhancement through Discrete Representations," 2026.
(Xue et al., 2023) Ren, S., et al., "Low-latency Speech Enhancement via Speech Token Generation," 2023.
(Pandey et al., 2024) Yang, S., et al., "Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement," 2024.
(Hsieh et al., 2024) Yang, S., et al., "On the Importance of Neural Wiener Filter for Resource Efficient Multichannel Speech Enhancement," 2024.
(Wang et al., 2023) Li, J., et al., "Neural Speech Enhancement with Very Low Algorithmic Latency and Complexity via Integrated Full- and Sub-Band Modeling," 2023.
(Westhausen et al., 2023) Ng, L., et al., "Low bit rate binaural link for improved ultra low-latency low-complexity multichannel speech enhancement in Hearing Aids," 2023.
(Zheng et al., 2022) Zheng, Y., et al., "Low-latency Monaural Speech Enhancement with Deep Filter-bank Equalizer," 2022.
(Wood et al., 2019) Leigh, K., et al., "Unsupervised Low Latency Speech Enhancement with RT-GCC-NMF," 2019.
(Romaniuk et al., 2020) Braun, S., et al., "Efficient Low-Latency Speech Enhancement with Mobile Audio Streaming Networks," 2020.
(Wang et al., 2021) Pandey, A., et al., "Deep neural network Based Low-latency Speech Separation with Asymmetric analysis-Synthesis Window Pair," 2021.
(Wang et al., 2022) Zhang, J., et al., "STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency," 2022.