---
title: Ultra-Low Latency Speech Enhancement
url: https://www.emergentmind.com/topics/ultra-low-latency-speech-enhancement
type: topic
---

# Ultra-Low Latency Speech Enhancement

Ultra-low-latency speech enhancement refers to algorithmic pipelines and model architectures engineered to process noisy speech signals and return an intelligible, high-quality enhanced version with an end-to-end processing delay constrained to a few milliseconds or less. Such stringent latency requirements are dictated by immersive, interactive domains including hearing assistive devices, true wireless stereo (TWS) earbuds, telepresence, and AR/VR, wherein total processing delays above 5–10 ms begin to degrade usability, speech intelligibility, and user comfort. The design of these systems is shaped by a multifaceted optimization problem spanning algorithmic latency, model footprint, computational complexity, and enhancement fidelity, under power and memory constraints typical of embedded hardware.

## 1. Fundamental Latency Constraints and Measurement

Ultra-low-latency regimes are typically defined as algorithmic delays below 5 ms, with recent literature targeting 1–3 ms or even sub-millisecond (sample-level) operation [2409.18239], [2409.18705], [2411.02019]. The precise measurement of end-to-end latency ($L_\text{total}$) must account for:

- **Analysis and Synthesis Windows:** In transform-domain methods, algorithmic latency is dictated by the synthesis (output) window length $L_s$, not the analysis (input) window: 
  $$
  L_\text{algo} = \frac{L_s - P}{f_s}
  $$
  where $P$ is hop size and $f_s$ is sampling rate [2409.10358].
- **Internal Model Buffering/Lookahead:** Any explicit future-frame context or model-induced delay.
- **Hardware Processing Time:** The time required for model inference, typically depending on MACs/s and the available FLOPs.

Many studies report both the **algorithmic** and **total (hardware + algorithmic)** latency, with a hard upper bound (often 4 or 5 ms) imposed by application domain requirements [2409.18705],[2307.08858].

## 2. Signal Representation and Architectural Strategies

A central design axis in ultra-low-latency speech enhancement is the choice of forward/inverse transforms and model topology. The most prominent approaches include:

- **Short-Time Fourier Transform (STFT) with Asymmetric Windows:** Asymmetric analysis-synthesis windowing enables high-frequency-resolution analysis with ultra-short, low-latency synthesis (e.g., 32 ms analysis, 4–8 ms synthesis). This reduces algorithmic latency to the synthesis window length while retaining mask learning performance [2106.11794], [2204.09911], [2409.10358], [1904.03130].
- **Learnable or Adaptive Filterbanks:** Deep filter-bank equalizers and adaptive trainable filterbanks provide shorter fixed-latency synthesis and allow the network to approximate frequency-domain manipulation in the time domain [2202.06764], [2409.10358].
- **Time-domain End-to-end Models:** Causal U-Nets with LSTM or SSM bottlenecks operate on raw waveform frames (1–3 ms), avoiding transform buffer delays entirely [2409.18705], [2411.02019].
- **Token-based Approaches:** Autoregressive or non-autoregressive speech generation via discrete audio tokens or vector quantized codes provides direct mapping from noisy to clean speech codes with minimal lookahead [2603.20242], [2310.08981].
- **Minimum-phase FIR Filtering:** In sample-by-sample enhancement, minimum-phase FIR filters generated by LSTM models are updated every millisecond or less, achieving mean algorithmic latency as low as 0.32 ms in hardware [2409.18239].

A summary comparison of representation strategies and minimum achievable algorithmic latency is given below:

| Approach                   | Min. Algo. Latency | Exemplary Systems                    |
|----------------------------|:------------------:|--------------------------------------|
| Asym. STFT Dual Windows    | 2–8 ms            | [2106.11794], [2204.09911], [1904.03130] |
| Deep/time-domain FBE       | 1–4 ms            | [2202.06764], [2409.10358]           |
| Token/quantized methods    | 1–5 ms            | [2603.20242], [2310.08981]           |
| Causal time-domain DNNs    | 1–3 ms            | [2409.18705], [2411.02019]           |
| Min-phase FIR (samplewise) | 0.3–1.2 ms        | [2409.18239]                         |


## 3. Model Architectures and Complexity Management

Achieving sub-5 ms latency under resource constraints requires both minimal buffering and highly efficient model architectures:

- **Depthwise Separable and Pointwise Convolutions:** Used to reduce compute (GMACs/s) while maintaining receptive field coverage [2008.07244], [2409.18705], [2601.14925].
- **Fast and Quantized Recurrence:** Efficient gated RNN variants (e.g., FastGRNN) with state-drift correction mechanisms [2601.14925].
- **Group-communication and Weight Sharing:** Extensive weight sharing and group-communication (GCFSnet) for low-complexity, low-latency multichannel enhancement [2307.08858].
- **Separation of Spatial and Temporal Processing:** Decoupled blocks for spatial filtering and LSTM-based temporal processing achieve both low complexity and 2 ms latency [2401.07879].
- **State-Space Model Modulation:** SlowFast frameworks modulate lightweight fast-branch SSMs with episodic summaries from a slow branch, allowing compute reductions of up to 70% at <2 ms latency [2411.02019].
- **Model Pruning and Quantization:** Structured pruning (e.g., SPDY+OBC) and per-layer quantization bring time-domain U-Nets with LSTM bottleneck to <0.3 GMAC and 3 ms latency on-die [2409.18705], while neural Wiener filters and recurrent DNNs achieve similar complexity/latency on multi-channel inputs [2401.07882].
- **Tokenization/Detokenization and Non-autoregressive Parallelism:** Non-autoregressive, parallelized token-based systems enable latencies below 10 ms on commodity hardware [2603.20242].

A prevalent pattern is the reduction of buffer-induced delay via architectural decisions, and the use of highly parallelizable, quantizable, and/or prunable network blocks aligned with targeted hardware.

## 4. Performance and Trade-offs: Latency, Complexity, and Enhancement Quality

The trade-off between latency, computational load, and enhancement quality is empirically characterized in all major ultra-low-latency studies:

- **Model Size vs. Window/Frame Size:** Shrinking the frame/hop/window size raises computational demands per second and degrades enhancement unless compensated by scaling up model size (parameters) [2409.10358], [2304.08707].
- **Complexity-Latency Tuning:** Systems such as MASnet [2008.07244] and D-LL-RNN [2401.07879] demonstrate linear scaling of GMAC/s with decreasing latency; performance plateaus when further reductions create bottlenecks in either spatial or temporal context modeling.
- **Objective Metrics:** PESQ, STOI, SI-SDR, and DNSMOS OVRL are most commonly reported. In all scenarios, meticulous balancing of model size and frame settings can recover enhancement performance at 2–4 ms latency to within ≲0.05–0.1 MOS or ≲0.3 dB SI-SDR of 20 ms baselines [2409.10358], [2304.08707], [2401.07879].
- **Subjective and Real-data Testing:** Ultra-low-latency models, when trained and evaluated on large open-domain datasets, generalize to unseen noise and reverberation conditions with only minor degradation compared to larger-latency systems [2409.10358], [2401.07882], [2409.18239].

| System                                   | Latency | SI-SDR (dB) | PESQ | DNSMOS OVRL | Compute      |
|-------------------------------------------|:-------:|:-----------:|:----:|:------------:|:------------:|
| Deep FIR [2409.18239]                     | 0.32–1.25 ms | 4.1 (mean SI-SDRi) | –    | 2.01         | 388 MIPS     |
| Wave-U-Net+LSTM pruned [2409.18705]       | 3 ms    | –           | 3.01 | 3.90 (UTMOS) | 0.21 GMAC/s  |
| GCFSnet (uni/lowB/bin) [2307.08858]       | 2 ms    | 0–1.5 (SI-SDR) | 1.18–1.23 | –  | 0.36 GMAC/s  |
| FSB-LSTM 6ch [2304.08707]                 | 4 ms    | 7.8         | 2.61 | –           | 3.4 GMAC/s   |
| D-LL-RNN-128-8-8 [2401.07879]             | 2 ms    | 5.8         | 2.60 | –           | 3.67 GFLOP/s |

## 5. Applications, Hardware Constraints, and Implementation

Targeted applications for ultra-low-latency speech enhancement demand strict power, memory, and compute budgets:

- **Hearing Aids / TWS Hearables:** End-to-end latency must be below 5 ms (preferred 2–3 ms); compute must fit into 0.2–0.5 GMAC/s, model size ≪ 1 MB, with real-time fixed-point or SIMD-accelerated pipelines and quantization-aware training [2409.18705], [2307.08858], [2409.18239].
- **Telephony / Conferencing / AR/VR:** Margins can be more relaxed, but end-to-end latency must remain sub-10 ms; model scaling and cross-device deployment highlighted in transmission+enhancement pipelines [2404.19375].
- **Embedded Implementation:** Block-structured pruning, depthwise convolution, group-communication, and modular SSM/LSTM core blocks are proven strategies for matching MACs/s and RAM budget on commercial DSPs and ASICs [2409.18705], [2411.02019].

Characteristic implementation strategies include structured pruning with per-layer MAC budget optimization (SPDY+OBC [2409.18705]), fixed-point quantization, and delayed-feature links (for binaural hearing aids [2307.08858]).

## 6. Comparative Analysis and Research Frontiers

Cross-system evaluation shows:

- **Frequency-domain vs. Time-domain:** Both can attain <5 ms latencies when using short synthesis windows or direct waveform modeling, with neither dominating across all hardware and quality metrics [2409.10358], [2204.09911].
- **Non-autoregressive Tokenization vs. Masking:** Token-based methods achieve similar MOS and SI-SDR at reduced compute and latency, especially when combined with parallelizable quantization steps [2603.20242], [2310.08981].
- **Resource-efficient Spatial-Temporal Decoupling:** Multi-channel systems benefit from decoupling spatial and temporal modeling; dense group communication and element-wise fusion are key enablers [2401.07879].

No single architecture is optimal for all constraints; design choices are dictated by the precise mix of latency, compute, memory, enhancement targets, and application-specific acoustic scenarios. A major research trend is the further reduction of latency toward the sample-level, matched to ever-stricter constraints from next-generation hearables and ultra-low-power processors [2411.02019], [2409.18239].

## 7. Key Open Problems and Directions

While practical ultra-low-latency enhancement is now achievable, the literature notes several ongoing challenges:

- **Maintaining Quality Under Extreme Constraints:** Preserving enhancement performance and intelligibility at <1 ms latency and ≪1 MB model size, especially in non-stationary or unseen noise environments [2409.18239].
- **Long-term State Stability:** Drift in RNN state over long sequences is an issue in efficient architectures; methods such as trainable complementary filters mitigate this [2601.14925].
- **Generalization and Adaptivity:** Ensuring robustness across users, speakers, and acoustic contexts under small-buffer and low-resourced conditions.
- **Evaluation Standardization:** Past work is often confounded by inconsistent benchmarks, noise types, and model reporting. Unified large-scale real-world evaluation is necessary [2409.10358].

A plausible implication is that further progress will hinge on co-optimization of transforms, model sparsity/quantization, and on-chip/hardware-aware inference, tightly coupled to the constraints and opportunities of next-generation personal audio platforms.

---

**References**

- [2409.18239] Dementyev, A., et al., "Towards Sub-millisecond Latency Real-Time Speech Enhancement Models on Hearables," 2024.
- [2409.18705] Kim, J., et al., "Speech Boosting: Low-Latency Live Speech Enhancement for TWS Earbuds," 2024.
- [2411.02019] Takamichi, S., et al., "Modulating State Space Model with SlowFast Framework for Compute-Efficient Ultra Low-Latency Speech Enhancement," 2024.
- [2409.10358] Martin, R., et al., "Ultra-Low Latency Speech Enhancement - A Comprehensive Study," 2024.
- [2601.14925] Srivastava, A., et al., "Fast-ULCNet: A fast and ultra low complexity network for single-channel speech enhancement," 2026.
- [2603.20242] Shi, F., et al., "LL-SDR: Low-Latency Speech enhancement through Discrete Representations," 2026.
- [2310.08981] Ren, S., et al., "Low-latency Speech Enhancement via Speech Token Generation," 2023.
- [2401.07879] Yang, S., et al., "Decoupled Spatial and Temporal Processing for Resource Efficient Multichannel Speech Enhancement," 2024.
- [2401.07882] Yang, S., et al., "On the Importance of Neural Wiener Filter for Resource Efficient Multichannel Speech Enhancement," 2024.
- [2304.08707] Li, J., et al., "Neural Speech Enhancement with Very Low Algorithmic Latency and Complexity via Integrated Full- and Sub-Band Modeling," 2023.
- [2307.08858] Ng, L., et al., "Low bit rate binaural link for improved ultra low-latency low-complexity multichannel speech enhancement in Hearing Aids," 2023.
- [2202.06764] Zheng, Y., et al., "Low-latency Monaural Speech Enhancement with Deep Filter-bank Equalizer," 2022.
- [1904.03130] Leigh, K., et al., "Unsupervised Low Latency Speech Enhancement with RT-GCC-NMF," 2019.
- [2008.07244] Braun, S., et al., "Efficient Low-Latency Speech Enhancement with Mobile Audio Streaming Networks," 2020.
- [2106.11794] Pandey, A., et al., "Deep neural network Based Low-latency Speech Separation with Asymmetric analysis-Synthesis Window Pair," 2021.
- [2204.09911] Zhang, J., et al., "STFT-Domain Neural Speech Enhancement with Very Low Algorithmic Latency," 2022.

Source: https://www.emergentmind.com/topics/ultra-low-latency-speech-enhancement