Feasibility of Time-Domain DNN-Based Speech Enhancement on Embedded FPGA for Hearing Aid

Abstract: Hearing aids impose strict latency and power constraints that current DNN-based speech enhancement systems struggle to meet on embedded hardware. We characterize this gap by deploying both speech separation and denoising using the lightweight SuDoRM-RF++ architecture on the AMD-Xilinx Kria KV260, evaluated at FP32 and 16-bit fixed-point precision for each task. Across these configurations, first-sample latency tracks with on-chip parameter caching rather than arithmetic throughput, identifying data movement as the primary bottleneck. Precision reduction halves the model memory footprint without compromising objective speech quality. The fixed-point denoising accelerator achieves a first-sample latency of 9.7~ms, meeting the 10~ms clinical threshold, while speech separation reaches 16.0~ms. These measurements establish concrete resource requirements for embedded DNN-based speech enhancement and quantify the remaining gap to hearing aid deployment.

• The paper shows that fixed-point (F16) quantization achieves denoising within clinical latency (<10 ms) while halving memory requirements.
• The study employs the SuDoRM-RF++ architecture with causal convolutional operations to efficiently perform speech separation and denoising on an embedded AMD-Xilinx Kria KV260 FPGA.
• Hardware analysis reveals that aggressive on-chip parameter caching minimizes latency and energy-to-first-sample, informing future custom low-power designs.

Time-Domain DNN-Based Speech Enhancement on Embedded FPGA for Hearing Aid Applications

Introduction and Motivation

Hearing aids impose stringent constraints on latency, power, and memory, rendering the direct deployment of state-of-the-art DNN-based speech enhancement models highly challenging. Traditional DSP algorithms, while power-efficient and suitable for stationary noise, fail in rapidly changing, non-stationary environments due to their strong statistical modeling assumptions. Time-domain DNNs outperform classical and time-frequency domain methods but demand substantial computational and memory resources, outstripping what is available in hearing prosthetics, where total processing latency must not exceed 10 ms to avoid perceptual artifacts and ensure user comfort.

This work systematically investigates the feasibility of deploying efficient, time-domain speech enhancement models on an embedded FPGA (AMD-Xilinx Kria KV260), focusing on both speech separation and denoising tasks at varying numerical precisions. The study quantifies the algorithmic performance, hardware resource requirements, and, crucially, the achievable first-sample latency—the key metric in clinical deployment scenarios.

Model Architecture and Datasets

The SuDoRM-RF++ 0.25x architecture serves as the backbone for both speech separation and denoising tasks. This model utilizes exclusively time-domain, causal convolutional operations with a compact parameterization (1.5–1.6M parameters, $\approx$5.6–6.1 MB), supporting low-latency streaming and efficient hardware mapping. For separation, the model outputs two speech sources from a single-channel mixture; for denoising, it reconstructs a clean signal from noisy input using an appropriately adapted output head.

Training leverages WSJ0-2mix for separation and the Valentini-Botinhao corpus for denoising, with standard data augmentations and multi-resolution loss functions, aligning with current best practices.

Figure 1: Schematic of the end-to-end SuDoRM-RF++ speech enhancement pipeline, demonstrating both source separation (A) and denoising (B) model variants.

FPGA Implementation Methodology

Deployment centers on manual conversion of trained PyTorch models to HLS-compliant C++ for synthesis on the Kria KV260 PL fabric. Designs feature extensive pipelining, circular buffers for history/state management, and AXI interfaces for both streaming data and model parameters. The key hardware distinction lies in the placement of model weights—either on BRAM/URAM (on-chip) or external DDR4 memory—the central determinant of achievable inference latency.

Two numerical precisions are considered: full precision (FP32) and quantized fixed-point (F16; Q4.12). The F16 implementation halves memory needs and allows more aggressive parameter caching.

Algorithmic Performance

Speech separation and denoising models deliver competitive objective quality:

• Speech Separation: SI-SDRi $\approx$6 dB (SEP32/SEP16), STOI: 0.75, PESQ: 2.18. The F16 model matches or surpasses FP32 in quality and occupies half the memory.
• Speech Denoising: PESQ: 2.41, STOI: 0.93, COVL: 3.35, HASPI: 0.90. Denoising results indicate strong intelligibility and quality retention over unprocessed audio, with background noise suppression performance matching resource-intensive alternatives at a fraction of the model size.

No observable degradation is induced by F16 quantization, confirming robust resilience of the SuDoRM-RF++ architecture to reduced precision in this context, and supporting compact deployments for resource-constrained platforms.

Hardware Results: Resource Utilization and Latency

Separation Task

Iterative architectural refinements demonstrate that on-chip model parameter residency is the dominant factor controlling latency, rather than raw arithmetic throughput or DSP utilization:

• SEP32: First-sample latency reduces from 157.6 ms (all parameters off-chip) to 44 ms (aggressive BRAM/URAM caching).
• SEP16: Latency reaches 16.0 ms in the most aggressive design (SEP16-v2), solely limited by available BRAM/URAM.

Denoising Task

Single-output denoising reduces the memory bottleneck, allowing even more aggressive on-chip caching:

• DEN32: 41.2 ms latency with moderate on-chip parameter caching.
• DEN16: Achieves 9.7 ms, the only implementation crossing below the critical 10 ms clinical threshold for real-time auditory processing.

  Figure 2: First-sample latency comparison for best FPGA (FP32 and F16) and CPU baselines on the Kria KV260, referenced to the clinical 10 ms threshold; only DEN16 surpasses this criterion.

Comparative Perspective and Power Analysis

Relative to prior FPGA-based approaches, this implementation is the first DNN-based, time-domain speech denoising solution on a resource-constrained SoC-FPGA to report first-sample latency within clinical bounds. Competing solutions are either untimed, non-real-time (RTF>1), or operate at lower quality and/or higher resource usage.

Power consumption remains a limiting aspect for wearable deployment: FPGA implementation draws 3–4 W (on-chip), far exceeding the 1–3 mW envelope of hearing aids. However, analysis shows that aggressive parameter caching directly reduces energy-to-first-sample, with DEN16 consuming 41 mJ—nearly 4x more efficient than corresponding FP32 and earlier designs for the same task and qualitative outcome.

Implications and Future Directions

Practical and Theoretical Implications

• Latency Optimization: Success in meeting clinical latency constraints for denoising implicates on-chip memory provisioning and precision reduction as critical design levers for embedded DNN deployment.
• Precision Selection: Fixed-point quantization to 16-bit is sufficient for high-quality speech enhancement, but further gains from INT8 or pruning require quantization-aware retraining due to the spectral sensitivity of time-domain weights.
• Hardware Constraints: Current embedded FPGAs (e.g., KV260) are effective for feasibility studies but are not appropriate for direct hearing aid inclusion. Custom ASIC or ASIP designs must internalize on-chip state-budget/latency tradeoffs identified here.
• Separation vs. Denoising Task Difficulty: Denoising is more tractable for extreme resource-constrained streaming inference than multi-source separation, reflecting the relative algorithmic complexity (single vs. multiple output signals, label permutation issues, etc.).

Prospects for AI-Hearing Integration

• Model Compression: Structured pruning and sub-INT8 quantization are required for deploying source separation on hearing-scale silicon while meeting sub-10 ms latency.
• Binaural and Multi-Mic Models: Integrating spatial or multi-channel front ends may enable improved separation capabilities with minimal parameter increase.
• Custom Hardware: Architecture-aware compilers and cross-layer optimization, including co-design of model and hardware pipeline, will be crucial, as highlighted by the dependency on selective caching and pipelined structures.

Conclusion

This work demonstrates that state-of-the-art time-domain DNN-based speech denoising is feasible on current-generation resource-constrained embedded FPGAs at clinically acceptable latency. Precision reduction and maximal on-chip parameter residency are the primary drivers for latency and energy efficiency, with fixed-point denoising executing in 9.7 ms on the Kria KV260. These findings precisely quantify the resource, architectural, and algorithmic conditions necessary for future hearing-aid-ready, neural network-accelerated speech enhancement systems. Remaining challenges for source separation, multi-microphone processing, and ultra-low-power implementation persist but are now addressable within a well-characterized design envelope.

---

References

For full reference list, see original paper (2606.04221).