FastEnhancer: Real-Time Enhancement

• FastEnhancer is a family of architectures optimized for real-time audio and image enhancement by minimizing architectural overhead and enabling causal, streamable inference.
• It utilizes a unique RNNFormer block that combines unidirectional GRU for temporal modeling and multi-head self-attention for frequency analysis, avoiding frame-caching delays.
• Empirical results demonstrate state-of-the-art real-time factors and high-quality enhancement metrics, ideal for applications like online meetings, smart devices, and hearing aids.

FastEnhancer designates a family of architectures and engineering principles for speed-optimized enhancement of audio or image signals, with explicit focus on ultra-low processing latency, resource frugality, and deployment feasibility for real-time streaming tasks. The term is currently most prominently associated with streaming neural speech enhancement and real-time perceptual image enhancement models, especially those integrating minimalistic structures with highly efficient deep learning blocks.

1. Definitional Overview and Theoretical Underpinnings

FastEnhancer, as formalized in "FastEnhancer: Speed-Optimized Streaming Neural Speech Enhancement" (Ahn et al., 26 Sep 2025), refers to architectures engineered to minimize wall-clock latency for real-time enhancement, typically in online meetings, smart home devices, and hearing aids. The fundamental theoretical strategy is twofold:

• Aggressive reduction of architectural overhead (e.g., restricting kernel size along time axes; minimizing high-dimensional sequential computation).
• Strategic use of modules that facilitate parallelization and cache-less computation, most notably via the RNNFormer block—the critical building unit which melds unidirectional GRU processing along the temporal axis with multi-head self-attention (MHSA) along the frequency axis.

FastEnhancer models strictly avoid architectural choices that introduce unduly large context windows or require persistent frame-caching, thus ensuring causal and streamable inference.

2. Architectural Principles and Block Structures

The primary FastEnhancer architecture exemplifies an encoder–decoder model. The input audio $x$ undergoes short-time Fourier transform (STFT), yielding a complex spectrogram $X$. Preprocessing includes power compression: $X_c = |X|^c \cdot e^{j \cdot \text{angle}(X)}$ with $c=0.3$, enhancing dynamic range characteristics akin to the auditory system.

The encoder commences with a strided convolution reducing input frequency resolution from $N_{\text{fft}}/2$ to $N_{\text{fft}}/8$ and inflating channel dimension from 2 to $C_1$. A stack of $L$ encoder blocks further abstracts features. The decoder reconstructs full resolution using a symmetrical series of decoder blocks and strided transposed convolutions; skip connections are used throughout to preserve fine-grained details.

A unique architectural element is the Pre-/Post-RNNFormer layer sequence:

• Pre-RNNFormer: linear filterbank initialization, followed by convolution to compress channel count.
• Post-RNNFormer: reverses these operations while ensuring recovery of spatial structure for waveform synthesis.

RNNFormer Block

The RNNFormer block contains:

• Time-axis module: unidirectional GRU to model temporal dependencies causally.
• Frequency-axis module: four-headed MHSA block capturing global frequency relationships.

Each module includes convolutional processing, batch normalization (optimized for inference via fusion), and a residual skip connection. The output mask $M$ is predicted and applied multiplicatively to $X_c$ prior to inverse processing for waveform reconstruction.

LaTeX schematic: $$\text{Time module: } h_t = \text{GRU}(x_t) \ \text{Frequency module: } z_f = \text{MHSA}(h) \ \text{Block output: } y = x + \text{Conv}(\text{BN}(z_f))$$

3. Performance Metrics and Empirical Evaluation

FastEnhancer is quantitatively assessed using both speech quality and intelligibility metrics:

• DNSMOS (P.808, P.835): non-intrusive speech quality assessment
• PESQ: perceptual evaluation of speech quality
• SCOREQ: reference-based natural speech assessment
• SISDR: scale-invariant signal-to-distortion ratio
• STOI, ESTOI: short-time objective intelligibility indexes
• WER: downstream word error rates with ASR engines (e.g., Whisper)

Processing speed is reported as real-time factor (RTF), measured in seconds per second of audio on single-thread CPU. FastEnhancer achieves state-of-the-art RTF values, especially in compact configurations (e.g., FastEnhancer-T), outperforming prior neural speech enhancement models.

4. Technical Innovations Enabling Fast Processing

Key FastEnhancer technical innovations include:

• Hybrid Block Design: Time-axis GRU delivers low-latency sequential modeling; frequency-axis MHSA enables concurrent global band interactions.
• Latency-Aware Convolutions: All convolutions along time axis have kernel size one, obviating streaming frame caching and yielding immediate output per frame.
• BatchNorm Fusion: BN layers (unlike LayerNorm) are fused at inference, removing redundant computation and reducing runtime operations.
• Mask Application: Final output synthesizes enhanced audio through element-wise mask multiplication and inverse STFT.

Other existing models leverage sub-band or dual-path designs to reduce MAC operations; FastEnhancer achieves comparable or superior results via architectural simplicity.

5. Implementation and Deployment Considerations

FastEnhancer codebase and pre-trained models are available (https://github.com/aask1357/fastenhancer). The model runs efficiently on commodity CPUs (Intel Xeon Gold, Apple M1) and is compatible with inference via ONNXRuntime, facilitating industrial deployment.

Training employs a composite loss: $$\mathcal{L} = \lambda_1 \mathcal{L}_{\text{mag}} + \lambda_2 \mathcal{L}_{\text{comp}} + \lambda_3 \mathcal{L}_{\text{con}} + \lambda_4 \mathcal{L}_{\text{wav}} + \lambda_5 \mathcal{L}_{\text{pesq}}$$ where $\mathcal{L}_{\text{mag}}$ is magnitude loss, $\mathcal{L}_{\text{comp}}$ is complex spectrogram loss, $\mathcal{L}_{\text{con}}$ is consistency loss, $\mathcal{L}_{\text{wav}}$ is waveform loss, and $\mathcal{L}_{\text{pesq}}$ is differentiable PESQ loss (with typical weights $\lambda_1 = 0.3$, $\lambda_2 = 0.2$, etc.).

6. Real-World and Industrial Applications

FastEnhancer is optimized for:

• Streaming noise suppression in online meetings and conferencing
• Smart speaker and home appliance audio enhancement
• Low-power, near-real-time hearing aid processing

Every layer and block is designed for causal, on-device inference with constrained memory, ensuring viable deployment on hardware without specialized accelerators.

7. Limitations and Scope of FastEnhancer Designs

While FastEnhancer models have set new baselines for speech enhancement latency and efficiency, architectural minimalism may entail upper bounds on expressivity for extremely degraded signals or highly nonlinear tasks. Nonetheless, the design choices reflect a principled trade-off between resource consumption and enhancement quality, steering the field toward practical, deployable enhancement under real-world constraints.

FastEnhancer architectures assert a new paradigm for speed-centric neural enhancement, achieving state-of-the-art results in both processing efficiency and perceptual metrics for real-time applications (Ahn et al., 26 Sep 2025).