Dynamic Noise Embedding: Noise Aware Training and Adaptation for Speech Enhancement

Abstract: Estimating noise information exactly is crucial for noise aware training in speech applications including speech enhancement (SE) which is our focus in this paper. To estimate noise-only frames, we employ voice activity detection (VAD) to detect non-speech frames by applying optimal threshold on speech posterior. Here, the non-speech frames can be regarded as noise-only frames in noisy signal. These estimated frames are used to extract noise embedding, named dynamic noise embedding (DNE), which is useful for an SE module to capture the characteristic of background noise. The DNE is extracted by a simple neural network, and the SE module with the DNE can be jointly trained to be adaptive to the environment. Experiments are conducted on TIMIT dataset for single-channel denoising task and U-Net is used as a backbone SE module. Experimental results show that the DNE plays an important role in the SE module by increasing the quality and the intelligibility of corrupted signal even if the noise is non-stationary and unseen in training. In addition, we demonstrate that the DNE can be flexibly applied to other neural network-based SE modules.