An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation (2008.09586v2)

Abstract: Speech enhancement and speech separation are two related tasks, whose purpose is to extract either one or more target speech signals, respectively, from a mixture of sounds generated by several sources. Traditionally, these tasks have been tackled using signal processing and machine learning techniques applied to the available acoustic signals. Since the visual aspect of speech is essentially unaffected by the acoustic environment, visual information from the target speakers, such as lip movements and facial expressions, has also been used for speech enhancement and speech separation systems. In order to efficiently fuse acoustic and visual information, researchers have exploited the flexibility of data-driven approaches, specifically deep learning, achieving strong performance. The ceaseless proposal of a large number of techniques to extract features and fuse multimodal information has highlighted the need for an overview that comprehensively describes and discusses audio-visual speech enhancement and separation based on deep learning. In this paper, we provide a systematic survey of this research topic, focusing on the main elements that characterise the systems in the literature: acoustic features; visual features; deep learning methods; fusion techniques; training targets and objective functions. In addition, we review deep-learning-based methods for speech reconstruction from silent videos and audio-visual sound source separation for non-speech signals, since these methods can be more or less directly applied to audio-visual speech enhancement and separation. Finally, we survey commonly employed audio-visual speech datasets, given their central role in the development of data-driven approaches, and evaluation methods, because they are generally used to compare different systems and determine their performance.

Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation

The paper "An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation" presents a comprehensive review of techniques for extracting target speech signals from complex acoustic environments. Speech enhancement (SE) and speech separation (SS) are critical tasks in automatic speech recognition (ASR) and hearing aid development. Traditional methods rely on signal processing frameworks and statistical models. However, leveraging visual information such as lip movements and facial expressions—intrinsically resistant to acoustic noise—has demonstrated significant potential in enhancing speech processing tasks.

Deep Learning Approaches in Audio-Visual Systems

The integration of audio and visual information, optimized through deep learning, forms the cornerstone of this domain. Deep learning models provide a data-driven, flexible framework to fuse multimodal signals. This paper systematically surveys various elements characterizing audio-visual speech systems, including:

• Acoustic and Visual Features: Explored are robust feature extraction methods—ranging from raw waveform and phase information to embeddings and landmarks—to provide rich data representations.
• Deep Learning Architectures: Techniques employing convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their combinations are highlighted for their efficacy in SE and SS.
• Fusion Techniques: The paper discusses diverse fusion methodologies, from early to intermediate fusion, spotlighting the integration of acoustic and visual cues within neural network layers to enhance signal processing.
• Training Targets and Objectives: A variety of training paradigms, such as direct mapping and mask approximation, are reviewed alongside regularization methods like permutation invariant training and multitask learning.

Data Resources and Evaluation

A crucial section is dedicated to current datasets and performance evaluation methodologies. It insists on large-scale dataset availability like VoxCeleb2 for training robust models and common assessment metrics like PESQ and SDR for objective evaluation.

Implications and Future Directions

The research emphasizes the transformative impact of audio-visual processing, particularly in mitigating issues inherent to audio-only approaches such as the "cocktail party problem." The significantly reduced source permutation ambiguity in SS due to visual input exemplifies this benefit. Furthermore, the paper anticipates future research directions, suggesting:

1. Integration of Real-World Conditions: Emphasizes datasets reflecting genuine acoustic contexts to close the simulation-reality gap.
2. Advanced Fusion Techniques: Proposes exploration of more sophisticated methods, such as attention mechanisms, to address modality dominance issues elegantly.
3. End-to-End System Development: Highlights the potential of fully integrated systems that process raw inputs, streamlining architecture complexity and enhancing generalizability.

The paper's systematic review provides a clear lens on the current landscape of deep-learning-based audio-visual speech enhancement and separation, highlighting key challenges and promising research avenues in achieving seamless human-machine interaction via improved speech intelligibility and quality.