VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency (2101.03149v2)

Abstract: We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers. Whereas existing methods focus on learning the alignment between the speaker's lip movements and the sounds they generate, we propose to leverage the speaker's face appearance as an additional prior to isolate the corresponding vocal qualities they are likely to produce. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement, and generalizes well to challenging real-world videos of diverse scenarios. Our video results and code: http://vision.cs.utexas.edu/projects/VisualVoice/.

Overview of "VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency"

The paper "VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency" authored by Ruohan Gao and Kristen Grauman presents a novel approach for audio-visual (AV) speech separation. The core objective is to disentangle mixed speech signals in video data, isolating the speech associated with a specific face despite the presence of background noise or interference from other speakers. Unlike previous methods that primarily focus on lip-reading to align speaker lip movements with generated sounds, this paper introduces an innovative method that leverages both the speaker's lip motion and facial appearance to predict the vocal qualities likely produced.

Methodology

The proposed VisualVoice framework is a multi-task learning approach that jointly learns AV speech separation and cross-modal speaker embeddings from unlabeled video data. The main components of the framework include:

1. Audio-Visual Speech Separator Network: This network employs a combination of lip motion analysis and facial attribute analysis to generate features that guide the separation process. The authors utilize a U-Net architecture for processing audio inputs and a combination of 3D convolutions and ShuffleNet v2 for lip motion capture. A ResNet-18 network extracts static facial features, providing additional context about likely voice characteristics.
2. Cross-Modal Consistency: The paper argues that the correlation between the visual appearance of a speaker's face and their voice offers complementary information for isolation tasks. They deploy a triplet loss mechanism to learn robust face-voice embeddings, which enhances separation precision by ascertaining the facial attributes align with auditory properties.
3. Training Paradigm: The training employs a "mix-and-separate" paradigm, where speech segments are synthetically mixed to create training samples with known ground truth, facilitating effective supervision in model training. Importantly, despite the lack of identity labels, the network can separate speakers solely based on facial and vocal characteristics through learned embeddings.

Results

The VisualVoice approach demonstrates state-of-the-art performance across multiple benchmark datasets, achieving notable improvements in signal-to-distortion ratio (SDR), signal-to-interference ratio (SIR), and speech intelligibility metrics like PESQ and STOI. The proposed method significantly surpasses traditional and recent AV speech separation methods on challenging datasets such as VoxCeleb2, Mandarin, TCD-TIMIT, CUAVE, and LRS2.

The authors empirically validate that incorporating facial appearance cues alleviates the dependence on reliable lip motion, thus making the model robust to scenarios where lip movement data may be incomplete or unreliable due to occlusions or non-frontal views.

Implications and Future Directions

The approach outlined in this paper broadens the scope of audio-visual speech processing by integrating cross-modal embeddings into the separation task, enabling improved speaker verification capabilities without requiring labeled face-voice pairs. This innovation positions VisualVoice to potentially enhance applications in user authentication, transcription services in noisy environments, and assistive technologies for those with hearing impairments.

For future work, expanding the methodology to explicitly model more granular cross-modal attributes such as specific facial features and voice timbral properties, could further refine separation accuracy and open new applications in audio-visual learning contexts, including enhanced virtual meeting experiences and improved audio forensics.

This research enriches the domain of AV speech separation by demonstrating that sophisticated multi-task networks can exploit various cues within video data to achieve significantly higher separation efficacy, setting a benchmark for future innovations in the field.