V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow

Abstract: In this paper, we introduce V2SFlow, a novel Video-to-Speech (V2S) framework designed to generate natural and intelligible speech directly from silent talking face videos. While recent V2S systems have shown promising results on constrained datasets with limited speakers and vocabularies, their performance often degrades on real-world, unconstrained datasets due to the inherent variability and complexity of speech signals. To address these challenges, we decompose the speech signal into manageable subspaces (content, pitch, and speaker information), each representing distinct speech attributes, and predict them directly from the visual input. To generate coherent and realistic speech from these predicted attributes, we employ a rectified flow matching decoder built on a Transformer architecture, which models efficient probabilistic pathways from random noise to the target speech distribution. Extensive experiments demonstrate that V2SFlow significantly outperforms state-of-the-art methods, even surpassing the naturalness of ground truth utterances. Code and models are available at: https://github.com/kaistmm/V2SFlow

• The paper introduces a novel speech decomposition strategy that isolates linguistic content, pitch, and speaker traits for enhanced video-to-speech synthesis.
• It employs HuBERT, VQ-VAE, and a pre-trained speaker model alongside a Rectified Flow Matching decoder to reconstruct natural and intelligible speech.
• Experimental results on LRS3-TED and LRS2-BBC demonstrate superior perceptual quality and reduced errors, underscoring its potential for real-world silent communication applications.

Insights into V2SFlow: Refining Video-to-Speech Conversion

The paper "V2SFlow: Video-to-Speech Generation with Speech Decomposition and Rectified Flow" presents a novel framework, V2SFlow, designed to address the challenges of Video-to-Speech (V2S) synthesis. The authors tackle the inherent variability and complexity present in speech signals by decomposing speech into distinct subspaces. This decomposition approach simplifies the synthesis problem, as it allows the model to separately estimate these subspaces from silent video frames, subsequently leading to higher quality and more intelligible speech synthesis.

Core Contributions and Methodology

The core of V2SFlow lies in its multi-faceted decomposition strategy and the subsequent reconstruction using advanced machine learning techniques. The paper identifies three primary attributes of speech decomposition: linguistic content, pitch, and speaker characteristics. This factorization aids in capturing the nuanced dynamics of speech generation, which is crucial for improving the system's generalization to unconstrained, real-world scenarios.

1. Content Estimation: The content, representing the linguistic information, is derived using the self-supervised speech model HuBERT. This approach avoids the need for additional textual annotations, enhancing the model's efficiency.
2. Pitch Extraction: A distinct approach is used to acquire pitch information by employing VQ-VAE to quantize features indicative of pitch from the audio, thus capturing essential prosodic elements.
3. Speaker Embedding: The extraction of speaker embeddings via a pre-trained speaker verification model allows the system to maintain speaker characteristics across the speech outputs, ensuring a coherent voice identity.

The reconstruction of the speech attributes into coherent audio is facilitated by a Rectified Flow Matching (RFM) decoder utilizing a Transformer architecture. This design is strengthened with the integration of Classifier-Free Guidance (CFG), which aids in refining the sampling trajectory and achieving high-quality speech with a minimal number of sampling steps.

Experimental Results and Comparisons

Through rigorous experimentation on datasets like LRS3-TED and LRS2-BBC, V2SFlow demonstrates superior performance over existing methods. It notably offers an increase in perceptual naturalness, even surpassing ground truth measures in some UTMOS assessments. The capability to produce speech that is indistinguishable or even better than the ground truth highlights its robustness in real-world applications.

The paper also provides comprehensive ablation studies, which reinforce the importance of each component within the V2SFlow system. Notably, the decomposition of speech attributes is shown to be critical for improving both intelligibility (as shown by reductions in WER) and naturalness (UTMOS improvements).

Implications and Future Directions

The advancements presented in V2SFlow have substantial implications for practical deployments where audio signals are not available or are perturbed, such as in silent communication applications or assistive technologies for speech impairments. The use of decomposition allows the system to address the ambiguity typically present in mapping visual to auditory data, enhancing the V2S pipeline's robustness and scalability.

Future research in this domain may explore further integrations with other modalities, such as visual gestures or contextual semantics, to enrich the depth and accuracy of synthesized speech. Additionally, further investigations into model optimization and efficiency could refine the system for real-time applications, expanding its applicability in live scenarios.

In summary, V2SFlow represents a significant step towards more accurate and natural V2S synthesis, providing a foundational framework for future innovations in this burgeoning field.