2000 character limit reached
Let There Be Sound: Reconstructing High Quality Speech from Silent Videos
Published 29 Aug 2023 in eess.AS, cs.AI, cs.LG, and cs.SD | (2308.15256v2)
Abstract: The goal of this work is to reconstruct high quality speech from lip motions alone, a task also known as lip-to-speech. A key challenge of lip-to-speech systems is the one-to-many mapping caused by (1) the existence of homophenes and (2) multiple speech variations, resulting in a mispronounced and over-smoothed speech. In this paper, we propose a novel lip-to-speech system that significantly improves the generation quality by alleviating the one-to-many mapping problem from multiple perspectives. Specifically, we incorporate (1) self-supervised speech representations to disambiguate homophenes, and (2) acoustic variance information to model diverse speech styles. Additionally, to better solve the aforementioned problem, we employ a flow based post-net which captures and refines the details of the generated speech. We perform extensive experiments on two datasets, and demonstrate that our method achieves the generation quality close to that of real human utterance, outperforming existing methods in terms of speech naturalness and intelligibility by a large margin. Synthesised samples are available at our demo page: https://mm.kaist.ac.kr/projects/LTBS.
- Unsupervised Speech Recognition. In NeurIPS.
- Wav2Vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In NeurIPS.
- Statistical Parametric Speech Synthesis. In Proc. ICASSP.
- Analysis of Emotional Speech Prosody in Terms of Part of Speech Tags. In Proc. Interspeech.
- DistilHuBERT: Speech Representation Learning by Layer-Wise Distillation of Hidden-Unit BERT. In Proc. ICASSP.
- A Simple Framework for Contrastive Learning of Visual Representations. In Proc. ICML.
- Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations. In NeurIPS.
- Lip Reading Sentences in the Wild. In Proc. CVPR.
- An Audio-Visual Corpus for Speech Perception and Automatic Speech Recognition. The Journal of the Acoustical Society of America, 120(5): 2421–2424.
- Daubechies, I. 1988. Orthonormal Bases of Compactly Supported Wavelets. Communications on pure and applied mathematics, 41(7): 909–996.
- Parallel Tacotron: Non-Autoregressive and Controllable TTS. In Proc. ICASSP.
- Vid2Speech: Speech Reconstruction from Silent Video. In Proc. ICASSP.
- Exploring Wav2Vec 2.0 on Speaker Verification and Language Identification. In Proc. Interspeech.
- Signal Estimation from Modified Short-Time Fourier Transform. IEEE Transactions on acoustics, speech, and signal processing, 32(2): 236–243.
- Conformer: Convolution-Augmented Transformer for Speech Recognition. In Proc. Interspeech.
- CMT: Convolutional Neural Networks Meet Vision Transformers. In Proc. CVPR.
- Flow-Based Unconstrained Lip to Speech Generation. In Proc. AAAI.
- Deep Residual Learning for Image Recognition. In Proc. CVPR.
- HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units. IEEE/ACM Trans. on Audio, Speech, and Language Processing, 29: 3451–3460.
- Unit Selection in a Concatenative Speech Synthesis System Using a Large Speech Database. In Proc. ICASSP.
- 3D Convolutional Neural Networks for Human Action Recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 35(1): 221–231.
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. NAACL.
- FluentTTS: Text-Dependent Fine-Grained Style Control for Multi-Style TTS. In Proc. Interspeech.
- Fre-GAN: Adversarial Frequency-Consistent Audio Synthesis. In Proc. Interspeech.
- Lip to Speech Synthesis with Visual Context Attentional GAN. In NeurIPS.
- Lip-to-Speech Synthesis in the Wild with Multi-Task Learning. In Proc. ICASSP.
- Distinguishing Homophenes Using Multi-Head Visual-Audio Memory for Lip Reading. In Proc. AAAI.
- Hifi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis. In NeurIPS.
- Textless Speech Emotion Conversion using Discrete and Decomposed Representations. In Proc. EMNLP.
- Lipper: Synthesizing Thy Speech Using Multi-View Lipreading. In Proc. AAAI.
- On Generative Spoken Language Modeling from Raw Audio. Transactions of the Association for Computational Linguistics, 9: 1336–1354.
- Łańcucki, A. 2021. Fastpitch: Parallel Text-to-Speech with Pitch Prediction. In Proc. ICASSP.
- Reconstructing Intelligible Audio Speech from Visual Speech Features. In Proc. Interspeech.
- Generating Intelligible Audio Speech from Visual Speech. IEEE/ACM Trans. on Audio, Speech, and Language Processing, 25(9): 1751–1761.
- VoiceMixer: Adversarial Voice Style Mixup. In NeurIPS.
- HierSpeech: Bridging the Gap between Text and Speech by Hierarchical Variational Inference using Self-supervised Representations for Speech Synthesis. In NeurIPS.
- Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis. In Proc. AAAI.
- Completer: Incomplete Multi-view Clustering via Contrastive Prediction. In Proc. CVPR.
- DiffSinger: Singing Voice Synthesis via Shallow Diffusion Mechanism. In Proc. AAAI.
- Decoupled Weight Decay Regularization. In Proc. ICLR.
- Visual Speech Recognition for Multiple Languages in the Wild. Nature Machine Intelligence, 4(11): 930–939.
- pYIN: A Fundamental Frequency Estimator Using Probabilistic Threshold Distributions. In Proc. ICASSP.
- Midgley, N. 2006. New Technology Catches Hitler Off Guard. Telegraph.
- SVTS: Scalable Video-to-Speech Synthesis. In Proc. Interspeech.
- End-to-End Audiovisual Speech Recognition. In Proc. ICASSP.
- Speech Resynthesis from Discrete Disentangled Self-Supervised Representations. In Proc. Interspeech.
- Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech. In Proc. ICML.
- Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis. In Proc. CVPR.
- WaveGlow: A Flow-based Generative Network for Speech Synthesis. In Proc. ICASSP.
- Robust Speech Recognition via Large-Scale Weak Supervision. In Proc. ICML.
- FastSpeech 2: Fast and High-Quality End-to-End Text to Speech. In Proc. ICLR.
- PortaSpeech: Portable and High-Quality Generative Text-to-Speech. In NeurIPS.
- Revisiting Over-Smoothness in Text to Speech. In Proc. ACL.
- Natural TTS Synthesis by Conditioning Wavenet on Mel Spectrogram Predictions. In Proc. ICASSP.
- Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron. In Proc. ICML.
- Smith, J. 1987. When Lip Reading between the Lines Had the Subtitles Beat by a Long Sight. LA Times.
- Fully-Hierarchical Fine-grained Prosody Modeling for Interpretable Speech Synthesis. In Proc. ICASSP.
- Attention is All You Need. In NeurIPS.
- SUPERB: Speech Processing Universal Performance Benchmark. In Proc. Interspeech.
- Speech Prediction in Silent Videos Using Variational Autoencoders. In Proc. ICASSP.
- Parallel WaveGAN: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram. In Proc. ICASSP.
- Investigation of Enhanced Tacotron Text-to-Speech Synthesis Systems with Self-Attention for Pitch Accent Language. In Proc. ICASSP.
- Mixed-Phoneme BERT: Improving BERT with Mixed Phoneme and Sup-Phoneme Representations for Text to Speech. In Proc. Interspeech.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.