Vid2Voc: Video-to-Speech Synthesis
- Vid2Voc is a family of audiovisual speech generation tasks that convert visual articulatory cues into corresponding acoustic signals.
- Techniques range from regression-based LPC synthesis to modular spectrogram prediction and hierarchical token diffusion, enhancing intelligibility and expressiveness.
- Multimodal approaches integrate video, text, and reference audio to control content, speaker identity, and emotion in applications like visual voice cloning and movie dubbing.
Vid2voc denotes a family of audiovisual speech generation problems in which visual input constrains or determines vocal output. In the narrow sense, it refers to video-to-speech synthesis from silent talking-face video, where a model predicts an acoustic representation or waveform from facial motion alone (Ephrat et al., 2017, Michelsanti et al., 2020, Mira et al., 2022, Ye et al., 17 Apr 2026). In a broader usage found in visual voice cloning and movie dubbing, the task additionally conditions on text and a reference audio, using video to specify emotion, speaking rate, and lip synchronization rather than lexical content (Chen et al., 2021, Cong et al., 2022). Across these formulations, the central technical problem is the same: mapping visually observable articulatory and affective cues to a speech signal whose content, identity, prosody, and timing are only partially determined by the video stream.
1. Task scope and conceptual variants
The literature uses closely related names for distinct formulations. Silent-video systems take a talking-face clip with no audio and synthesize speech directly from visual evidence. Visual Voice Cloning (V2C) and movie dubbing systems take a triplet and must preserve the voice from while matching the emotion implied by (Chen et al., 2021). Voice conversion systems, by contrast, are audio-to-audio models; they are adjacent to vid2voc research but are not themselves video-conditioned. One paper states this explicitly for O_O-VC, noting that it is “not a video-to-voice conversion (“vid2voc”) method” (Tu et al., 10 Oct 2025).
| Formulation | Inputs | Representative work |
|---|---|---|
| Video-to-speech / lip-to-speech | Silent video | (Ephrat et al., 2017, Michelsanti et al., 2020, Mira et al., 2022, Ye et al., 17 Apr 2026) |
| Visual Voice Cloning / movie dubbing | Text + reference audio + reference video | (Chen et al., 2021, Cong et al., 2022) |
| Related voice conversion | Source audio + target-speaker conditioning | (Tu et al., 10 Oct 2025, Maman et al., 14 Jul 2026) |
This taxonomy matters because the ambiguity structure differs across settings. In silent-video reconstruction, the model must infer linguistic content, voicing, pitch, and timbre from visual evidence that is intrinsically incomplete. In V2C, lexical content is supplied by text and target identity by reference audio, so video primarily governs emotion, prosody, and synchrony. The V2C formulation therefore emphasizes disentangling identity and emotion across different modalities, whereas silent-video systems emphasize articulatory-to-acoustic inversion (Chen et al., 2021).
2. Early silent-video speech reconstruction
Early vid2voc systems treated the problem as regression from visual frames to compact acoustic features. “Vid2speech: Speech Reconstruction from Silent Video” predicts an 18-dimensional per-frame speech target derived from 8th-order LPC analysis converted to Line Spectrum Pairs (LSPs). Audio is downsampled to 8 kHz, framed into 40 ms windows with 20 ms overlap, and each video frame is associated with two successive overlapping audio feature vectors, concatenated into the 18-dimensional target. The model uses a temporal context window of grayscale frames, resized to , and maps them to with a VGG-style CNN comprising five consecutive conv3-conv3-maxpool blocks, two fully connected layers with 512 neurons each, and a final output layer of size 18. Training uses mean squared error, and synthesis is performed by an LPC-based synthesizer driven by Gaussian white noise excitation (Ephrat et al., 2017).
That system established several empirical points. On speaker S4 of GRID, it reported audio-only intelligibility of 82.6% and audio-visual intelligibility of 79.9%, compared with 40.0% and 51.9% for the Le Cornu and Milner baseline; for out-of-vocabulary evaluation, it reported 51.6% audio-visual accuracy versus a 10.0% chance level. It also found that using the entire face rather than only the mouth yielded roughly 40% lower error, implying that cues beyond the lips contribute to acoustic reconstruction (Ephrat et al., 2017).
“Vocoder-Based Speech Synthesis from Silent Videos” moved from LPC/LSP regression to structured vocoder prediction. Its vid2voc pipeline is silent video frames visual encoder temporal model acoustic feature predictors 0 WORLD vocoder synthesis. A 3D convolutional visual encoder processes a 7-frame context window, a single-layer GRU with hidden size 128 models temporal structure, and separate decoders predict spectral envelope, aperiodicity, fundamental frequency, and voiced/unvoiced state. The system also includes an auxiliary visual speech recognition branch trained with CTC, and it evaluates four variants: vid2voc-M, vid2voc-F, vid2voc-M-VSR, and vid2voc-F-VSR (Michelsanti et al., 2020).
The shift to full WORLD parameterization made the synthesis problem more structured. In the speaker-dependent setting on GRID, vid2voc-M-VSR reported PESQ 1.90, ESTOI 0.455, and WER 15.1%, compared with PESQ 1.71 and ESTOI 0.329 for the Vougioukas et al. baseline. In the speaker-independent setting, performance remained substantially harder, with vid2voc-M-VSR at PESQ 1.23, ESTOI 0.227, and WER 51.6%. The paper reports that adding the VSR decoder improves both PESQ and ESTOI in almost all conditions, while mouth-only input often performs better than full-face input for reconstruction, with full-face input sometimes slightly improving WER. This indicates that the choice of visual crop interacts with the target objective rather than admitting a single universally best preprocessing strategy (Michelsanti et al., 2020).
3. Scalable spectrogram prediction and hierarchical discrete generation
A later line of work replaced direct waveform or vocoder training with modular spectrogram prediction. “SVTS: Scalable Video-to-Speech Synthesis” is a two-stage system consisting of video-to-spectrogram prediction and spectrogram-to-waveform synthesis with a neural vocoder. Its visual frontend uses a 3D convolutional stem followed by ResNet-18, temporal dependencies are modeled by a Conformer, and a frozen speaker encoder supplies an external speaker embedding concatenated with the video features before the Conformer. The predictor outputs a hidden vector of size 320 per video frame, reshaped into 4 × 80 spectrogram frames, matching 20 fps video and 80 fps spectrogram extraction. Training uses a combination of 1 loss and spectral convergence loss, and waveform synthesis is performed by a frozen Parallel WaveGAN vocoder pre-trained on LibriTTS for 1 million iterations (Mira et al., 2022).
The central contribution of SVTS is scalability. It evaluates on GRID, LRW, LRS3, and LRS3 + English-only VoxCeleb2, totaling about 1,550 hours for the largest training setup. On GRID in the seen-speaker setting, SVTS-S achieved WER 2.36%. On LRW, SVTS-M reported PESQ 1.49, STOI 0.649, ESTOI 0.483, and WER 13.4%. On LRS3, the paper states that SVTS-L produced intelligible speech and, according to the authors, was the first to show such results on that dataset (Mira et al., 2022). This suggests that decoupling spectrogram prediction from waveform synthesis was a decisive architectural step for scaling vid2voc beyond small, highly constrained corpora.
“Hierarchical Codec Diffusion for Video-to-Speech Generation” reformulates vid2voc as hierarchical masked token prediction over discrete RVQ codec tokens. Speech is encoded into 12 RVQ levels with codebook size 1024 per level, partitioned into low-level tokens 2 and high-level tokens 3. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Lip features come from AV-HuBERT-Large, identity features from ArcFace aligned to GE2E embeddings with an 4 loss, and emotion features from PosterV2. The model is trained with denoising score entropy over all 12 RVQ levels plus an identity-alignment term, and inference starts from all-[MASK] tokens, uses Euler sampling for 64 steps, and decodes the recovered tokens into waveform speech (Ye et al., 17 Apr 2026).
HiCoDiT makes the speech hierarchy itself a conditioning prior. Trained on 261.5 hours of VoxCeleb2 after preprocessing and evaluated on LRS2 and LRS3 without training on them, it reported on LRS3 with video-only guidance WER 29.41, DNSMOS 3.50, UTMOS 3.84, LSE-C 7.15, and EmoAcc 79.41. The paper attributes these gains to matching visual cues to distinct speech levels rather than injecting all visual information into a single monolithic latent. Ablations reported that removing hierarchical modeling or replacing DS-AdaILN with simpler conditioning degrades WER, synchronization, and emotion accuracy, supporting the claim that coarse-to-fine speech structure is operationally useful for vid2voc (Ye et al., 17 Apr 2026).
4. Visual voice cloning and movie dubbing
In the V2C formulation, vid2voc is no longer pure silent-video inversion. “V2C: Visual Voice Cloning” defines a task in which text determines what is said, reference audio determines who is speaking, and reference video determines how it is emotionally delivered. The paper argues that movie dubbing requires emotion and prosody consistent with the scene, and that these properties are often better inferred from visual context than from an unrelated reference audio clip. Its dataset, V2C-Animation, contains 10,217 aligned video clips, 10,217 audio clips, and 10,217 subtitles/texts from 26 animated movies and 153 speakers / characters. It includes 8 emotion labels—angry, disgust, fear, happy, neutral, sad, surprise, and others—with an average subtitle length of 6.51 words and average audio/video duration of 2.40 s (Chen et al., 2021).
The baseline V2C-Net uses a FastSpeech2-style synthesizer. Each phoneme embedding is formed by element-wise addition of a text-derived phoneme embedding, a speaker embedding from a 3-layer LSTM encoder applied to the reference audio, and an emotion embedding from an I3D encoder applied to the reference video. The synthesizer predicts duration, pitch, energy, and mel-spectrogram, and a pretrained HiFi-GAN vocoder converts mel to waveform. The paper also proposes MCD-DTW-SL, which weights MCD-DTW by speech length to penalize severe length mismatch (Chen et al., 2021).
The reported results establish the difficulty of V2C relative to conventional VC. V2C-Net achieved MCD 11.79, MCD-DTW 10.09, MCD-DTW-SL 10.05, identity accuracy 62.50%, emotion accuracy 56.25%, MOS-naturalness 3.97, and MOS-similarity 3.90. An ablation reported identity and emotion accuracies of 25.00% and 47.61% for video only, 59.38% and 53.13% for audio only, and 62.50% and 56.25% for audio + video. This demonstrates that reference audio is crucial for speaker identity and reference video improves emotion alignment (Chen et al., 2021).
“Learning to Dub Movies via Hierarchical Prosody Models” extends this line by decomposing visual guidance into lip, face, and scene levels. Its Duration Aligner uses lip embeddings as queries and phoneme embeddings as keys and values to align phonemes with visible mouth motion, then upsamples the aligned representation to mel length with Conv-Transpose. Its Prosody Adaptor estimates valence and arousal from faces with EmoFAN and maps arousal to energy and valence to pitch via attention. Its Scene Atmosphere Booster extracts a global scene embedding with I3D and fuses it with prosodic features. The resulting representation is decoded to mel-spectrogram and converted to waveform by HiFi-GAN (Cong et al., 2022).
On the V2C benchmark, this model reported LSE-D 8.036, LSE-C 5.608, MCD 15.66, Id. Acc. 37.75, Emo. Acc. 61.46, MOS-N 4.03 ± 0.08, and MOS-S 3.89 ± 0.07. On Chem, it reported LSE-D 6.975, LSE-C 7.176, AQ 3.84 ± 0.11, and AV Sync 3.97 ± 0.08. Its ablations are especially informative: removing the Duration Aligner degraded synchronization most strongly, removing the Prosody Adaptor caused large drops in identity and emotion accuracy, and removing the Atmosphere Booster hurt performance more modestly. The paper further reports that removing valence hurts more than removing arousal, that valence/arousal outperform raw facial features, that multi-head attention outperforms dot-product attention, and that Conv-Transpose is better than simple duplication for duration expansion (Cong et al., 2022).
5. Datasets, metrics, and recurring empirical patterns
Vid2voc research spans tightly controlled corpora and in-the-wild audiovisual datasets. GRID remains the canonical controlled benchmark for silent-video reconstruction, with 1000 sentences, 34 talkers, a fixed six-word grammar, 51 distinct words, 3-second clips, and 25 FPS video (Ephrat et al., 2017). LRW contributes about 150 hours of single-word utterances recorded in the wild, while LRS3 contains 312 hours, more than 50,000 words, thousands of speakers, and long sentences (Mira et al., 2022). V2C-Animation contributes multimodal dubbing supervision with text + audio + video + identity + emotion together, and the V2C benchmark used by hierarchical prosody models contains 26 Disney cartoon movies, 153 characters, and 10,217 video clips with paired audio and subtitles (Chen et al., 2021, Cong et al., 2022).
Evaluation protocols differ by formulation. Early silent-video work emphasized human word intelligibility accuracy on GRID (Ephrat et al., 2017). Vocoder-based and scalable lip-to-speech systems used PESQ, ESTOI, STOI, and WER (Michelsanti et al., 2020, Mira et al., 2022). V2C introduced objective measures for speech quality, identity, and emotion correctness, including MCD, MCD-DTW, MCD-DTW-SL, identity accuracy, emotion accuracy, and MOS-naturalness / MOS-similarity (Chen et al., 2021). Recent VTS systems added LSE-C and LSE-D for lip synchronization, DNSMOS and UTMOS for perceptual quality, EmoAcc for emotion, and SpkSim for speaker similarity (Ye et al., 17 Apr 2026).
Several patterns recur across papers. First, direct visual evidence is sufficient for nontrivial intelligibility, but full naturalness is harder than content recovery. Vid2Speech reported strong intelligibility despite white-noise excitation and explicitly prioritized intelligibility over naturalness (Ephrat et al., 2017). Second, structured acoustic targets matter: predicting full WORLD features improved the quality-intelligibility balance relative to partial parameterizations or direct waveform generation in controlled settings (Michelsanti et al., 2020). Third, scaling benefits from modularity: SVTS argues that once waveform synthesis is delegated to a fixed pretrained vocoder, the model can exploit much larger datasets efficiently (Mira et al., 2022). Fourth, recent systems increasingly factor visual conditioning by speech attribute rather than treating video as a single embedding; HiCoDiT aligns lip motion with low-level speaker-aware semantics and facial expression with high-level prosody, while hierarchical prosody models separate lip, face, and scene cues (Ye et al., 17 Apr 2026, Cong et al., 2022).
A common misconception is that “more face” is always uniformly better than “mouth only.” The evidence is mixed and task-dependent. Vid2Speech reported that the entire face gave roughly 40% lower error than mouth-only input (Ephrat et al., 2017), whereas vid2voc reported that mouth-only input often performed better for reconstruction, with full-face input sometimes slightly improving WER (Michelsanti et al., 2020). This indicates that the optimal visual field depends on the target representation, decoder structure, and whether the objective emphasizes acoustic fidelity or linguistic recognition.
6. Relation to voice conversion and current frontier
Vid2voc intersects with, but is not reducible to, voice conversion. The difference is methodological as well as definitional. V2C systems use video to infer emotion and timing while preserving speaker identity from reference audio (Chen et al., 2021). By contrast, O_O-VC is described as an audio-to-audio conversion framework trained on synthetic one-to-one aligned utterance pairs and is explicitly characterized as “not a vid2voc system” (Tu et al., 10 Oct 2025). This distinction is important because it separates visual conditioning problems from speaker/content disentanglement problems.
Recent voice conversion work nevertheless points toward a broader synthesis framework relevant to vid2voc. “Adapting a Diffusion-Based Music Synthesis Model to Human Voice Conversion” shows that a diffusion-based multi-instrument music model can be adapted to speech VC and singing VC by replacing note conditioning with PPGs and pitch, reinterpreting timbre conditioning as speaker or singer identity via FiLM, and relying on off-the-shelf feature extractors such as CREPE, wav2vec 2.0 / XLS-R, TRILL, HTDemucs, and an Onsets and Frames-style piano-roll extractor. It reported that T5-Voc is competitive with or better than a dedicated VC baseline on naturalness and performer similarity, while also showing that training on instrumental data degrades vocal quality and that the VC-specialized baseline remains stronger on phonetic fidelity (Maman et al., 14 Jul 2026).
The relevance to vid2voc lies in system design rather than direct input modality. This suggests that future vid2voc systems may increasingly combine visual encoders with generic audio generation backbones—diffusion models, codec token models, or unified speech-singing-music models—rather than relying exclusively on bespoke lip-to-speech architectures. The recent literature already exhibits this movement: from LPC regression to WORLD vocoding, from frozen neural vocoders to discrete diffusion over RVQ tokens, and from monolithic visual conditioning to explicit factorization of identity, content, prosody, and scene context (Michelsanti et al., 2020, Mira et al., 2022, Ye et al., 17 Apr 2026, Maman et al., 14 Jul 2026).
The field therefore comprises two converging agendas. One agenda seeks progressively better reconstruction from silent video, with gains in intelligibility, synchronization, and expressiveness. The other seeks multimodal control, where text, reference audio, and video jointly determine content, identity, and affective delivery. Vid2voc, understood broadly, is the point at which these agendas meet: audiovisual speech generation conditioned by the visible face, the scene, or both.