Text2Lip: Advances in Lip Synthesis
- Text2Lip is a family of methods that transform text into viseme sequences, lip landmarks, or complete talking-face videos using diverse architectures.
- Key techniques include modular pipelines, parallel non-autoregressive decoders, and Tacotron-based models that align text with dynamic lip movements.
- Empirical results show robust low-resource adaptation and high lip-sync quality validated by metrics like PSNR, SSIM, and MSE.
Text-to-lip (T2L) generation is the task of synthesizing talking lip according to text, and in the cited literature it appears both as conversion of text into lip landmarks and as a key component of talking face generation (Chen et al., 2020, Oneata et al., 2022, Liu et al., 2021). The term also names a 2025 viseme-centric framework, "Text2Lip," which converts text to viseme sequences, predicts landmarks, and then renders facial video via a landmark-guided renderer (Wang et al., 4 Aug 2025). Across these works, the output space ranges from normalized landmark trajectories to cropped lip images and full facial video, and the model families range from modular text-to-speech plus speech-to-lip pipelines to direct text-conditioned decoders and viseme-guided renderers (Han et al., 2022, Wang et al., 4 Aug 2025).
1. System families and problem formulations
The cited literature exhibits several distinct formulations of T2L. DualLip formulates lip generation jointly with lip reading and uses both paired and unpaired data through a dual-learning loop (Chen et al., 2020). ParaLip formulates T2L as a parallel non-autoregressive decoding problem with explicit duration prediction (Liu et al., 2021). FlexLip decomposes Text2Lip conversion into two independent, controllable modules, Text-to-Speech and Speech-to-Lip (Oneata et al., 2022). The Tacotron-based metaverse system extends Tacotron 2 so that one forward pass predicts both mel-spectrogram frames and a 60-dimensional lip-landmark displacement vector (Han et al., 2022). The 2025 Text2Lip framework is viseme-guided and couples landmark prediction with EchoMimic-based rendering (Wang et al., 4 Aug 2025).
| System | Primary output | Distinctive mechanism |
|---|---|---|
| DualLip | 128×64 lip frames | supervised + unsupervised dual learning |
| ParaLip | 160×80 lip frames | non-autoregressive parallel decoding |
| FlexLip | lip landmarks | modular TTS + STL, controllability |
| Tacotron-based Text2Lip | 20 lip landmarks × 3D | shared Tacotron 2 encoder/decoder |
| Text2Lip (2025) | landmarks + talking-face video | viseme-guided pseudo-audio + EchoMimic |
A recurrent misconception is that T2L has a single canonical pipeline. The published systems do not support that view. Some methods are frame generators, some are landmark predictors, and some are complete talking-face stacks. This suggests that "Text2Lip" is better understood as a family of text-conditioned articulatory generation problems than as one fixed architecture.
2. Linguistic conditioning and visual representations
The linguistic side of T2L is heterogeneous. DualLip uses plain characters for GRID and phonemes for TCD-TIMIT, with special `<#>,<p_1,\dots,p_n\hat{F_0}\hat{\Delta}\mathbf{h}_i(x,y)40l \in \mathbb{R}^{40}D=8\alpha = V^\top (l-\mu),l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.and predicts mel-spectrogram frames, pitch contour$0 from static shape $and predicts mel-spectrogram frames, pitch contour$1 (Oneata et al., 2022). The Tacotron-based model instead uses 20 lip landmarks as 3D world-coordinate displacements, yielding a 60-dimensional vector after pose normalization and subtraction of a speaker-specific reference shape (Han et al., 2022). The 2025 Text2Lip framework predicts 2D coordinates of $and predicts mel-spectrogram frames, pitch contour$2 landmarks, embeds landmark histories and MFCC audio into a 512-dimensional space, and conditions prediction on viseme features produced by a 2-layer, 4-head Transformer (Wang et al., 4 Aug 2025).
These representational choices have methodological consequences. Landmark-based systems factor articulation away from rendering, while frame-based systems couple articulation and appearance earlier. A plausible implication is that landmark parameterizations ease adaptation and controllability, whereas pixel-space generators expose image-quality metrics such as PSNR, SSIM, LPIPS, FID, and FVD more directly.
3. Alignment, decoding, and objective functions
A central design axis in T2L is how text tokens are aligned to output frames. DualLip has a “with duration” variant, where each input character is repeated exactly $and predicts mel-spectrogram frames, pitch contour$3 times by an Expander, and a “without duration” variant, where a location-sensitive attention mechanism learns a monotonic alignment between characters and frames (Chen et al., 2020). ParaLip also predicts token-level durations $and predicts mel-spectrogram frames, pitch contour$4 and uses a Length Regulator that repeats hidden states according to ground-truth durations in training and predicted durations at inference (Liu et al., 2021). FlexLip inherits duration modeling from FastPitch, where upsampling expands mel frames according to $and predicts mel-spectrogram frames, pitch contour$5 before decoding to natural speech patterns (Oneata et al., 2022). The Tacotron-based system uses location-sensitive attention exactly as Tacotron 2, with teacher forcing throughout fine-tuning (Han et al., 2022).
Autoregression is another fault line. DualLip’s duration-free variant is autoregressive and feeds back the decoder’s last prediction during generation (Chen et al., 2020). ParaLip was proposed precisely because autoregressive decoding “inherently hinders the inference speed” and has “a detrimental effect on the quality of generated lip frames due to error propagation”; it therefore generates all frames in parallel (Liu et al., 2021). The 2025 Text2Lip framework introduces a different bridge: during training, the real-audio stream is dropped with probability
$and predicts mel-spectrogram frames, pitch contour$6
with $and predicts mel-spectrogram frames, pitch contour$7 and $and predicts mel-spectrogram frames, pitch contour$8, and pseudo-audio is reconstructed from enhanced viseme features via cross-modal attention when audio is absent (Wang et al., 4 Aug 2025).
The loss functions reflect these architectural choices. FlexLip uses the standard FastPitch-style objective
$and predicts mel-spectrogram frames, pitch contour$9
and the speech-to-lip loss
$, and duration predictions$0
DualLip uses $, and duration predictions$1 reconstruction for lip generation, CTC for lip reading, and a dual objective
$, and duration predictions$2
with no adversarial, perceptual, or feature-matching losses in text-to-lip (Oneata et al., 2022, Chen et al., 2020). ParaLip supplements $, and duration predictions$3 reconstruction and duration loss with SSIM loss and LSGAN adversarial loss, with
$, and duration predictions$4
and weights $, and duration predictions$5 (Liu et al., 2021). The Tacotron-based model uses Smooth L1 with $, and duration predictions$6 on the 60-dimensional landmark vector only during fine-tuning (Han et al., 2022). The 2025 Text2Lip framework states a joint loss over landmark regression, audio reconstruction, frame reconstruction, GAN, perceptual, and sync terms, but the weighting hyperparameters are not explicitly listed in the paper summary (Wang et al., 4 Aug 2025).
4. Modularity, identity control, and low-resource adaptation
FlexLip makes modularity and controllability explicit. Its architecture permits component swap-in/out, separate adaptation of TTS or STL, and explicit control over phoneme durations $, and duration predictions$7, pitch $, and duration predictions$8, and lip shape $, and duration predictions$9 (Oneata et al., 2022). For speaker adaptation, it disentangles dynamic motion from static identity shape: for a new speaker, the mean lip shape $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$0 is recomputed on a few frames and substituted at inference,
$[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$1
without updating network weights (Oneata et al., 2022).
The low-resource results are a defining feature of this line of work. FlexLip reports that as little as 20 min of data can be used for the audio generation component and as little as 5 min for the speech-to-lip component, while objective measures remain comparable with those obtained using a larger set of training samples (Oneata et al., 2022). In the Obama-to-Trump speech-to-lip adaptation experiment, fine-tuning from 5 min improves $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$2 from approximately $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$3 to approximately $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$4, near full-data performance; in zero-shot lip adaptation, replacing $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$5 with $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$6 reduces $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$7 from $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$8 to $[2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states$9 (Oneata et al., 2022).
Other systems control identity differently. DualLip uses a single reference lip frame $(x,y)$0, sampled randomly during training and fixed to the first frame at test time, to produce a global style or identity vector $(x,y)$1 (Chen et al., 2020). ParaLip uses an identity lip image $(x,y)$2 processed by an identity encoder whose intermediate features are passed to the video decoder through skip-connections (Liu et al., 2021). The 2025 Text2Lip renderer takes one reference identity image together with predicted landmarks and audio features (Wang et al., 4 Aug 2025). The Tacotron-based metaverse model instead relies on transfer learning: encoder and gate weights are pre-trained on LJ Speech 1.1 and frozen, while the decoder is retrained on 93 TED-talk clips from LRS 3 totaling less than 5 min of video; training converged in 7 hours (Han et al., 2022).
This body of results suggests that T2L systems repeatedly separate articulatory dynamics from identity, but they do so with different primitives: a mean shape, a reference frame, an identity encoder, or a reference image.
5. Evaluation protocols and empirical results
Evaluation depends strongly on the representation being predicted. FlexLip evaluates TTS quality with ASR-based WER and cosine speaker similarity, and evaluates lip accuracy with $(x,y)$3 in PCA space and $(x,y)$4 after reconstruction; end-to-end alignment uses either DTW on MFCC or forced phoneme durations for 1:1 frame correspondence (Oneata et al., 2022). DualLip and ParaLip evaluate lip or face generation with PSNR, SSIM, and Landmark Distance (LMD), while DualLip additionally reports CER/WER or PER for lip reading (Chen et al., 2020, Liu et al., 2021). The 2025 Text2Lip paper extends the metric suite to LPIPS, FID, FVD, Sync-C, Sync-D, BLEU-1/4, WER, DTW-P, and MPJPE (Wang et al., 4 Aug 2025).
Several numerical results are repeatedly cited. In FlexLip, TTS fine-tuned on 20 min achieves WER $(x,y)$5 and cosine $(x,y)$6, versus natural WER $(x,y)$7 and cosine $(x,y)$8; the ASR-initialized STL encoder reaches $(x,y)$9 on the Obama test set, and the end-to-end Text2Lip pipeline using phone durations yields $40$0, close to the STL oracle with natural audio at $40$1 (Oneata et al., 2022). In DualLip on GRID with duration, a model trained with only 10% paired data gives CER $40$2, WER $40$3, PSNR $40$4, SSIM $40$5, and LMD $40$6, while adding 90% unpaired data improves this to CER $40$7, WER $40$8, PSNR $40$9, SSIM $l \in \mathbb{R}^{40}$0, and LMD $l \in \mathbb{R}^{40}$1; the 10% paired model thereby surpasses the 100% paired-only model in lip generation quality, whose PSNR is $l \in \mathbb{R}^{40}$2, SSIM is $l \in \mathbb{R}^{40}$3, and LMD is $l \in \mathbb{R}^{40}$4 (Chen et al., 2020).
ParaLip emphasizes speed-quality tradeoffs. On GRID without ground-truth duration at inference, DualLip reports PSNR $l \in \mathbb{R}^{40}$5, SSIM $l \in \mathbb{R}^{40}$6, LMD $l \in \mathbb{R}^{40}$7, whereas ParaLip reports PSNR $l \in \mathbb{R}^{40}$8, SSIM $l \in \mathbb{R}^{40}$9, LMD $D=8$0; on TCD-TIMIT, DualLip reports $D=8$1, $D=8$2, $D=8$3, whereas ParaLip reports $D=8$4, $D=8$5, $D=8$6 (Liu et al., 2021). Its inference time is reported as approximately $D=8$7/clip for autoregressive DualLip versus $D=8$8 for ParaLip on GRID, and $D=8$9 versus $\alpha = V^\top (l-\mu),$0 on TCD-TIMIT, corresponding to $\alpha = V^\top (l-\mu),$1 and $\alpha = V^\top (l-\mu),$2 speedups (Liu et al., 2021).
The 2025 Text2Lip paper positions viseme-guided generation as competitive with audio-driven methods. On GRID it reports SSIM $\alpha = V^\top (l-\mu),$3, PSNR $\alpha = V^\top (l-\mu),$4, LPIPS $\alpha = V^\top (l-\mu),$5, FID $\alpha = V^\top (l-\mu),$6, FVD $\alpha = V^\top (l-\mu),$7, and Sync-C $\alpha = V^\top (l-\mu),$8, with Landmark WER $\alpha = V^\top (l-\mu),$9 and BLEU-1 $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$0 (Wang et al., 4 Aug 2025). Its ablations state that removing viseme mapping drops BLEU-1 from $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$1 to $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$2, and removing the pseudo-audio module increases FID from $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$3 to $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$4 (Wang et al., 4 Aug 2025). The Tacotron-based model reports a best validation loss of $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$5 at epoch $l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.$6, and its ablation study shows that removing Post-Net and Pre-Net substantially improves convergence on landmarks, while not using a pretrained encoder yields the lowest numeric validation loss but qualitatively the lips do not move at inference (Han et al., 2022).
6. Limitations, unresolved issues, and cited future directions
The literature identifies several recurring limitations. DualLip states that the “without duration” variant lags behind the “with duration” variant in absolute image quality, that lip reading is sensitive to the quality of generated pseudo-videos, and that the absence of adversarial or perceptual losses means subtle lip textures or teeth may still appear slightly blurry (Chen et al., 2020). The Tacotron-based model states that it has no explicit modeling of prosody, emotion, or speaking style, assumes nearly static head pose because clips with large head motion were dropped, and notes that there is no common benchmark for lip-sync quality and that human perceptual studies remain future work (Han et al., 2022). ParaLip’s motivation also identifies error propagation as a structural drawback of autoregressive decoders (Liu et al., 2021).
The cited future directions are correspondingly diverse. DualLip proposes incorporating GAN or perceptual losses, harvesting large-scale unconstrained text such as subtitles and face videos such as YouTube, extending to fully unsupervised lip-reading in the wild or zero-shot talking-face generation with novel identities, and jointly optimizing the full TTS + text-to-lip + lip-to-face pipeline end-to-end (Chen et al., 2020). The Tacotron-based work proposes adding pitch, energy, or emotion embeddings, expanding to full 68-landmark plus eye-gaze and head-pose prediction, and using self-supervised pre-training on large unlabeled talking-head video corpora (Han et al., 2022).
A further misconception is that audio must be present at inference for high-quality lip synchronization. The published record does not sustain that as a universal requirement. FlexLip explicitly uses a TTS intermediary, but the Tacotron-based model predicts lip landmarks directly from text, and the 2025 Text2Lip curriculum drives the model toward entirely text-based generation by letting 7 late in training (Oneata et al., 2022, Han et al., 2022, Wang et al., 4 Aug 2025). A plausible implication is that future work will continue to treat audio as optional context rather than as a mandatory source modality, especially when the target is articulatory structure rather than waveform reconstruction.
