---
title: 'Text2Lip: Advances in Lip Synthesis'
url: https://www.emergentmind.com/topics/text2lip
type: topic
---

# Text2Lip: Advances in Lip Synthesis

Text-to-lip (T2L) generation is the task of synthesizing talking lip according to text, and in the cited literature it appears both as conversion of text into lip landmarks and as a key component of talking face generation [2009.05784] [2206.03206] [2107.06831]. The term also names a 2025 viseme-centric framework, "Text2Lip," which converts text to viseme sequences, predicts landmarks, and then renders facial video via a landmark-guided renderer [2508.02362]. Across these works, the output space ranges from normalized landmark trajectories to cropped lip images and full facial video, and the model families range from modular text-to-speech plus speech-to-lip pipelines to direct text-conditioned decoders and viseme-guided renderers [2209.14698] [2508.02362].

## 1. System families and problem formulations

The cited literature exhibits several distinct formulations of T2L. DualLip formulates lip generation jointly with lip reading and uses both paired and unpaired data through a dual-learning loop [2009.05784]. ParaLip formulates T2L as a parallel non-autoregressive decoding problem with explicit duration prediction [2107.06831]. FlexLip decomposes Text2Lip conversion into two independent, controllable modules, Text-to-Speech and Speech-to-Lip [2206.03206]. The Tacotron-based metaverse system extends Tacotron 2 so that one forward pass predicts both mel-spectrogram frames and a 60-dimensional lip-landmark displacement vector [2209.14698]. The 2025 Text2Lip framework is viseme-guided and couples landmark prediction with EchoMimic-based rendering [2508.02362].

| System | Primary output | Distinctive mechanism |
|---|---|---|
| DualLip | 128×64 lip frames | supervised + unsupervised dual learning |
| ParaLip | 160×80 lip frames | non-autoregressive parallel decoding |
| FlexLip | lip landmarks | modular TTS + STL, controllability |
| Tacotron-based Text2Lip | 20 lip landmarks × 3D | shared Tacotron 2 encoder/decoder |
| Text2Lip (2025) | landmarks + talking-face video | viseme-guided pseudo-audio + EchoMimic |

A recurrent misconception is that T2L has a single canonical pipeline. The published systems do not support that view. Some methods are frame generators, some are landmark predictors, and some are complete talking-face stacks. This suggests that "Text2Lip" is better understood as a family of text-conditioned articulatory generation problems than as one fixed architecture.

## 2. Linguistic conditioning and visual representations

The linguistic side of T2L is heterogeneous. DualLip uses plain characters for GRID and phonemes for TCD-TIMIT, with special `<#>,<$>` tokens for silence [2009.05784]. ParaLip uses characters on GRID and phonemes on TCD-TIMIT, and obtains ground-truth durations from the P2FA forced aligner [2107.06831]. FlexLip’s TTS component ingests a phoneme sequence $p_1,\dots,p_n$ and predicts mel-spectrogram frames, pitch contour $\hat{F_0}$, and duration predictions $\hat{\Delta}$ [2206.03206]. The Tacotron-based model is character-level and inherits Tacotron 2 location-sensitive attention over encoder states $\mathbf{h}_i$ [2209.14698]. The 2025 Text2Lip framework first applies a text-to-phoneme tool or dictionary to obtain an IPA phoneme sequence and then maps each phoneme to a viseme class via a fixed lookup table, specifically the Microsoft SAPI mapping [2508.02362].

The visual side is equally varied. FlexLip extracts 68 facial landmarks per frame with dlib, keeps 20 lip points $(x,y)$, normalizes them by translation, rotation, and scale, and then projects the normalized $40$-dimensional vector $l \in \mathbb{R}^{40}$ to $D=8$ via PCA,
$$
\alpha = V^\top (l-\mu),
$$
with reconstruction
$$
l_t = \mu + \sum_{i=1}^8 \alpha_{t,i} v_i.
$$
This explicitly separates dynamic PCA coefficients $\alpha$ from static shape $\mu$ [2206.03206]. The Tacotron-based model instead uses 20 lip landmarks as 3D world-coordinate displacements, yielding a 60-dimensional vector after pose normalization and subtraction of a speaker-specific reference shape [2209.14698]. The 2025 Text2Lip framework predicts 2D coordinates of $K=68$ landmarks, embeds landmark histories and MFCC audio into a 512-dimensional space, and conditions prediction on viseme features produced by a 2-layer, 4-head Transformer [2508.02362].

These representational choices have methodological consequences. Landmark-based systems factor articulation away from rendering, while frame-based systems couple articulation and appearance earlier. A plausible implication is that landmark parameterizations ease adaptation and controllability, whereas pixel-space generators expose image-quality metrics such as PSNR, SSIM, LPIPS, FID, and FVD more directly.

## 3. Alignment, decoding, and objective functions

A central design axis in T2L is how text tokens are aligned to output frames. DualLip has a “with duration” variant, where each input character is repeated exactly $d_i$ times by an Expander, and a “without duration” variant, where a location-sensitive attention mechanism learns a monotonic alignment between characters and frames [2009.05784]. ParaLip also predicts token-level durations $d_i^\*$ and uses a Length Regulator that repeats hidden states according to ground-truth durations in training and predicted durations at inference [2107.06831]. FlexLip inherits duration modeling from FastPitch, where upsampling expands mel frames according to $\hat{\Delta}$ before decoding to natural speech patterns [2206.03206]. The Tacotron-based system uses location-sensitive attention exactly as Tacotron 2, with teacher forcing throughout fine-tuning [2209.14698].

Autoregression is another fault line. DualLip’s duration-free variant is autoregressive and feeds back the decoder’s last prediction during generation [2009.05784]. ParaLip was proposed precisely because autoregressive decoding “inherently hinders the inference speed” and has “a detrimental effect on the quality of generated lip frames due to error propagation”; it therefore generates all frames in parallel [2107.06831]. The 2025 Text2Lip framework introduces a different bridge: during training, the real-audio stream is dropped with probability
$$
p_{drop}(t)=p_{start}+(p_{end}-p_{start})\cdot t/T,
$$
with $p_{start}=0$ and $p_{end}=1$, and pseudo-audio is reconstructed from enhanced viseme features via cross-modal attention when audio is absent [2508.02362].

The loss functions reflect these architectural choices. FlexLip uses the standard FastPitch-style objective
$$
L_{TTS}=L_{mel}+\lambda_{dur}L_{dur}+\lambda_{pitch}L_{pitch},
$$
and the speech-to-lip loss
$$
L_{STL}=E_t[\|\alpha_t-\hat{\alpha}_t\|_2^2].
$$
DualLip uses $L_1$ reconstruction for lip generation, CTC for lip reading, and a dual objective
$$
L_{total}=L^p_{lg}+L^p_{lr}+\alpha(L^u_{lg}+L^u_{lr}),
$$
with no adversarial, perceptual, or feature-matching losses in text-to-lip [2206.03206] [2009.05784]. ParaLip supplements $L_1$ reconstruction and duration loss with SSIM loss and LSGAN adversarial loss, with
$$
L_{total}=\lambda_1L_{rec}+\lambda_2L_{dur}+\lambda_3L_{ssim}+\lambda_4L_{adv}^G,
$$
and weights $\lambda_1=1,\lambda_2=1,\lambda_3=1,\lambda_4=5$ [2107.06831]. The Tacotron-based model uses Smooth L1 with $\beta=1.0$ on the 60-dimensional landmark vector only during fine-tuning [2209.14698]. The 2025 Text2Lip framework states a joint loss over landmark regression, audio reconstruction, frame reconstruction, GAN, perceptual, and sync terms, but the weighting hyperparameters are not explicitly listed in the paper summary [2508.02362].

## 4. Modularity, identity control, and low-resource adaptation

FlexLip makes modularity and controllability explicit. Its architecture permits component swap-in/out, separate adaptation of TTS or STL, and explicit control over phoneme durations $\Delta$, pitch $F_0$, and lip shape $\mu$ [2206.03206]. For speaker adaptation, it disentangles dynamic motion from static identity shape: for a new speaker, the mean lip shape $\mu'$ is recomputed on a few frames and substituted at inference,
$$
l'_t=\mu'+\sum_{i=1}^8 \alpha_{t,i} v_i,
$$
without updating network weights [2206.03206].

The low-resource results are a defining feature of this line of work. FlexLip reports that as little as 20 min of data can be used for the audio generation component and as little as 5 min for the speech-to-lip component, while objective measures remain comparable with those obtained using a larger set of training samples [2206.03206]. In the Obama-to-Trump speech-to-lip adaptation experiment, fine-tuning from 5 min improves $MSE_{8D}$ from approximately $0.109$ to approximately $0.096$, near full-data performance; in zero-shot lip adaptation, replacing $\mu$ with $\mu'$ reduces $MSE_{40D}$ from $0.071$ to $0.034$ [2206.03206].

Other systems control identity differently. DualLip uses a single reference lip frame $I$, sampled randomly during training and fixed to the first frame at test time, to produce a global style or identity vector $z^I$ [2009.05784]. ParaLip uses an identity lip image $l_I \in \mathbb{R}^{160\times80\times3}$ processed by an identity encoder whose intermediate features are passed to the video decoder through skip-connections [2107.06831]. The 2025 Text2Lip renderer takes one reference identity image together with predicted landmarks and audio features [2508.02362]. The Tacotron-based metaverse model instead relies on transfer learning: encoder and gate weights are pre-trained on LJ Speech 1.1 and frozen, while the decoder is retrained on 93 TED-talk clips from LRS 3 totaling less than 5 min of video; training converged in 7 hours [2209.14698].

This body of results suggests that T2L systems repeatedly separate articulatory dynamics from identity, but they do so with different primitives: a mean shape, a reference frame, an identity encoder, or a reference image.

## 5. Evaluation protocols and empirical results

Evaluation depends strongly on the representation being predicted. FlexLip evaluates TTS quality with ASR-based WER and cosine speaker similarity, and evaluates lip accuracy with $MSE_{8D}$ in PCA space and $MSE_{40D}$ after reconstruction; end-to-end alignment uses either DTW on MFCC or forced phoneme durations for 1:1 frame correspondence [2206.03206]. DualLip and ParaLip evaluate lip or face generation with PSNR, SSIM, and Landmark Distance (LMD), while DualLip additionally reports CER/WER or PER for lip reading [2009.05784] [2107.06831]. The 2025 Text2Lip paper extends the metric suite to LPIPS, FID, FVD, Sync-C, Sync-D, BLEU-1/4, WER, DTW-P, and MPJPE [2508.02362].

Several numerical results are repeatedly cited. In FlexLip, TTS fine-tuned on 20 min achieves WER $\approx 8.6\%$ and cosine $\approx 0.69$, versus natural WER $\approx 9.3\%$ and cosine $\approx 0.68$; the ASR-initialized STL encoder reaches $MSE_{8D}=0.064$ on the Obama test set, and the end-to-end Text2Lip pipeline using phone durations yields $MSE_{8D}\approx0.094$, close to the STL oracle with natural audio at $0.064$ [2206.03206]. In DualLip on GRID with duration, a model trained with only 10% paired data gives CER $=10.9\%$, WER $=24.8\%$, PSNR $=29.87$, SSIM $=0.881$, and LMD $=1.374$, while adding 90% unpaired data improves this to CER $=2.66$, WER $=6.25$, PSNR $=30.91$, SSIM $=0.902$, and LMD $=1.234$; the 10% paired model thereby surpasses the 100% paired-only model in lip generation quality, whose PSNR is $30.68$, SSIM is $0.895$, and LMD is $1.288$ [2009.05784].

ParaLip emphasizes speed-quality tradeoffs. On GRID without ground-truth duration at inference, DualLip reports PSNR $=29.13$, SSIM $=0.872$, LMD $=1.809$, whereas ParaLip reports PSNR $=28.74$, SSIM $=0.875$, LMD $=1.675$; on TCD-TIMIT, DualLip reports $27.38$, $0.809$, $2.351$, whereas ParaLip reports $27.64$, $0.816$, $2.084$ [2107.06831]. Its inference time is reported as approximately $0.30\,s$/clip for autoregressive DualLip versus $0.022\,s$ for ParaLip on GRID, and $0.65\,s$ versus $0.034\,s$ on TCD-TIMIT, corresponding to $13.1\times$ and $19.1\times$ speedups [2107.06831].

The 2025 Text2Lip paper positions viseme-guided generation as competitive with audio-driven methods. On GRID it reports SSIM $=0.740$, PSNR $=19.023\,dB$, LPIPS $=0.238$, FID $=32.109$, FVD $=277.66$, and Sync-C $=4.641$, with Landmark WER $=39.43\%$ and BLEU-1 $=54.81\%$ [2508.02362]. Its ablations state that removing viseme mapping drops BLEU-1 from $54.81$ to $47.23$, and removing the pseudo-audio module increases FID from $32.1$ to $120.4$ [2508.02362]. The Tacotron-based model reports a best validation loss of $7.466\times10^{-3}$ at epoch $458$, and its ablation study shows that removing Post-Net and Pre-Net substantially improves convergence on landmarks, while not using a pretrained encoder yields the lowest numeric validation loss but qualitatively the lips do not move at inference [2209.14698].

## 6. Limitations, unresolved issues, and cited future directions

The literature identifies several recurring limitations. DualLip states that the “without duration” variant lags behind the “with duration” variant in absolute image quality, that lip reading is sensitive to the quality of generated pseudo-videos, and that the absence of adversarial or perceptual losses means subtle lip textures or teeth may still appear slightly blurry [2009.05784]. The Tacotron-based model states that it has no explicit modeling of prosody, emotion, or speaking style, assumes nearly static head pose because clips with large head motion were dropped, and notes that there is no common benchmark for lip-sync quality and that human perceptual studies remain future work [2209.14698]. ParaLip’s motivation also identifies error propagation as a structural drawback of autoregressive decoders [2107.06831].

The cited future directions are correspondingly diverse. DualLip proposes incorporating GAN or perceptual losses, harvesting large-scale unconstrained text such as subtitles and face videos such as YouTube, extending to fully unsupervised lip-reading in the wild or zero-shot talking-face generation with novel identities, and jointly optimizing the full TTS + text-to-lip + lip-to-face pipeline end-to-end [2009.05784]. The Tacotron-based work proposes adding pitch, energy, or emotion embeddings, expanding to full 68-landmark plus eye-gaze and head-pose prediction, and using self-supervised pre-training on large unlabeled talking-head video corpora [2209.14698].

A further misconception is that audio must be present at inference for high-quality lip synchronization. The published record does not sustain that as a universal requirement. FlexLip explicitly uses a TTS intermediary, but the Tacotron-based model predicts lip landmarks directly from text, and the 2025 Text2Lip curriculum drives the model toward entirely text-based generation by letting $p_{drop}\to1$ late in training [2206.03206] [2209.14698] [2508.02362]. A plausible implication is that future work will continue to treat audio as optional context rather than as a mandatory source modality, especially when the target is articulatory structure rather than waveform reconstruction.

Source: https://www.emergentmind.com/topics/text2lip