VisualTTS: Visual-Conditioned Speech Synthesis

• VisualTTS is a multimodal speech synthesis paradigm that uses visual cues—such as lip movements, facial expressions, and environmental context—to refine speech timing, prosody, and acoustic features.
• The framework spans multiple formulations including automatic voice-over, immersive environmental speech, and face-conditioned synthesis, each exploiting visual input to add expressive and spatial nuances.
• Recent systems demonstrate that dedicated alignment and fusion modules converting visual features into meaningful speech control signals can significantly improve synchronization, naturalness, and expressiveness.

Visual Text-to-Speech (VisualTTS) denotes a family of speech synthesis problems in which visual input is used as a conditioning signal for generating speech. In the arXiv literature, the term spans several related formulations: speech synchronized to a silent talking video for automatic voice over or video dubbing, reverberant speech conditioned on an environmental image for immersive acoustics, speech conditioned on a face image for speaker identity and emotion control, and visually grounded text-to-speech that treats text itself as an image rather than a sequence of discrete symbols (Lu et al., 2021, Liu et al., 2023, Lee et al., 2023, Nakano et al., 2022). Across these formulations, the defining property is that speech is not determined by text alone; visual evidence constrains timing, prosody, room acoustics, speaker traits, or expressive style.

1. Task scope and problem formulations

The literature does not use “VisualTTS” for a single canonical task. Instead, it names several technically distinct settings in which speech generation is conditioned on visual context.

Formulation	Visual input	Representative papers
Automatic voice over / video dubbing	Silent talking video, usually lip or face frames	(Lu et al., 2021, Hassid et al., 2021, Wang et al., 27 Nov 2025, Cheng et al., 29 Sep 2025)
Immersive environmental VTTS	Environmental RGB or RGB-D image, speaker position, scene semantics	(Liu et al., 2023, He et al., 2024, Liu et al., 2024)
Face-conditioned voice synthesis	Face image, sometimes with emotion intensity	(Lee et al., 2023, Chu et al., 2024)
Visual-context prosody modeling	General video context or listener visual feedback	(Que et al., 31 Jan 2025, Zhou et al., 2023)
Visually grounded text encoding	Rendered text images or character image patches	(Nakano et al., 2022, Arigala et al., 5 Jun 2026)

In the original automatic voice over formulation, the task is to synthesize natural-sounding speech that is precisely synchronized with the lip motions in a silent pre-recorded video, using both the video and its text script as input (Lu et al., 2021). A later line of work explicitly equates VisualTTS with video dubbing and adds reference speech timbre cloning to the task definition (Wang et al., 27 Nov 2025). A distinct immersive line defines VTTS as taking an environmental image as the prompt to synthesize reverberant speech for the spoken content, with the goal of matching room acoustics and spatial configuration (Liu et al., 2023, He et al., 2024). Face-conditioned systems widen the scope further: Face-TTS conditions speech generation on face images for zero-shot speaker synthesis, while FEIM-TTS conditions on face image and emotion intensity and states that it demonstrates capability for VisualTTS (Lee et al., 2023, Chu et al., 2024).

This breadth of usage suggests that VisualTTS is better understood as a multimodal conditioning paradigm than as a single benchmarked problem. What unifies the area is the attempt to map visual evidence into acoustically meaningful control signals.

2. Lip-synchronized dubbing and silent-video speech synthesis

The 2021 VisualTTS model formulates automatic voice over as a multimodal TTS problem and introduces two mechanisms for lip-speech synchronization: textual-visual attention and visual fusion during acoustic decoding (Lu et al., 2021). Lip images are cropped from video frames, encoded by a visual encoder composed of a Conv3D layer and a ResNet-18 block, and fused with a textual encoder through scaled dot-product attention,

$$C(Q_T, K_V, V_V) = \text{softmax}\left(\frac{Q_T K_V^T}{\sqrt{d_{K_V}}}\right)V_V.$$

The model then concatenates visual embedding, predicted mel-spectrogram features, and speaker embedding inside the decoder. On the reported benchmark, VisualTTS improved synchronization over Tacotron and Tacotron with TVA, achieving LSE-C $5.87$, LSE-D $8.45$, and FD $5.92$, while MOS remained $4.17 \pm 0.06$, essentially unchanged relative to the attention-only baseline (Lu et al., 2021).

VDTTS extends the dubbing formulation from lip crops to full-face video and emphasizes visually driven prosody as well as synchronization (Hassid et al., 2021). Its video encoder is a modified VGG3D-style CNN operating on full-face crops, and its decoder uses independent GMM-based attention heads over video and text:

$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$

followed by a linear projection of the concatenated contexts. The paper reports that removing video input or restricting input to the mouth region degrades synchronization and prosody metrics, and that the model approaches ground-truth synchronization quality on GRID while generalizing to in-the-wild VoxCeleb2 (Hassid et al., 2021). A central empirical claim is that full-face input carries prosodic and paralinguistic cues beyond visemes alone.

VSpeechLM shifts the dubbing architecture toward large-capacity discrete-speech modeling by inserting a text-video aligner in front of a SpeechLLM-based decoder (Wang et al., 27 Nov 2025). Text is converted to a phoneme sequence, video is encoded into lip embeddings with pretrained AV-HuBERT, and reference or target speech is discretized with a hierarchical codec. The aligner computes a similarity matrix,

$$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$

and outputs an expanded phoneme sequence aligned to video frames. Decoder training uses cross-entropy on codec tokens, with the total loss

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$

On Chem, VSpeechLM reports WER $12.5$, Spk Sim $78.9$, UTMOS $5.87$0, MCD-DTW $5.87$1, and MCD-DTW-SL $5.87$2, outperforming DSU, HPMDubbing, StyleDubber, and EmoDubber on most objective metrics (Wang et al., 27 Nov 2025). Its ablation further states that a visual prefix prompt degrades performance relative to the proposed text-video aligner.

3. Visual cues for prosody, interaction, and fine-grained speech control

A separate branch of work uses visual input not merely to enforce lip closure timing, but to resolve the under-specification of prosody in text-only TTS. VisualSpeech is the clearest statement of this agenda: it argues that visual context is underutilized for prosody prediction and proposes a multimodal extension of FastSpeech2 with a visual encoder and a cross-attention fusion module (Que et al., 31 Jan 2025). Text encoder output $5.87$3 attends to visual encoder output $5.87$4 through

$5.87$5

On CMD2, FastSpeech2 achieved pitch $5.87$6, energy $5.87$7, and duration $5.87$8, whereas VisualSpeech with Omnivore features achieved $5.87$9, $8.45$0, and $8.45$1, and VisualSpeech with ResNet50 features achieved $8.45$2, $8.45$3, and $8.45$4 (Que et al., 31 Jan 2025). The same paper reports lower MCD for both visual variants than for FastSpeech2 and states that visual features alone are strongly predictive of prosody. It also notes that no formal MOS or user study is reported.

VA-TTS moves the visual conditioning signal from the speaker to the listener, defining a task in which speech is conditioned on textual input and sequential visual feedback such as nods and smiles from the listener in face-to-face communication (Zhou et al., 2023). Listener video frames are converted into 3DMM expression and pose coefficients, encoded with an LSTM, and fused with phoneme identity, speaker ID, and audio context through stacked MHSA blocks:

$8.45$5

$8.45$6

$8.45$7

The model predicts pitch, energy, and duration refinements and passes them to FastSpeech2. On ViCo-X, the baseline TTS system reported GPE $8.45$8, VDE $8.45$9, FFE $5.92$0, MCD$5.92$1 $5.92$2, pitch MAE $5.92$3, energy MAE $5.92$4, and duration MAE $5.92$5, whereas VA-TTS reduced these to $5.92$6, $5.92$7, $5.92$8, $5.92$9, $4.17 \pm 0.06$0, $4.17 \pm 0.06$1, and $4.17 \pm 0.06$2, respectively (Zhou et al., 2023). The model is causal up to a one-frame lookahead.

These systems collectively challenge a common simplification that visual input is useful only for viseme timing. The results instead indicate that visual signals can also guide pauses, emphasis, pitch contour, and interactional adaptation.

4. Immersive VTTS and spatial environment understanding

Environmental VisualTTS treats speech quality as a function not only of linguistic content and speaker characteristics, but also of physical space. ViT-TTS is the first visual TTS model with scalable diffusion transformers and defines the task as generating speech from phoneme sequence plus an environmental image so that the output reflects room acoustics such as reverberation (Liu et al., 2023). The model uses a ResNet-18 image encoder, cross-attention-based visual-text fusion,

$4.17 \pm 0.06$3

a FastSpeech2-style variance adaptor, and a transformer denoiser with adaptive layer normalization, followed by BigVGAN. It also introduces self-supervised pre-training for both the encoder and denoiser. On SoundSpaces-Speech, ViT-TTS reported, for seen scenes, MOS $4.17 \pm 0.06$4, RTE $4.17 \pm 0.06$5, and MCD $4.17 \pm 0.06$6, and for unseen scenes, MOS $4.17 \pm 0.06$7, RTE $4.17 \pm 0.06$8, and MCD $4.17 \pm 0.06$9, outperforming DiffSpeech, Visual-DiffSpeech, and cascaded baselines (Liu et al., 2023).

MS$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$0KU-VTTS argues that RGB-only environmental modeling overlooks depth, speaker position, and environmental semantics (He et al., 2024). Its framework extracts RGB features $$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$1, depth features $$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$2, speaker position features $$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$3 from YOLOv8 detections using periodic embeddings,

$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$4

and semantic features $$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$5 from Gemini Pro Vision captions with BERT. The core “Dominant-Supplement Serial Interaction” first performs RGB-depth interaction, then speaker-position-enhanced interaction, then RGB-semantic interaction. A dynamic fusion module weights the knowledge paths using entropy,

$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$6

$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$7

$$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$8

With ViT-TTS as backbone, it reported unseen MOS $$c_{x}^t = \textrm{Att}_x(q^t, c_{x}^{t-1}, h_x), \quad c_{y}^t = \textrm{Att}_y(q^t, c_{y}^{t-1}, h_y),$$9, RTE $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$0, MCD $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$1, and seen MOS $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$2, RTE $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$3, MCD $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$4, with $$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$5 improvement over all baselines (He et al., 2024). The ablation states that removing any modality or replacing dynamic fusion with feature concatenation degrades all metrics.

M$$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$6SE-VTTS further emphasizes multi-modal and multi-scale spatial environment understanding, using RGB and depth patches, Gemini-generated environment captions, top-$$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$7 patch selection, and local-aware global spatial understanding (Liu et al., 2024). RGB and depth images are partitioned into patches, CLIP extracts patch and global [CLS] representations, caption embeddings guide attention to important RGB patches,

$$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$8

and the selected RGB indices are used to retrieve corresponding depth patches. Local-global interaction and semantic-guided fusion yield a final multimodal representation,

$$\mathbf{A} = \operatorname{Softmax}\left(\frac{\mathbf{P}_{\text{enc}}\mathbf{L}_{\text{enc}}^\top}{\sqrt{d}}\right),$$9

Reported results were MOS $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$0, RT60 Error $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$1, and MCD $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$2, improving over ProDiff, DiffSpeech, VoiceLDM, ViT-TTS-ResNet18, and ViT-TTS-CLIP (Liu et al., 2024).

This environmental line redefines “visual speech synthesis” in acoustic rather than articulatory terms: the visual signal describes where the utterance takes place, not how lips move.

5. Face-conditioned voice, emotion control, and visually grounded text

Face-TTS introduces face images as a direct conditioning signal for zero-shot TTS and states that this is the first time face images are used as a condition to train a TTS model (Lee et al., 2023). Built on Grad-TTS, it uses a pre-trained visual network $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$3 and audio network $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$4 to produce matching 512-dimensional embeddings and adds a speaker feature binding loss,

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$5

The full objective combines prior, duration, diffusion, and speaker losses. On LRS3, Face-TTS reported MOS $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$6 for unseen speakers under face conditioning, objective cross-modal matching accuracy $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$7 for unseen speakers with binding loss versus $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$8 without speaker loss and $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{CE}}^{\text{align}} + \mathcal{L}_{\text{CE}}^{\text{dec}}.$$9 for Grad-TTS with audio embedding, and human AB/ABX matching of about $12.5$0 (Lee et al., 2023). The system also reports voice generation for virtual faces from image generators.

FEIM-TTS extends face conditioning to emotion intensity control in a zero-shot setting and explicitly claims capability for VisualTTS (Chu et al., 2024). It uses a face network that extracts a 512-dimensional embedding from a $12.5$1 face image, an emotion embedding of dimension 128, a score-based diffusion model for mel-spectrogram generation, and classifier-free guidance:

$12.5$2

where $12.5$3 yields neutral speech, $12.5$4 yields standard emotional embedding, and $12.5$5 yields over-emphasized emotion. The model is trained on CREMA-D, MELD, and LRS3. Human raters gave FEIM-TTS MOS $12.5$6 on seen text and $12.5$7 on unseen text for naturalness, compared with FACE-TTS at $12.5$8 and $12.5$9, and average MOS $78.9$0 for emotion perception (Chu et al., 2024).

Another use of the adjective “visual” concerns the representation of text itself. vTTS synthesizes speech from visual text, meaning text rendered as an image, and reports that it can transfer emphasis and emotion attributes in visual text to speech without additional labels and architectures (Nakano et al., 2022). It uses CNN-extracted visual features as replacements for conventional text embeddings inside a FastSpeech2-inspired non-autoregressive model. Reported MOS values were $78.9$1 for Japanese with $78.9$2, $78.9$3 for Korean with $78.9$4, and $78.9$5 for English with $78.9$6, matching or exceeding conventional TTS. For rare and OOV characters, vTTS reported better MOS and CER than conventional TTS, for example OOV MOS $78.9$7 versus $78.9$8 and OOV CER $78.9$9 versus $5.87$00 (Nakano et al., 2022).

Pixel-TTS pushes this visually grounded text line further by rendering each character as a $5.87$01 grayscale patch, projecting the resulting image with a single Conv2D layer into a sequence of 512-dimensional embeddings, and passing them through four ConvNeXtV2 blocks (Arigala et al., 5 Jun 2026). It states that this design eliminates embedding matrix expansion during fine-tuning. On LibriSpeech-PC test at 300k updates, Pixel-TTS reported WER $5.87$02 and CER $5.87$03 versus Text-TTS at $5.87$04 and $5.87$05. In zero-shot German, Pixel-TTS reported WER $5.87$06 and CER $5.87$07 versus Text-TTS at $5.87$08 and $5.87$09, and under heavy Unicode homoglyph noise its WER increased from $5.87$10 to $5.87$11, whereas Text-TTS increased from $5.87$12 to $5.87$13 (Arigala et al., 5 Jun 2026).

6. Architectures, evaluation regimes, and active research directions

VisualTTS research currently spans several architectural families. Early lip-synchronization systems rely on encoder-decoder attention, as in VisualTTS and VDTTS, where alignment between text and video is learned through cross-modal attention or dual attention heads (Lu et al., 2021, Hassid et al., 2021). FastSpeech2-style non-autoregressive systems dominate prosody-focused settings such as VisualSpeech, VA-TTS, and vTTS (Que et al., 31 Jan 2025, Zhou et al., 2023, Nakano et al., 2022). Diffusion and flow-matching models dominate immersive acoustics and some newer unified systems, including ViT-TTS, Face-TTS, FEIM-TTS, and VSSFlow (Liu et al., 2023, Lee et al., 2023, Chu et al., 2024, Cheng et al., 29 Sep 2025). VSpeechLM introduces a SpeechLLM decoder with discrete codec tokens (Wang et al., 27 Nov 2025).

Evaluation is correspondingly fragmented. Lip-synchronized dubbing papers use LSE-C and LSE-D from SyncNet, Frame Disturbance, WER, speaker similarity, UTMOS, MCD, MCD-DTW, and MCD-DTW-SL (Lu et al., 2021, Wang et al., 27 Nov 2025). Prosody-oriented systems emphasize pitch, energy, duration errors, GPE, VDE, FFE, and MCD$5.87$14 (Zhou et al., 2023, Que et al., 31 Jan 2025). Environmental VTTS evaluates naturalness with MOS and room-acoustics similarity with RT60 Error or RTE, alongside MCD (Liu et al., 2023, He et al., 2024, Liu et al., 2024). A broader review of audiovisual speech synthesis places these evaluation choices within the older distinction between TTS quality metrics and talking-head synchronization metrics, and stresses that synchronization remains an open problem because even small audiovisual misalignments are perceptually salient (Georgiou et al., 2021).

Several recurrent findings cut across subfields. First, visually conditioned speech synthesis is not reducible to mouth ROI conditioning: VDTTS reports that full-face crops improve prosodic metrics, VisualSpeech reports gains from non-lip-centric scene features, and environmental VTTS shows that RGB alone can be insufficient for reverberation modeling (Hassid et al., 2021, Que et al., 31 Jan 2025, He et al., 2024). Second, dedicated alignment and fusion modules repeatedly outperform naive fusion. VisualTTS improves over Tacotron by combining textual-visual attention with decoder-time visual fusion; VSpeechLM reports that a visual prefix prompt causes confusion and degraded performance; MS$5.87$15KU-VTTS reports that replacing dynamic fusion with feature concatenation hurts performance (Lu et al., 2021, Wang et al., 27 Nov 2025, He et al., 2024). Third, joint learning across related audio tasks need not degrade performance. VSSFlow explicitly argues, contrary to the prevailing belief associated with earlier unified attempts, that end-to-end joint learning of V2S, VisualTTS, and TTS improves convergence, conditional generation, and classifier-free guidance stability (Cheng et al., 29 Sep 2025). Its conditioning strategy uses cross-attention for ambiguous video conditions and self-attention via concatenation for deterministic transcript conditions, inside a 10-layer flow-matching DiT with 1D RoPE. On Chem, it reports WER $5.87$16, Spk.Sim $5.87$17, UTMOS $5.87$18, MCD $5.87$19, LSE-C $5.87$20, and LSE-D $5.87$21 (Cheng et al., 29 Sep 2025).

The field also borders on text-to-audiovisual generation, where speech audio and facial motion are generated jointly from text rather than speech alone. NEUTART uses a shared Transformer feature space for mel spectrograms and FLAME coefficients, plus a lip-reading loss for visual supervision, while AV-Flow uses two parallel diffusion transformers with highway connections and flow matching to generate speech and photo-realistic 4D avatar motion from text (Milis et al., 2023, Chatziagapi et al., 18 Feb 2025). These systems are adjacent to VisualTTS rather than identical to it, but they indicate an increasingly common research direction: replacing cascaded audio-first pipelines with jointly optimized audiovisual generative models.

The resulting picture is of a rapidly diversifying area. “VisualTTS” now names a set of multimodal speech synthesis paradigms whose common premise is that visual evidence constrains latent speech structure in ways that text alone does not specify. The main open technical question is therefore not whether visual input helps, but which visual variables matter for which speech attribute, and what architectural bias most effectively converts those variables into alignment, acoustics, timbre, or prosody.