---
title: Conversational Speech-Visual Synthesis
url: https://www.emergentmind.com/topics/conversational-speech-visual-synthesis-csvs
type: topic
---

# Conversational Speech-Visual Synthesis

Searching arXiv for recent papers on conversational speech-visual synthesis and closely related multimodal conversational generation.
Conversational Speech-Visual Synthesis (CSVS) denotes a family of multimodal generation tasks in which conversational context is used to synthesize speech, visual behavior, or both in a coordinated manner. In current arXiv literature, the term is not restricted to a single benchmark or output format. It includes mixed-audio generation of co-located 3D dyadic facial performances [2603.08674], visually grounded conversational speech models that converse about images [2503.15633], listener-conditioned speech synthesis that adapts prosody to visual feedback [2306.12020], multi-turn conversational head generation with explicit speaking and listening roles [2307.02090], end-to-end spoken dialogue models that map audio-visual input to audio-visual response without intermediate text [2406.07867], unified text-driven frameworks that jointly synthesize conversational speech and interactive faces [2512.20296], and multimodal large-language-model systems that predict emotion, speech, and facial animation together [2508.04585]. This breadth suggests that CSVS is best understood as a research umbrella for conversationally coupled speech–visual generation rather than as a single narrowly defined task.

## 1. Problem formulations and scope

A central distinction in the literature is whether CSVS generates visual behavior from speech, speech from visual context, or both modalities jointly. In the co-located dyadic formulation of “Talking Together: Synthesizing Co-Located 3D Conversations from Audio,” the input is one mixed audio waveform $A \in \mathbb{R}^T$, and the system outputs two complete 3D facial animation streams, one for each participant, including facial expression coefficients, skeletal joint rotations for neck, head, left eye, and right eye, and global head translations over $L$ frames [2603.08674]. In that formulation, each participant’s state is represented as $\psi \in \mathbb{R}^{L\times 63}$, $\theta \in \mathbb{R}^{L\times 4\times 3}$, $t \in \mathbb{R}^{L\times 3}$, and $x=\mathrm{concat}(\psi,\theta,t)\in\mathbb{R}^{L\times 78}$ [2603.08674].

Other formulations invert the conditioning direction. “Visual-Aware Text-to-Speech” defines a listener-aware speech synthesis task in which the model takes phonemes $Q=\{q_1,\dots,q_n\}$ and a streaming listener video sequence $V=\{v_1,\dots,v_m\}$ and predicts acoustic features and waveform conditioned on both inputs, formalized as $\mathcal{A}'=\mathbf{G}'_A(\mathcal{Q},\mathcal{V})$ [2306.12020]. The emphasis there is not visual rendering but prosodic adaptation to listener feedback.

A third class of formulations produces both modalities jointly. TAVID takes a text dialogue $x$ and a reference image $I$ and outputs synchronized conversational speech and interactive facial video for dyadic conversation, with dual-channel semantic token sequences $S=[s_1,s_2]$ mediating both pipelines [2512.20296]. UniTalker similarly defines CSVS as prediction of the next utterance’s emotion, speech waveform, and facial animation from multimodal conversational history plus the current target utterance’s speaker and text [2508.04585]. The Face-to-Face spoken dialogue model of MultiDialog processes audio-visual speech from a user and generates synchronized audio-visual spoken response without using intermediate text during generation [2406.07867].

A related but narrower usage appears in MoshiVis, where CSVS is framed as generating conversational speech conditioned on a visual input while preserving low latency and prosody and remaining able to switch to non-visual topics [2503.15633]. Although this is not dyadic avatar synthesis, it retains the core CSVS concern of conversational coordination across modalities.

| Formulation | Inputs | Outputs / emphasis |
|---|---|---|
| Co-located dyadic 3D conversation | Mixed audio waveform | Two spatially aware 3D facial performances |
| Visual-aware TTS | Phonemes and listener video | Speech with listener-conditioned prosody |
| Interactive conversational head generation | Audio, roles, dialog cues, partner visuals, identity image | Multi-turn speaking/listening head behavior |
| End-to-end audio-visual dialogue | User audio-visual speech | Audio-visual spoken response |
| Unified text-driven dialogue generation | Text dialogue and reference image | Conversational speech and interactive facial video |

A common misconception is to equate CSVS with conventional talking-head lip synchronization. The literature consistently distinguishes conversational synthesis from one-way “talking head” generation by introducing listener reactions, turn-taking, prosody adaptation, role alternation, mutual gaze, or joint audio–visual generation [2603.08674].

## 2. Representations and model families

The field uses several representational regimes. One line relies on parametric 3D face models. The co-located dyadic system uses a 3DMM-based parametric face model with geometry
$$
M(\boldsymbol{\beta}, \boldsymbol{\psi}) = \mathbf{S} + \sum_i \beta_i \mathbf{S}_i + \sum_j \psi_j \mathbf{E}_j,
$$
combined with linear blend skinning, axis-angle rotations, and translations in metric meters in a camera/world coordinate system [2603.08674]. Interactive conversational head generation also decouples identity and motion through 3DMM coefficients, with fixed identity-dependent features $\mathcal{I}=(\alpha,\delta,\gamma)$ and dynamic motion features $m=(\beta,p)$ rendered by PIRenderer [2307.02090].

A second line relies on landmark or video-token abstractions. UniTalker introduces LmkCodec, a neural landmark codec that tokenizes per-frame 2D facial landmarks at 25 Hz with one token per frame, using $\text{fsq\_levels}=[8,5,5,5]$ and a face vocabulary size of 1,000 [2508.04585]. MultiDialog instead treats audio-visual speech as discrete AV tokens extracted by AV-HuBERT and quantized with 500 HuBERT clusters at 25 Hz, using mouth crops of $96\times 96$ and resampled 16 kHz audio [2406.07867].

A third line is organized around conversational semantic tokens. TAVID predicts dual-stream semantic tokens from text, where the tokens are derived from the 35th layer of XLS-R and discretized into 10k clusters; these tokens condition both an acoustic denoiser and a latent-diffusion video generator [2512.20296]. This shared-token design is meant to tie lip timing, speaking–listening alternation, and speech generation to the same intermediate representation.

Architecturally, the literature spans diffusion, autoregressive LLMs, CNNs, LSTMs, and conditional flow models. The dyadic 3D conversation model uses a denoising diffusion generator with a shared U-Net backbone, dual noisy streams, inter-speaker cross-attention, role embeddings, and FiLM conditioning [2603.08674]. UniTalker builds its EVSLM module on Qwen2.5-0.5B and performs multi-task token prediction over emotion, facial landmarks, and speech tokens [2508.04585]. MoshiVis augments a pretrained speech LLM with visual cross-attention modules inserted between MHSA and FFN, plus a dynamic gate [2503.15633]. The ViCo/ViCo-X baseline uses streaming LSTM decoders for listener and speaker motion generation, coupled by a role transformer for multi-turn alternation [2307.02090]. “Learning Speech-driven 3D Conversational Gestures from Video” uses a temporal 1D U-Net-like CNN with three modality-specific decoders and a conditional discriminator over body and hand motion [2102.06837].

These differences reflect distinct operating points. Parametric 3D representations support metric spatial reasoning and rig retargeting; token-based LLM formulations support long-context dialogue modeling; sequence-to-sequence LSTMs and temporal CNNs emphasize streaming or lightweight inference; diffusion and flow models prioritize synthesis fidelity [2603.08674][2307.02090][2406.07867][2512.20296].

## 3. Interaction modeling, synchronization, and conversational structure

Interaction modeling is the defining technical problem of CSVS. In mixed-audio dyads, the difficulty arises from overlapping speech, frequent turn-taking, and the need to synthesize both speaker and listener behavior from a single waveform [2603.08674]. The dyadic diffusion model addresses this through speaker probability masks, role embeddings $e_{\text{speak}}$ and $e_{\text{listen}}$, and bidirectional inter-speaker cross-attention in decoder layers so that each participant’s features incorporate the other’s current state [2603.08674]. The same system adds a selective eye gaze loss on samples in the top 20% rotation variance, weighted by $\lambda_{\text{gaze}}=5$, to encourage realistic gaze dynamics [2603.08674].

Synchronization is implemented differently across papers. UniTalker enforces a bimodal speech-visual hard alignment decoding strategy in which facial and speech tokens are generated in a strict interleaved sequence
$$
\langle T_{N,1}^{v}, T_{N,1}^{s}, T_{N,2}^{v}, T_{N,2}^{s}, \ldots \rangle,
$$
with $T=\max^v=\max^s$ at 25 tokens/second [2508.04585]. TAVID instead uses the same dual-stream semantic tokens to condition both speech and video pipelines, arguing that shared semantics align timing, lip motion, and interaction patterns across modalities [2512.20296]. MultiDialog also uses a shared AV token sequence for both speech and lip generation, but without an explicit monotonic alignment module beyond token-level conditioning and a Wav2Lip-style renderer [2406.07867].

Listener modeling is explicit in several systems. ViCo/ViCo-X defines responsive listening head generation, expressive talking head generation, and integrated conversational head generation as separate tasks, and its baseline fuses speaker audio, speaker dynamics, listener attitude, and listener reference features in a streaming LSTM [2307.02090]. VA-TTS imposes a causality constraint at the phoneme level: for phoneme $q_i$, only past listener frames $\mathcal{V}_{a_i}=\{v_1,\dots,v_{a_i}\}$ are used, with $\phi=1$ sufficient at 30 FPS given per-phoneme prosody generation time $\mathcal{T}\approx 2.67$ ms [2306.12020]. MoshiVis addresses a different form of conversational switching: its gate $g_t=\sigma(W_2\phi(W_1 h_t))$ modulates whether visual evidence influences the speech stream, allowing the model to move between image-grounded content and unrelated conversational topics without explicit gate supervision [2503.15633].

Spatial awareness is a further differentiator. The dyadic 3D conversation paper trains on absolute first-frame translations and predicts motion deltas, embedding both participants in the same 3D coordinate frame; at test time, Gemini 1.5 is few-shot prompted with example text-to-translation pairs to output initial 3D coordinates from prompts such as “arguing across a table” or “intimate conversation” [2603.08674]. This is a stronger notion of interaction than two isolated heads rendered as if in separate video-call windows.

A plausible implication is that CSVS systems can be organized by what they synchronize: phoneme-to-lip timing, prosody-to-expression coupling, speaker-to-listener responsiveness, or shared 3D scene geometry. The recent literature progressively broadens the synchronization target from local articulation to whole-conversation coordination [2306.12020][2603.08674][2512.20296].

## 4. Data resources and curation pipelines

Progress in CSVS is closely tied to dataset construction. The largest reported scale in the supplied literature appears in the co-located dyadic 3D conversation work, which curates a Dyadic Conversation Dataset of over 2 million interacting pairs totaling 50,000+ hours with 10k+ identities, plus a Synthetic Dubbing Dataset of another 50,000+ hours constructed from high-quality single-person frontal videos [2603.08674]. Its dyad pipeline includes scenario filtering to remove “two-scene” video-call layouts, quality control for occluded, tiny, or blurred faces, Looking-to-Listen for per-speaker audio separation, WebRTC VAD for frame-level speaking probability masks, GLEAN-like face super-resolution, and 3DMM reconstruction with metric scale estimated from average inter-ocular distance [2603.08674].

At smaller but still substantial scale, MultiDialog provides approximately 340 hours of audio-visual dialogues derived from TopicalChat, with 8,733 dialogues, 187,859 utterances, 339.71 hr total length, parallel recordings of both interlocutors, seven emotion classes used at recording time, and turn timestamps logged during capture [2406.07867]. The dataset was recorded in a professional studio with green screen, two cameras, and two microphones; recordings missing audio or visual streams were filtered out, and any audio–video misalignment was manually adjusted by sliding start times [2406.07867].

ViCo and ViCo-X target conversational heads rather than full speech–video dialogue. ViCo contains 50 source videos and 483 valid clips spanning 95 min 22 s, with 92 unique people and three attitude labels—positive, neutral, and negative—cross-validated by at least three annotators [2307.02090]. ViCo-X is a staged multi-turn corpus based on filtered Chinese e-commerce conversations from JDDC, recorded with two actors two meters apart and annotated with ISO 24617-2 dialog acts, frame-resolution sentence timings, and speaker position [2307.02090]. VA-TTS uses ViCo-X as a multimodal, multi-speaker Mandarin conversational dataset with 10 speakers, 48 kHz 16-bit audio, 30 FPS video, and 3DMM-derived listener dynamics $m=(\beta,p)\in\mathbb{R}^{1\times 70}$ [2306.12020].

Other works emphasize scalable pseudo-annotation. The 3D gesture synthesis paper constructs more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people using monocular 3D face performance capture, upper-body pose estimation, and 3D hand pose estimation, followed by confidence-based filtering and interpolation for hand gaps up to 8 frames [2102.06837]. TAVID aggregates approximately 500 hours of video data and approximately 2000 hours of speech data from HDTF, ViCo, Seamless Interaction, LibriTTS-R, DailyTalk, Fisher, and Seamless Interaction, with preprocessing that excludes frames with occlusions, excessive movement, or head rotations above $30^\circ$ [2512.20296]. UniTalker combines DailyTalk and NCSSD for 113 hours of speech-only dialogue, MultiDialog for 307 hours of visual-spoken dialogue, and RAVDESS, MEAD, and CelebV-HQ for 89 hours of single-utterance audio-visual data [2508.04585].

These curation strategies reveal a recurrent pattern: conversational data are scarce in native audio-visual form, so the literature repeatedly supplements them with synthetic dubbing, pseudo-dialogue assembly, in-the-wild reconstruction, or mixed single-role and conversational corpora [2603.08674][2503.15633][2512.20296].

## 5. Objectives, metrics, and empirical evidence

Training objectives vary by formulation but are consistently structured around reconstruction plus interaction-specific constraints. The dyadic diffusion model minimizes
$$
\mathcal{L}=E_{x_0,t,x_t}\left[\|x_0-G(x_t,t,c)\|_2^2\right]
$$
with weighted $L_2$ losses on expression, rotation, and translation using $\lambda_{\text{expr}}=1$, $\lambda_{\text{rot}}=8$, and $\lambda_{\text{trans}}=1$, a vertex velocity smoothness term with $\lambda_{\text{vel}}=1$, and an auxiliary gaze loss weighted by $\lambda_{\text{gaze}}=5$ [2603.08674]. VA-TTS minimizes the $L_2$ distance between predicted and ground-truth phoneme-wise prosody features in log scale [2306.12020]. The 3D gesture paper combines modality-specific reconstruction losses with a conditional adversarial objective and a total weight $w=5$ on the adversarial term [2102.06837]. UniTalker uses MSE for landmark reconstruction, cross-entropy for emotion and token prediction, conditional flow matching for speech rendering, and an Echomimic-based talking-face renderer [2508.04585]. TAVID uses separate objectives for visual diffusion, text-to-semantic cross-entropy, acoustic flow matching, and Speaker Mapper regression [2512.20296].

Evaluation protocols are correspondingly heterogeneous. The dyadic 3D conversation paper uses Fréchet Distance, Paired FD, parameter MSE and vertex MSE for FULL, EXP, ROT, TRANSL, EYE, and LIP, plus SID diversity, and reports that the proposed method achieves the best performance across nearly all metrics relative to DualTalk and single-speaker baselines [2603.08674]. Its forced-choice study with 19 participants and 14 clips finds preferences for the proposed model in lip quality (79.3%), speaker movements (73.8%), listener movements (73.0%), interaction quality (71.4%), and eye gaze quality (68.3%) over SelfTalk and DualTalk [2603.08674].

VA-TTS reports consistent objective improvements over a FastSpeech2 baseline: GPE 20.23 to 20.09, VDE 9.54 to 9.23, FFE 20.90 to 20.58, and $MCD_{13}$ 6.11 to 5.97; phoneme-wise mean absolute error improves from Pitch 37.02 to 35.60, Energy 5.32 to 5.08, and Duration 143.73 ms to 135.97 ms [2306.12020]. In the conversational gesture work, a user study with 67 participants rates the audio+pose adversarial model at naturalness $4.05\pm 0.85$ and synchrony $4.00\pm 0.91$, compared with $3.54\pm 1.11$ and $3.78\pm 1.08$ for direct regression CNN and $3.15\pm 1.03$ and $3.21\pm 1.11$ for LSTM; a second study finds naturalness $4.01\pm 0.82$ and synchrony $3.93\pm 0.92$ for the proposed model versus $2.88\pm 1.02$ and $3.11\pm 1.13$ for MoGlow [2102.06837].

ViCo/ViCo-X emphasizes motion distances and lip synchronization. For responsive listening head generation on ViCo-X, ExpFD improves from 13.739 and 17.635 for Random and Mirror to 10.656 for the responsive model, AngleFD from 5.913 to 4.350, and TransFD from 6.316 to 5.129 [2307.02090]. For expressive talking head generation on ViCo-X, AVOffset improves from 0.636 to 0.083 and AVConf from 2.524 to 2.755 when listener signals are added [2307.02090]. Human evaluation on ViCo-X reports $3.35\pm 0.12$ for the Conversational Agent, $3.02\pm 0.35$ for the Listener + Speaker baseline, and $4.4\pm 0.10$ for ground truth [2307.02090].

In joint speech-and-video generation, TAVID reports strong gains. On Seamless Interaction, TAVID (Text) achieves Visual Quality $3.75\pm 0.23$, Lip Sync $3.80\pm 0.23$, Turn-taking $3.84\pm 0.24$, FID 16.625, FVD 179.305, LPIPS 0.056, LSE-C 6.457, LSE-D 8.403, RPCC 0.031, $\Delta$SID 0.489, and $\Delta$Var 0.011, improving over audio-driven and TTS-cascaded DIM baselines [2512.20296]. On VoxCeleb2 unseen speakers, TAVID (face-stylized) reports Naturalness $4.20\pm 0.20$, Face Matching $3.87\pm 0.19$, UTMOS 3.530, and VoxSim 0.380 [2512.20296]. UniTalker on MultiDialog reports SIM\_SPK 0.902, PDTW 42.014, ACC\_SE 0.743, MOS\_SN 4.128, and MOS\_SE 4.103 for synthesized speech with visual context, and FID 24.214, PSNR 19.513, LPIPS 0.204, SSIM 0.741, ACC\_VE 0.813, LSE-C 6.386, LSE-D 8.189, MOS\_VC 4.148, MOS\_VN 4.254, and MOS\_VE 4.323 for talking-face animation [2508.04585].

MoshiVis uses conventional visual-understanding benchmarks rather than avatar metrics, but its numbers still illustrate a CSVS trade-off between speech supervision and visual grounding. With $p_{\text{audio}}=0\%$, it reports OCR-VQA 38.5%, VQAv2 49.3%, and COCO CIDEr 113, while noting that audio quality is degraded at 0% audio and improves quickly with small $p_{\text{audio}}$; the best trade-off often occurs at $p_{\text{audio}}\approx 25\%$ [2503.15633].

Because these metrics target different subproblems, direct cross-paper ranking is not meaningful. The empirical record instead shows repeated local gains when conversational structure, listener information, shared semantics, or multimodal context are modeled explicitly [2306.12020][2307.02090][2508.04585][2512.20296].

## 6. Applications, misconceptions, limitations, and open directions

The applications proposed across the literature are consistent: immersive VR and telepresence, digital humans, virtual agents, social robots, avatar chatbots, dubbing, accessibility, and communication aids [2603.08674][2102.06837][2307.02090][2406.07867]. These applications depend not only on lip sync but also on interaction coherence, role alternation, responsive listening, and identity preservation.

Several misconceptions are addressed implicitly by the body of work. First, CSVS is not solely a speech-driven mouth animation problem. The cited systems model head pose, gaze, facial expression, prosody, listener feedback, body and hand gestures, turn-taking, dialog acts, or joint speech generation [2102.06837][2306.12020][2307.02090][2603.08674]. Second, better articulation alone is not sufficient for conversational realism: multiple papers report that listener-aware conditioning, role embeddings, motion continuity losses, gaze losses, or shared semantics improve perceived naturalness and interaction quality [2307.02090][2603.08674][2508.04585][2512.20296]. Third, “multimodal” does not always mean end-to-end joint generation; some systems are decoupled into motion prediction and rendering, or speech modeling with visual conditioning, rather than fully unified synthesis [2307.02090][2503.15633].

The principal technical limitations are also recurrent. Heavy overlapping speech remains challenging in mixed-audio dyads, where occasional lip-sync misattribution can occur [2603.08674]. Occlusions, fast motion, and in-the-wild footage degrade reconstruction quality for both lip and gesture learning [2102.06837][2603.08674][2512.20296]. Listener behavior can remain generic, with limited control over fine-grained states such as nod types or confusion [2603.08674]. More than two speakers are generally out of scope, and multi-party extension would require new stream coupling and spatial layout control [2603.08674][2307.02090]. MultiDialog does not yet exploit its emotion labels in generation, and its current model generates only one responding face rather than both interlocutors jointly [2406.07867]. UniTalker remains offline or semi-real-time, reporting roughly 2 seconds for speech and roughly 5 seconds per 25 video frames at $512\times 512$ on a local RTX 4080 [2508.04585]. TAVID does not claim real-time streaming and does not profile latency under very rapid turn-switching [2512.20296].

Ethical concerns are treated as integral rather than peripheral. The literature repeatedly identifies deepfake risk, impersonation, privacy, licensing, consent, sensitive-content avoidance, demographic bias, watermarking, provenance tracking, misuse detection, and usage restrictions as necessary considerations for deployment [2603.08674][2307.02090][2508.04585][2512.20296]. This suggests that future CSVS systems will likely be evaluated not only on realism and synchrony but also on controllability, provenance, and governance.

A plausible implication is that the next major step for CSVS lies in combining four properties that are currently distributed across separate systems: real-time duplex interaction, strong multimodal dialogue reasoning, explicit spatial and social coordination, and robust safeguards. The present literature establishes these ingredients individually, but no single formulation in the supplied corpus simultaneously solves all of them [2503.15633][2603.08674][2508.04585][2512.20296].

Source: https://www.emergentmind.com/topics/conversational-speech-visual-synthesis-csvs