Conversational Speech-Visual Synthesis
- Conversational Speech-Visual Synthesis (CSVS) is a multimodal framework that generates coordinated speech and visual behavior based on conversational context.
- It leverages diverse methods such as diffusion models, autoregressive LLMs, and token-based representations to model turn-taking, prosody adaptation, and visual cues.
- Empirical evidence shows that CSVS improves lip sync, gaze dynamics, and interaction quality, making it promising for digital avatars and immersive communications.
Searching arXiv for papers on conversational speech-visual synthesis and closely related multimodal conversational generation. Conversational Speech-Visual Synthesis (CSVS) denotes a family of multimodal generation tasks in which conversational context is used to synthesize speech, visual behavior, or both in a coordinated manner. In current arXiv literature, the term is not restricted to a single benchmark or output format. It includes mixed-audio generation of co-located 3D dyadic facial performances (Shan et al., 9 Mar 2026), visually grounded conversational speech models that converse about images (Royer et al., 19 Mar 2025), listener-conditioned speech synthesis that adapts prosody to visual feedback (Zhou et al., 2023), multi-turn conversational head generation with explicit speaking and listening roles (Zhou et al., 2023), end-to-end spoken dialogue models that map audio-visual input to audio-visual response without intermediate text (Park et al., 2024), unified text-driven frameworks that jointly synthesize conversational speech and interactive faces (Kim et al., 23 Dec 2025), and multimodal large-language-model systems that predict emotion, speech, and facial animation together (Hu et al., 6 Aug 2025). This breadth suggests that CSVS is best understood as a research umbrella for conversationally coupled speech–visual generation rather than as a single narrowly defined task.
1. Problem formulations and scope
A central distinction in the literature is whether CSVS generates visual behavior from speech, speech from visual context, or both modalities jointly. In the co-located dyadic formulation of “Talking Together: Synthesizing Co-Located 3D Conversations from Audio,” the input is one mixed audio waveform , and the system outputs two complete 3D facial animation streams, one for each participant, including facial expression coefficients, skeletal joint rotations for neck, head, left eye, and right eye, and global head translations over frames (Shan et al., 9 Mar 2026). In that formulation, each participant’s state is represented as , , , and (Shan et al., 9 Mar 2026).
Other formulations invert the conditioning direction. “Visual-Aware Text-to-Speech” defines a listener-aware speech synthesis task in which the model takes phonemes and a streaming listener video sequence and predicts acoustic features and waveform conditioned on both inputs, formalized as (Zhou et al., 2023). The emphasis there is not visual rendering but prosodic adaptation to listener feedback.
A third class of formulations produces both modalities jointly. TAVID takes a text dialogue and a reference image 0 and outputs synchronized conversational speech and interactive facial video for dyadic conversation, with dual-channel semantic token sequences 1 mediating both pipelines (Kim et al., 23 Dec 2025). UniTalker similarly defines CSVS as prediction of the next utterance’s emotion, speech waveform, and facial animation from multimodal conversational history plus the current target utterance’s speaker and text (Hu et al., 6 Aug 2025). The Face-to-Face spoken dialogue model of MultiDialog processes audio-visual speech from a user and generates synchronized audio-visual spoken response without using intermediate text during generation (Park et al., 2024).
A related but narrower usage appears in MoshiVis, where CSVS is framed as generating conversational speech conditioned on a visual input while preserving low latency and prosody and remaining able to switch to non-visual topics (Royer et al., 19 Mar 2025). Although this is not dyadic avatar synthesis, it retains the core CSVS concern of conversational coordination across modalities.
| Formulation | Inputs | Outputs / emphasis |
|---|---|---|
| Co-located dyadic 3D conversation | Mixed audio waveform | Two spatially aware 3D facial performances |
| Visual-aware TTS | Phonemes and listener video | Speech with listener-conditioned prosody |
| Interactive conversational head generation | Audio, roles, dialog cues, partner visuals, identity image | Multi-turn speaking/listening head behavior |
| End-to-end audio-visual dialogue | User audio-visual speech | Audio-visual spoken response |
| Unified text-driven dialogue generation | Text dialogue and reference image | Conversational speech and interactive facial video |
A common misconception is to equate CSVS with conventional talking-head lip synchronization. The literature consistently distinguishes conversational synthesis from one-way “talking head” generation by introducing listener reactions, turn-taking, prosody adaptation, role alternation, mutual gaze, or joint audio–visual generation (Shan et al., 9 Mar 2026).
2. Representations and model families
The field uses several representational regimes. One line relies on parametric 3D face models. The co-located dyadic system uses a 3DMM-based parametric face model with geometry
2
combined with linear blend skinning, axis-angle rotations, and translations in metric meters in a camera/world coordinate system (Shan et al., 9 Mar 2026). Interactive conversational head generation also decouples identity and motion through 3DMM coefficients, with fixed identity-dependent features 3 and dynamic motion features 4 rendered by PIRenderer (Zhou et al., 2023).
A second line relies on landmark or video-token abstractions. UniTalker introduces LmkCodec, a neural landmark codec that tokenizes per-frame 2D facial landmarks at 25 Hz with one token per frame, using 5 and a face vocabulary size of 1,000 (Hu et al., 6 Aug 2025). MultiDialog instead treats audio-visual speech as discrete AV tokens extracted by AV-HuBERT and quantized with 500 HuBERT clusters at 25 Hz, using mouth crops of 6 and resampled 16 kHz audio (Park et al., 2024).
A third line is organized around conversational semantic tokens. TAVID predicts dual-stream semantic tokens from text, where the tokens are derived from the 35th layer of XLS-R and discretized into 10k clusters; these tokens condition both an acoustic denoiser and a latent-diffusion video generator (Kim et al., 23 Dec 2025). This shared-token design is meant to tie lip timing, speaking–listening alternation, and speech generation to the same intermediate representation.
Architecturally, the literature spans diffusion, autoregressive LLMs, CNNs, LSTMs, and conditional flow models. The dyadic 3D conversation model uses a denoising diffusion generator with a shared U-Net backbone, dual noisy streams, inter-speaker cross-attention, role embeddings, and FiLM conditioning (Shan et al., 9 Mar 2026). UniTalker builds its EVSLM module on Qwen2.5-0.5B and performs multi-task token prediction over emotion, facial landmarks, and speech tokens (Hu et al., 6 Aug 2025). MoshiVis augments a pretrained speech LLM with visual cross-attention modules inserted between MHSA and FFN, plus a dynamic gate (Royer et al., 19 Mar 2025). The ViCo/ViCo-X baseline uses streaming LSTM decoders for listener and speaker motion generation, coupled by a role transformer for multi-turn alternation (Zhou et al., 2023). “Learning Speech-driven 3D Conversational Gestures from Video” uses a temporal 1D U-Net-like CNN with three modality-specific decoders and a conditional discriminator over body and hand motion (Habibie et al., 2021).
These differences reflect distinct operating points. Parametric 3D representations support metric spatial reasoning and rig retargeting; token-based LLM formulations support long-context dialogue modeling; sequence-to-sequence LSTMs and temporal CNNs emphasize streaming or lightweight inference; diffusion and flow models prioritize synthesis fidelity (Shan et al., 9 Mar 2026, Zhou et al., 2023, Park et al., 2024, Kim et al., 23 Dec 2025).
3. Interaction modeling, synchronization, and conversational structure
Interaction modeling is the defining technical problem of CSVS. In mixed-audio dyads, the difficulty arises from overlapping speech, frequent turn-taking, and the need to synthesize both speaker and listener behavior from a single waveform (Shan et al., 9 Mar 2026). The dyadic diffusion model addresses this through speaker probability masks, role embeddings 7 and 8, and bidirectional inter-speaker cross-attention in decoder layers so that each participant’s features incorporate the other’s current state (Shan et al., 9 Mar 2026). The same system adds a selective eye gaze loss on samples in the top 20% rotation variance, weighted by 9, to encourage realistic gaze dynamics (Shan et al., 9 Mar 2026).
Synchronization is implemented differently across papers. UniTalker enforces a bimodal speech-visual hard alignment decoding strategy in which facial and speech tokens are generated in a strict interleaved sequence
0
with 1 at 25 tokens/second (Hu et al., 6 Aug 2025). TAVID instead uses the same dual-stream semantic tokens to condition both speech and video pipelines, arguing that shared semantics align timing, lip motion, and interaction patterns across modalities (Kim et al., 23 Dec 2025). MultiDialog also uses a shared AV token sequence for both speech and lip generation, but without an explicit monotonic alignment module beyond token-level conditioning and a Wav2Lip-style renderer (Park et al., 2024).
Listener modeling is explicit in several systems. ViCo/ViCo-X defines responsive listening head generation, expressive talking head generation, and integrated conversational head generation as separate tasks, and its baseline fuses speaker audio, speaker dynamics, listener attitude, and listener reference features in a streaming LSTM (Zhou et al., 2023). VA-TTS imposes a causality constraint at the phoneme level: for phoneme 2, only past listener frames 3 are used, with 4 sufficient at 30 FPS given per-phoneme prosody generation time 5 ms (Zhou et al., 2023). MoshiVis addresses a different form of conversational switching: its gate 6 modulates whether visual evidence influences the speech stream, allowing the model to move between image-grounded content and unrelated conversational topics without explicit gate supervision (Royer et al., 19 Mar 2025).
Spatial awareness is a further differentiator. The dyadic 3D conversation paper trains on absolute first-frame translations and predicts motion deltas, embedding both participants in the same 3D coordinate frame; at test time, Gemini 1.5 is few-shot prompted with example text-to-translation pairs to output initial 3D coordinates from prompts such as “arguing across a table” or “intimate conversation” (Shan et al., 9 Mar 2026). This is a stronger notion of interaction than two isolated heads rendered as if in separate video-call windows.
A plausible implication is that CSVS systems can be organized by what they synchronize: phoneme-to-lip timing, prosody-to-expression coupling, speaker-to-listener responsiveness, or shared 3D scene geometry. The recent literature progressively broadens the synchronization target from local articulation to whole-conversation coordination (Zhou et al., 2023, Shan et al., 9 Mar 2026, Kim et al., 23 Dec 2025).
4. Data resources and curation pipelines
Progress in CSVS is closely tied to dataset construction. The largest reported scale in the supplied literature appears in the co-located dyadic 3D conversation work, which curates a Dyadic Conversation Dataset of over 2 million interacting pairs totaling 50,000+ hours with 10k+ identities, plus a Synthetic Dubbing Dataset of another 50,000+ hours constructed from high-quality single-person frontal videos (Shan et al., 9 Mar 2026). Its dyad pipeline includes scenario filtering to remove “two-scene” video-call layouts, quality control for occluded, tiny, or blurred faces, Looking-to-Listen for per-speaker audio separation, WebRTC VAD for frame-level speaking probability masks, GLEAN-like face super-resolution, and 3DMM reconstruction with metric scale estimated from average inter-ocular distance (Shan et al., 9 Mar 2026).
At smaller but still substantial scale, MultiDialog provides approximately 340 hours of audio-visual dialogues derived from TopicalChat, with 8,733 dialogues, 187,859 utterances, 339.71 hr total length, parallel recordings of both interlocutors, seven emotion classes used at recording time, and turn timestamps logged during capture (Park et al., 2024). The dataset was recorded in a professional studio with green screen, two cameras, and two microphones; recordings missing audio or visual streams were filtered out, and any audio–video misalignment was manually adjusted by sliding start times (Park et al., 2024).
ViCo and ViCo-X target conversational heads rather than full speech–video dialogue. ViCo contains 50 source videos and 483 valid clips spanning 95 min 22 s, with 92 unique people and three attitude labels—positive, neutral, and negative—cross-validated by at least three annotators (Zhou et al., 2023). ViCo-X is a staged multi-turn corpus based on filtered Chinese e-commerce conversations from JDDC, recorded with two actors two meters apart and annotated with ISO 24617-2 dialog acts, frame-resolution sentence timings, and speaker position (Zhou et al., 2023). VA-TTS uses ViCo-X as a multimodal, multi-speaker Mandarin conversational dataset with 10 speakers, 48 kHz 16-bit audio, 30 FPS video, and 3DMM-derived listener dynamics 7 (Zhou et al., 2023).
Other works emphasize scalable pseudo-annotation. The 3D gesture synthesis paper constructs more than 33 hours of annotated body, hand, and face data from in-the-wild videos of talking people using monocular 3D face performance capture, upper-body pose estimation, and 3D hand pose estimation, followed by confidence-based filtering and interpolation for hand gaps up to 8 frames (Habibie et al., 2021). TAVID aggregates approximately 500 hours of video data and approximately 2000 hours of speech data from HDTF, ViCo, Seamless Interaction, LibriTTS-R, DailyTalk, Fisher, and Seamless Interaction, with preprocessing that excludes frames with occlusions, excessive movement, or head rotations above 8 (Kim et al., 23 Dec 2025). UniTalker combines DailyTalk and NCSSD for 113 hours of speech-only dialogue, MultiDialog for 307 hours of visual-spoken dialogue, and RAVDESS, MEAD, and CelebV-HQ for 89 hours of single-utterance audio-visual data (Hu et al., 6 Aug 2025).
These curation strategies reveal a recurrent pattern: conversational data are scarce in native audio-visual form, so the literature repeatedly supplements them with synthetic dubbing, pseudo-dialogue assembly, in-the-wild reconstruction, or mixed single-role and conversational corpora (Shan et al., 9 Mar 2026, Royer et al., 19 Mar 2025, Kim et al., 23 Dec 2025).
5. Objectives, metrics, and empirical evidence
Training objectives vary by formulation but are consistently structured around reconstruction plus interaction-specific constraints. The dyadic diffusion model minimizes
9
with weighted 0 losses on expression, rotation, and translation using 1, 2, and 3, a vertex velocity smoothness term with 4, and an auxiliary gaze loss weighted by 5 (Shan et al., 9 Mar 2026). VA-TTS minimizes the 6 distance between predicted and ground-truth phoneme-wise prosody features in log scale (Zhou et al., 2023). The 3D gesture paper combines modality-specific reconstruction losses with a conditional adversarial objective and a total weight 7 on the adversarial term (Habibie et al., 2021). UniTalker uses MSE for landmark reconstruction, cross-entropy for emotion and token prediction, conditional flow matching for speech rendering, and an Echomimic-based talking-face renderer (Hu et al., 6 Aug 2025). TAVID uses separate objectives for visual diffusion, text-to-semantic cross-entropy, acoustic flow matching, and Speaker Mapper regression (Kim et al., 23 Dec 2025).
Evaluation protocols are correspondingly heterogeneous. The dyadic 3D conversation paper uses Fréchet Distance, Paired FD, parameter MSE and vertex MSE for FULL, EXP, ROT, TRANSL, EYE, and LIP, plus SID diversity, and reports that the proposed method achieves the best performance across nearly all metrics relative to DualTalk and single-speaker baselines (Shan et al., 9 Mar 2026). Its forced-choice study with 19 participants and 14 clips finds preferences for the proposed model in lip quality (79.3%), speaker movements (73.8%), listener movements (73.0%), interaction quality (71.4%), and eye gaze quality (68.3%) over SelfTalk and DualTalk (Shan et al., 9 Mar 2026).
VA-TTS reports consistent objective improvements over a FastSpeech2 baseline: GPE 20.23 to 20.09, VDE 9.54 to 9.23, FFE 20.90 to 20.58, and 8 6.11 to 5.97; phoneme-wise mean absolute error improves from Pitch 37.02 to 35.60, Energy 5.32 to 5.08, and Duration 143.73 ms to 135.97 ms (Zhou et al., 2023). In the conversational gesture work, a user study with 67 participants rates the audio+pose adversarial model at naturalness 9 and synchrony 0, compared with 1 and 2 for direct regression CNN and 3 and 4 for LSTM; a second study finds naturalness 5 and synchrony 6 for the proposed model versus 7 and 8 for MoGlow (Habibie et al., 2021).
ViCo/ViCo-X emphasizes motion distances and lip synchronization. For responsive listening head generation on ViCo-X, ExpFD improves from 13.739 and 17.635 for Random and Mirror to 10.656 for the responsive model, AngleFD from 5.913 to 4.350, and TransFD from 6.316 to 5.129 (Zhou et al., 2023). For expressive talking head generation on ViCo-X, AVOffset improves from 0.636 to 0.083 and AVConf from 2.524 to 2.755 when listener signals are added (Zhou et al., 2023). Human evaluation on ViCo-X reports 9 for the Conversational Agent, 0 for the Listener + Speaker baseline, and 1 for ground truth (Zhou et al., 2023).
In joint speech-and-video generation, TAVID reports strong gains. On Seamless Interaction, TAVID (Text) achieves Visual Quality 2, Lip Sync 3, Turn-taking 4, FID 16.625, FVD 179.305, LPIPS 0.056, LSE-C 6.457, LSE-D 8.403, RPCC 0.031, 5SID 0.489, and 6Var 0.011, improving over audio-driven and TTS-cascaded DIM baselines (Kim et al., 23 Dec 2025). On VoxCeleb2 unseen speakers, TAVID (face-stylized) reports Naturalness 7, Face Matching 8, UTMOS 3.530, and VoxSim 0.380 (Kim et al., 23 Dec 2025). UniTalker on MultiDialog reports SIM_SPK 0.902, PDTW 42.014, ACC_SE 0.743, MOS_SN 4.128, and MOS_SE 4.103 for synthesized speech with visual context, and FID 24.214, PSNR 19.513, LPIPS 0.204, SSIM 0.741, ACC_VE 0.813, LSE-C 6.386, LSE-D 8.189, MOS_VC 4.148, MOS_VN 4.254, and MOS_VE 4.323 for talking-face animation (Hu et al., 6 Aug 2025).
MoshiVis uses conventional visual-understanding benchmarks rather than avatar metrics, but its numbers still illustrate a CSVS trade-off between speech supervision and visual grounding. With 9, it reports OCR-VQA 38.5%, VQAv2 49.3%, and COCO CIDEr 113, while noting that audio quality is degraded at 0% audio and improves quickly with small 0; the best trade-off often occurs at 1 (Royer et al., 19 Mar 2025).
Because these metrics target different subproblems, direct cross-paper ranking is not meaningful. The empirical record instead shows repeated local gains when conversational structure, listener information, shared semantics, or multimodal context are modeled explicitly (Zhou et al., 2023, Zhou et al., 2023, Hu et al., 6 Aug 2025, Kim et al., 23 Dec 2025).
6. Applications, misconceptions, limitations, and open directions
The applications proposed across the literature are consistent: immersive VR and telepresence, digital humans, virtual agents, social robots, avatar chatbots, dubbing, accessibility, and communication aids (Shan et al., 9 Mar 2026, Habibie et al., 2021, Zhou et al., 2023, Park et al., 2024). These applications depend not only on lip sync but also on interaction coherence, role alternation, responsive listening, and identity preservation.
Several misconceptions are addressed implicitly by the body of work. First, CSVS is not solely a speech-driven mouth animation problem. The cited systems model head pose, gaze, facial expression, prosody, listener feedback, body and hand gestures, turn-taking, dialog acts, or joint speech generation (Habibie et al., 2021, Zhou et al., 2023, Zhou et al., 2023, Shan et al., 9 Mar 2026). Second, better articulation alone is not sufficient for conversational realism: multiple papers report that listener-aware conditioning, role embeddings, motion continuity losses, gaze losses, or shared semantics improve perceived naturalness and interaction quality (Zhou et al., 2023, Shan et al., 9 Mar 2026, Hu et al., 6 Aug 2025, Kim et al., 23 Dec 2025). Third, “multimodal” does not always mean end-to-end joint generation; some systems are decoupled into motion prediction and rendering, or speech modeling with visual conditioning, rather than fully unified synthesis (Zhou et al., 2023, Royer et al., 19 Mar 2025).
The principal technical limitations are also recurrent. Heavy overlapping speech remains challenging in mixed-audio dyads, where occasional lip-sync misattribution can occur (Shan et al., 9 Mar 2026). Occlusions, fast motion, and in-the-wild footage degrade reconstruction quality for both lip and gesture learning (Habibie et al., 2021, Shan et al., 9 Mar 2026, Kim et al., 23 Dec 2025). Listener behavior can remain generic, with limited control over fine-grained states such as nod types or confusion (Shan et al., 9 Mar 2026). More than two speakers are generally out of scope, and multi-party extension would require new stream coupling and spatial layout control (Shan et al., 9 Mar 2026, Zhou et al., 2023). MultiDialog does not yet exploit its emotion labels in generation, and its current model generates only one responding face rather than both interlocutors jointly (Park et al., 2024). UniTalker remains offline or semi-real-time, reporting roughly 2 seconds for speech and roughly 5 seconds per 25 video frames at 2 on a local RTX 4080 (Hu et al., 6 Aug 2025). TAVID does not claim real-time streaming and does not profile latency under very rapid turn-switching (Kim et al., 23 Dec 2025).
Ethical concerns are treated as integral rather than peripheral. The literature repeatedly identifies deepfake risk, impersonation, privacy, licensing, consent, sensitive-content avoidance, demographic bias, watermarking, provenance tracking, misuse detection, and usage restrictions as necessary considerations for deployment (Shan et al., 9 Mar 2026, Zhou et al., 2023, Hu et al., 6 Aug 2025, Kim et al., 23 Dec 2025). This suggests that future CSVS systems will likely be evaluated not only on realism and synchrony but also on controllability, provenance, and governance.
A plausible implication is that the next major step for CSVS lies in combining four properties that are currently distributed across separate systems: real-time duplex interaction, strong multimodal dialogue reasoning, explicit spatial and social coordination, and robust safeguards. The present literature establishes these ingredients individually, but no single formulation in the supplied corpus simultaneously solves all of them (Royer et al., 19 Mar 2025, Shan et al., 9 Mar 2026, Hu et al., 6 Aug 2025, Kim et al., 23 Dec 2025).