MoCha: Towards Movie-Grade Talking Character Synthesis (2503.23307v1)

Abstract: Recent advancements in video generation have achieved impressive motion realism, yet they often overlook character-driven storytelling, a crucial task for automated film, animation generation. We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. In this paper, we propose MoCha, the first of its kind to generate talking characters. To ensure precise synchronization between video and speech, we propose a speech-video window attention mechanism that effectively aligns speech and video tokens. To address the scarcity of large-scale speech-labeled video datasets, we introduce a joint training strategy that leverages both speech-labeled and text-labeled video data, significantly improving generalization across diverse character actions. We also design structured prompt templates with character tags, enabling, for the first time, multi-character conversation with turn-based dialogue-allowing AI-generated characters to engage in context-aware conversations with cinematic coherence. Extensive qualitative and quantitative evaluations, including human preference studies and benchmark comparisons, demonstrate that MoCha sets a new standard for AI-generated cinematic storytelling, achieving superior realism, expressiveness, controllability and generalization.

• The paper introduces a novel framework that synthesizes movie-grade full-portrait talking character animations using speech and text inputs.
• It employs a specialized speech-video window attention mechanism to achieve precise lip synchronization and expressive character movements.
• The joint training strategy leveraging both speech-labeled and text-labeled video datasets enhances realism and cinematic coherence in multi-character interactions.

MoCha introduces a framework for synthesizing "Talking Characters," moving beyond traditional talking head generation to create full-portrait animations of one or more characters driven by speech and text inputs. The goal is to produce movie-grade animations suitable for automated film and animation, emphasizing character-driven storytelling and cinematic coherence.

Methodology

The core of MoCha is likely built upon a latent video diffusion model architecture, adapted to incorporate speech as a primary driving modality alongside text prompts. Key innovations address the challenges of precise audio-visual synchronization, data scarcity, and multi-character interaction.

Speech-Video Window Attention: To achieve accurate lip synchronization and alignment between speech features and generated video frames, MoCha proposes a specialized attention mechanism. This mechanism operates over windows of speech and video tokens.

Let $S = \{s_1, s_2, ..., s_M\}$ be the sequence of speech tokens (e.g., extracted audio features like MFCCs or learned representations) and $V = \{v_1, v_2, ..., v_N\}$ be the sequence of latent video tokens for a generated clip. The attention mechanism likely computes attention scores between speech tokens within a local window and corresponding video tokens. This focused attention allows the model to correlate specific phonemes or speech patterns ($s_i$) with the necessary visual changes (e.g., lip movements, facial expressions) in the relevant video frames ($v_j$). The windowing aspect helps manage computational complexity and grounds the synchronization locally, which is crucial for speech. The formulation might resemble a cross-attention mechanism where video tokens attend to speech tokens within a defined temporal window:

$$Attention(Q_{video}, K_{speech}, V_{speech}) = softmax(\frac{Q_{video} K_{speech}^T}{\sqrt{d_k}}) V_{speech}$$

Here, $Q_{video}$ are queries derived from video tokens, while $K_{speech}$ and $V_{speech}$ are keys and values derived from speech tokens within the relevant window. This ensures that the visual generation process is tightly conditioned on the fine-grained temporal structure of the input speech.

Joint Training Strategy: Large-scale video datasets with synchronized speech labels are rare. To mitigate this, MoCha employs a joint training strategy leveraging both speech-labeled video data (Speech-Vid) and more abundant text-labeled video data (Text-Vid). The training likely involves alternating batches or using a combined loss function.

$$L_{total} = \lambda_{speech} L_{speech-vid} + \lambda_{text} L_{text-vid}$$

During training steps using Speech-Vid data, the model learns the speech-to-video mapping, focusing on synchronization via the speech-video window attention. During steps using Text-Vid data, the model learns broader semantic understanding, action generation, and scene composition from text descriptions. This joint approach allows the model to learn accurate lip-sync from limited speech data while benefiting from the diversity and scale of text-video datasets for generating complex actions and character appearances, improving overall generalization.

Structured Prompt Templates and Multi-Character Control: To handle multi-character scenes and dialogues, MoCha introduces structured text prompts incorporating character tags. These tags allow users to specify which character is speaking or performing an action at different times. For example, a prompt might look like:

"[Character A]: Hello! How are you? [Scene: Park bench, daytime] [Character B]: I'm doing well, thank you. [Action: Character B smiles]"

The model is trained to parse these tags and associate speech segments or actions with the corresponding designated character. This enables the generation of turn-based conversations where characters interact contextually, maintaining identity consistency and contributing to cinematic coherence. This is presented as the first successful attempt at generating multi-character conversations with turn-taking directly from dialogue inputs in this manner.

Implementation and Training Details

MoCha likely utilizes a pre-trained video generation model (e.g., a latent diffusion model similar to Stable Video Diffusion or others) as its backbone. The speech encoder could be a standard model like Wav2Vec 2.0 or HuBERT, potentially fine-tuned. The key modifications involve integrating the speech encoder's outputs into the diffusion model's conditioning mechanism (e.g., via cross-attention) and implementing the novel speech-video window attention layer.

Training requires curated datasets. The speech-labeled data might come from sources like VFHQ or HDTF, while text-labeled data could be drawn from large-scale datasets like WebVid-10M or Panda-70M. The joint training regimen requires careful balancing of the two data types and loss components ($\lambda_{speech}, \lambda_{text}$). The structured prompts necessitate specific preprocessing to parse character tags and align them temporally with speech segments or action descriptions. Training would be computationally intensive, requiring significant GPU resources, typical for large video generation models.

Evaluation and Results

The paper reports extensive evaluations, comparing MoCha against existing talking head and video generation methods.

Quantitative Metrics: Standard video quality metrics (e.g., FVD, IS) and potentially lip-sync metrics (e.g., SyncNet scores) are likely used. MoCha is reported to outperform prior methods on these metrics, demonstrating higher fidelity and better synchronization.

Qualitative Assessment: Qualitative examples showcase MoCha's ability to generate realistic full-portrait character animations with accurate lip-sync, expressive facial movements, and coherent body language driven by speech. The ability to handle diverse character appearances, actions described in text, and multi-character dialogues with consistent identities is highlighted.

Human Preference Studies: Head-to-head comparisons were conducted where human evaluators rated videos generated by MoCha and baseline methods based on criteria like realism, expressiveness, audio-visual synchronization, and adherence to prompts. MoCha reportedly achieved significantly higher preference scores, indicating a substantial improvement in perceived quality and alignment with user intent for cinematic storytelling. The results suggest MoCha sets a new benchmark in the field.

Applications and Implications

MoCha represents a significant step towards automated generation of narrative content. Potential applications include:

• Automated Film/Animation: Generating dialogue scenes, character animations for movies, cartoons, or pre-visualization.
• Virtual Reality & Metaverse: Creating interactive and responsive virtual characters driven by real-time speech.
• Digital Avatars: Enhancing the realism and expressiveness of digital avatars for communication or entertainment.
• Accessibility: Tools for generating sign language interpretations or visualizing speech for hearing-impaired individuals.

The ability to handle multi-character interactions and maintain cinematic coherence directly from dialogue and text prompts opens possibilities for more complex AI-driven storytelling systems. The joint training strategy also provides a practical approach for leveraging diverse datasets effectively.

Conclusion

MoCha advances the state-of-the-art in talking character synthesis by moving beyond constrained talking heads to full-portrait, multi-character animation generation. Through its novel speech-video window attention for synchronization, joint training strategy for data efficiency, and structured prompting for multi-character control, it demonstrates superior performance in realism, expressiveness, and controllability. The work provides a strong foundation for future research in AI-powered cinematic storytelling and character animation.