UniTalker: Unified Audiovisual Speech Models

• UniTalker is a research area unifying audio-driven 3D facial animation and conversational speech-visual synthesis via multi-head architectures and robust multimodal modeling.
• It overcomes challenges of heterogeneous 3D annotations and dialogue contexts by integrating pre-trained audio encoders, PCA stabilization, and advanced tokenization techniques.
• Empirical results show reduced error rates and faster inference, positioning UniTalker as a promising foundation for cross-dataset transfer and interactive audiovisual synthesis.

“UniTalker” denotes two distinct but conceptually related research systems in audio-visual speech technology. In one usage, it refers to a unified model for audio-driven 3D facial animation that is designed to learn across heterogeneous 3D annotation formats and large multi-dataset supervision (Fan et al., 2024). In another, it denotes a Conversational Speech-Visual Synthesis (CSVS) system that models multimodal dialogue context and generates empathetic speech together with natural talking-face animations (Hu et al., 6 Aug 2025). Across both usages, the name is associated with unified modeling of speech, facial motion, and multimodal interaction, but the underlying tasks, architectures, and evaluation protocols are different.

1. Terminology and scope

The name “UniTalker” appears in multiple papers with different technical objectives. The two explicit uses are summarized below.

Name	Primary task	Defining characteristics
UniTalker (Fan et al., 2024)	Audio-driven 3D facial animation	Multi-head architecture, A2F-Bench, PCA, model warm-up, pivot identity embedding
UniTalker (Hu et al., 6 Aug 2025)	Conversational Speech-Visual Synthesis	Multimodal dialogue modeling, LmkCodec, bimodal hard alignment, emotion-guided rendering
UniTalk (Nguyen et al., 28 May 2025)	Active speaker detection dataset	44.5 hours, frame-level annotations, 48,693 speaking identities
UniTalking (Li et al., 2 Mar 2026)	Talking portrait generation	Unified diffusion framework, MM-DiT, shared self-attention, voice cloning

The 2024 UniTalker paper addresses the problem of mapping input audio to realistic 3D facial motion despite inconsistent annotation schemes across datasets. Its central claim is that a unified multi-head architecture can absorb supervision from vertices, blendshape weights, and model parameters within one system (Fan et al., 2024). The 2025 UniTalker paper broadens the problem definition from speech-only response generation to CSVS, where multimodal dialogue context is used to produce coherent audiovisual responses rather than speech alone (Hu et al., 6 Aug 2025).

A common source of confusion is that adjacent literature uses closely related names for different subfields. “UniTalk” is a benchmark for active speaker detection rather than an overview model, and “UniTalking” is a unified audio-video generative framework rather than a dialogue-conditioned response system (Nguyen et al., 28 May 2025, Li et al., 2 Mar 2026).

2. UniTalker for audio-driven 3D facial animation

The 2024 UniTalker is an audio-driven 3D facial animation system built to overcome the restriction that previous models were typically tied to one annotation convention and therefore to small, isolated datasets. Its core is a unified multi-head architecture that can learn from and output multiple types of 3D facial annotation simultaneously (Fan et al., 2024).

The system follows an encoder-decoder design with several specific components. The audio encoder uses pre-trained speech models, specifically Wav2Vec2 and WavLM, to extract contextualized audio representations. A frequency adaptor is placed after the transformer to align audio feature frequencies with the output face-motion frame rate. A Temporal Convolutional Network then acts as the motion decoder, and the paper characterizes this decoder as non-autoregressive, faster, and as effective as autoregressive models. Multiple output heads correspond to different annotation conventions, including direct vertices, blendshape weights, and model parameters (Fan et al., 2024).

This architectural choice is motivated by annotation heterogeneity rather than by multimodal dialogue modeling. The model is intended to unify training over datasets that use different 3D representations, so the shared backbone learns a common audio-to-motion representation while the heads specialize to dataset-specific output spaces. In the paper’s formulation, high-dimensional vertex outputs can be reconstructed from a PCA subspace via

$$\hat{\mathbf{y}}_v = \hat{\mathbf{y}}_{PCA} \times \mathbf{W}_L^T$$

and the training objective combines losses across the native vertex space, the PCA space, and parameterized outputs:

$$\mathcal{L} = l(\hat{\mathbf{y}}_v, \mathbf{y}_v) + \alpha \cdot l(\hat{\mathbf{y}}_{PCA}, \mathbf{y}_{PCA}) + \beta \cdot l(\hat{\mathbf{y}}_\theta, \mathbf{y}_\theta)$$

(Fan et al., 2024)

The significance of this UniTalker lies in scale and transfer. Rather than normalizing all datasets into a single annotation target before training, it keeps the representational plurality explicit and learns across it. This makes the model closer to a shared pre-trained backbone with dataset-specific decoders than to a conventional single-dataset animation regressor.

3. Training stabilization, A2F-Bench, and empirical behavior

The 2024 paper pairs the multi-head design with three training strategies intended to stabilize optimization and enforce cross-head consistency: PCA, model warm-up, and pivot identity embedding (Fan et al., 2024).

PCA is applied to high-dimensional vertex labels so that vertex-based supervision becomes comparable in dimensionality and optimization difficulty to lower-dimensional parameter-based heads. The paper describes Incremental-PCA compression to a fixed lower dimension, with 512 components given as an example. Model warm-up is a two-stage procedure in which the pre-trained audio encoder is first frozen while the decoder is updated, after which joint training is performed. Pivot identity embedding introduces an extra pseudo-speaker embedding and randomly switches some labels to this pivot identity with 10% probability, reducing dataset-specific bias in identity conditioning (Fan et al., 2024).

To support large-scale unified training, the authors assemble A2F-Bench from five publicly available datasets and three newly curated datasets. The benchmark comprises 934 speakers, 8,654 sequences, and 18.53 hours of annotated 3D audiovisual data. It includes BIWI, Vocaset, Multiface, 3D-ETF-HDTF, 3D-ETF-RAVDESS, FaceForensics++, a Chinese speech dataset, and a multilingual song dataset. The annotation space spans raw mesh vertices, FLAME parameters, and ARKit blendshapes, and the audio sources include both speech and songs (Fan et al., 2024).

The reported quantitative results emphasize lip articulation accuracy and transfer. With a single trained UniTalker model, the paper reports lip vertex error reductions of 9.2% on BIWI and 13.7% on Vocaset. Fine-tuning the pre-trained UniTalker on seen datasets yields an average error reduction of 6.3% on A2F-Bench. For unseen-dataset adaptation, the paper states that fine-tuning UniTalker on an unseen dataset with only half the data surpasses prior state-of-the-art models trained on the full dataset. The same study also reports inference times of 0.024–0.054 seconds for 10-second audio, compared with 0.7–16.5 seconds for prior autoregressive and diffusion baselines (Fan et al., 2024).

These results position the 2024 UniTalker less as a narrowly scoped benchmark model and more as a pre-training and transfer substrate for audio-to-3D-face tasks. The paper explicitly describes the pre-trained model as showing promise as a foundation model for audio-driven facial animation (Fan et al., 2024).

4. UniTalker for Conversational Speech-Visual Synthesis

The 2025 UniTalker introduces a different problem setting: Conversational Speech-Visual Synthesis. CSVS is defined as an extension of traditional Conversational Speech Synthesis in which the system leverages multimodal dialogue context and returns coherent audiovisual responses rather than speech alone (Hu et al., 6 Aug 2025).

The paper motivates this formulation by arguing that existing CSS systems perceive only text and speech within the dialogue context and therefore miss crucial cues such as listening behavior and eye contact. It further argues that speech-only responses constrain the interactive experience. UniTalker is proposed as a unified model that integrates multimodal perception and multimodal rendering capabilities. It uses a large-scale LLM to understand speaker, text, speech, and talking-face animations in the dialogue context, then applies multi-task sequence prediction to infer the target utterance’s emotion and generate empathetic speech together with natural talking-face animations (Hu et al., 6 Aug 2025).

Its pipeline begins with multimodal tokenization. Text is represented with Byte Pair Encoding tokens. Speech is tokenized at frame level using an FSQ-based speech tokenizer integrated with ASR models. Facial animations are tokenized at frame level by a specialized neural landmark codec called LmkCodec. Speaker information is represented through voiceprint-based speaker embeddings. These serialized multimodal tokens are processed by EVSLM, an LLM-based module built on Qwen2.5-0.5B (Hu et al., 6 Aug 2025).

The task formulation explicitly conditions target prediction on dialogue history and current context:

$$\left\{ U_N^{e'}, U_N^{v'}, U_N^{s'} \right\} \mid \mathcal{H}, \mathcal{C} = \left\{ U_N^{e'}, U_N^{v'}, U_N^{s'} \mid \{ P_i, U_i^v, U_i^s, U_i^t, U_i^e \}_{i=1}^{N-1} \cup \{ P_N, U_N^t \} \right\}$$

where the system predicts the current utterance’s emotion, talking-face animation, and speech from multimodal dialogue context (Hu et al., 6 Aug 2025).

The rendering stage is likewise bimodal. The speech renderer is an emotion-guided conditional flow matching model conditioned on predicted emotion, speaker embedding, speech tokens, and a context spectrogram reference, followed by a HiFi-GAN vocoder. The talking-face animation renderer is based on Echomimic and uses predicted facial landmarks, speech, and reference images to produce high-fidelity animations (Hu et al., 6 Aug 2025).

5. Synchronization, landmark tokenization, and emotion-guided rendering

The 2025 UniTalker emphasizes three optimizations intended to keep the generated speech and visual output consistent in emotion, content, and duration: a specialized neural landmark codec, bimodal speech-visual hard alignment decoding, and emotion-guided rendering (Hu et al., 6 Aug 2025).

LmkCodec is designed to tokenize and reconstruct facial expression sequences efficiently. It processes Mediapipe-extracted landmarks, applies FSQ down-projection, and reconstructs them with a neural decoder. The paper states that it produces one token per frame at 25 Hz, using a 1000-entry codebook. Its tokenization and reconstruction are written as

$$T_{i,<\max}^v = \text{ROUND}\big(\text{FSQ-down}(\text{Encoder}(U_{i, 1 \rightarrow k}^f))\big)$$

and

$$U_{i, 1 \rightarrow k}^{f'} = \text{Decoder}(\text{FSQ-up}(T_{i,<\max}^v))$$

with training based on mean squared error between original and reconstructed landmarks (Hu et al., 6 Aug 2025).

The bimodal speech-visual hard alignment strategy exploits the fact that both the speech and landmark tokenizers operate at the same frame rate, 25 Hz. EVSLM therefore predicts a frame-wise interleaved sequence of visual and speech tokens,

$$\langle T_{N,1}^v, T_{N,1}^s, T_{N,2}^v, T_{N,2}^s, \ldots \rangle$$

using teacher forcing during training and a joint cross-entropy loss over emotion class, facial tokens, speech tokens, and the EOS marker. The paper presents this design as the mechanism that guarantees temporal and content alignment between mouth motion, expression, and speech (Hu et al., 6 Aug 2025).

Emotion guidance is applied to both renderers. For speech, predicted emotion conditions the conditional flow matching model so that the output is empathetic and emotionally congruent. For facial animation, predicted landmarks already encode emotional cues, and the animation renderer preserves this alignment with the speech signal (Hu et al., 6 Aug 2025).

The empirical evaluation spans DailyTalk, NCSSD, MultiDialog, RAVDESS, MEAD, and CelebV-HQ. Objective metrics include FID, PSNR, LPIPS, SSIM, LMD, LSE-C, LSE-D, speaker similarity, pitch dynamics, and emotion accuracy; subjective metrics include MOS for speech naturalness, speech expressiveness, animation quality, animation expression, and lip sync. For LmkCodec validity, the paper reports that LmkCodec with FSQ uses one token per frame and obtains FID 4.15, PSNR 26.1, LPIPS 0.0052, and LMD 0.0018, while alternative codecs use higher token rates with different quality trade-offs. At the system level, the paper states that UniTalker outperforms GPT-Talker and Empatheia in speaker similarity, emotional accuracy, and perceived naturalness, and that it achieves the lowest FID and LPIPS and the highest SSIM and animation emotion accuracy among the compared animation baselines while maintaining lip synchronization (Hu et al., 6 Aug 2025).

6. Relation to neighboring research, misconceptions, and limitations

The term “UniTalker” sits within a broader cluster of work on audiovisual speech understanding and generation, but several neighboring systems address different problems. “UniTalk” is a dataset for active speaker detection rather than a response-generation model. It contains 44.5 hours of video, 48,693 annotated speaking face tracks, and 4 million face crops, and is explicitly curated for underrepresented languages, noisy backgrounds, and crowded scenes (Nguyen et al., 28 May 2025). That benchmark is relevant because active speaker detection is an upstream capability for identifying which visible face is speaking, but it is not itself a speech-visual synthesis system.

“UniTalking” is also distinct. It is a unified, end-to-end diffusion framework for generating synchronized speech audio and lip-synced video, built on Multi-Modal Diffusion Transformer Blocks with shared self-attention over audio and video latent tokens. It supports text prompts, identity images, and reference audio for personalized voice control, and uses a pre-trained Wan2.2-5B video prior together with an MMAudio 1D VAE and BigVGAN vocoder (Li et al., 2 Mar 2026). This makes it closer to general talking portrait generation than to the dialogue-conditioned CSVS formulation of UniTalker (Hu et al., 6 Aug 2025).

Another misconception is to conflate “UniTalker” with the generic single-speaker case in multi-talker ASR. Multi-talker recognition papers on Whisper adaptation, target-talker ASR, and SURT 2.0 address overlapped speech recognition, speaker counting, or streaming unmixing rather than audiovisual response synthesis (Li et al., 2023, Meng et al., 2024, Raj et al., 2023). In SURT 2.0, the summary explicitly notes that the model generalizes to the single-talker case, but this is a usage of “UniTalker” as a scenario description rather than as the title of a specific system (Raj et al., 2023).

The two explicit UniTalker systems also expose different limitations. The 2024 animation model is framed around annotation heterogeneity and data scale, not conversational interactivity, so its contributions are strongest in cross-dataset pre-training and annotation transfer (Fan et al., 2024). The 2025 CSVS model states that current animation synthesis is offline or semi-real-time because of high rendering costs, and it identifies susceptibility to misuse, including deepfakes, as an ethical concern to be addressed by watermarking and responsible usage restrictions (Hu et al., 6 Aug 2025). Taken together, these limitations indicate that “UniTalker” research spans both representation unification and multimodal interaction, but the operational constraints remain task-specific.