AvatarSync: Synchronized Avatar Behavior

• AvatarSync is a research paradigm ensuring temporal coherence between speech, gestures, facial dynamics, and full-body transforms for effective multimodal communication.
• Implementations range from a Mozilla Hubs extension in educational metaverses to high-fidelity Unreal Engine avatars and autoregressive talking-head animation frameworks.
• Evaluations using latency, FID, FVD, and other metrics emphasize the importance of timing consistency, perceptual fidelity, and computational efficiency in these systems.

AvatarSync denotes a line of research concerned with keeping avatar behavior temporally coherent with its driving signals, especially speech, gesture, facial motion, and networked state updates. In the literature provided here, the name is used for multiple systems rather than a single standardized framework: a Mozilla Hubs extension for synchronizing full-body avatar transforms through WebRTC DataChannel in an educational metaverse (Hu et al., 2023), a high-fidelity open embodied avatar built in Unreal Engine 4 and controlled through a simple Python interface (Aneja et al., 2019), and an autoregressive talking-head animation framework that generates phoneme-aligned facial video from a single reference image and text or audio input (Deng et al., 15 Sep 2025). Across these usages, the shared technical concern is that an avatar is not merely a rendered character; it is a multimodal communicative instrument whose effectiveness depends on the alignment of verbal and non-verbal cues.

1. Conceptual scope

A recurring misconception is to equate avatar synchronization with lip synchronization alone. The cited works take a broader view. In the educational-metaverse setting, the central requirement is consistency between voices and gestures, because communicative quality depends on the temporal alignment of speech and body motion (Hu et al., 2023). In the open embodied avatar system, synchronization includes phoneme control, head gesture, facial expression, and retargeted facial behavior driven from webcam input (Aneja et al., 2019). In the autoregressive talking-head system, synchronization is framed as generation over phoneme-aligned visual tokens so that lip motion, facial state, and temporal continuity are modeled directly rather than added after the fact (Deng et al., 15 Sep 2025).

Related work strengthens this interpretation. AV-Flow jointly generates speech, synchronized lip motion, facial expressions, and head pose from text only through two parallel diffusion transformers with intermediate highway connections (Chatziagapi et al., 18 Feb 2025). SyncAnimation addresses audio-driven head pose, upper-body motion, facial expression, blinking, and lip movement in a unified NeRF-based renderer (Liu et al., 24 Jan 2025). AvatarForcing targets real-time, long-form talking-avatar generation under simultaneous constraints of low latency, high visual fidelity, and minute-scale temporal stability (Cui et al., 15 Mar 2026). This suggests that AvatarSync, as a research theme, has evolved from isolated lip alignment toward joint temporal modeling of speech, facial dynamics, body motion, and long-horizon consistency.

2. Full-body transform synchronization in an educational metaverse

In "Synchronizing Full-Body Avatar Transforms with WebRTC DataChannel on Educational Metaverse" (Hu et al., 2023), AvatarSync is a system-level extension to a Mozilla Hubs–based educational metaverse. The starting limitation is that the original platform only supports simplified upper-body avatars with a hard-coded bone set and hierarchy. That design keeps client-side computation low and helps accessibility, especially on low-end devices, but it constrains non-verbal communication, particularly the body gestures that matter in teaching, discussion, and collaboration.

The extension has two technical components. First, it generalizes avatar support from the fixed Hubs hierarchy to arbitrary or full-body avatar skeletons. Bone mapping is implemented by matching bones by name similarity to body parts; the example given is LowerArm.R mapped to the right elbow. The system also adds Cyclic Coordinate Descent Inverse Kinematics (CCD IK), allowing a full-body avatar to reflect user poses naturally even when only limited pose inputs are available. Second, avatar transform synchronization is delegated from the platform’s original WebSocket-based mechanism to WebRTC DataChannel. Audio remains transmitted through the WebRTC-based real-time communication stack, while avatar transforms move over a direct browser communication channel.

The rationale for this networking choice is twofold. WebRTC DataChannel enables real-time browser-to-browser transmission, runs on top of UDP while retaining reliability properties closer to TCP, uses DTLS to encrypt or encode the transmission, and avoids routing avatar transform updates through a central server (Hu et al., 2023). The authors therefore argue for both reduced voice-gesture inconsistency and improved security or privacy, since full-body transforms can reveal sensitive personal information through body pose and movement. The synchronized state is explicitly the avatar’s per-bone transform state: position, orientation, and size of bones.

The preliminary validation compares the original implementation with the proposed DataChannel-based implementation on a self-hosted Mozilla Hubs instance. One Apple MacBook serves as sender and one Windows 10 desktop as observer; both are connected to the same room. Over 5 minutes, the sender repeatedly plays a 3-second audio clip while synchronizing circular movements of the avatar’s left hand, and the observer records timestamps for received audio and observed avatar motion (Hu et al., 2023).

Condition	Average latency	Standard deviation
Proposed implementation	257.64 ms	16.10 ms
Original implementation	184.04 ms	49.81 ms

The reported result is nuanced. In both conditions, avatar transforms arrived faster than audio. The proposed implementation did not demonstrate a smaller absolute latency gap, but it showed more consistent timing between audio and avatar transforms, indicated by the lower standard deviation (Hu et al., 2023). The authors explicitly note limitations: the avatar motion was triggered when the audio clip was played rather than exactly when audio transmission began, only two devices were used, both were in the same room, and larger client loads may better expose architectural differences. For that reason, the study is best characterized as a proof-of-concept rather than a conclusive performance benchmark.

3. Embodied avatar control, lip syncing, and expression retargeting

A different use of the name appears in "A High-Fidelity Open Embodied Avatar with Lip Syncing and Expression Capabilities" (Aneja et al., 2019). Here AvatarSync is a high-fidelity open embodied avatar system built in Unreal Engine 4 inside the AirSim environment and controlled through a simple Python interface. Once the avatar executable is running, a Python client connects to it and can update avatar controls as frequently as needed. The release includes code, models, and scripts on GitHub and is distributed under the MIT license plus a responsible AI license.

The avatar exposes several control channels: 38 bone positions for facial landmarks and upper-body or head-related parts, 24 facial action unit presets, 19 phonetic mouth-shape presets, head rotation in yaw, pitch, and roll, camera pose with 6 degrees of freedom, and skin tone and age texture parameters (Aneja et al., 2019). Each bone position has 6 degrees of freedom, with position $(x_b, y_b, z_b)$ and orientation $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$. This makes the system an expressive rig rather than a single-purpose lip-sync model.

Its lip-syncing pipeline is intentionally explicit rather than end-to-end learned. In the conversational-agent example, text is generated using a public natural language API from Microsoft, converted to speech using a text-to-speech service, converted into a sequence of visemes using PocketSphinx, and then synchronously played while the avatar’s phoneme presets are driven to match it (Aneja et al., 2019). The system supports 19 phoneme presets even though English has about 44 phonemes; the paper states that the smaller set still works effectively for most dialogue. A practical caveat is that lip syncing and expression retargeting can issue competing commands to the mouth. The proposed workaround is to control only the upper face while lip syncing, leaving the mouth region for phoneme-driven animation.

Facial expression control is available either through FACS action unit controls or through basic emotional expression control created as combinations of action units or via expression retargeting. The AU recognizer is a CNN trained on Emotionet in PyTorch with binary cross-entropy and predicts 12 AUs as a 12-dimensional probability vector; the reported average F1-score is 0.78 (Aneja et al., 2019). A second retargeting path uses ExprGen and a Character Multi-Layer Perceptron trained with mean square error to transfer expression from a primary character to the avatar. These two retargeting modes show that the system can be driven either by semantic facial attributes or by learned geometric transfer.

The limitations are explicit. FACS action unit intensity is not validated, composite expressions are rule-based, concurrent lip sync and expression control can cause implausible facial movement, and future evaluation is suggested using FACS and perceptual studies (Aneja et al., 2019). The significance of this system lies less in a novel synchronization loss than in providing a reusable research platform that unifies phoneme control, AU control, expression retargeting, head pose control, and real-time scripting.

4. Autoregressive phoneme-aligned talking-head generation

"AvatarSync: Rethinking Talking-Head Animation through Autoregressive Perspective" (Deng et al., 15 Sep 2025) reframes synchronized talking-head generation as autoregressive generation over phoneme-aligned visual tokens. The task is defined as generating a realistic talking-head video from a single reference portrait image plus either text or speech audio, with correct lip sync, identity preservation, and smooth facial or head motion. The paper’s critique of prior methods is twofold: GAN-based methods are fast but often produce artifacts and weak identity preservation, while diffusion-based methods are visually sharper but rely on multi-step denoising and remain prone to inter-frame flicker and identity drift.

The autoregressive alternative is formalized as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$

with frame-token generation conditioned on prior visual context and conditioning input $c$ (Deng et al., 15 Sep 2025). The core architectural choice is to operate at the phoneme level. Text or speech is converted into phoneme sequences, while the reference image is encoded into visual tokens using a pretrained visual tokenizer such as Open-MAGVIT2 or an LFQ/VQ-style tokenizer. Facial regions are localized with MMPose for better facial alignment and cropped appropriately.

The model follows a two-stage "divide and conquer" design. Facial Keyframe Generation (FKG) focuses on phoneme-level semantic representation by generating sparse key facial frames aligned to phoneme units. Inter-frame interpolation then fills in the motion between keyframes, focusing on temporal coherence and visual smoothness. The paper introduces Phoneme-to-Visual Mapping to anchor abstract phonemes to character-level visual units, and a customized Text-Frame Causal Attention Mask so that each keyframe attends only to its corresponding phoneme. Among several attention masks examined—non-causal global attention, causal accumulative attention, limited history attention, and one-to-one attention—the proposed one-to-one design is presented as the best causal choice for synchronized generation (Deng et al., 15 Sep 2025).

Training combines token-level reconstruction and pixel-space refinement. The reconstruction-style objective is a token-level cross-entropy over visual tokens, and the second stage adds LPIPS perceptual loss, identity consistency loss, and facial similarity loss:

$$\mathcal{L}_{\mathit{total}} = \lambda_{\text{1}} \cdot \mathcal{L}_{\mathit{recon}} + \lambda_{\text{2}} \cdot \mathcal{L}_{\mathit{lpips}} + \lambda_{\text{3}} \cdot \mathcal{L}_{\mathit{Id}} + \lambda_{\text{4}} \cdot \mathcal{L}_{\mathit{FS}}.$$

Interpolation uses phoneme-timestamp pairs together with a timestamp-aware adaptive strategy based on a selective state space model inspired by VFIMamba, enabling efficient bidirectional context reasoning and parallel processing of different keyframe intervals (Deng et al., 15 Sep 2025).

Evaluation is reported on CMLR and HDTF with a 95:5 train-test split. On CMLR, AvatarSync achieves FID 17.11, FVD 189.24, LPIPS 0.07, PSNR 24.14, SSIM 0.86, and Sync-D 0.98. On HDTF, it reports FID 23.69, FVD 251.89, LPIPS 0.21, PSNR 23.34, SSIM 0.89, and Sync-D 1.26 (Deng et al., 15 Sep 2025). The paper also states that generation time is near-linear in phoneme count and that, at 20 phonemes, AvatarSync is 2.4× faster than Hallo. In this usage, AvatarSync is not a network-transport system or a general embodied-avatar toolkit; it is a phoneme-driven autoregressive video generator aimed at visual fidelity, temporal consistency, controllability, and computational efficiency.

5. Adjacent synchronization paradigms

Several nearby systems illuminate how the broader synchronization problem is being decomposed. AV-Flow is a text-to-audio-visual generation system for photo-realistic 4D talking avatars that uses two parallel diffusion transformers, one generating a mel-spectrogram $S \in \mathbb{R}^{n \times 80}$ and the other generating head pose $H$ and facial encodings $F$, coupled through intermediate highway connections (Chatziagapi et al., 18 Feb 2025). Its training objective is

$$\mathcal{L}_{\text{AV-Flow}} = \lambda_s \mathcal{L}_s + \lambda_h \mathcal{L}_h + \lambda_f \mathcal{L}_f,$$

with the paper noting that L1 is used rather than L2 in practice. Quantitatively, it reports $F1_{\text{lips}} = 0.964$, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$0, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$1, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$2, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$3, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$4, $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$5, and $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$6 (Chatziagapi et al., 18 Feb 2025). In dyadic conversation, audio-visual guidance improves smile F1 from 0.611 to 0.685.

SyncAnimation approaches the problem through a three-module pipeline: AudioPose Syncer, AudioEmotion Syncer, and High-Synchronization Human Renderer (Liu et al., 24 Jan 2025). It explicitly models audio-driven head pose, facial expression coefficients, and blinking dynamics, then renders head and upper body together through a NeRF field rather than pasting a synthesized face back onto an original torso. The renderer is defined as

$(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$7

and the method reports real-time inference at 41 FPS. Representative quantitative results include, for SyncAnimation-One, PSNR 21.2323, LPIPS 0.1543, SSIM 0.7343, FID 20.0567, LMD 3.2215, AUE 3.5485, Diversity 0.2570, and EAR 0.0357 (Liu et al., 24 Jan 2025). The zero-shot version uses Gaussian noise instead of reference pose or blendshape templates and remains comparable, which the authors interpret as evidence of genuinely audio-driven generation.

AvatarForcing targets minute-scale temporal stability in streaming talking-avatar generation from a reference image plus streaming audio (Cui et al., 15 Mar 2026). It introduces one-step streaming diffusion over a fixed local-future window $(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$8, dual-anchor temporal forcing, and two-stage streaming distillation. The core window formulation is

$(\sigma_{b1}, \sigma_{b2}, \sigma_{b3})$9

On a new long-form benchmark of nearly 400 videos, it reports FID 56.22, FVD 737.92, CSIM 0.91, Sync-C 5.64, Sync-D 9.26, and latency 0.034 s/frame = 34 ms/frame using a 1.3B-parameter student model (Cui et al., 15 Mar 2026). The ablations indicate that removing the style anchor, temporal anchor, anchor-audio zero-padding, or RoPE re-indexing degrades long-form stability. Taken together, these systems show that the technical center of gravity in avatar synchronization has shifted toward explicit temporal modeling, multi-channel conditioning, and deployment-aware inference design.

6. Data foundations, evaluation regimes, and relation to avatar creation

The dataset bottleneck is addressed directly by Allo-AVA, a large-scale multimodal dataset for allocentric avatar gesture animation (Punjwani et al., 2024). It contains 7,500 videos, ~1,250 hours, 135 billion extracted keypoints, 15 million transcribed words, 3 million+ annotations, and 2.46 TB total size. The construction pipeline uses yt-dlp, moviepy, OpenAI Whisper base with word-level timestamps, OpenPose, MediaPipe, and Dynamic Time Warping for alignment. The multimodal input is formalized as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$0

and the target gesture sequence as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$1

The paper reports correlations such as hand movement vs. speech rate at 0.72 and gesture amplitude vs. emotional intensity at 0.68, while also noting that speech rate and movement intensity have only a very weak correlation of 0.0285 (Punjwani et al., 2024). This directly supports the view that synchronized avatar behavior requires more than a simplistic lip-sync or speech-rate heuristic.

A complementary evaluation perspective is provided by AVATAR, a video-centric Audio-Visual Localization benchmark, and TAVLO, its temporal-aware model (Choi et al., 7 Jul 2025). AVATAR defines four scenarios—Single-sound, Mixed-sound, Multi-entity, and Off-screen—and a Cross-event subset where the active sound source changes over time. TAVLO preserves temporal structure by representing visual features as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$2

audio features as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$3

and inference-time localization maps as

$P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$4

On Cross-event videos, conventional methods show drops as large as up to $P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$5 CIoU and up to $P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$6 AUC, whereas TAVLO shows only $P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$7 CIoU and $P(X) = \prod_{i=1}^{N} P(x_i \mid x_{<i}, c),$8 AUC (Choi et al., 7 Jul 2025). Although this is localization rather than avatar animation, it sharpens the temporal criterion implicit in AvatarSync: the relevant question is not merely what object could plausibly be speaking, but what is making that sound right now.

Avatar synchronization should also be distinguished from avatar creation. SwiftAvatar addresses selfie-to-avatar auto-creation on arbitrary avatar engines through dual-domain generators, GAN inversion, semantic augmentation, and a lightweight avatar vector estimator (Wang et al., 2023). AvatarStudio creates high-fidelity, animatable 3D human avatars from text alone through a coarse-to-fine pipeline combining a NeRF stage, DMTet mesh refinement, SMPL-guided articulation, and DensePose-conditioned SDS (Zhang et al., 2023). Arc2Avatar generates expressive 3D head avatars from a single image via identity guidance, 3D Gaussian Splatting, FLAME correspondence, and optional SDS-based correction for expressions (Gerogiannis et al., 9 Jan 2025). These works solve representation, parameterization, and animatability problems rather than the timing problem itself, but they provide the avatar assets and control structures on which synchronization systems can operate.

Across the literature, the central implication is consistent: realistic avatars require aligned timing between speech, motion, and visual state, yet the implementation locus varies sharply. In some works the synchronization problem is network transport and browser-native state dissemination (Hu et al., 2023); in others it is real-time control over phonemes, action units, and retargeted expressions (Aneja et al., 2019); in still others it is autoregressive generation, diffusion coupling, NeRF-based rendering, or long-form streaming stability (Deng et al., 15 Sep 2025). The term "AvatarSync" therefore denotes not one canonical architecture, but a technical agenda centered on temporally coherent avatar embodiment.