---
title: 'JoyAvatar: Real-time Infinite-Length Avatar Synthesis'
url: https://www.emergentmind.com/topics/joyavatar
type: topic
---

# JoyAvatar: Real-time Infinite-Length Avatar Synthesis

JoyAvatar is an audio-driven autoregressive generative model for real-time and infinite-length avatar video synthesis using block-wise diffusion Transformers. It is designed to address computational and quality limitations of prior diffusion Transformer (DiT) methods and autoregressive approaches in audio-driven avatar generation. Key innovations include Progressive Step Bootstrapping (PSB) for improved stability, Motion Condition Injection (MCI) for enhanced temporal coherence, and Unbounded Rotary Position Embeddings via Cache-Resetting (URCR) enabling video generation of arbitrary duration. The architecture is optimized for high performance, achieving real-time synthesis speeds, and is benchmarked on custom out-of-distribution (OOD) test sets for metrics such as lip-sync, temporal consistency, and visual fidelity [2512.11423].

## 1. Block-wise Autoregressive Diffusion Framework

JoyAvatar generates video as a sequence of fixed-length, non-overlapping blocks of latent frames, with each block autoregressively conditioned on all prior blocks and audio input. Given latent frames $x^1,\ldots,x^N$, the model factorizes the video likelihood as:
$$
p(x^{1:N}) = \prod_{i} p(x^i \mid x^{1:i-1}, a)
$$
where $a$ represents the audio driving features. For each block, a few-step denoising DiT instantiates $p(x^i \mid \cdot)$:
- The block’s latents are initially corrupted with Gaussian noise at the maximum noise level.
- $T_i$ denoising steps are performed to recover the clean data, conditioning on:
  - KV (Key/Value) cache of prior block representations (including motion conditions, see MCI).
  - Cross-attention to projected Wav2Vec audio embeddings.
  - Cross-attention to static ArcFace-based identity embeddings.
- After denoising, the generated block is appended to the cache, and the cache is truncated to a sliding window (e.g., 4 blocks) to bound computation. Generation continues sequentially in this manner, supporting infinite-length decoding [2512.11423].

## 2. Progressive Step Bootstrapping (PSB)

PSB addresses error accumulation during autoregressive roll-outs by allocating more denoising steps to the initial blocks, which serve as critical context. Training is performed with a window of 4 blocks:
- Main denoising steps: noise timesteps at [1000, 750, 500, 250].
- An auxiliary “sub-step” branch (midpoints [875, 625, 375, 125]) is stochastically activated during training, supplying denser learning signals for early blocks via optional extra supervision.
- During inference, the number of denoising steps $T_{i}$ per block is {8, 7, 6, 5} for blocks 1–4, reverting to the baseline 4 steps for $i > 4$.

Pseudocode (inference schedule):

```python
for i in 1…block_count:
    if i <= 4:
        Ti = 9 - i    # yields 8,7,6,5
    else:
        Ti = 4
    noisy_block = sample_noise()
    for t in denoise_schedule[1:Ti]:
        noisy_block = DiT_denoise(noisy_block, t, context=KV_cache, audio, id)
    append_block_to_cache(noisy_block)
```

PSB results in stabilized early generation, reducing the compounding of noise and enhancing output consistency [2512.11423].

## 3. Motion Condition Injection (MCI)

MCI is designed to mitigate block-boundary flicker artifacts and improve temporal coherence. For each block $i$:
- The last clean latent $y_{i-1}$ from the preceding block is used to construct a motion condition frame at the start of block $i$ denoising, for noise timestep $t_1$:
$$
c = \sqrt{\bar{\alpha}_{t_1}}\, y_{i-1} + \sqrt{1-\bar{\alpha}_{t_1}}\, \epsilon,\, \epsilon \sim \mathcal{N}(0, I)
$$
where $\bar{\alpha}_{t}$ is the cumulative noise factor.
- This noisy frame $c$ is prepended to the KV-cache, enabling the model to attend to both the actual historical latent frames and this noise-corrupted condition.
- During training, MCI is randomly applied with probability $p=0.7$; at inference, it is always applied [2512.11423].

## 4. Unbounded RoPE via Cache-Resetting (URCR)

Standard Rotary Position Embeddings (RoPE) are limited by fixed positional index capacity, leading to overflow or degradation beyond a sequence length threshold (e.g., 1024 tokens). URCR resolves this, supporting infinite-length video synthesis by:
- “Sink frames” (first block) anchoring global reference via relative embeddings.
- KV states are cached before RoPE application; at each attention invocation, RoPE is dynamically re-applied using the true current block position.
- When the block index $I > N$ (e.g., $N=27$): sink frame indices are reset to zero; subsequent relative positions are renumbered such that no index ever exceeds $N$.

This dynamic resetting maintains all positional indices within a bounded window, preventing index overflow and preserving model performance for arbitrarily long sequences [2512.11423].

## 5. Model Implementation and Quantitative Performance

Major implementation parameters include:
- Model: Causal student, 1.3B parameters; audio-driven bidirectional teacher, 14B parameters.
- Input preprocessing uses Wav2Vec 1.0 for audio features and ArcFace for identity embeddings, with both projected into the DiT via cross-attention.
- Latent frame organization: 3 per block, 4 blocks in window (total up to 27 latents at training, with 3 sink frames).
- Diffusion schedule: [1000, 750, 500, 250] (main), [875, 625, 375, 125] (subs), AdamW optimizer ($\beta_1=0.9, \beta_2=0.999$), learning rates $2 \times 10^{-6}$ (generator), $4 \times 10^{-7}$ (fake score).
- Distributed training: 64 GPUs, window size 4, 10k steps, batch size 1 per GPU.

Quantitative results on a custom OOD test set (100 avatars):

| Metric       | Value       | Interpretation          |
|--------------|-------------|------------------------|
| Sync-C (↑)   | 5.25        | Lip sync (higher better)|
| Sync-D (↓)   | 8.30        | Lip sync deviation (lower better)|
| IDC (↑)      | 0.72        | Temporal consistency    |
| Q_score (↑)  | 0.84        | Visual quality          |
| Q_face (↑)   | 0.53        | Facial fidelity         |
| Inference FPS| 16.2 (DiT)  | Real-time speed         |
| Inference FPS (E2E)| >30 (multi-GPU)| Streaming speed|

JoyAvatar outperforms causal DiT models such as LiveAvatar 14B (4.3 FPS) on lip-sync and temporal metrics while operating approximately 4× faster [2512.11423].

## 6. Known Limitations and Prospective Developments

- The 1.3B parameter causal student model is capacity-limited for scenes with extreme facial details or complexity, notably in close-up generation.
- Extended roll-outs (many minutes) reveal gradual drift in facial appearance, and highly expressive avatars may amplify these effects.
- Rapid head movements can occasionally produce localized motion blur artifacts.
- Ongoing research includes scaling the generator to 5–10B parameters for fine-grained detail, refining controls for semantic attributes (e.g., emotion, gaze), and adopting adaptive denoising step allocations based on online error tracking [2512.11423].

Source: https://www.emergentmind.com/topics/joyavatar