Rolling Sink Frame Mechanism (RSFM)

• Rolling Sink Frame Mechanism (RSFM) is a framework that stabilizes real-time streaming diffusion video by maintaining an invariant sink frame and synchronized RoPE schedule.
• It mitigates identity drift and color artifacts by fixing the reference latent vector after the first block and advancing joint time indices in lock-step.
• Empirical results show RSFM enhances performance metrics such as ASE and IQA while sustaining high throughput (~20 FPS) for infinite-length avatar synthesis.

The Rolling Sink Frame Mechanism (RSFM) is a synchronization and reference management framework designed to address long-horizon temporal fidelity and artifact accumulation in real-time, streaming diffusion-based video generation. As introduced in the context of the Live Avatar paradigm for high-fidelity, infinite-length audio-driven avatar synthesis using large diffusion models, RSFM ensures the stability of identity and appearance by maintaining an invariant appearance latent, precisely aligned through time with model-internal positional embeddings. RSFM was specifically introduced to mitigate the challenges of identity drift and color instability inherent to conventional blockwise autoregressive video generation (Huang et al., 4 Dec 2025).

1. Motivation: Long-Horizon Inconsistency and Failure Cases

RSFM directly addresses two principal failure modes observed in blockwise autoregressive diffusion models deployed for avatar generation over extended sequences:

• Identity drift: Small mismatches in attention or denoising at each autoregressive step cause subtle but compounding deviations in facial features or characteristics, leading to a gradual loss of resemblance to the original reference over tens of seconds to minutes.
• Color artifacts: Shifts in exposure, white-balance, and vividness introduce perceptible unrealistic color casts or temporal flickering effects.

Standard frame-to-frame diffusion sampling relies on a "KV cache" memory and generates each block conditional only on recent context, so accumulated errors are not self-correcting. Furthermore, the rotary positional embedding (RoPE) used to encode the relative position of the static reference image in training is fixed; during inference, as stream length increases, the reference’s effective RoPE offset drifts beyond the distribution observed during training, compounding mismatch of identity cues.

2. Mathematical Structure and Core Mechanisms

RSFM is built upon two tightly coupled constructs:

2.1 Sink Frame Definition

Let $S$ denote the “sink frame,” a latent vector that embodies the canonical identity reference of the avatar. After generation of the first block, RSFM sets $S \leftarrow \hat x^1$, where $\hat x^1$ is the denoised latent for the initial output. This vector is held fixed for all subsequent blocks.

2.2 Rolling RoPE Time Coordination

For each denoising step $j$ within a block of size $T$ steps:

• The model maintains two time indices: $t_j$ for the current latent’s diffusion time, and $s_j$ for the sink frame.
• The RoPE schedule enforces $s_0 = t_T$ at the start; at each subsequent step, both $t_j$ and $s_j$ are incremented by $\Delta t = -\frac{1}{T}$. Thus, $t_j - s_j$ remains invariant, ensuring consistency with training-time RoPE offset.

This locked-step advancement ensures the reference’s relative RoPE phase mirrors the positional dynamics learned during training, preventing drift-induced attention degradation.

2.3 Adaptive Attention Sink (AAS) Update

The sink frame is updated only once, after denoising the first block. For $i > 1$, $S$ remains unchanged, precluding semantic drift and aligning the model’s internal reference with its own generation manifold.

3. Algorithmic Workflow

The generation process incorporating RSFM can be summarized as follows:

1. Initialization:
   • Encode reference image $R$ into latent $S$.
   • Set up empty per-step KV caches and define RoPE and denoising hyperparameters $(T, \Delta t, L)$.
2. Per-Block Generation:
   • For each output block, sample random noise and run $T$ denoising steps, applying RoPE with rolling time $s_j$ to the sink.
   • The model predicts velocity and new keys/values for the KV cache and proceeds to the next denoising time.
   • Post-denoising, decode the latent and output the video frame(s).
   • Upon completing the first block, update $S$ to the clean latent and reset $s_0$.
3. KV Cache Management:
   • Key/value caches are truncated and updated each denoising step, with a typical length $L = 4$ blocks.

Pseudocode:

S = encode(R) Initialize KV_1...KV_T as empty for i in range(1, M+1): # M blocks x = sample_noise() for j in range(T, 0, -1): S_j = apply_RoPE(S, s_j) v, kv = vθ(x, t_j, KV_j, c_i, S_j) x = x + v * Δt KV_j = truncate_and_push(KV_j, kv, L) output = Dec(x) if i == 1: S = x s_0 = t_T

Only factual details from (Huang et al., 4 Dec 2025) are shown; see primary text for the full workflow.

4. Appearance Recalibration and Error Correction

The recalibration effect of RSFM is realized via two means:

• AAS (Adaptive Attention Sink): Synchronizes the reference representation with the first block’s output, ensuring it lies within the model’s generative distribution and stabilizes identity reproduction in all subsequent frames.
• Rolling RoPE: By strictly coupling the reference’s temporal coordinate to the evolving diffusion time within each block, the attention mechanism always resolves identity cues with the same relative offset encountered during training. This eliminates the inference-time drift typically observed in naive autoregressive diffusion pipelines.
• Training-time History Corrupt: The model is further regularized by “History Corrupt,” wherein noisy key/value caches perturb context, compelling the model to use the sink frame for static identity retention. This ties training and inference behaviors, though RSFM introduces no additional losses specifically.

5. Integration in Multi-GPU Streaming and System Overheads

Within systems employing Timestep-forcing Pipeline Parallelism (TPP), denoising steps are distributed across multiple GPUs, with each GPU assigned a fixed subset of the total denoising schedule. RSFM operates identically on each GPU, as sunk frame updates and RoPE phase advances are purely local and incur negligible additional computational or communication overhead. The sink frame is broadcast only once (post–block 1), after which all processing is GPU-local. RSFM thus preserves high throughput: ~20 FPS on 5×H800 GPUs is reported for a 14B-parameter model, with no appreciable impact on latency or memory usage.

6. Implementation Hyperparameters and Resource Considerations

Key hyperparameter values for practical deployments include:

Parameter	Value(s)	Notes
Denoising steps $T$	4–6	Real-time/quality trade-off
Step size $\Delta t$	$-1/T$	Uniform schedule
KV cache length $L$	4–8 blocks	Context window
Block size	3 frames
Sink frame update	After block 1	Once only

Hardware requirements are ~20GB for single-GPU (4 steps/14B) or 5×80GB for TPP. The computation introduced by RoPE shifts and sink maintenance is negligible relative to the diffusion machinery.

7. Empirical Results and Ablation Analyses

Quantitative benchmarks on GenBench-LongVideo (7 min) demonstrate:

• Appearance/Style Error (ASE): Improves from 3.00 → 3.38 with RSFM’s full configuration.
• Image Quality Assessment (IQA): Increases from 4.66 → 4.73.
• DINO-S: Remains 0.94, with substantially diminished identity drift.
• Frames/second (FPS): Sustained at ~20.9; time-to-first-frame 2.89 s.

Ablation results highlight RSFM’s importance:

• Without AAS: ASE falls to 3.13, IQA drops to 4.44.
• Without Rolling RoPE: DINO-S declines from 0.93 to 0.86, showing RoPE’s role in identity preservation.
• Without History Corrupt: ASE 2.90, IQA 3.88; reliance on the sink is reduced.

Long-horizon stress tests up to 10,000 seconds show all primary metrics (ASE, IQA, Sync-C, DINO-S) remain stable within ±0.01, empirically establishing RSFM’s efficacy for infinite streaming scenarios.

---

RSFM thus systematically eliminates long-horizon degradation in streaming diffusion-based video synthesis by ensuring an invariant, correctly synchronized reference for conditional attention. Its principles of adaptive reference updating and RoPE schedule maintenance are now foundational for practical, real-time avatar-generation systems using large diffusion models (Huang et al., 4 Dec 2025).