Wan-Streamer: Real-Time Multimodal Streaming

• Wan-Streamer is a native-streaming interactive foundation model that integrates language, audio, and video in a unified and low-latency processing framework.
• It employs a single Transformer with block-causal attention to jointly model and synchronize multimodal tokens, ensuring robust cross-modal interaction.
• Empirical evaluations demonstrate reduced word error rates and minimal audio-visual synchronization errors compared to traditional cascaded architectures.

Wan-Streamer is an end-to-end, native-streaming interactive foundation model architected for real-time, low-latency, full-duplex audio-visual interaction. Its hallmark is a single Transformer that jointly models language, audio, and video as both input and output in a unified, incrementally streamed sequence without reliance on modularized subsystems for ASR, TTS, avatar animation, or video generation. All perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned within a single model, designed specifically for sub-second interactive performance (Huang et al., 23 Jun 2026).

1. Model Architecture and Token Representation

Wan-Streamer processes each 160 ms interaction window (the "streaming unit") as an interleaved sequence of tokens: visual ($v$), audio ($a$), and text ($t$), on both the user ($u$) and agent ($y$) sides. The system defines three distinct token vocabularies:

• $\mathcal{V}$: discrete visual latents from a causal video VAE,
• $\mathcal{A}$: continuous audio latents from a causal audio VAE (quantized or embedding indices),
• $\mathcal{T}$: text tokens from a standard tokenizer.

At streaming unit $k$, observations are:

• User: $$u_k = (u_k^v, u_k^a, u_k^t) \in \mathcal{V}^* \times \mathcal{A}^* \times \mathcal{T}^*$$
• Agent: $a$0.

The cumulative causal sequence up to unit $a$1 is:

$a$2

which is input directly into a single Transformer. Streaming causality is enforced via a block-causal attention mask $a$3 partitioned such that each block corresponds to one of six token streams per unit. The mask allows tokens in the present or any previous block to attend while future blocks are strictly masked, ensuring incremental processing suitable for native streaming.

2. Causal Encoders, Decoders, and Inference Pipeline

Strictly causal encoders transform each 160 ms audio or video chunk into latents: $a$4 for audio and $a$5 for video. Incoming tokens and latents are appended to the Transformer's key/value (KV) cache, supporting real-time extension of the sequence. Output is generated as three interleaved streams:

• Text tokens by next-token autoregression,
• Audio/video latents by a conditional flow-matching diffusion head $a$6.

Denoised audio and video latents are decoded to waveform ($a$7) and video frames ($a$8), respectively.

The inference pipeline is composed of a "thinker" (manages encoders, Transformer, decoders) and a "performer" (executes the flow-matching diffusion solver):

1. Thinker receives $a$9, encodes to latents, updates KV cache, and performs a short Transformer pass.
2. Concurrently, Performer denoises the next modal latent $t$0 using the previous KV context.
3. Thinker decodes $t$1 returned by Performer, emitting audio and video output.
4. Performer receives the next KV slice and continues the denoising iteration.

This pipeline enables overlapping of perception, state propagation, and emission with prediction at each 160 ms boundary, maintaining continuous 25 FPS operation provided total performer and inter-process communication time does not exceed the per-chunk window.

3. Joint Training Objectives and Procedures

Training employs a multitask objective over a mixture of understanding, generation, and fully duplex interactive data:

• Text cross-entropy for language modeling: $t$2,
• Connectionist Temporal Classification (CTC) loss for auxiliary speech-to-text pretraining,
• Visual classification/captioning via standard cross-entropy for multimodal understanding,
• Flow matching for audio/video latents:

For modality $t$3 and noise level $t$4, the stochastic latent is defined as $t$5, with the diffusion head $t$6 predicting the velocity field, producing loss:

$t$7

• The overall loss is $t$8, with $t$9 aggregating all visual objectives.

Training on end-to-end duplex data, where text/audio/video are simultaneously present on input and output, enables the model to learn inference-time behaviors such as turn management, response timing, interruption, and cross-modal synchronization inherent to interactive dialogues. A final distillation stage creates a student model using rolling self-forcing and embedded classifier-free guidance for student diffusion, compressing the latency envelope to within real-time targets.

4. Latency and Performance Analysis

Latency is measured using a two-GPU thinker-performer pipeline. Model-side latency, defined as the interval from input chunk arrival to emission-ready output, comprises: | Process | Approximate Latency (ms) | |------------------------------------|--------------------------| | Audio/video encoding + tokenization| 30 | | Transformer think pass + KV update | 40 | | Performer (8-step diffusion solve) | 100 | | Audio/video decoding | 30 | | Total model-side latency | 200 |

Inclusion of standard 350 ms two-way network latency yields $u$0 ms total round-trip interaction—encompassing perception, reasoning, and multimodal emission. This latency is signal-to-signal, from user input onset to reception of fully decoded audio-visual output, and is substantiated as the lowest known aligned latency for an end-to-end audio-visual agent, outperforming both speech-only and cascaded partial-multimodal systems (Huang et al., 23 Jun 2026).

5. Comparison with Modular Interactive Architectures

Traditional real-time agents operate as cascades: voice activity detection (VAD) → ASR → LLM → TTS → avatar rendering, with each boundary introducing buffering, non-differentiable timing constraints, and composite error propagation:

• ASR errors produce downstream text misinterpretation,
• TTS enforces fixed output timing decoupled from upstream control,
• Avatar generation or lip sync repairs must be post-processed,

In Wan-Streamer, attention over all modalities is unified, learning joint semantics and co-generation of audio-visual content without intermediate bottlenecks or handover-induced errors. Block-causal attention aligns all tokens per 160 ms streaming unit, eliminating scheduling heuristics and synchronizing video, audio, and text output in one forward pass.

Empirical evaluation highlights tangible benefits:

• Word error rate (WER) on the WebSpeech test set drops from $u$1 (standard ASR-LLM cascade) to $u$2,
• Visual speech-audio synchronization error is reduced from $u$3 ms (post-processed repair) to $u$4 ms native model synchronization.

6. Extensibility and Application Domains

The current system produces 192p video at 25 FPS. Upgrading to HD or 4K requires scaling the causal video VAE and diffusion solver without architectural change. Prolonged interaction, spanning tens of minutes, is achievable by hierarchical memory augmentation. Additional modalities such as depth, gaze, or full-body pose may be incorporated as extra token sequences with minor adjustment to the block-causal attention schema.

Applications include:

• Embodied agents for synchronous visual response and interruption management during user input,
• Live broadcasting avatars capable of in-line translation and animation with sub-second visual feedback,
• Teleoperation interfaces integrating real-time speech, gesture, and visual feedback for remote robotics.

Wan-Streamer establishes a new paradigm in interactive foundation modeling, characterized by joint learning of listening, perception, reasoning, generation, and synchronized multimodal output within a single streaming Transformer framework, enabling sub-second, full-duplex audio-visual interaction previously unattainable via cascaded systems (Huang et al., 23 Jun 2026).