Wan-Streamer v0.1: End-to-End Streaming Transformer

• Wan-Streamer v0.1 is a unified model that integrates perception, reasoning, and generation for synchronized text, audio, and video interactions.
• It employs block-causal attention with causal encoders/decoders to achieve sub-second latency and reduce error accumulation across modalities.
• The system achieves a model-side response latency of ≈200 ms and a total interaction latency of ≈550 ms, enabling real-time duplex communication.

Wan-Streamer v0.1 is a native-streaming, end-to-end interactive foundation model that addresses high-fidelity, real-time, full-duplex audio-visual interaction using a single unified Transformer architecture. Unlike previous cascaded interactive agent systems that rely on separate modules for voice activity detection (VAD), automatic speech recognition (ASR), text-to-speech (TTS), language modeling, or video generation, Wan-Streamer unifies perception, reasoning, generation, response timing, turn-taking, and cross-modal synchrony within one joint model, thereby reducing pipeline latency and error accumulation. The system is designed to support sub-second total interaction latency (≈550 ms) and model-side response latency of ≈200 ms for audio-video-text conversational agents, facilitated by innovations in block-causal attention, strictly causal encoders/decoders, and low-latency multimodal scheduling (Huang et al., 23 Jun 2026).

1. Unified Model Architecture

Wan-Streamer employs a single autoregressive, interleaved Transformer that encodes and generates sequences of interleaved tokens across three primary modalities: text, audio, and video. Each "streaming unit" represents a temporally aligned block (160 ms at 25 fps) containing user input tokens (text, audio VAE latents, video VAE latents) and agent output tokens (text, audio, video).

Stream Representation

For each streaming unit $k$, the model appends $$u_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}$$ as input and predicts $$y_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}$$ as output, treating the entire conversation as a unified causal sequence over both input and output modalities.

Block-Causal Attention

To support streaming, the attention mechanism divides the sequence into non-overlapping blocks of size $L$ (aligned with the 160 ms streaming unit). The attention mask $M_{ij}$ is defined as:

$$M_{ij} = \begin{cases} 0 & \text{if } \mathrm{block}(j) < \mathrm{block}(i) \ 0 & \text{if } \mathrm{block}(j) = \mathrm{block}(i) \ -\infty & \text{if } \mathrm{block}(j) > \mathrm{block}(i) \end{cases}$$

with weights

$$A_{ij} = \mathrm{softmax}\left[ \frac{Q_i \cdot K_j^T}{\sqrt{d_k}} + M_{ij} \right]$$

This approach allows all tokens within each streaming unit (block) to attend to each other for joint token synchronization, while ensuring strict autoregressive causality between blocks.

Causal Encoders and Decoders

• Input: Audio and video inputs are processed by strictly causal VAEs (1D temporal conv for audio, 3D causal conv for video), producing quantized latent token streams.
• Processing: Transformer encoder layers employ block-causal attention, supporting incremental streaming and immediate cache updates.
• Output: Text decoding uses a standard next-token cross-entropy head. Audio and video generation leverage flow-matching decoders implementing a compact diffusion solver to denoise latents blockwise, restricted to causal and same-block contexts.

2. Streaming Token Scheduling and Inference

The system operates at a granularity of 160 ms per streaming unit (25 fps), synchronizing user audio, video, and potentially text. The inference scheduling follows this loop:

Pseudocode Sketch

$$u_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}$$2

This scheduling, split across two GPUs ("thinker" and "performer"), allows decoder and generator operations for adjacent units to overlap, maintaining per-unit throughput below the 160 ms audio-video block duration.

3. End-to-End Joint Learning

Multi-Task Regimen

Wan-Streamer trains in three principal stages:

• Stage 1 (Multi-task pretraining):
  • Text language modeling via cross-entropy
  • ASR/TTS via cross-entropy on speech tokens
  • Image/video/audio discriminative tasks (classification/regression)
  • Generative modeling via diffusion and flow matching
• Stage 2 (End-to-end duplex interaction):
  • Direct interleaving of user and agent text/audio/video in a single stream
  • Optimization objectives:

  $$L_{\text{text}} = -\sum_t \log p(\text{text}_t | \text{history})$$

  $$L_{\text{FM}} = \mathbb{E}_{t, \text{noise}}\, \|v_\theta(z_t,\text{history}) - (z_t - z_0)/\sqrt{1-T}\|^2$$

  where $z_0$ is the clean target latent and $$u_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}$$0, $$u_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}$$1.

• Stage 3 (Distillation for latency):

  • Teacher uses classifier-free guidance and many flow solver steps; student uses fewer steps
  • Rolling self-forcing aligns student distribution to teacher via sequential rollout.

Joint Perceptual and Generative Synchronization

By avoiding traditional cascades (e.g., VAD → ASR → LLM → TTS → avatar), the model learns joint perception, turn-taking, dialogue planning, and multimodal generation within a single stream. This enables learned synchronization of lip motion, prosody, facial gesture, and text generation without explicit module boundaries.

4. Latency Metrics and System Benchmarks

Latency Results

• Model-side response latency: ≈200 ms from receipt of a 160 ms streaming unit to ready output (using dual A100 GPUs).
• Total interaction latency: ≈550 ms including typical 350 ms bidirectional network round-trip.

Comparative System Performance

System	Scope	Model-Side Latency	Total Interaction Latency
Seeduplex	Full-duplex speech	250 ms	≈550 ms (API)
GPT-4o Realtime API	Speech+vision→speech	232–320 ms	≈500 ms (API)
Qwen3-Omni	AV+text in → speech/text out	234/547 ms	N/R
Wan-Streamer v0.1	AV+text in/out, AV generation	200 ms	550 ms

Unlike many avatar generators, Wan-Streamer achieves full end-to-end audio-visual conversational loop at 25 fps, with results not predicated on separate external modules for dialogue or speech synthesis. Other systems often report only rendering FPS or evaluate only speech modules.

5. Implementation Details and Operational Constraints

Tokenization and Modal Interfaces

• Text: Standard BPE tokenization, leveraging preexisting LLM checkpoints.
• Audio: Streaming causal VAE outputs 16 latents per 160 ms (10 ms hop), discretized via learned codebooks.
• Video: Causal VAE operates at 25 fps with per-frame small latents, similarly discretized.

Hardware and Kernel Optimizations

• Deploys across two GPUs linked via NVLink or PCIe for KV-cache and latent tensor exchange.
• CUDA graph capture applied for efficient per-block execution.
• Custom Triton/Torch kernels implement block-causal attention and compact diffusion solvers.
• KV-cache retention supports continuous streaming for 100s of seconds.

Limitations and Prospective Directions

• Prototype v0.1 operates at 192×192 video resolution; scaling to HD/4K is straightforward but contingent on larger VAE models and increased GPU memory consumption.
• Network latency remains a limiting factor for remote deployments, suggesting future potential in on-device or edge computation.
• Further compression (quantization, pruning) is required for single-GPU real-time operation.
• Expanded capabilities, such as richer turn-management, plug-in skills, and extended memory modules, can be integrated in the same block-causal framework.

6. Position Among Interactive Foundation Models

Wan-Streamer v0.1 represents the first fully end-to-end, native-streaming model to integrate listening (text, audio, video), reasoning (multimodal cognition), decision-making (response timing, interruption), and multimodal output generation within a block-causal Transformer backbone. By interleaving multimodal tokens at 160 ms granularity, leveraging block-causal attention and specialized two-GPU scheduling, it achieves low latency and mutual synchronization of audio-visual streams, addressing bottlenecks endemic to cascaded pipelines (Huang et al., 23 Jun 2026). This architectural integration frames a path toward continuous, low-latency, duplex audio-visual agents capable of richer interactive behavior than current speech-only or staged avatar systems.