---
title: 'Wan-Streamer v0.1: End-to-End Streaming Transformer'
url: https://www.emergentmind.com/topics/wan-streamer-v0-1
type: topic
---

# Wan-Streamer v0.1: End-to-End Streaming Transformer

Wan-Streamer v0.1 is a native-streaming, end-to-end interactive foundation model that addresses high-fidelity, real-time, full-duplex audio-visual interaction using a single unified Transformer architecture. Unlike previous cascaded interactive agent systems that rely on separate modules for voice activity detection (VAD), automatic speech recognition (ASR), text-to-speech (TTS), language modeling, or video generation, Wan-Streamer unifies perception, reasoning, generation, response timing, turn-taking, and cross-modal synchrony within one joint model, thereby reducing pipeline latency and error accumulation. The system is designed to support sub-second total interaction latency (≈550 ms) and model-side response latency of ≈200 ms for audio-video-text conversational agents, facilitated by innovations in block-causal attention, strictly causal encoders/decoders, and low-latency multimodal scheduling [2606.25041].

## 1. Unified Model Architecture

Wan-Streamer employs a single autoregressive, interleaved Transformer that encodes and generates sequences of interleaved tokens across three primary modalities: text, audio, and video. Each "streaming unit" represents a temporally aligned block (160 ms at 25 fps) containing user input tokens (text, audio VAE latents, video VAE latents) and agent output tokens (text, audio, video).

### Stream Representation

For each streaming unit $k$, the model appends 
$$
u_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}
$$ 
as input and predicts 
$$
y_k = \{\text{text}_k, \text{audio}_k, \text{video}_k\}
$$ 
as output, treating the entire conversation as a unified causal sequence over both input and output modalities.

### Block-Causal Attention

To support streaming, the attention mechanism divides the sequence into non-overlapping blocks of size $L$ (aligned with the 160 ms streaming unit). The attention mask $M_{ij}$ is defined as:
$$
M_{ij} = \begin{cases}
0 & \text{if } \mathrm{block}(j) < \mathrm{block}(i) \\
0 & \text{if } \mathrm{block}(j) = \mathrm{block}(i) \\
-\infty & \text{if } \mathrm{block}(j) > \mathrm{block}(i)
\end{cases}
$$
with weights 
$$
A_{ij} = \mathrm{softmax}\left[ \frac{Q_i \cdot K_j^T}{\sqrt{d_k}} + M_{ij} \right]
$$
This approach allows all tokens within each streaming unit (block) to attend to each other for joint token synchronization, while ensuring strict autoregressive causality between blocks.

### Causal Encoders and Decoders

- **Input**: Audio and video inputs are processed by strictly causal VAEs (1D temporal conv for audio, 3D causal conv for video), producing quantized latent token streams. 
- **Processing**: Transformer encoder layers employ block-causal attention, supporting incremental streaming and immediate cache updates.
- **Output**: Text decoding uses a standard next-token cross-entropy head. Audio and video generation leverage flow-matching decoders implementing a compact diffusion solver to denoise latents blockwise, restricted to causal and same-block contexts.

## 2. Streaming Token Scheduling and Inference

The system operates at a granularity of 160 ms per streaming unit (25 fps), synchronizing user audio, video, and potentially text. The inference scheduling follows this loop:

#### Pseudocode Sketch

```python
for each streaming unit k:
    # THINKER (GPU 0)
    encode_audio_video(u_k)                     # causal encoders
    KV_k = Transformer_token_causal(KV_{k-1}, u_k)
    send KV_k → Performer

    receive {audio_latents,video_latents}_{k-1} from Performer
    decode_audio_video({audio_latents,video_latents}_{k-1})  # causal decoders
    emit text tokens for unit k if available

    # Meanwhile, PERFORMER (GPU 1)
    receive KV_k from Thinker
    append = KV_k
    y_k_latents = flow_matching_solver(KV_k)    # next block's audio/video latents
    send y_k_latents → Thinker
```

This scheduling, split across two GPUs ("thinker" and "performer"), allows decoder and generator operations for adjacent units to overlap, maintaining per-unit throughput below the 160 ms audio-video block duration.

## 3. End-to-End Joint Learning

### Multi-Task Regimen

Wan-Streamer trains in three principal stages:

- **Stage 1 (Multi-task pretraining):** 
  - Text language modeling via cross-entropy
  - ASR/TTS via cross-entropy on speech tokens
  - Image/video/audio discriminative tasks (classification/regression)
  - Generative modeling via diffusion and flow matching

- **Stage 2 (End-to-end duplex interaction):**
  - Direct interleaving of user and agent text/audio/video in a single stream
  - Optimization objectives:
    $$
    L_{\text{text}} = -\sum_t \log p(\text{text}_t | \text{history})
    $$
    $$
    L_{\text{FM}} = \mathbb{E}_{t, \text{noise}}\, \|v_\theta(z_t,\text{history}) - (z_t - z_0)/\sqrt{1-T}\|^2
    $$
    where $z_0$ is the clean target latent and $z_t = (1-T)z_0 + T\epsilon$, $\epsilon \sim \mathcal{N}(0, I)$.

- **Stage 3 (Distillation for latency):**
  - Teacher uses classifier-free guidance and many flow solver steps; student uses fewer steps
  - Rolling self-forcing aligns student distribution to teacher via sequential rollout.

### Joint Perceptual and Generative Synchronization

By avoiding traditional cascades (e.g., VAD → ASR → LLM → TTS → avatar), the model learns joint perception, turn-taking, dialogue planning, and multimodal generation within a single stream. This enables learned synchronization of lip motion, prosody, facial gesture, and text generation without explicit module boundaries.

## 4. Latency Metrics and System Benchmarks

### Latency Results

- **Model-side response latency:** ≈200 ms from receipt of a 160 ms streaming unit to ready output (using dual A100 GPUs).
- **Total interaction latency:** ≈550 ms including typical 350 ms bidirectional network round-trip.

### Comparative System Performance

| System                   | Scope                                | Model-Side Latency | Total Interaction Latency |
|--------------------------|--------------------------------------|--------------------|--------------------------|
| Seeduplex                | Full-duplex speech                   | 250 ms             | ≈550 ms (API)            |
| GPT-4o Realtime API      | Speech+vision→speech                 | 232–320 ms         | ≈500 ms (API)            |
| Qwen3-Omni               | AV+text in → speech/text out         | 234/547 ms         | N/R                      |
| Wan-Streamer v0.1        | AV+text in/out, AV generation        | 200 ms             | 550 ms                   |

Unlike many avatar generators, Wan-Streamer achieves full end-to-end audio-visual conversational loop at 25 fps, with results not predicated on separate external modules for dialogue or speech synthesis. Other systems often report only rendering FPS or evaluate only speech modules.

## 5. Implementation Details and Operational Constraints

### Tokenization and Modal Interfaces

- **Text:** Standard BPE tokenization, leveraging preexisting large language model checkpoints.
- **Audio:** Streaming causal VAE outputs 16 latents per 160 ms (10 ms hop), discretized via learned codebooks.
- **Video:** Causal VAE operates at 25 fps with per-frame small latents, similarly discretized.

### Hardware and Kernel Optimizations

- Deploys across two GPUs linked via NVLink or PCIe for KV-cache and latent tensor exchange.
- CUDA graph capture applied for efficient per-block execution.
- Custom Triton/Torch kernels implement block-causal attention and compact diffusion solvers.
- KV-cache retention supports continuous streaming for 100s of seconds.

### Limitations and Prospective Directions

- Prototype v0.1 operates at 192×192 video resolution; scaling to HD/4K is straightforward but contingent on larger VAE models and increased GPU memory consumption.
- Network latency remains a limiting factor for remote deployments, suggesting future potential in on-device or edge computation.
- Further compression (quantization, pruning) is required for single-GPU real-time operation.
- Expanded capabilities, such as richer turn-management, plug-in skills, and extended memory modules, can be integrated in the same block-causal framework.

## 6. Position Among Interactive Foundation Models

Wan-Streamer v0.1 represents the first fully end-to-end, native-streaming model to integrate listening (text, audio, video), reasoning (multimodal cognition), decision-making (response timing, interruption), and multimodal output generation within a block-causal Transformer backbone. By interleaving multimodal tokens at 160 ms granularity, leveraging block-causal attention and specialized two-GPU scheduling, it achieves low latency and mutual synchronization of audio-visual streams, addressing bottlenecks endemic to cascaded pipelines [2606.25041]. This architectural integration frames a path toward continuous, low-latency, duplex audio-visual agents capable of richer interactive behavior than current speech-only or staged avatar systems.

Source: https://www.emergentmind.com/topics/wan-streamer-v0-1