Papers
Topics
Authors
Recent
Search
2000 character limit reached

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Published 23 Jun 2026 in cs.CV, cs.AI, cs.GR, and cs.SD | (2606.25041v1)

Abstract: We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.

Summary

  • The paper introduces a fully unified end-to-end Transformer that synchronously processes language, audio, and video.
  • It employs block-causal attention and a thinker-performer pipeline to achieve sub-second response latency with 25 FPS video.
  • The work challenges cascaded paradigms by integrating multimodal perception, reasoning, and generation in a single causal stream.

End-to-End Real-Time Interactive Foundation Models: An Analysis of Wan-Streamer v0.1

Model Architecture and Technical Innovations

Wan-Streamer v0.1 proposes a fully unified, end-to-end Transformer-based architecture for real-time, full-duplex audio-visual interaction. All modalities—language, audio, and video—are encoded as input and output tokens within a single temporally causal sequence, processed by a Transformer with block-causal attention. This allows Wan-Streamer to synchronously handle perception, reasoning, generation, response timing, and cross-modal synchronization. The model explicitly rejects the cascaded module paradigm (VAD, ASR, TTS, avatar animation), which typically incurs additional latency and error accumulation at step boundaries. Causality is imposed throughout: all VAEs (for audio and video), encoders, decoders, and the central Transformer operate in strictly causal fashion, enabling immediate emission and commitment of each streaming unit (160 ms, 25 FPS) to history.

The system is deployed via a "thinker-performer" pipeline on two GPUs, where the thinker handles perceptual encoding, KV-cache update, and output decoding, while the performer conducts generation of the next audio-visual latent response via a flow-matching solver. The overlap between current-frame perception/state update, previous-frame output decoding, and next-frame latent computation ensures minimal wasted compute and latency.

Training Methodology

Training proceeds in three stages:

  1. Independent-Task Pretraining: The Transformer core is initialized from a LLM and trained with a mixture of multimodal understanding and generation data. Audio, video, and image VAEs, and their encoders/decoders, are optimized jointly with the language backbone for alignment across modalities.
  2. End-to-End Interaction Training: Duplex interaction data interleaves user and agent text/audio/video streams on a single timeline, enabling the model to learn realistic response timing, turn-taking, visible listening behaviors, and interruption handling under a unified causal format.
  3. Low-Latency Distillation: A high-capacity teacher with classifier-free guidance (CFG) and more solver steps is distilled into a deployment-efficient student. Rolling distillation and self-forcing are used to preserve long-context consistency and minimize train-test mismatch, as the student is periodically rolled out and trained on its own generated history.

The audio and video response streams are generated jointly with flow-matching diffusion, conditional on the complete causal context, and are committed back to the context for future generation.

Empirical Results

Wan-Streamer v0.1 achieves a model-side response latency of approximately 200 ms and a total interaction latency of approximately 550 ms when including an example 350 ms network round-trip time. This is achieved with concurrent real-time 25 FPS video and synchronized audio output, suitable for conversational avatars and digital humans.

Relative to prior work, Wan-Streamer makes the explicit claim that audio-visual perception, language reasoning, dialogue management, speech generation, and visual generation are all optimized and executed by a single model, rejecting hidden intermediate representations or module boundaries. Systems such as GPT-4o, Seeduplex, and Moshi deliver speech-driven dialogue or speech-to-speech interfaces with similar model-only latency, but do not generate synchronized visual content. Other digital human frameworks either still depend on cascaded pipelines (e.g., MIDAS, LPM 1.0, X-Streamer, MAVID) or act solely as audio/video renderers, with latencies not including semantic reasoning and response timing.

The model's design also supports highly naturalistic behavior. During non-speaking intervals, the agent maintains subtle visual activity (posture, gaze, expressions); during listening states, it produces temporally aligned feedback (nods, micro-expressions). Proactive and reactive turn-taking—including interruption handling—emerges via data-driven learning on interleaved, full-duplex interaction streams, rather than hand-coded policies.

Theoretical and Practical Implications

Wan-Streamer v0.1 embodies a shift from traditional cascaded dialogue agents toward natively streaming, full-duplex, and fully multimodal interactive agents. The system's architecture and empirical latency support direct deployment in applications requiring sub-second agent response with real-time visual feedback: digital assistants, live broadcast avatars, and interactive entertainment.

From a modeling perspective, the work demonstrates that large foundation models can handle not only multimodal perception and generation but also temporal management (turn-taking, interruption, continual feedback) in a fashion more akin to human conversation. The seamless synchronization of modalities, avoidance of intermediate text bottlenecks, and direct optimization for low-latency output constitute the main contributions. The use of block-causal attention and causal autoregressive streams ensures that each emitted output meaningfully extends the ongoing interactive history, supporting long-duration coherence and identity preservation.

Future Directions

The paper identifies several avenues for scaling and further research:

  • Output Resolution: The current v0.1 experiment is at 192p; scaling to higher resolutions is projected as straightforward but not yet demonstrated.
  • Broader Data Regimes: Expanding the diversity of supervised, semi-supervised, and unsupervised end-to-end interaction data will likely further increase robustness and behavioral diversity.
  • Integrating Action and World Models: Extending the paradigm with world interaction/action output streams could provide direct online control over embodied agents or avatars, enabling both simulated and real-world actuation.
  • Model Compression / Edge Deployment: Achieving similar perceptual and low-latency performance with smaller, more resource-efficient models will be critical for practical device-side applications.

Conclusion

Wan-Streamer v0.1 introduces a tightly integrated, causal architecture for real-time, multimodal dialogue agents that is directly optimized for full-duplex, low-latency, and naturalistic audio-visual interaction. By eliminating modular boundaries and modeling the interaction as an interleaved causal process across all modalities, it achieves sub-second round-trip latency with synchronized linguistic, vocal, and visual feedback. The system validates the feasibility and advantages of designing multimodal agents from first principles as streaming, unified foundation models, and opens new possibilities for both research and deployment of truly interactive AI systems.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

Wan-Streamer v0.1: A simple explanation for teens

1) What is this paper about?

The paper introduces Wan-Streamer, a new kind of AI that can see, listen, think, and respond in real time—using text, speech, and video—all at once. Instead of having many separate parts (like one for speech recognition, one for writing answers, one for making a talking face), it’s a single model that handles everything together, like one “brain” for a digital character. The goal is to make conversations with AI feel fast, natural, and human-like, including things like nodding while listening, speaking with lip-sync, and reacting instantly to what you do or say.

2) What questions were the researchers trying to answer?

Here are the main questions in simple terms:

  • Can we build one unified AI that listens and speaks at the same time (full‑duplex), across text, audio, and video, without stitching together lots of separate tools?
  • Can this AI react quickly enough to feel natural—under a second from your action to its response?
  • Can it keep everything in sync (voice, lips, facial expressions, gaze) while also understanding and planning what to say?
  • Can it learn good “conversation behavior” (when to talk, when to pause, how to show it’s listening) directly from data, instead of hard-coded rules?

3) How does Wan-Streamer work? (Methods in everyday language)

Think of a live conversation as a stream of tiny “chunks” of information: little pieces of sound, frames of video, and words. Wan-Streamer processes and produces these chunks continuously, so it can listen and respond at the same time.

  • One model for everything:
    • It uses a single “Transformer” (a popular AI architecture) to handle text, audio, and video together. You can imagine the Transformer as a very fast conductor keeping all players (sound, vision, language) in rhythm.
    • It treats inputs and outputs as interleaved tokens—small pieces of sound, video frames, and words—so it can track and update the conversation moment by moment.
  • “Causal” design for streaming:
    • Causal just means “only use the past and present, not the future.” Like talking live: you can’t use information that hasn’t happened yet.
    • The encoders (which understand your speech and video) and decoders (which create the AI’s speech and video) are built to work this way, ensuring low delay.
  • Block‑causal attention:
    • Attention is how the model focuses on the right parts of its memory. “Block‑causal” means it looks at recent chunks in order, like scanning back through the last few moments of a conversation, to keep things fast and consistent.
  • Latent spaces and “flow matching” (simple analogy):
    • Audio and video are first compressed into a “latent” form (think of it like a compact summary or a sketch).
    • Flow matching is like cleaning noise off a photo: the model starts from a slightly noisy version of what it wants to say/show and learns how to “denoise” it into clear speech and motion, guided by the current context.
  • Training in three steps: 1) Pretraining on many tasks (understanding images/audio/video, generating them, and normal dialogue) so the model develops broad skills. 2) End‑to‑end interaction training where it learns to listen and respond across text, audio, and video simultaneously—picking up timing, turn-taking, and “visible listening” (like nods or gaze). 3) Distillation for speed: a larger “teacher” model teaches a smaller “student” model to be fast while keeping quality, so it can run in real time.
  • Thinker–Performer setup (for fast serving):
    • Imagine two teammates passing a baton each 160 milliseconds (about the blink of an eye):
    • The “thinker” quickly understands your latest audio/video and updates the conversation state.
    • The “performer” focuses on producing the next slice of the AI’s voice and video.
    • They overlap their work so responses come out smoothly and quickly.

4) What did they find, and why is it important?

  • Sub‑second responsiveness:
    • The model-side delay is around 200 milliseconds (0.2 seconds). Including a typical internet delay (~350 ms), total interaction is about 550 milliseconds—just over half a second. That’s fast enough to feel natural in a live conversation.
  • Full‑duplex behavior:
    • The AI can “listen” and “speak” at the same time. While the AI is talking, it still sees/hears you and can adapt—like stopping when you interrupt or changing what it says if you react.
  • Natural, synchronized responses:
    • Because voice and video are generated from the same shared context, lips, facial expressions, and tone match naturally, instead of being glued together after the fact.
    • During quiet moments, it doesn’t freeze; it maintains subtle motions (breathing, gaze, posture), making it feel “present.”
  • Fewer moving parts, fewer errors:
    • Since everything runs inside one model, there’s less waiting between modules and fewer chances for misalignment (like speech not matching lips, or delayed reactions).
  • Real-time video output:
    • It streams at 25 frames per second with streaming “units” as short as 160 ms, enabling smooth, continuous visuals.

Overall, these results show that a single, end‑to‑end model can deliver fast, realistic audio‑visual conversations better than stitching together separate systems.

5) Why does this matter, and what’s next?

  • Practical impact:
    • More lifelike digital assistants that can look and sound engaged.
    • Better live broadcasting or interactive entertainment, where characters can see you, listen, and respond instantly.
    • Training and education tools that feel like communicating with a real person, not a delayed script.
    • Early steps toward embodied AI that can interact with the real world in real time.
  • Limitations and future work:
    • This v0.1 version focuses on lower video resolution (192p) as a proof of concept. The authors say scaling to higher resolution is straightforward and plan to improve it.
    • As with any AI system, quality, safety, and robustness will improve with more data, better training, and broader testing.

In short, Wan‑Streamer shows that building a digital “person” that can see, hear, think, and respond smoothly isn’t just about speed—it’s about designing the whole system to stream like real human conversation, with all senses and actions working together in one place.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge Gaps, Limitations, and Open Questions

Below is a focused list of what remains missing, uncertain, or unexplored in the paper that future work could concretely address.

Architecture and Modeling

  • Quantify and ablate the block-causal attention design (e.g., block size, interleaving schedule, token rates) to measure trade-offs among latency, quality, and cross-modal synchronization.
  • Provide ablations comparing the unified end-to-end model against strong cascaded baselines at matched quality to isolate where end-to-end learning yields gains beyond latency (e.g., fewer sync errors, better interruption handling).
  • Detail and evaluate the causal audio/video VAE design (compression rate, receptive fields, codebook size) and its impact on identity preservation, prosody, and visual fidelity.
  • Analyze long-horizon stability under full-duplex streaming (30–60+ min): quantify drift in identity, prosody, speaking rhythm, and scene state; test mitigation beyond rolling distillation.
  • Formalize and evaluate the turn-taking policy learned implicitly (e.g., metrics for barge-in responsiveness, talk-over rate, false interruptions), and explore controllable knobs (aggressiveness, responsiveness).
  • Study multi-party settings (overlapping speakers, multiple faces): diarization-free handling, role attribution, and turn management in shared audio-visual contexts.
  • Investigate how the model handles conflicting cross-modal cues (e.g., incongruent text vs audio vs video), and whether it prioritizes certain modalities; expose controllable cross-modal weighting.
  • Assess generalization to unseen visual embodiments (beyond head-and-shoulders), occlusions, hands/gestures, and non-frontal poses; define requirements to support full-body behavior.

Data and Training

  • Disclose the composition of the duplex interaction dataset (hours, modality balance, demographics, languages, domains) and licensing; quantify distributional coverage and gaps.
  • Evaluate bias and fairness across accents, languages (including code-switching/tonal languages), demographics, and cultural nonverbal behaviors; report failure modes and mitigation strategies.
  • Clarify the proportion of synthetic vs real interaction data and its effect on naturalness and stability; investigate domain adaptation to in-the-wild video chat.
  • Provide details of the teacher model used for distillation (architecture, CFG scale, solver steps) and quantify student-teacher gaps across tasks and horizons.
  • Explore curriculum or preference optimization to align subjective behaviors (gaze, nodding, backchannels) with human preferences; assess whether RLHF or data-driven preference models improve perceived naturalness.

Evaluation and Metrics

  • Add standardized audiovisual sync metrics (e.g., LSE-C/LSE-D, onset asynchrony) under varied conditions (speaking, listening, idling, interruptions).
  • Report audio quality/comprehensibility metrics (e.g., MOS, PESQ, STOI) and video quality/temporal consistency metrics (e.g., FVD, LPIPS, identity embedding drift).
  • Conduct controlled user studies (blinded A/B) for perceived latency, naturalness, responsiveness, and interruption handling; publish protocols for reproducibility.
  • Establish and release an open benchmark/protocol for end-to-end duplex latency that covers input acquisition, processing, and synchronized audio-visual output (beyond first-packet/first-token).
  • Quantify benefits of “visible listening” (nonverbal feedback) on user experience and task outcomes; measure rates of appropriate vs distracting feedback.

Latency, Throughput, and Scaling

  • Specify hardware details used for the ~200 ms model latency (GPU type, memory, clocks); report sensitivity to batch size and concurrency.
  • Characterize the cost/throughput envelope (tokens/s, FLOPs, energy) and cost-per-minute for single-user and multi-user serving; evaluate carbon/energy footprint.
  • Analyze scalability to higher resolutions (e.g., 720p/1080p) and higher FPS (30/60): compute budget, decoder limits, VAE bandwidth, and whether real-time constraints remain satisfied.
  • Evaluate single-GPU operation and consumer hardware viability (laptops, gaming GPUs); test quantization, sparsity, and KV cache compression on latency/quality.
  • Justify the 160 ms streaming unit choice; compare against finer granularity (40–80 ms) for perceived responsiveness and prosody, and quantify scheduling overheads.
  • Detail KV-cache exchange bandwidth/latency requirements (PCIe vs NVLink) and limits as history grows; propose cache pruning/compaction strategies for long sessions.

Robustness, Safety, and Security

  • Test robustness to real-world conditions: background noise, reverb, music/TV interference, motion blur, occlusions, camera switches, and varied lighting; include stress tests.
  • Evaluate resilience to network jitter, packet loss, and fluctuating bandwidth; design A/V jitter buffers, concealment, and graceful degradation strategies.
  • Address privacy for always-on A/V capture: on-device processing, PII redaction, ephemeral storage, consent mechanisms, and data minimization policies.
  • Develop safeguards against impersonation/deepfake misuse: speaker/face consent checks, provenance and watermarking for generated audio/video, and detectable markers.
  • Assess vulnerability to adversarial A/V attacks and prompt injection via audio/video; harden the thinker–performer interface against KV or latent tampering and leakage.
  • Provide content moderation for both inputs and outputs (e.g., harmful gestures or speech), including multilingual moderation coverage and latency impact.

Capabilities and System Integration

  • Measure proactive speaking accuracy: rates of correct vs spurious event-triggered comments; introduce thresholds and user controls to avoid over-talkative behavior.
  • Support explicit user controls for persona, voice timbre, speaking rate, and nonverbal style; expose stable APIs for dynamic adjustment during a session.
  • Explore tool-use/action grounding within the same causal stream (e.g., APIs, robot control), and evaluate coordination with speech and visual expression.
  • Investigate cross-session persistence (identity, memory of user preferences) with privacy-preserving storage; measure continuity vs drift across sessions.
  • Extend to multilingual and code-switching speech synthesis with accurate lip motions across languages; quantify cross-lingual lip-sync and prosody alignment.

Reproducibility and Transparency

  • Release code/models or provide sufficient training/inference details (hyperparameters, data scales, optimizer schedules) for replication; include ablations referenced qualitatively.
  • Publish a thorough latency breakdown (encode, KV update, solver, decode, communication) and its variance under load; include end-to-end traces.
  • Open-source representative duplex interaction datasets or simulators, with clear licenses and demographic balance, to catalyze standardized evaluation.
View Paper Prompt  View All Prompts 

Practical Applications

Immediate Applications

Below are deployable use cases that leverage Wan-Streamer’s native full-duplex, sub-second, audio-visual streaming with synchronized speech and video (≈200 ms model-side; ≈550 ms including typical network), causal encoders/decoders, and block-causal attention. Each item lists sector links, potential products/workflows, and feasibility notes.

  • Customer support video agents (retail, finance, telecom)
    • What: On-website or in-branch kiosk avatars that see and hear users and respond with synchronized speech and facial/body cues; proactive listening behaviors (nods, gaze) during user speech reduce perceived wait time and increase trust.
    • Tools/products/workflows: “Video Contact Center Agent” SDK; WebRTC-based thinker-performer serving; CRM integration for context; small-window 192p avatar tiles in chat widgets.
    • Assumptions/dependencies: Two-GPU serving or equivalent; low-latency links; brand-safety and disclosure policies; v0.1 resolution (192p) is acceptable in small UI; content moderation.
  • E-commerce virtual shopping assistants (retail/e-commerce)
    • What: Product explainers that can watch the shopper (e.g., show item to camera), provide recommendations, and demonstrate usage via expressive avatar, maintaining natural turn-taking and interruptions.
    • Tools/products/workflows: “Storefront Avatar” plugin for Shopify/Custom CMS; product catalog grounding; analytics for conversion A/B tests.
    • Assumptions/dependencies: SKU grounding and safe retrieval; localization; guardrails against hallucinated specs.
  • VTuber co-hosts and live stream sidekicks (media/entertainment)
    • What: Always-on, responsive co-presenters that react to chat and streamer activity, showing idle/listening behaviors between utterances to appear present.
    • Tools/products/workflows: OBS/Streamlabs plugin; scene capture as video input; performer on cloud GPU; thinker on local GPU to reduce latency.
    • Assumptions/dependencies: Streaming platform policies on synthetic media; watermarking/disclosure; 25 FPS at 192p may suffice for picture-in-picture.
  • Language learning tutors with conversational avatars (education)
    • What: Full-duplex practice partners that react to learner speech and facial cues; prosody/lip-sync alignment helps pronunciation coaching.
    • Tools/products/workflows: LMS integration; session recording/review; pronunciation scoring overlay.
    • Assumptions/dependencies: Multilingual ASR-like competence via end-to-end encoders must be adequate for target languages; pedagogy and accuracy checks.
  • Meeting companions and real-time translators (enterprise software)
    • What: Avatars that attend calls, summarize, react visually while others speak, and interject with clarifications; optional cross-lingual speech-to-speech with synchronized visual output.
    • Tools/products/workflows: Plug-ins for Zoom/Meet/Teams; enterprise KMS grounding; diarization-free full-duplex flow.
    • Assumptions/dependencies: Corporate privacy/compliance; network QoS; disclosure to participants.
  • Telehealth intake and navigation assistants (healthcare)
    • What: Clinic kiosks or portals with an empathetic avatar that triages symptoms, collects forms, and reacts to patient affect; visible listening improves adherence and satisfaction.
    • Tools/products/workflows: EHR integration for structured handoff; scripted guardrails; offline fallback scripts.
    • Assumptions/dependencies: HIPAA/GDPR compliance; medical disclaimers; rigorous safety filters; escalation workflow to clinicians.
  • Banking/insurance branch kiosks and remote teller (finance)
    • What: In-lobby digital representatives that handle FAQs, appointment setting, and form walkthroughs; can detect user confusion from visual cues and adjust pacing.
    • Tools/products/workflows: Kiosk OS with local thinker; cloud performer; workflow engines for KYC steps (non-binding guidance).
    • Assumptions/dependencies: Regulatory constraints; identity and fraud safeguards; recorded consent for audio/video capture.
  • In-car infotainment avatars (automotive)
    • What: Hands-free co-pilots that proactively comment on navigation states and driver commands while maintaining continuous perception and visible responses.
    • Tools/products/workflows: Edge thinker on cabin SoC; cloud performer via 5G; voice+video UI on center stack.
    • Assumptions/dependencies: Connectivity robustness; on-device fallback; driver distraction policies; smaller models for embedded.
  • Accessibility: expressive TTS avatars for text users (public sector, assistive tech)
    • What: For users who type/gesture instead of speaking, the avatar renders expressive, lip-synced speech with visible cues to aid social interaction.
    • Tools/products/workflows: AAC device integration; customizable voices/faces; quick phrase banks.
    • Assumptions/dependencies: Personalization ethics; prevent impersonation misuse; no guaranteed sign-language ability in v0.1.
  • Research testbed for full-duplex interaction (academia)
    • What: A platform to study turn-taking, interruption handling, and social cue timing across modalities using one causal model.
    • Tools/products/workflows: Open benchmarks for signal-to-signal latency; ablations on block-causal attention and KV exchange; logging of timing features.
    • Assumptions/dependencies: Availability of weights/API; experimental IRB approvals for human studies.
  • Real-time broadcast captioning with avatar summaries (media/education)
    • What: Avatar presents bite-sized summaries during live events while monitoring the speaker to time interjections and yield control smoothly.
    • Tools/products/workflows: Broadcast mixer integration; schedule-aware prompting; latency budget enforcement.
    • Assumptions/dependencies: Rights for rebroadcast; accuracy QA; ephemeral storage policies.

Long-Term Applications

These require further research, scaling (e.g., higher resolution than 192p), domain adaptation, or additional safety/tooling before production deployment.

  • High-fidelity digital humans for customer-facing roles (retail, hospitality, public services)
    • Future: 720p–4K realistic agents with consistent identity across sessions, multi-person scene awareness, and fine-grained non-verbal social cues.
    • Needed: Higher-res causal VAEs/decoders; stronger distillation for quality vs. latency; identity and brand persona memory; photoreal safety and watermarking standards.
  • Full-duplex multimodal therapists and coaches (healthcare/mental health, wellness)
    • Future: Agents that continuously perceive facial/voice affect, deliver CBT-style interventions, and adapt in the moment with safe escalation.
    • Needed: Clinical validation; bias and harm audits; regulated deployment frameworks; explainability; robust affect detection; stronger guardrails.
  • Classroom co-teachers and lab demonstrators (education)
    • Future: Avatars that watch the class/lab via cameras, proactively address confusion, demonstrate multi-step procedures, and manage interruptions.
    • Needed: On-prem deployments with multiple camera feeds; pedagogy-aligned prompting; FERPA/GDPR compliance; classroom trials.
  • Real-time multimodal copilots for robots and IoT (robotics, smart spaces)
    • Future: Agents that perceive the robot’s sensors, converse with users, and coordinate actions, using the same causal stream to couple language and world state.
    • Needed: Action grounding and safety layers; integration with planners/controllers; failure-handling; latency bounds for control loops.
  • AR/VR telepresence avatars with spatial awareness (XR)
    • Future: Spatially anchored avatars that react to 3D context and users’ gaze/gestures, maintaining continuity in immersive meetings or training.
    • Needed: Causal 3D/NeRF encoders; SLAM integration; low-latency edge rendering; higher-res streaming.
  • Real-time cross-lingual mediators with cultural cues (government services, diplomacy, travel)
    • Future: Agents that bridge languages with culturally appropriate non-verbal behaviors and turn-taking norms in high-stakes settings.
    • Needed: Multilingual/end-to-end training for target locales; evaluation of cultural non-verbal appropriateness; secure on-prem hosting.
  • Personalized companions with long-horizon memory (consumer, wellness)
    • Future: Avatars that maintain consistent persona, recall long-term context across sessions, and adapt non-verbal style to user preferences.
    • Needed: Memory architectures (vector DB + causal state), privacy-preserving storage, preference learning with consent.
  • Agentic field technicians with see-and-say guidance (energy, manufacturing)
    • Future: On-site assistants that watch tasks via headcams, proactively instruct with synchronized audio-visual cues, and stop when the user speaks or risks arise.
    • Needed: Domain fine-tuning, safety-critical checklists, offline/edge modes, PPE detection, certification with industry standards.
  • Secure government kiosks for services (public sector)
    • Future: Always-on multi-service kiosks that manage documents, forms, and interviews with natural full-duplex interaction, including visibility into user affect to pace and clarify.
    • Needed: Procurement standards, accessibility compliance, robust identity verification paired with explicit anti-impersonation safeguards.
  • Game NPCs with live, duplex social presence (gaming)
    • Future: NPCs that perceive player actions in real time (video feed/controller state) and respond with expressive, interruptible dialogue and animation.
    • Needed: Engine integration (Unreal/Unity) with sandboxes; deterministic behavior budgets; content controls.
  • Standards and policy frameworks for full-duplex AI (policy, standards bodies)
    • Future: Sector-wide metrics and compliance frameworks for signal-to-signal latency, interruption handling, disclosure, watermarking, and safe deployment of real-time audio-visual agents.
    • Needed: Multi-stakeholder working groups; reference implementations; legally recognized disclosure/watermark standards; privacy-by-design templates.
  • Fraud-resistant identity-preserving avatars (finance, healthcare, legal)
    • Future: Verified avatars for authenticated individuals (e.g., remote notarization) that can be trusted despite generative capabilities.
    • Needed: Cryptographic signatures/watermarks on audio-video; liveness detection; binding to legal identity; regulatory approval.

Cross-cutting assumptions and dependencies (impacting both categories)

  • Compute and deployment: Current v0.1 relies on a two-GPU thinker-performer pipeline with per-unit generation within 160 ms; edge-cloud split and cost optimizations are necessary for scale.
  • Video fidelity: v0.1 validated at 192p, 25 FPS; many premium uses require higher resolution and improved realism without breaking causal, low-latency constraints.
  • Network: ≈350 ms bidirectional latency assumed in reported totals; QoS and edge deployments are critical for consistent sub-second experiences.
  • Safety and compliance: Always-on AV capture requires consent, disclosure, and data minimization; content moderation and bias mitigation are necessary, especially in health/finance/education.
  • Misuse risks: Synthetic audio/video necessitate disclosure, watermarking, and anti-impersonation safeguards; policy alignment will shape feasible deployments.
  • Data and domain grounding: High-quality duplex interaction data and domain-specific grounding (EHR, CRM, manuals) are needed for reliability; multilingual/localization support must be validated.
  • Integration: Toolchains around WebRTC, SDKs for mobile/web/kiosk, and APIs for CRM/EHR/LMS are required to productize the model’s capabilities.
View Paper Prompt  View All Prompts 

Glossary

  • ASR: Automatic Speech Recognition; converting speech audio into text. "Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules,"
  • autoregressive streaming: Generating outputs step-by-step where each step conditions on the entire past context in a streaming setting. "block-causal multimodal attention, and full-history autoregressive streaming."
  • bidirectional generation: Offline generation that uses both past and future context, unsuitable for low-latency streaming. "replacing offline bidirectional generation with cached context and rolling prediction"
  • bidirectional network latency: Round-trip network delay between user and server. "approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency"
  • block-causal attention: An attention mechanism that restricts attention to past blocks to enable low-latency incremental decoding. "block-causal attention for incremental streaming."
  • causal decoders: Decoders that emit outputs using only past information, enabling real-time streaming. "causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling,"
  • causal encoders: Encoders that process inputs using only past observations to maintain causality for streaming. "causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling,"
  • classifier-free guidance (CFG): A technique to guide generative models by mixing conditional and unconditional predictions to improve fidelity. "A stronger teacher with classifier-free guidance (CFG) and more flow-matching solver steps is distilled into an efficient student"
  • conditional flow matching: A training method where a model learns velocity fields that transform noise into data conditioned on context. "generated jointly with conditional flow matching."
  • CUDA graph capture: Capturing and replaying GPU execution graphs to reduce kernel launch overhead and improve throughput. "Together with CUDA graph capture, compilation, and optimized kernels, Wan-Streamer reaches approximately 200 ms model-side response latency."
  • distribution matching: A distillation objective aligning the student’s output distribution with a teacher’s to preserve quality. "using a self-forcing strategy [17] with distribution matching [45, 46] to align the student trajectory"
  • endpointing: Detecting utterance boundaries or response cutoffs in streaming speech interactions. "Reported numbers mix model response, API TTFB, endpointing, and network."
  • flow-matching solver: The numerical procedure that integrates learned velocity fields to denoise latents into clean samples. "the performer uses the newly received full-history KV context to run only the flow-matching solver for the next audio-visual latent unit."
  • full-duplex: Simultaneous bidirectional communication—listening and speaking at the same time. "real-time, low-latency, full-duplex audio-visual interaction."
  • KV-cache: Cached key/value tensors from Transformer attention used to speed up incremental decoding. "This produces the new KV-cache slice for the current interaction state."
  • latent: A compact continuous representation in which data (e.g., audio/video) are encoded for generation or decoding. "The audio and video responses are represented in continuous latent spaces"
  • native-streaming: Designed from the ground up to operate incrementally with causal constraints, not adapted post hoc. "a native-streaming, end-to-end interactive foundation model"
  • prefill: The initial step that populates caches and state before steady-state streaming begins. "After system prefill, the thinker broadcasts the initial KV cache"
  • prosody: The rhythm, stress, and intonation of speech that conveys expressive timing and emphasis. "lip motion, facial dynamics, and prosody are synchronized natively rather than repaired by post-hoc alignment."
  • rolling distillation: Distilling a teacher into a student while rolling out the student over consecutive steps to reduce long-horizon drift. "We also use rolling distillation to mitigate long-horizon degradation"
  • rolling forcing: A strategy for autoregressive diffusion to sustain long video generation; referenced as related streaming technique. "Rolling forcing: Autoregressive long video diffusion in real time."
  • RTF: Real-Time Factor; the ratio of processing time to media duration (lower is faster than real time). "0.58 s first-token; RTF 0.20-0.27."
  • self-forcing: Training the model on its own generated history to narrow the train–test gap in streaming rollouts. "using a self-forcing strategy [17] with distribution matching [45, 46]"
  • streamability: The property that a model/stack can process and emit data incrementally under causal constraints. "These requirements make streamability a modeling constraint rather than a serving optimization."
  • streaming unit: A fixed-duration chunk that defines the granularity of incremental processing and emission. "streaming units as short as 160 ms at 25 fps."
  • thinker-performer pipeline: A split execution design where a “thinker” handles encoding/state/decoding while a “performer” generates latents. "we deploy it as a thinker-performer streaming pipeline to maximize overlap and hardware utilization."
  • token-causal Transformer pass: A short, causal decoding pass over tokens for language prediction and state updates. "runs the short token-causal Transformer pass for language prediction and state update."
  • TTFB: Time To First Byte; a networking/serving latency metric for the first response packet. "Reported numbers mix model response, API TTFB, endpointing, and network."
  • TTFC: Time To First Chunk; latency until the first chunk of a streaming output is produced. "TTFC ~0.7 s; ~ 25 FPS"
  • variational autoencoder (VAE): A latent-variable generative model with encoder/decoder trained via variational inference. "strictly causal audio and video variational autoencoders (VAEs) for streaming latent coding"
  • velocity fields: Vector fields defining instantaneous change in latent variables used to integrate flows from noise to data. "estimate the velocity fields for both noisy audio and video latents"
  • VAD: Voice Activity Detection; detecting speech presence in audio streams. "Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules,"
  • TTS: Text-To-Speech; generating speech audio from text. "Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules,"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 50 likes about this paper.

Sign Up for Free