X-Streamer: Unified Multimodal Digital Human Framework

• X-Streamer is a multimodal framework that fuses text, speech, and video inputs to generate synchronized digital human interactions using a Thinker–Actor architecture.
• It employs a frozen pretrained language–speech model (Thinker) for real-time reasoning and a trainable multimodal generator (Actor) for coherent audiovisual rendering.
• The design ensures temporal continuity and robust identity preservation through chunk-wise video diffusion, cross-modal attention, and efficient streaming processing.

X-Streamer is an end-to-end multimodal human world modeling framework for building digital human agents capable of infinite interactions across text, speech, and video within a single unified architecture. Starting from a single portrait, it enables real-time, open-ended video calls driven by streaming multimodal inputs. The system is organized around a Thinker–Actor dual-transformer architecture in which a pretrained language–speech model performs streaming perception and reasoning, while a trainable multimodal generator converts the resulting hidden states into synchronized text, speech, and video outputs in real time (Xie et al., 25 Sep 2025).

1. System definition and architectural scope

The framework is defined by the unified modeling relation

$(T_o, A_o, V_o) = \mathcal{M}( T_i, A_i, I_s ),$

where the outputs are text, audio, and video conditioned on streaming text input $T_i$, streaming audio input $A_i$, and a reference portrait $I_s$ (Xie et al., 25 Sep 2025). In operational terms, X-Streamer turns a static portrait into persistent and intelligent audiovisual interactions by coupling multimodal understanding and multimodal generation inside one architecture.

Its central architectural distinction is the Thinker–Actor split. The Thinker is a frozen pretrained language–speech transformer, specifically GLM-4-Voice, that ingests streaming text and speech queries and produces hidden embeddings $h^c$ for chunk $c$. The Actor is a trainable multimodal generator transformer, initialized from GLM-4-Voice weights, that consumes $h^c$ and autoregressively produces an interleaved sequence of discrete text tokens $t_o^c$, discrete audio tokens $a_o^c$, and continuous video latent tokens $v_o^c$ (Xie et al., 25 Sep 2025).

This division of labor makes the system “unified” in the sense of coordinated multimodal reasoning and generation rather than modality homogenization. A plausible implication is that X-Streamer preserves strong pretrained dialog competence in the Thinker while allowing the Actor to specialize for synchronized audiovisual rendering.

2. Input representation and training formulation

X-Streamer processes text, speech, portrait identity, and video output through distinct but coordinated representations. Text is tokenized into discrete semantic tokens $T_i$0 by a pretrained text tokenizer. Speech is encoded at 12.5 Hz into discrete audio tokens $T_i$1 via the GLM-4-Voice speech tokenizer. The reference portrait is embedded via a visual encoder into a fixed latent $T_i$2. Video output is generated as continuous VAE latents $T_i$3 at 8 × spatiotemporal compression using LTX-Video (Xie et al., 25 Sep 2025).

The autoregressive training objective is given as

$T_i$4

This objective makes explicit that each chunk is conditioned not only on the input portrait and prior user queries, but also on prior Actor outputs across all three modalities (Xie et al., 25 Sep 2025). The formulation therefore treats conversation, speech realization, and video evolution as a coupled history-dependent process.

A useful consequence of this design is that temporal continuity is not delegated solely to the video pathway. Instead, the conditioning context includes prior text and audio outputs as well, which suggests that discourse state, prosody, and facial motion are intended to co-evolve under one autoregressive factorization.

3. Thinker module: streaming reasoning and multimodal context formation

The Thinker module uses GLM-4-Voice as a pretrained language–speech backbone. It is frozen at inference to preserve conversational intelligence. Its streaming paradigm alternates between blocks of 13 text tokens and 26 speech tokens, corresponding to approximately 2 s of audio for low-latency streaming. It maintains a rolling cache of up to 8 K tokens, approximately 10 min of combined text/audio context (Xie et al., 25 Sep 2025).

The hidden-state formulation is defined over a concatenated multimodal sequence $T_i$5, where

$T_i$6

At chunk $T_i$7, the Thinker produces

$T_i$8

where $T_i$9 is the total token count from the reference image, user queries, and prior Actor outputs up to chunk $A_i$0 (Xie et al., 25 Sep 2025). These hidden states capture long-range, time-aligned multimodal context and are used as keys and values by the Actor.

This arrangement gives the Thinker a role that is broader than automatic speech recognition or text-only dialogue management. It reasons over interleaved linguistic, acoustic, and identity-conditioned context. The inclusion of prior Actor outputs in $A_i$1 indicates that the system’s internal state depends on its own generated conversational history rather than only on external inputs.

4. Actor module: interleaved generation and chunk-wise video diffusion

The Actor performs multimodal generation chunk by chunk. For each approximately 2 s chunk $A_i$2, the interleaving schedule is 13 text tokens, followed by 26 audio tokens, followed by 6 video chunks, each containing 8 frames, for a total of 384 video tokens (Xie et al., 25 Sep 2025). Text and audio are trained with standard cross-entropy under teacher forcing, and audio is converted to waveform through a HiFi-GAN vocoder.

Video generation is handled as continuous latent prediction with velocity-parameterized diffusion on latent chunk $A_i$3. The diffusion training loss is

$A_i$4

The paper describes this as chunk-wise autoregressive diffusion for video and states that the Actor employs cross-attention to $A_i$5 for tight semantic and temporal alignment across modalities (Xie et al., 25 Sep 2025).

A key training device is chunk-wise diffusion forcing. Each chunk $A_i$6 is noised independently with timestep $A_i$7, and during training all chunks and their histories are jointly denoised under varied noise levels. The stated purpose is robustness to imperfect self-generated histories. At inference, a “pyramid” scheduler $A_i$8 organizes denoising so that only $A_i$9 forward passes are needed instead of $I_s$0 (Xie et al., 25 Sep 2025).

The paper attributes long-horizon stability to this forcing scheme, stating that it guarantees coherent motion across chunk boundaries and prevents drift over long horizons. Within the system’s own terminology, this is a mechanism for maintaining temporal coherence in a streaming diffusion process rather than a post hoc temporal smoothing stage.

5. Temporal alignment, context retention, and identity persistence

Temporal and cross-modal alignment are implemented through inter- and intra-chunk attention combined with time-aligned multimodal positional embeddings. The positional encoding for token $I_s$1 of modality $I_s$2 at time $I_s$3 is $I_s$4. X-Streamer applies 1D RoPE on text and audio tokens along the temporal axis and 3D RoPE on video tokens across time, height, and width for spatiotemporal localization (Xie et al., 25 Sep 2025).

Within each video chunk, the model uses bidirectional self-attention among the 8-frame spatial tokens, giving full spatiotemporal context inside the chunk. It also applies cross-attention to aligned text/audio hidden states $I_s$5 through $I_s$6 alignment. Across chunks, it uses causal autoregressive attention on preceding video tokens $I_s$7 to enforce temporal causality (Xie et al., 25 Sep 2025). The resulting attention pattern is therefore mixed: bidirectional locally within a chunk, causal globally across chunks.

Cross-modality attention in each Actor block is written as

$I_s$8

with $I_s$9 given by video-token queries and $h^c$0 given by Thinker hidden states $h^c$1, including global identity tokens (Xie et al., 25 Sep 2025). The identity mechanism embeds $h^c$2 as a constant prefix token sequence at the very start of the context, allowing every cross-attention layer to attend to the source portrait.

The paper states that this global identity reference empirically stabilizes facial identity even over hours of interaction. A plausible implication is that identity consistency is treated as a persistent conditioning signal accessible throughout the entire generation stack, rather than as a one-time initialization of the visual pathway.

6. Real-time deployment, context limits, and interpretive boundaries

The reported model size is 18 B parameters, with peak VRAM of approximately 53 GB on one GPU. On a single GPU, runtime is described as approximately 51.2 s for the Thinker plus 58.2 s for the Actor per 1 min of output, which is not real time. The paper reports that a two-GPU deployment, with the Thinker on GPU 0 and the Actor on GPU 1, together with KV caching and the chunk-wise pyramid scheduler, achieves 25 fps end-to-end multimodal streaming (Xie et al., 25 Sep 2025).

The deployed context windows are heterogeneous by modality. Video self-attention is limited to up to 2 K video tokens, approximately 10 s, while the Thinker retains 8 K tokens, approximately 10 min, for text and audio context (Xie et al., 25 Sep 2025). The latency optimizations named in the paper are interleaved chunk generation, which avoids buffering full speech before video; Key–Value cache, which eliminates recomputation of previous context; and chunk-wise denoising, which reduces diffusion passes to $h^c$3.

The phrase “infinite interactions” should therefore be interpreted carefully. The paper explicitly claims infinite-horizon, synchronized audiovisual conversations with a persistent digital human, but it also reports finite rolling context limits for text/audio and finite self-attention horizons for video (Xie et al., 25 Sep 2025). This suggests that “infinite” refers to open-ended streaming operation with mechanisms for long-horizon stability, not to unbounded full-history retention in all modalities.

Within the paper’s framing, the principal significance of X-Streamer is the integration of streaming reasoning, interleaved multimodal generation, chunk-wise video diffusion, and persistent identity conditioning into a single Thinker–Actor framework. The system is presented as a step toward unified world modeling of interactive digital humans (Xie et al., 25 Sep 2025).