MultiStream-LLM Architectures

• MultiStream-LLM is a family of architectures that use concurrent streams to process asynchronous inputs from various roles, modalities, or models.
• It synchronizes parallel streams—such as serving-time prompts, multimodal signals, and source model probabilities—to optimize latency and throughput.
• The approach enhances system robustness and security by structurally separating inputs and outputs, enabling efficient inference and improved monitoring.

MultiStream-LLM denotes an emerging family of large-language-model architectures, serving systems, and training frameworks that replace a single sequential context with multiple concurrent streams. In the literature considered here, those streams take several distinct forms: evolving prompt fragments arriving from retrievers during inference, parallel modalities such as signing, fingerspelling, and lip movements, separate computational roles such as user input, visible output, thinking, audit, and actions, or token-level probability distributions emitted by multiple source models during continual training (Bachkaniwala et al., 29 Mar 2026, Thomas et al., 20 Aug 2025, Su et al., 12 May 2026, Kong et al., 28 May 2025). The common design objective is to make LLM behavior natively concurrent under asynchrony, memory pressure, or modality mismatch rather than forcing all information through a single monolithic sequence.

1. Conceptual scope

In the supplied literature, “MultiStream-LLM” is not restricted to one canonical definition. A systems line uses it to describe multi-tenant serving in which multiple concurrent requests have prompts that evolve over time as retrieval context arrives; an architectural line uses it to describe decoder-only models that simultaneously read and write across several streams; a multimodal line uses it for separate recognition streams fused before language generation; and a training-time integration line treats the outputs of several source LLMs as parallel probability streams (Bachkaniwala et al., 29 Mar 2026, Su et al., 12 May 2026, Thomas et al., 20 Aug 2025, Kong et al., 28 May 2025).

Interpretation	Streams	Representative work
Serving-time streaming prompts	Append-mode or update-mode prompt chunks	"Stream2LLM: Overlap Context Streaming and Prefill for Reduced TTFT" (Bachkaniwala et al., 29 Mar 2026)
Parallel internal/external roles	User, system, documents, thinking, audit, actions, tools	"Multi-Stream LLMs: Unblocking LLMs with Parallel Streams of Thoughts, Inputs and Outputs" (Su et al., 12 May 2026)
Multimodal translation	Continuous signing, fingerspelling, lipreading	"MultiStream-LLM: Bridging Modalities for Robust Sign Language Translation" (Thomas et al., 20 Aug 2025)
Multi-model knowledge aggregation	Token-level probability distributions from source LLMs	"Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation" (Kong et al., 28 May 2025)

A related formulation is ReaLLM, defined as a real-time paradigm in which a decoder-only multimodal LLM generates after every input token received in real time rather than only after a fully prefixed prompt. In Speech ReaLLM, this is instantiated for ASR with BLANK-gated emissions after each audio embedding. The paper explicitly states that the same principle can extend beyond audio because the input token could be audio, vision, sensor readings, or text updates, although it does not implement concurrent multi-stream inputs beyond audio (Seide et al., 2024). This suggests that a unifying abstraction across the literature is not the modality itself, but the conversion of asynchronous evidence into causally structured streams.

2. Parallel-stream computation inside decoder-only LLMs

The most explicit architectural formalization appears in "Multi-Stream LLMs: Unblocking LLMs with Parallel Streams of Thoughts, Inputs and Outputs" (Su et al., 12 May 2026). There, the model operates over $H$ parallel streams, each corresponding to a role such as user input, visible assistant output, reasoning, audit, or tool interaction. At each synchronous timestep, the model reads tokens from all input streams that have arrived so far and emits one token per output stream, with an idle “-” token representing no output on a stream for that tick. The multi-stream factorization is written as

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$

This formulation imposes intra-stream autoregression and cross-stream causal dependence on earlier timesteps. The paper uses stream embeddings, per-stream RoPE position counters, and interleaved packing so that tokens are ordered row-wise across streams rather than stream-by-stream. Under the basic mask, a query at $(h,t)$ can attend only to positions with $\tau < t$; an inference-friendly variant additionally permits same-step visibility from lower-indexed streams. Training uses a summed cross-entropy loss over supervised output positions, and a stream-contrastive weighting based on log-probability shift further upweights tokens that benefit most from cross-stream context (Su et al., 12 May 2026).

The operational consequence is synchronous decoding across roles. The model can read while writing, think while acting, or audit while solving because every forward pass advances all streams simultaneously. The theoretical speedup is stated as

$S \approx \frac{\sum_{m=1}^M T_m}{\max_m T_m},$

so end-to-end latency is governed by the maximum stream length rather than the sum of all output lengths. Empirically, the paper reports that Token Number to First Target Token drops to zero across multiple benchmarks, including GSM8K, MATH500, LogicNLI, SQuAD, PubMedQA, and ProofWriter. For example, on Qwen3-4B and MATH500, the stream model reports Acc 64.00, TNFT 0, Tokens 742.26, and Delay 26.51, compared with Vanilla values of Acc 60.80, TNFT 103.16, Tokens 1363.95, and Delay 45.82 (Su et al., 12 May 2026).

The same paper also links stream separation to security and monitorability. Separate streams for system prompts, user inputs, and documents provide a stronger structural signal about provenance and authority than a single shared sequence, and dedicated internal streams allow the model to externalize evaluation awareness or task concerns independently of the visible answer. On prompt-injection-style benchmarks, the multi-stream variants reduce Attack Success Rate relative to single-stream instruction-tuned baselines on TensorTrust, Purple, RuLES, StruQ-ID, and StruQ-OOD, while IFEval remains comparable or better (Su et al., 12 May 2026).

3. Streaming prompts and concurrent serving

A distinct use of MultiStream-LLM concerns inference serving rather than model architecture. "Stream2LLM: Overlap Context Streaming and Prefill for Reduced TTFT" (Bachkaniwala et al., 29 Mar 2026) addresses the setting in which an inference engine must accept multiple concurrent requests whose prompts evolve over time as context arrives from external retrievers. The paper formalizes two retrieval patterns. In append-mode, prompt growth is monotonic:

$P_t = P_{t^-} \oplus \Delta_t,$

so previously computed KV blocks remain valid and only the appended chunk requires prefill. In update-mode, the new prompt $\tilde{P}_t$ may replace a suffix of the old prompt, so STREAM2LLM computes

$LCP(P_{t^-}, \tilde{P}_t) = k,$

preserves KV for tokens $1 \ldots k$, invalidates the suffix, and resumes prefill from position $k+1$.

The implementation extends vLLM while retaining PagedAttention’s block-based KV cache. Its core systems contribution is a two-phase scheduler. Phase 1 computes priority ordering and feasibility without allocating memory or mutating request state. Phase 2 performs resource acquisition and adaptive preemption, selecting recomputation or swapping according to hardware-specific profiles. The paper defines preemption costs as

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$0

and uses

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$1

This cost-aware design is coupled to GPU KV block pools, CPU swap pools, and LCP-aware invalidation that frees only the invalid suffix (Bachkaniwala et al., 29 Mar 2026).

The empirical objective is reduced time-to-first-token under disaggregated prefill–decode deployment. TTFT is defined as

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$2

Across real workloads, the paper reports up to 10.8–11.0× faster median TTFT than non-streaming on the crawler workload at high load, and 3.9–4.3× at low load, on H200 with Llama-3.1-8B-Instruct. On the ANNS update-mode workload, it reports 2.49–2.63× P95 TTFT speedups at QPS 1.0 and a retained 1.30–1.42× P50 advantage at higher load even with significant invalidation. Throughput parity is maintained, with trace completion times differing within approximately 1% across methods (Bachkaniwala et al., 29 Mar 2026). The paper also reports that naive streaming can suffer catastrophic P99 tails under memory pressure, whereas FCFS and LCAS with cost-based preemption avoid this behavior.

This serving-oriented line is narrower than the architectural line above. The model weights themselves need not change; instead, the inference engine is modified so that asynchronous prompt evolution becomes a first-class scheduling primitive. In that sense, the “streams” are not roles inside the transformer but temporally evolving prompt states across many requests.

4. Multimodal streams and temporal alignment

In sign language translation, MultiStream-LLM refers to a modular multimodal architecture that separates recognition problems before fusion. "MultiStream-LLM: Bridging Modalities for Robust Sign Language Translation" (Thomas et al., 20 Aug 2025) argues that monolithic gloss-free SLT models break down on high-speed fingerspelling and asynchronous non-manual cues. Its architecture uses three specialized predictors: continuous signing, fingerspelling, and lipreading. Hand crops and face crops are extracted with MediaPipe and resized to $$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$3; a ViT-based sequence-type classifier routes frames among continuous sign, fingerspelling, or rest using hard Gumbel–Softmax. The manual experts share a DINOv2-style backbone and output CTC logits for pseudo-gloss or letter vocabularies, while the lipreading branch uses a masked ViT-base encoder, a temporal adapter, and a CTC phoneme head (Thomas et al., 20 Aug 2025).

Fusion occurs in a shared space after modality-specific projections. The unified manual stream is

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$4

and manual–lip fusion is controlled by

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$5

A transformer encoder with relative positional bias then resolves temporal misalignment before passing contextualized embeddings to a decoder-only LLM, either through prefix tuning or full fine-tuning. The paper states that learned alignment outperforms fixed temporal shifts and that the gate plus relative positional attention serve as soft alignment variables, allowing mouthings to lead or lag manual signs (Thomas et al., 20 Aug 2025).

The reported performance is task-specific rather than generic. On How2Sign, the model achieves BLEU-1/2/3/4 of 46.0/33.5/26.8/23.5 and sets a new state of the art with BLEU-4 = 23.5. On ChicagoFSWildPlus, it achieves 73.2% letter accuracy. Ablations attribute a large part of the gain to explicit stream separation: without lipreading, BLEU-4 drops from 23.5 to 16.6 and letter accuracy from 73.2% to 54.3%; without the LLM, BLEU-4 drops to 7.8 and letter accuracy to 58.8% (Thomas et al., 20 Aug 2025). Here, “multi-stream” is therefore literal multimodal decomposition, not merely parallel prompting.

Speech ReaLLM presents a related but different stream-based formulation for ASR. It interleaves streaming audio embeddings with text tokens in a single causal decoder sequence, predicts a BLANK token when it should wait, and emits recognized words when confident. The decoder runs after each new audio embedding arrives, with no explicit endpointing required. On Librispeech test, the 80M Speech ReaLLM achieves WERs of 3.0% and 7.4% in real time without an external LM or auxiliary loss (Seide et al., 2024). The paper explicitly states that it does not implement multi-stream beyond audio, but it frames ReaLLM as modality-agnostic and describes a path to interleaving tokens from multiple concurrent streams into one causal sequence. This suggests that the sign-language and ReaLLM lines share a concern with temporal flow and asynchronous evidence, even though one uses modular fusion before language generation and the other uses a single decoder with BLANK-gated incremental emission.

5. Multi-model knowledge streams and routing

Another interpretation treats each source model as a stream. "Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation" (Kong et al., 28 May 2025) represents each source LLM $$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$6 by a token-level probability distribution stream

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$7

An Adaptive Selection Network consumes normalized distribution matrices, produces logits

$$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$8

and selects candidates via a threshold $$p_\theta(\mathbf{y}^{(1)}, \ldots, \mathbf{y}^{(H)}) = \prod_{h=1}^{H} \prod_{t=1}^{T_h} p_\theta\!\left( y^{(h)}_t \,\big|\, \mathbf{y}^{(h)}_{<t},\, \{\mathbf{y}^{(h')}_{<t}\}_{h' \neq h} \right).$$9. Selected streams are reweighted by normalized probabilities $(h,t)$0 and fused into $(h,t)$1 for supervision of a single target model. A feedback-driven loss based on the squared coefficient of variation,

$(h,t)$2

is used to avoid selector collapse. The paper reports up to 50% reduction in knowledge interference relative to FuseLLM, improvement across BBH, MMLU, CommonsenseQA, and MultiPL-E, and the same perplexity with 50% fewer training steps than baselines (Kong et al., 28 May 2025). Deployment remains inference-efficient because only the target LLM is used at inference.

Routing frameworks extend the same broad logic to query streams. MixLLM formulates query-to-LLM assignment as a dynamic contextual-bandit problem with per-LLM quality and length predictors, an uncertainty term $(h,t)$3, and a latency-aware penalty $(h,t)$4. Under a time constraint, it reports 97.25% of GPT-4’s quality at 24.18% of the cost, and with Llama 3.1–8B/70B added it reaches 98.55% of GPT-4’s quality at 16.79% cost (Wang et al., 9 Feb 2025). Online Cascade Learning addresses a different operating point: a cascade of smaller models ending in an LLM, with a learned deferral policy over streaming inputs. Across four benchmarks, it reports inference-cost reductions by as much as 90% while paralleling LLM accuracy and maintaining robustness against input distribution shifts (Nie et al., 2024).

These works broaden the meaning of “stream” from sensory or prompt tokens to probabilistic or decision-theoretic flows. A plausible implication is that MultiStream-LLM has become an umbrella for systems in which concurrency is delegated to selection, fusion, or arbitration mechanisms rather than to a single homogeneous context window.

6. Benchmarks, infrastructure, and unresolved issues

As the term has diversified, evaluation and infrastructure have also specialized. "X-Stream: Exploring MLLMs as Multiplexers for Multi-Stream Understanding" (Sun et al., 1 Jun 2026) is described as the first benchmark dedicated to multi-stream streaming understanding. It contains 4,220 QA pairs across 932 videos and evaluates 11 subtasks in multi-window, multi-view, and multi-device scenarios. The benchmark frames MLLMs as naive multiplexers operating under a fixed token-rate budget and studies spatial, time, and semantic division multiplexing. State-of-the-art performance remains limited: Gemini 3 Pro reaches Overall 49.60, Forward 20.77, and Early Response 73.13, while open-source streaming models perform substantially worse. The paper identifies cross-stream anti-interference, cross-stream reference alignment, cross-stream cooperation, and proactive behavior as core difficulties (Sun et al., 1 Jun 2026).

Infrastructure work has addressed a different bottleneck: how to stream tokens across heterogeneous compute domains. "STREAM: Multi-Tier LLM Inference Middleware with Dual-Channel HPC Token Streaming" (Nassar et al., 11 Jun 2026) unifies local, institutional HPC, and cloud inference through a three-tier router, a local complexity judge, and a dual-channel architecture that separates Globus Compute control from WebSocket relay data delivery. The reported median TTFT is 0.26 s for local inference, 0.54 s for HPC relay streaming, and 1.68 s for cloud, with the HPC relay delivering a 21.1× TTFT improvement over batch mode’s 11.40 s. The system uses end-to-end AES-256-GCM encryption for token payloads and reports 85.1% free-tier retention on a 1,200-query benchmark spanning ten domains (Nassar et al., 11 Jun 2026). Here again, the stream abstraction is infrastructural: control and data planes are decoupled, and per-query token streams are routed across trust and firewall boundaries.

A training-time analogue appears in StreamRL, which disaggregates generation and training for RL over LLMs and introduces sample-level stream generation. The Stream Generation Service returns completed samples immediately so that the Trainer can begin Reference Model inference, KL computation, reward derivation, and dynamic batching before the full generation batch finishes. Combined with skewness-aware scheduling for long-tail outputs, the paper reports throughput improvements of up to 2.66× over existing systems and cost-effectiveness improvements of up to 1.33× in a heterogeneous, cross-datacenter setting (Zhong et al., 22 Apr 2025). Although the paper is not itself framed as a canonical MultiStream-LLM model, it extends the same concurrency principle to post-training.

The literature also exposes recurring trade-offs. STREAM2LLM notes that highly volatile update-mode prompts can increase recomputation overhead and tail latency, and that starvation is possible if some requests rarely receive chunks (Bachkaniwala et al., 29 Mar 2026). The sign-language framework identifies domain shift, signer variability, and low-resource signs as open problems (Thomas et al., 20 Aug 2025). Fusion-based multi-LLM integration still incurs token-alignment overhead and can struggle with strongly conflicting sources (Kong et al., 28 May 2025). X-Stream shows that current models have especially poor proactive timing, causal reasoning, and decision-making under multi-stream online constraints (Sun et al., 1 Jun 2026). Across these lines, the unresolved question is not whether multiple streams can be engineered, but which decomposition of streams best matches the structure of the task: roles, modalities, prompts, models, systems tiers, or training stages.