X-Stream: Multi-Stream MLLM Benchmark

• X-Stream is a benchmark for evaluating online multi-stream video understanding in real-world applications like live sports, autonomous driving, and multi-screen collaboration.
• It features a rigorously dual-verified dataset of 4,220 timestamped QA pairs from 932 videos across 11 subtasks, ensuring cross-stream necessity and sufficiency.
• The framework employs spatial, temporal, and semantic multiplexing strategies to manage token budgets and tackle challenges such as cross-stream interference and proactive temporal reasoning.

X-Stream is a benchmark explicitly designed to evaluate the real-time, online understanding capabilities of multi-modal LLMs (MLLMs) when processing multiple, synchronized video streams. Targeting real-world demands such as live sports broadcasting, autonomous driving, and multi-screen collaboration, X-Stream addresses the absence of prior evaluation frameworks that require concurrent multi-stream reasoning, cross-stream reference, and proactive temporal awareness. The benchmark introduces a rigorously dual-verified dataset of 4,220 timestamped QA pairs across 932 videos and 11 highly heterogeneous subtasks, representing the first systematic platform for examining multi-stream streaming understanding in unconstrained, “in the wild” scenarios (Sun et al., 1 Jun 2026).

1. Motivation and Problem Definition

X-Stream formalizes multi-stream streaming understanding as the continuous online fusion and reasoning over $N$ time-aligned streams $\{S_1, \dots, S_N\}$ with shared clocks. The modeled agent must answer two primary query types at any time $t$:

• Instant questions: Answerable immediately based on past or present frames.
• Forward questions: Requiring the agent to delay response until a specified future event unfolds.

Prior benchmarks (e.g., StreamingBench, OVO-Bench, MVU-Bench) are constrained to $N=1$ or assume offline access, neglecting cross-stream interference (filtering misleading cues), cross-stream reference (event localization between views), and cross-stream cooperation (information synthesis). X-Stream fills these gaps by mandating synchronized streams ($N=2$–$5$), timestamped QA requiring cross-stream context, and an evaluation protocol scoring both accuracy and temporal appropriateness.

2. Dataset Construction and Taxonomy

X-Stream was curated from 857 hours of raw multi-stream video (2–10 feeds per take, 20+ sources, 8 domains), distilled into 160 hours spanning 451 takes (932 videos, 4,220 QA pairs). Videos average 15.8 minutes (range: 5–30 min); 80% of takes contain $N=2$ streams, while 20% include $N=3$–$5$. Approximately 30% of questions involve audio.

The dataset taxonomy covers 11 subtasks:

• Foundational perception: Visual/audio/temporal grounding, counting, saliency detection.
• Logical cognition: 3D spatial reasoning, counterfactual reasoning, causal inference, common-sense reasoning, anomaly detection.
• Agency: Behavior planning.

Curation leverages a dual-verification pipeline:

1. Pre-processing: All feeds resampled at 2 FPS; chunk size limited to <50 MB.
2. QA generation: MLLM (Gemini-3-Pro) plus structured templates generate timestamped QAs and rationales.
3. Multi-Stream Sufficiency: Each QA must be solvable by model $f$ when accessing all streams aligned at $\{S_1, \dots, S_N\}$0.
4. Multi-Stream Necessity: No single stream alone allows $\{S_1, \dots, S_N\}$1 to answer correctly.
5. Human review: 31 experts apply two-stage QA correction, achieving 94.5% dataset accuracy.

Error breakdown: 65.2% question issues, 21.7% answer errors, 13% timestamp errors.

3. Signal Multiplexing Theory: Theoretical Framework

X-Stream introduces a multiplexing-theoretic perspective, treating MLLMs as naive multiplexers with a fixed bandwidth limit $\{S_1, \dots, S_N\}$2 (tokens/sec). Three orthogonal division methods—derived from classical communications theory—govern simultaneous stream processing:

• Spatial Division Multiplexing: Streams are downsampled (ratios $\{S_1, \dots, S_N\}$3, $\{S_1, \dots, S_N\}$4) and concatenated per frame, subject to the global token constraint $\{S_1, \dots, S_N\}$5.
• Time Division Multiplexing: At each timestep $\{S_1, \dots, S_N\}$6, a single stream is selected ($\{S_1, \dots, S_N\}$7) such that $\{S_1, \dots, S_N\}$8; temporal metadata (tags/embeddings) preserve source identity.
• Semantic Division Multiplexing: Salient tokens ($\{S_1, \dots, S_N\}$9, $t$0) are selected per frame using a determinantal point process (DPP) kernel optimized for relevance and diversity. The token budget satisfies $t$1.

This framework enables systematic investigation of performance trade-offs under varying stream counts $t$2 and bandwidth constraints $t$3.

4. Experimental Protocol and Baseline Results

Online inference is conducted over every second of video (sliding window: 2s), enforcing a token cap $t$4 via adaptive resizing or sampling. Evaluation metrics include Instant, Backward, Forward, and Comprehensive accuracy scores ([0, 100]); Early Response (ER$t$5) and No Response (NR$t$6) measure temporal alignment. Rationales are scored by LLM-as-Judge (Qwen3-235B), correlating with humans at Spearman $t$7.

Examined models:

Model Type	Model Names	Peak Performance (%)
Proprietary	Gemini-3-Pro, GPT-5, GPT-4o, Doubao-Seed-1.8	49.6 (Gemini-3-Pro)
Open-source	Qwen2.5-VL-7B, Qwen2.5-Omni-7B, Qwen3-VL-8B, etc.	34.3
Streaming LLMs	Dispider, VideoLLM-online-8B, MMDuet2	<16
Human Upper	—	91.8

Forward (proactive) QA is most difficult: Gemini-3-Pro achieves 20.8%, with ER=73% and NR=0.23%. Foundational perception tasks score 60–75%; Causal reasoning and Behavior Planning remain below 45%, indicating these as current bottlenecks. Cross-stream ability metrics: Single-Stream ≈ 72%, Cross-Ref ≈ 75%, Cross-Interference ≈ 67%, Multi-Coop ≈ 71%.

Reported failure cases include:

• "Pseudo" multi-stream QA solvable using one stream
• Poor anticipation for forward questions (premature or missing response)
• Disalignment in temporal fusion across streams for both free-form and multiple-choice QAs

5. Comparative Analysis of Multiplexing Strategies

Empirical ablations using Gemini-3-Pro and Qwen3-Omni-30B-A3B highlight the characteristics of each multiplexing scheme:

• Spatial Division excels at cross-stream referencing (spatial layout preservation)
• Time Division optimal for $t$8 if token budget is ample (complete frames retained)
• Semantic Division outperforms for $t$9 or strict $N=1$0 (efficient salience filtering)

Scalability trends (Table 4): Spatial and Time Division schemes degrade rapidly as $N=1$1 increases; Semantic Division degrades gracefully. Analogies to classic telecom multiplexing include frequency wasting and time jitter.

Empirical guidance for further research includes:

• Learnable, dynamic multiplexers for token/pixel allocation
• Early cross-stream fusion layers
• Adaptive semantic token selection conditioned on queries
• Context window and hierarchical summarization enhancement to ease $N=1$2 limitations

6. Challenges, Implications, and Prospects

X-Stream exposes the substantial gap between human and MLLM performance (best SOTA ≈ 50% overall, <21% proactive), especially for cross-stream temporal and causal cognition. Major technical challenges include:

• Joint, end-to-end learnable multiplexers optimizing spatial, temporal, and semantic trade-offs
• Scaling model context windows and hierarchical memory for high-resolution, long-duration feeds
• Extending multiplexing to richer modalities (audio-visual fusion) under bandwidth constraints
• Enabling feedback-rich, multi-agent or fully multi-modal interaction

A plausible implication is that the trajectory of multi-stream agents will be determined by advances in dynamic resource allocation, cross-modal fusion, and online memory architectures. X-Stream provides both a demanding dataset and analytical foundation for systematic progress in these directions (Sun et al., 1 Jun 2026).