---
title: 'X-Stream: Multi-Stream MLLM Benchmark'
url: https://www.emergentmind.com/topics/x-stream
type: topic
---

# X-Stream: Multi-Stream MLLM Benchmark

X-Stream is a benchmark explicitly designed to evaluate the real-time, online understanding capabilities of multi-modal large language models (MLLMs) when processing multiple, synchronized video streams. Targeting real-world demands such as live sports broadcasting, autonomous driving, and multi-screen collaboration, X-Stream addresses the absence of prior evaluation frameworks that require concurrent multi-stream reasoning, cross-stream reference, and proactive temporal awareness. The benchmark introduces a rigorously dual-verified dataset of 4,220 timestamped QA pairs across 932 videos and 11 highly heterogeneous subtasks, representing the first systematic platform for examining multi-stream streaming understanding in unconstrained, “in the wild” scenarios [2606.02482].

## 1. Motivation and Problem Definition

X-Stream formalizes multi-stream streaming understanding as the continuous online fusion and reasoning over $N$ time-aligned streams $\{S_1, \dots, S_N\}$ with shared clocks. The modeled agent must answer two primary query types at any time $t$:

- **Instant questions:** Answerable immediately based on past or present frames.
- **Forward questions:** Requiring the agent to delay response until a specified future event unfolds.

Prior benchmarks (e.g., StreamingBench, OVO-Bench, MVU-Bench) are constrained to $N=1$ or assume offline access, neglecting cross-stream interference (filtering misleading cues), cross-stream reference (event localization between views), and cross-stream cooperation (information synthesis). X-Stream fills these gaps by mandating synchronized streams ($N=2$–$5$), timestamped QA requiring cross-stream context, and an evaluation protocol scoring both accuracy and temporal appropriateness.

## 2. Dataset Construction and Taxonomy

X-Stream was curated from 857 hours of raw multi-stream video (2–10 feeds per take, 20+ sources, 8 domains), distilled into 160 hours spanning 451 takes (932 videos, 4,220 QA pairs). Videos average 15.8 minutes (range: 5–30 min); 80% of takes contain $N=2$ streams, while 20% include $N=3$–$5$. Approximately 30% of questions involve audio.

The dataset taxonomy covers 11 subtasks:

- **Foundational perception:** Visual/audio/temporal grounding, counting, saliency detection.
- **Logical cognition:** 3D spatial reasoning, counterfactual reasoning, causal inference, common-sense reasoning, anomaly detection.
- **Agency:** Behavior planning.

Curation leverages a dual-verification pipeline:

1. **Pre-processing:** All feeds resampled at 2 FPS; chunk size limited to <50 MB.
2. **QA generation:** MLLM (Gemini-3-Pro) plus structured templates generate timestamped QAs and rationales.
3. **Multi-Stream Sufficiency:** Each QA must be solvable by model $f$ when accessing all streams aligned at $t$.
4. **Multi-Stream Necessity:** No single stream alone allows $f$ to answer correctly.
5. **Human review:** 31 experts apply two-stage QA correction, achieving 94.5% dataset accuracy.

Error breakdown: 65.2% question issues, 21.7% answer errors, 13% timestamp errors.

## 3. Signal Multiplexing Theory: Theoretical Framework

X-Stream introduces a multiplexing-theoretic perspective, treating MLLMs as naive multiplexers with a fixed bandwidth limit $C_{\text{max}}$ (tokens/sec). Three orthogonal division methods—derived from classical communications theory—govern simultaneous stream processing:

- **Spatial Division Multiplexing:** Streams are downsampled (ratios $r_m$, $r_n$) and concatenated per frame, subject to the global token constraint $\|\mathcal{T}(X_t)\| \leq C_{\text{max}}$.
- **Time Division Multiplexing:** At each timestep $t$, a single stream is selected ($\alpha_t, \beta_t \in \{0,1\}$) such that $\alpha_t\|\mathcal{T}(M_t)\| + \beta_t\|\mathcal{T}(N_t)\| \leq C_{\text{max}}$; temporal metadata (tags/embeddings) preserve source identity.
- **Semantic Division Multiplexing:** Salient tokens ($k_m$, $k_n$) are selected per frame using a determinantal point process (DPP) kernel optimized for relevance and diversity. The token budget satisfies $|\mathcal{S}(\mathcal{T}(M_t), k_m)| + |\mathcal{S}(\mathcal{T}(N_t), k_n)| \leq C_{\text{max}}$.

This framework enables systematic investigation of performance trade-offs under varying stream counts $N$ and bandwidth constraints $C_{\text{max}}$.

## 4. Experimental Protocol and Baseline Results

Online inference is conducted over every second of video (sliding window: 2s), enforcing a token cap $C_{\text{max}} = 250\ \text{tokens/sec}$ via adaptive resizing or sampling. Evaluation metrics include Instant, Backward, Forward, and Comprehensive accuracy scores ([0, 100]); Early Response (ER$\downarrow$) and No Response (NR$\downarrow$) measure temporal alignment. Rationales are scored by LLM-as-Judge (Qwen3-235B), correlating with humans at Spearman $\rho=0.62$.

Examined models:

| Model Type     | Model Names                                              | Peak Performance (%) |
|----------------|---------------------------------------------------------|---------------------|
| Proprietary    | Gemini-3-Pro, GPT-5, GPT-4o, Doubao-Seed-1.8            | 49.6 (Gemini-3-Pro) |
| Open-source    | Qwen2.5-VL-7B, Qwen2.5-Omni-7B, Qwen3-VL-8B, etc.       | 34.3                |
| Streaming LLMs | Dispider, VideoLLM-online-8B, MMDuet2                   | <16                 |
| Human Upper    | —                                                       | 91.8                |

Forward (proactive) QA is most difficult: Gemini-3-Pro achieves 20.8%, with ER=73% and NR=0.23%. Foundational perception tasks score 60–75%; Causal reasoning and Behavior Planning remain below 45%, indicating these as current bottlenecks. Cross-stream ability metrics: Single-Stream ≈ 72%, Cross-Ref ≈ 75%, Cross-Interference ≈ 67%, Multi-Coop ≈ 71%.

Reported failure cases include:

- "Pseudo" multi-stream QA solvable using one stream
- Poor anticipation for forward questions (premature or missing response)
- Disalignment in temporal fusion across streams for both free-form and multiple-choice QAs

## 5. Comparative Analysis of Multiplexing Strategies

Empirical ablations using Gemini-3-Pro and Qwen3-Omni-30B-A3B highlight the characteristics of each multiplexing scheme:

- **Spatial Division excels** at cross-stream referencing (spatial layout preservation)
- **Time Division optimal** for $N=2$ if token budget is ample (complete frames retained)
- **Semantic Division outperforms** for $N\geq3$ or strict $C_{\text{max}}$ (efficient salience filtering)

Scalability trends (Table 4): Spatial and Time Division schemes degrade rapidly as $N$ increases; Semantic Division degrades gracefully. Analogies to classic telecom multiplexing include frequency wasting and time jitter.

Empirical guidance for further research includes:

- Learnable, dynamic multiplexers for token/pixel allocation
- Early cross-stream fusion layers
- Adaptive semantic token selection conditioned on queries
- Context window and hierarchical summarization enhancement to ease $C_{\text{max}}$ limitations

## 6. Challenges, Implications, and Prospects

X-Stream exposes the substantial gap between human and MLLM performance (best SOTA ≈ 50% overall, <21% proactive), especially for cross-stream temporal and causal cognition. Major technical challenges include:

- Joint, end-to-end learnable multiplexers optimizing spatial, temporal, and semantic trade-offs
- Scaling model context windows and hierarchical memory for high-resolution, long-duration feeds
- Extending multiplexing to richer modalities (audio-visual fusion) under bandwidth constraints
- Enabling feedback-rich, multi-agent or fully multi-modal interaction

A plausible implication is that the trajectory of multi-stream agents will be determined by advances in dynamic resource allocation, cross-modal fusion, and online memory architectures. X-Stream provides both a demanding dataset and analytical foundation for systematic progress in these directions [2606.02482].

Source: https://www.emergentmind.com/topics/x-stream