SimpleStream: Efficient Streaming Video Baseline

• SimpleStream is a streaming baseline that uses a fixed-size sliding window to maintain the most recent frames for causal multimodal query answering.
• The approach replaces complex memory architectures with a recency-focused protocol, achieving state-of-the-art performance on benchmarks like OVO-Bench and StreamingBench.
• Its design ensures predictable computational costs and efficient resource use, balancing real-time visual perception with episodic recall in streaming applications.

SimpleStream is a formalized sliding-window baseline for streaming video understanding that maintains only the most recent $N$ frames as context for an off-the-shelf video LLM (VLM) to answer multimodal queries under strict causal constraints. Contrary to the increasing reliance on complex memory and retrieval mechanisms in recent literature, SimpleStream demonstrates that a carefully constructed recency window can match or surpass the performance of advanced streaming models on standard evaluation suites, with superior efficiency and a clear protocol for direct comparison (Shen et al., 2 Apr 2026).

1. Formal Definition and Protocol

SimpleStream operates on a theoretically infinite stream of video frames $\dots, f_1, f_2, \dots, f_t, \dots$ with queries $q_t$ arriving at each time step $t \in \mathbb{N}$. Under causal (streaming) assumptions, answers at time $t$ are permitted to use only frames $\{f_1,\dots,f_t\}$. To ensure practical computational and memory efficiency, SimpleStream restricts the input context at each step to a window

$C_t = \{ f_{\max(1, t-N+1)}, \dots, f_t \}.$

Given this working context and a text query $q_t$, the output is defined by direct invocation of a VLM:

$$\text{SimpleStream}(t) = \mathrm{VLM}(C_t, q_t) = \mathrm{VLM}(\{f_{t-N+1}, \dots, f_t\}, q_t),$$

where “VLM” references a pretrained multimodal transformer (e.g., Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct) without any extra memory, retrieval, or compression extensions. By definition, no frames prior to $t-N+1$ are ever revisited or stored after they leave the window. This operational simplicity is the essence of the protocol.

2. Implementation Workflow

Implementation proceeds as a tightly bounded inference pipeline:

• Frame Sampling & Buffering: The video stream is sampled at a fixed rate (typically 1 fps), with frames stored in a FIFO buffer to maintain the last $\dots, f_1, f_2, \dots, f_t, \dots$0 frames.
• Visual Encoding: Each frame $\dots, f_1, f_2, \dots, f_t, \dots$1 in the buffer is encoded by the frozen visual encoder of the target VLM (CLIP-style), resulting in a sequence of visual tokens per frame.
• Prompt Construction: Visual tokens from the buffered $\dots, f_1, f_2, \dots, f_t, \dots$2 frames are serialized in temporal order, followed by the text query $\dots, f_1, f_2, \dots, f_t, \dots$3, forming the multimodal prompt compatible with the VLM.
• Autoregressive Decoding: The VLM decoder attends over the concatenated sequence, producing the answer string in a single forward pass with greedy decoding via the standard language modeling head.

Latency and peak GPU memory at each step are strictly functions of the window size $\dots, f_1, f_2, \dots, f_t, \dots$4 and the backbone parameters, as there is no added memory, retrieval, or key-value compression subsystem involved. This ensures a predictable and stable computational profile as input sequences grow.

3. The Perception–Memory Trade-Off

Empirical findings reveal a nontrivial trade-off between exploiting a longer context and maintaining real-time visual perception. Two evaluation axes on OVO-Bench are distinguished:

• Real-time perception score ($\dots, f_1, f_2, \dots, f_t, \dots$5): Macro-averaged accuracy over six “Real-Time Visual Perception” tasks.
• Episodic recall score ($\dots, f_1, f_2, \dots, f_t, \dots$6): Mean score across two memory-centric backward tracks (Episodic Memory (EPM) and Action Sequence Identification (ASI)).

For any method versus SimpleStream, differences are computed as

$\dots, f_1, f_2, \dots, f_t, \dots$7

Empirical evidence shows that virtually all memory-centric models yield $\dots, f_1, f_2, \dots, f_t, \dots$8 (perception loss) when $\dots, f_1, f_2, \dots, f_t, \dots$9 (memory gain), signifying that integrating more historical frames tends to harm immediate scene understanding even as it improves recall (Shen et al., 2 Apr 2026).

4. Benchmark Results and Comparative Performance

SimpleStream attains state-of-the-art performance on both OVO-Bench and StreamingBench among streaming methods across multiple VLM backbones. The table below highlights key results from Table 1 using the optimal configuration (Qwen3-VL-8B, $q_t$0):

Model (streaming)	#Frames	StreamingBench (%)	OVO-Bench Avg (%)
HERMES-7B†	1 fps	79.44	59.20
StreamForest-7B	1 fps	77.26	56.60
Dispider-7B	1 fps	67.63	45.35
TimeChat-Online-7B	1 fps	75.28	51.80
SimpleStream (Qwen3-VL-8B+4f)	1 fps	80.59	67.70

†HERMES-7B with 4K token memory.

SimpleStream exceeds all existing streaming models by at least 8.5 percentage points on OVO-Bench, and matches the highest-reported performance on StreamingBench. This outcome holds while restricting context to the four most recent frames and not leveraging any hierarchical memory systems.

5. Controlled Ablation and Diagnostic Analyses

A series of controlled ablation experiments elucidate the limitations and optimal settings for SimpleStream and similar baselines:

• Recency Window Size: Accuracy on OVO-Bench peaks at $q_t$1 frames (67.7%) and plateaus or degrades at larger $q_t$2, invalidating the assumption of “monotonic gain from more frames.”
• Model Scale: The optimal $q_t$3 depends on the backbone; small and mid-sized models maximize performance at $q_t$4, while some larger models benefit from $q_t$5–$q_t$6. The capacity of the backbone moderates the effective use of extended context.
• Retrieval-Augmented Generation (Visual-RAG): Concatenating five CLIP-retrieved historical chunks to a $q_t$7-frame window yields episodic memory gains ($q_t$8 pp on EPM+ASI) but incurs a $q_t$9 pp deficit in real-time perception, confirming the segregation of perception and recall.
• Memory Bands and Compression Modules: Methods such as Flash-VStream, ReKV, HERMES, and StreamForest all induce higher peak memory footprints and, universally, negative $t \in \mathbb{N}$0 values compared to SimpleStream. Peak GPU memory for SimpleStream remains invariant as the video stream grows, in contrast to the sharply increasing memory requirements of competitors.

6. Implications and Recommendations for Benchmarking

Based on the evidence provided, several protocol recommendations are put forward:

1. Mandatory Baselines: All new streaming-video understanding architectures should be rigorously evaluated against SimpleStream using matched backbones and equivalent protocols to prevent spurious claims of progress through additional complexity.
2. Disaggregated Metrics: Reporting should decompose “real-time perception” ($t \in \mathbb{N}$1) and “long-range memory” ($t \in \mathbb{N}$2), eschewing single macro-averages that obscure the inherent perception–memory trade-off.
3. Explicit Task Separation: Benchmark suites are advised to distinctly evaluate recent-scene perception, episodic memory recall, and hallucination robustness.
4. Efficiency Reporting: Time-to-first-token and peak memory statistics must be co-reported with accuracy for all methods, ensuring that the resource cost of memory-centric innovations is justified by demonstrable recall improvements.

In synthesis, SimpleStream establishes a robust, efficient, and interpretable streaming baseline. The principal insight is that state-of-the-art performance can be achieved via recency-focused context windows, and that more elaborate memory solutions should be held to strict empirical standards, demonstrating advantages over SimpleStream in both recall and perceptual accuracy without disproportionate resource trade-offs (Shen et al., 2 Apr 2026).