Video Streaming Thinking: Online Reasoning

• Video Streaming Thinking (VST) is a framework for continuous video reasoning that processes frames incrementally with strict no-future access.
• VST emphasizes causal memory, selective retention, and event-triggered responses to balance real-time question answering with resource constraints.
• Recent research benchmarks, like StreamingEval, illustrate the trade-offs between accuracy, throughput, and memory usage in streaming video models.

Video Streaming Thinking (VST) denotes a causal, incremental regime of video understanding in which a model observes a stream over time, updates usable internal state as new frames or segments arrive, and answers under bounded memory, compute, and latency constraints rather than after seeing a complete prerecorded clip. Across recent work, VST is characterized by strict no-future access, persistent state maintenance, asynchronous question arrival, and online response timing; in this sense it is closely aligned with “streaming video reasoning,” “streaming online question answering,” and “Watch–Think–Speak” style formulations that replace retrospective clip-level inference with continuous reasoning over partial evidence (Tang et al., 23 Mar 2026, Liu et al., 13 Mar 2026).

1. Historical emergence and scope

Early video-LLM work on streaming established the basic departure from offline clip reasoning. "VideoLLM-online: Online Video LLM for Streaming Video" introduced the LIVE framework for temporally aligned, long-context, and real-time conversation within a continuous video stream, together with Streaming EOS prediction so the model can learn when to stay silent rather than decoding a response at every frame; on average, it supports streaming dialogue in a 5-minute video clip at over 10 FPS on an A100 GPU (Chen et al., 2024). "Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams" then made online video QA explicitly memory-centric by splitting execution into a frame handler and a question handler connected through STAR memory, with a fixed total memory size of 681 tokens and reported real-time answering within 1 second upon user query (Zhang et al., 2024). In parallel, "Streaming Long Video Understanding with LLMs" proposed Memory-Propagated Streaming Encoding and Adaptive Memory Selection, arguing that long-video understanding can be made compatible with a constant downstream visual-token budget by recurrently compressing history and retrieving only question-related memories (Qian et al., 2024).

Later work expanded the scope of VST from online dialogue to explicit reasoning control. "StreamMind: Unlocking Full Frame Rate Streaming Video Dialogue through Event-Gated Cognition" reframed the problem as continuous perception with sparse, event-triggered LLM invocation, reporting ultra-FPS streaming video processing at 100 FPS on a single A100 (Ding et al., 8 Mar 2025). "Thinking in Streaming Video" made incremental reasoning itself explicit by generating short > traces at every step and reusing them as memory (Liu et al., 13 Mar 2026). "An Efficient Streaming Video Understanding Framework with Agentic Control" treated streaming understanding as a cascaded control problem over remembering, responding, and escalating to stronger reasoning models (Liu et al., 18 May 2026). "What Should a Streaming Video Model Remember?" sharpened the central question from raw retention to selective evidence allocation under fixed budgets (Ge et al., 15 Jun 2026). This suggests that VST has evolved from simple online inference toward a joint study of causal memory, answer timing, retrieval, and adaptive compute.

2. Formal models of causal streaming inference

A canonical formalization appears in "StreamingEval: A Unified Evaluation Protocol towards Realistic Streaming Video Understanding" (Tang et al., 23 Mar 2026). In its streaming online question answering setting, at time $t$ the model may access prior dialogue context $C_t$, the newly arrived video frame $V_t$, and a user query $Q_t$, but when a query arrives it may condition only on frames that have already arrived and have already been encoded. When the encoded query finishes at time $t_1$, the response is generated by maximizing

$$p\!\left(R_{t_1}\mid C_{t_1}, V_{[0,t_1]}, Q_{t_1}\right).$$

To make this operational, StreamingEval implements an asynchronous, time-causal pipeline with three decoupled processes: a Frame Player emitting $\{(v_i,\tau_i)\}_{i=1}^{\infty}$, an Encoder-and-Memory Updater computing

$$z_i = g_\theta(v_i), \qquad M_{\tau_i^+} = \mathcal{U}\!\left(M_{\tau_i^-}, z_i;\, B, \pi\right),$$

and a Responder that answers queries from the memory snapshot available when query encoding completes,

$$R_{t_1} \sim p_\phi(\cdot \mid q_{t_0}, C_{t_1}, M_{t_1}).$$

The key distinction from offline video understanding is that there is no future access, no full-video buffering for later recomputation, and no hidden offline pass.

ThinkStream turns this causal setup into an explicit online reasoning loop (Liu et al., 13 Mar 2026). The stream is $\mathcal{V}=\{v_1,v_2,\dots\}$, and at each step the model outputs

$C_t$0

with

$C_t$1

The generation process is

$C_t$2

so perception, incremental reasoning, and response timing are unified in one autoregressive policy. "Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal LLMs" extends this logic to multi-turn segment streams $C_t$3 and dialogue history $C_t$4, enforcing strict online causality with a segment-level streaming causal mask and streaming positional encoding so that answers at turn $C_t$5 depend only on $C_t$6, $C_t$7, and prior turns (Wang et al., 12 Mar 2026).

3. Memory as the substrate of streaming thought

VST research converges on the idea that memory is not an auxiliary module but the primary substrate of online reasoning. One line of work retains multiple granularities of visual evidence. Flash-VStream’s STAR memory contains a feature buffer, spatial memory, temporal memory, abstract memory, and retrieved memory, with total capacity

$C_t$8

and the reported implementation fixes this at 681 tokens (Zhang et al., 2024). Temporal memory is updated by weighted K-means, abstract memory by semantic attention, and retrieved memory by nearest-neighbor recovery from the buffer. VideoStreaming adopts a related but recurrent formulation: for clip $C_t$9, the streaming encoder updates memory by

$V_t$0

then performs question-conditioned top-$V_t$1 memory selection so the downstream LLM sees a constant visual token budget, 256 tokens in the default setting (Qian et al., 2024).

A second line compresses history into structured semantic units. Vista performs scene-aware segmentation, scene-aware compression, and scene-aware recall for post-hoc streaming QA (Lu et al., 9 Feb 2026). Completed scenes are compressed into one GPU-resident scene token

$V_t$2

while full-resolution frames are offloaded to CPU memory or disk; when a query $V_t$3 arrives, the model computes $V_t$4, scores scenes by

$V_t$5

retrieves top-$V_t$6 scenes, and reconstructs the final visual evidence

$V_t$7

Event-VStream shifts the storage unit from scenes to events: it detects boundaries from semantic drift, motion intensity, and predictive error, then stores a persistent event memory bank

$V_t$8

with merge-or-append consolidation

$V_t$9

Language generation is triggered only at event boundaries, not at fixed intervals (Guo et al., 22 Jan 2026). Think While Watching similarly writes one memory note $Q_t$0 per segment and preserves a persistent segment-level bank $Q_t$1, directly targeting multi-turn consistency (Wang et al., 12 Mar 2026).

A third line treats reasoning itself as memory. ThinkStream’s Reasoning-Compressed Streaming Memory retains only a recent visual KV window and all prior reasoning/action tokens: $Q_t$2 Old dense visual tokens are evicted, but their semantic content survives as textual reasoning traces (Liu et al., 13 Mar 2026). SelectStream pushes this further into latent evidence allocation. Its memory state is a fixed-capacity latent graph

$Q_t$3

with explicit budgets for active nodes $Q_t$4, retrieved subgraph size $Q_t$5, and injected evidence tokens $Q_t$6. Surprise-driven adaptive windowing decides when to write, priority-preserving consolidation decides what to preserve, and query-conditioned graph reasoning decides what to expose to the frozen VLM (Ge et al., 15 Jun 2026). This suggests a broad shift from raw-token retention toward structured episodic, semantic, or latent evidence memories that are query-ready without replaying the full stream.

4. Response timing, adaptive computation, and concurrent watching

VST requires deciding not only what to remember, but when to think, when to answer, and how much compute to spend. StreamMind isolates this problem as event-gated cognition (Ding et al., 8 Mar 2025). Continuous perception runs on every frame through the Event-Preserving Feature Extractor (EPFE),

$Q_t$7

while a Cognition Gate receives the prompt and current perception token and outputs either </response> or </silence>. The paper’s core systems claim is that prior per-step LLM invocation yields $Q_t$8 total cost over $Q_t$9 frames, whereas event-gated invocation reserves heavyweight reasoning for query-relevant moments and reaches 100 FPS on a single A100.

R3-Streaming formulates this control problem explicitly (Liu et al., 18 May 2026). At each step it maintains memory $t_1$0, receives query $t_1$1, and chooses from

$t_1$2

Its Remember stage applies age-aware dual-zone compression,

$t_1$3

preserving recent context while aggressively compressing history. The Respond stage estimates readiness $t_1$4 and emits $t_1$5 if $t_1$6. The Reason stage routes between a fast model and a stronger slow model using TB-GRPO, which stabilizes escalation around a target band and prevents mode collapse to always-escalate behavior. Reported results reach 57.92 on OVO-Bench and 76.36 on StreamingBench while reducing visual token usage by 95 to 96 percent.

Think While Watching addresses a different bottleneck: serialized watching and answering (Wang et al., 12 Mar 2026). It introduces a dual-KV-cache pipeline that overlaps source ingestion and autoregressive generation, together with a segment-level streaming causal mask

$t_1$7

and decoupled positional offsets for received units and generated units. In the multi-round setting it maintains accuracy while reducing output tokens by 56 percent on StreamingBench. Across these systems, VST increasingly treats response timing and compute allocation as first-class online decisions rather than byproducts of a fixed decoding schedule.

5. Evaluation protocols and empirical trade-offs

StreamingEval provides the most explicit benchmark definition of realistic VST (Tang et al., 23 Mar 2026). It normalizes historical visual context with a fixed-capacity, byte-budgeted memory bank, feeds frames at 1 FPS, allows queries at arbitrary times, and jointly measures four aspects of deployability: question-answering accuracy, visual encoding efficiency via MaxFPS, text decoding latency via TTFT, and memory cost. These are summarized by

$t_1$8

with equal weights in the main leaderboard. The protocol matters because it avoids the common offline surrogate in which a video is truncated at a query timestamp and then processed as a full clip.

Its empirical findings are central to the encyclopedia of VST. On OVO-Bench, the best offline-adapted model in overall accuracy is Qwen3-VL-8B at 58.00, whereas the best online model is StreamForest-7B at 55.57; on StreamingBench, InternVL3.5-8B reaches 77.96, Qwen3-VL-8B 77.31, and StreamForest-7B 77.26. Efficiency does not align monotonically with “online” status: many models exceed 1 FPS, but VideoChatOnline-4B reaches only 0.14 FPS, while TTFT stays under 1.5 s for most models. The composite StreamingScore further shows that a low-accuracy model can dominate under severe resource weighting: on OVO-Bench, Flash-VStream-7B attains StreamingScore 2.34 despite 33.15 accuracy, largely because its TTFT is 0.12 s and memory use is 0.35 GB, whereas Qwen3-VL-8B reaches 2.21 with much higher accuracy but slightly higher resource cost. Task-cluster analysis reinforces a characteristic VST trade-off: offline or memory-richer systems are stronger on Backward Tracing, while some native online systems are more competitive on Forward Active Responding. Memory-budget ablations show near-saturation in the 1.0–1.5 GB regime, clear degradation below 1.0 GB, and convergence under the severe 0.1 GB budget.

Newer systems report stronger benchmark numbers under their own protocols, but they do so through different memory and control strategies. ThinkStream-3B reports 59.66 on OVO-Bench and 75.00 on StreamingBench Real-Time by turning reasoning traces into memory (Liu et al., 13 Mar 2026). SelectStream-Qwen3-VL-8B reaches 67.03 on OVO-Bench and 82.67 on StreamingBench through selective latent evidence retrieval (Ge et al., 15 Jun 2026). R3-Streaming-7B|4B-Thinking reaches 57.92 and 76.36 with age-aware forgetting and adaptive escalation (Liu et al., 18 May 2026). Think While Watching reports single-round gains of 2.6% on StreamingBench and 3.79% on OVO-Bench, together with large multi-round token reductions (Wang et al., 12 Mar 2026). Taken together, these results indicate that VST performance depends less on a simple offline-versus-online dichotomy than on how memory quality, retrieval selectivity, throughput, and answer timing are co-designed.

6. Misconceptions, limitations, and research directions

A recurrent misconception is that native online architectures necessarily dominate offline ones in realistic streaming deployments. StreamingEval shows the opposite can happen: mainstream offline Video-LLMs, retrofitted with a bounded-memory adapter, often achieve better task accuracy under the same causal protocol, though usually at greater resource usage (Tang et al., 23 Mar 2026). A second misconception is that more history is always better. SelectStream argues that indiscriminate history injection can dilute current-scene perception, and its evidence-budgeted retrieval is built precisely to avoid that failure mode (Ge et al., 15 Jun 2026). R3-Streaming reports that over-retaining stale evidence can hurt both accuracy and efficiency, and that preserving a small high-fidelity recent window while compressing history aggressively is often preferable (Liu et al., 18 May 2026).

Another limitation is that current memory abstractions are still coarse. Vista stores one compact scene token per completed scene and performs one-shot top-$t_1$9 recall, but the paper notes that retrieval is one-shot, scene tokens are coarse, and complex cross-scene reasoning or subtle transitions can remain difficult (Lu et al., 9 Feb 2026). ThinkStream bounds dense visual memory but not total reasoning-token growth; its RLVR stage is also limited to verifiable formats such as multiple-choice, yes/no, and counting (Liu et al., 13 Mar 2026). R3-Streaming explicitly states that TB-GRPO controls average escalation ratio but does not solve worst-case latency spikes during prolonged high-information-density intervals (Liu et al., 18 May 2026). StreamingEval itself is limited mainly to open-source 7B–8B models and warns against over-generalizing a single benchmark into a universal deployment standard (Tang et al., 23 Mar 2026).

VST is also a systems problem below the reasoning layer. CoStream shows that continuous video analytics can be made substantially cheaper by exploiting codec metadata already exposed by compression, applying codec-guided patch pruning before ViT encoding and selective KV-cache refresh during LLM prefilling; it reports up to $$p\!\left(R_{t_1}\mid C_{t_1}, V_{[0,t_1]}, Q_{t_1}\right).$$0 throughput improvement and 0–8% F1 drop at its default operating point (Zou et al., 7 Apr 2026). The paper is explicit that this does not improve the underlying reasoning capability of the VLM, but it does make continuous multimodal inference far more affordable. This suggests that practical VST will depend on joint progress in causal memory design, retrieval, concurrency, and serving infrastructure.

The dominant research directions follow directly from these limitations. StreamingEval advocates streaming, incrementally updatable representations and latency–accuracy controllable computation allocation (Tang et al., 23 Mar 2026). Vista implies richer scene memory and stronger online boundary detection (Lu et al., 9 Feb 2026). R3-Streaming points toward relevance-aware forgetting beyond age-only zoning, stronger readiness calibration, and spike-aware compute control (Liu et al., 18 May 2026). ThinkStream suggests further work on long-horizon memory mechanisms that keep semantic traces useful without unbounded growth (Liu et al., 13 Mar 2026). In aggregate, these directions define VST less as a single architecture than as a research program centered on causality, persistent memory, selective recall, adaptive reasoning depth, and realistic resource-bounded deployment.