Event-VStream: Event-Driven Real-Time Understanding for Long Video Streams

Abstract: Real-time understanding of long video streams remains challenging for multimodal LLMs (VLMs) due to redundant frame processing and rapid forgetting of past context. Existing streaming systems rely on fixed-interval decoding or cache pruning, which either produce repetitive outputs or discard crucial temporal information. We introduce Event-VStream, an event-aware framework that represents continuous video as a sequence of discrete, semantically coherent events. Our system detects meaningful state transitions by integrating motion, semantic, and predictive cues, and triggers language generation only at those boundaries. Each event embedding is consolidated into a persistent memory bank, enabling long-horizon reasoning while maintaining low latency. Across OVOBench-Realtime, and long-form Ego4D evaluations, Event-VStream achieves competitive performance. It improves over a VideoLLM-Online-8B baseline by +10.4 points on OVOBench-Realtime, achieves performance close to Flash-VStream-7B despite using only a general-purpose LLaMA-3-8B text backbone, and maintains around 70% GPT-5 win rate on 2-hour Ego4D streams.

• The paper introduces Event-VStream, an event-driven approach that reduces redundant computation and preserves long-term context in video streams by consolidating frames into event tokens.
• Event-VStream detects event boundaries using motion, semantic drift, and prediction error cues, significantly improving OCR and ATR tasks by +8.05 and +12.07 per-task score gains respectively.
• The method outperforms baseline models with a stable win rate of ~70% over 2-hour streams while maintaining low latency (0.05–0.08 s/token), even without retraining the underlying VLM.

Event-VStream addresses two persistent failure modes of streaming video-LLMs (VLMs): redundant computation from high-frequency decoding, and forgetting caused by fixed-interval KV-cache refresh. The paper's central claim is that both problems stem from a representational mismatch — frame-uniform processing assumes visual semantics evolve smoothly, whereas real video exhibits block-structured stability punctuated by abrupt transitions. The proposed remedy is to treat the stream as a sequence of discrete events: detect boundaries using motion, semantic drift, and prediction error; consolidate each event into a compact embedding stored in a persistent memory bank; and trigger language generation only at those boundaries. The framework is training-free and model-agnostic, instantiated on top of VideoLLM-Online-8B with a general-purpose LLaMA-3-8B text backbone.

Motivation and empirical grounding

The authors ground their design choice empirically rather than only by appeal to intuition. Analysis of frame-level embedding similarity matrices reveals block-structured recurrence — semantically coherent segments reappear over time rather than decaying smoothly — and temporal redundancy drops sharply at event boundaries instead of degrading gradually. They also observe that motion spikes precede semantic drift (drops in cosine similarity between consecutive embeddings) by roughly 2 seconds, suggesting motion serves as an early boundary cue while semantic prediction error confirms the transition. This pattern is connected to Event Segmentation Theory from cognitive science, in which humans update event models when perceptual predictions fail. The paper's boundary detector operationalizes exactly this principle.

A notable design consequence is that redundancy should be prevented before visual tokens enter the LLM, not merely pruned afterward. The paper argues that KV-cache sparsification methods (ReKV, Look-M, MeDA, StreamMem) act too late: they reduce memory growth but cannot recover information discarded before semantic consolidation, nor do they stop redundant embeddings from being processed in the first place.

Method

The system runs as a single causal forward loop over frames sampled at 2 FPS. For each frame $x_t$, an embedding $f_t$ is extracted and a running event representation $\bar{f}$ is maintained via exponential moving average. The event boundary score integrates three cues:

$$E_t = w_{\text{sem}}(1 - s_t) + w_{\text{mot}}\tilde{m}_t + w_{\text{pred}}c_t$$

where $s_t = \cos(f_t, \bar{f})$ measures semantic drift, $\tilde{m}_t$ is normalized motion magnitude (optical-flow norm or frame-diff energy), and $c_t$ is a causal one-step prediction error computed by a lightweight three-layer MLP $g_\theta$ trained self-supervised to predict $f_t$ from $f_{t-1}$, then frozen. A boundary fires when $f_t$0 exceeds an adaptive threshold modulated by short-term motion variance (a fixed threshold $f_t$1 already works robustly). On a boundary, segment embeddings are aggregated into an event token via temporally weighted pooling centered on the boundary time.

The event memory applies a merge-or-append rule: if a new event token is highly similar ($f_t$2) to the most recent entry, it is merged via EMA-style interpolation; otherwise it occupies a new slot. The event-driven decoder generates text $f_t$3 conditioned on the current event token and retrieved context, remaining silent within boundaries. A simple hysteresis policy coalesces boundaries closer than $f_t$4 and emits keep-alive updates if no boundary occurs within $f_t$5, preventing both bursty updates and excessive silence.

Results

OVOBench-Realtime. Event-VStream scores 28.15 average across six subtasks (OCR, ACR, ATR, STU, FPD, OJR), a +10.4 point absolute gain over its VideoLLM-Online-8B baseline (17.73) under identical 2 FPS input and vision encoder. It trails Flash-VStream-7B (28.37) by only 0.83 points despite Flash-VStream's vision-specialized LLaVA-based backbone. The largest per-task gains are on OCR (8.05 → 24.38) and ATR (12.07 → 27.59); the baseline retains an advantage on FPD (45.54 vs 33.66). The narrow gap to Flash-VStream supports the paper's claim that the event-centric streaming mechanism, not backbone capacity, accounts for most of the improvement.

Long-horizon stability on Ego4D. Using four unedited 102–120 minute egocentric streams evaluated in five sequential segments with GPT-5 as judge, Event-VStream maintains roughly 70% win rate throughout, peaking at 88.3% in the final segment. Baselines fail in characteristic ways: StreamingVLM degrades quickly due to fixed-interval cache refresh breaking temporal continuity; Flash-VStream shows unstable performance from re-encoding near-identical frames; VideoLLM-Online collapses into repetitive output and runs out of memory after roughly 300 seconds.

Latency. Per-token generation latency stays in the 0.05–0.08 s/token range across full 2-hour streams with no observable drift, below the 0.1 s/token real-time threshold. The system sustains approximately 17 FPS throughput on a single RTX 6000 Ada GPU. Flash-VStream attains lower steady-state latency (~0.03–0.04 s/token), so Event-VStream trades some raw speed for long-horizon coherence.

Ablations. Removing any boundary cue degrades caption quality and increases latency. Dropping the semantic term causes the largest collapse (caption win rate falls to 11.8%, latency rises to 0.190 s), because transient motion alone triggers false updates that fragment narration. Removing prediction error yields 46.7% win rate at 0.187 s latency; removing motion yields 37.5% at 0.173 s. The full three-cue model achieves the best quality–efficiency trade-off, confirming cue complementarity.

Limitations and open questions

The paper concedes several constraints explicitly. The GPT-5-as-judge protocol for long-stream evaluation may bias toward certain linguistic styles; bidirectional A/B testing partially mitigates this, but human studies are deferred. The ablation table reports incomplete results for the full model variant, leaving the exact headline numbers of the complete configuration under-specified relative to the ablation rows. The adaptive threshold mechanism is presented as optional, with fixed thresholds claimed sufficient — the marginal benefit of adaptivity is not quantified. Evaluation rests on four selected Ego4D sequences and one benchmark suite, so generalization to other domains (e.g., surveillance, third-person instructional video) remains untested. The method is currently vision-only; audio and speech cues are identified as future extensions. Finally, the merge-or-append memory rule compares only against the most recent entry, which may discard older recurring content; whether this suffices for retrieval over very long horizons is left open.

Conclusion

Event-VStream demonstrates that shifting the unit of streaming video processing from frames to detected semantic events simultaneously reduces redundant computation, preserves long-horizon context, and maintains real-time latency. The combination of a three-cue causal boundary detector, a lightweight event-level memory with merge-or-append updates, and event-triggered decoding with pacing control delivers a +10.4 point gain over its backbone on OVOBench-Realtime and stable ~70% GPT-5 win rate over 2-hour streams, all without retraining the underlying VLM. The evidence indicates that where and when a streaming VLM chooses to update matters as much as how much context it can retain.