OmniStarPro: Proactive Streaming Video Benchmark

• OmniStarPro is a benchmark and dataset for online and streaming video understanding that supports proactive narration, grounding, dense QA, and long-horizon memory tasks.
• It features two partitions—OmniStarPro-Live for short-horizon tasks and OmniStarPro-Long for long-term memory evaluation—covering 15 real-world scenarios with expert-verified annotations.
• The benchmark emphasizes causal processing, autonomous response timing, and memory stress challenges, pushing Video-LLMs to recall evicted history in real time.

OmniStarPro is a benchmark and dataset for online and streaming video understanding with Video-LLMs, introduced together with LiveStarPro to evaluate proactive behavior under causal access, autonomous response timing, and long-horizon memory requirements (Yang et al., 16 Jun 2026). It is designed for models that must process continuous video streams without future frames, decide when to respond versus remain silent in real time, and preserve information over minute- to hour-scale streams. In practical terms, OmniStarPro combines a short-horizon partition, OmniStarPro-Live, with a long-form partition, OmniStarPro-Long; together they span 15 diverse real-world scenarios, support eight tasks, provide expert-verified second-level temporal annotations, and enforce a unified online evaluation protocol in which models operate under streaming constraints rather than fixed offline decoding schedules (Yang et al., 16 Jun 2026).

1. Definition, scope, and relation to prior streaming benchmarks

OmniStarPro was proposed to support realistic evaluation of proactive streaming assistants. Its primary purpose is to assess systems that must narrate in real time, ground events as they occur, and answer questions that depend on either recent context or temporally distant events. The benchmark is tightly coupled to the LiveStarPro architecture, where it functions as the main training and evaluation substrate for Streaming Verification Decoding (SVeD), Streaming Causal Attention Masks (SCAM), and Tree-Structured Hierarchical Memory (TSHM), but it is presented as a general-purpose benchmark for proactive streaming video understanding (Yang et al., 16 Jun 2026).

Conceptually, the benchmark targets three requirements that are often separated in earlier work: causal stream processing, autonomous response scheduling, and long-term memory. Existing streaming benchmarks cited in the same work—SVBench, OVO-Bench, and StreamBench—are described as focusing mostly on streaming QA on limited domains, often Ego4D first-person videos. OmniStarPro extends that setting by covering 15 real-world scenarios rather than primarily egocentric data, by including narration, grounding, dense QA, and long-term recall tasks, and by introducing hour-scale streams with explicit memory stress (Yang et al., 16 Jun 2026).

A common misconception is to treat streaming evaluation as equivalent to running an offline model left-to-right. OmniStarPro is stricter than that. Its online protocol requires causal access only, prohibits lookahead, and requires the model to choose when to speak. This makes the benchmark diagnostic არა only of semantic prediction quality but also of response gating and memory management. A plausible implication is that OmniStarPro shifts evaluation from clip-level recognition toward temporally situated decision-making, where silence, latency, redundancy, and recall beyond the active context window become first-class variables.

2. Dataset composition, scenarios, and annotation pipeline

OmniStarPro has two partitions. OmniStarPro-Live is the short-horizon component and contains minute-scale videos with five streaming tasks. OmniStarPro-Long is the long-form component and contains streams from 10 minutes to beyond 1 hour with three memory-centric tasks. Across both partitions, the benchmark contains 22,245 videos or streams (Yang et al., 16 Jun 2026).

Partition	Scale and duration	Primary role
OmniStarPro-Live	20,137 validated streams; source videos $\leq 6$ minutes	Short-horizon online tasks
OmniStarPro-Long	2,108 long-form streams; 10 minutes to $>1$ hour	Long-term memory tasks

The short-horizon partition is derived from YouTube short videos of at most 6 minutes. After filtering and annotation, it contains 20,137 validated streams with real-time narrations, split into 19,137 training streams and 1,000 evaluation streams. Many videos are 51–150 seconds long, and 45.54% exceed 100 seconds. Annotation density is high: on average there are 14.5 QA pairs and 8.2 caption segments per video (Yang et al., 16 Jun 2026).

The long-form partition is built from YouTube videos longer than 10 minutes drawn from the same scenarios. Starting from 9,260 candidates, the curation pipeline yields 2,108 high-quality long-form streams. Their duration distribution is 1,396 streams between 10 and 30 minutes, 331 streams between 30 and 60 minutes, and 381 streams beyond 60 minutes, with an average duration of 34.7 minutes. For memory evaluation, the partition contains 12,704 queries for LMR, CDQ, and TBR, each paired with supporting timestamp information and a computed memory span (Yang et al., 16 Jun 2026).

Scenario coverage is a central feature. The benchmark spans 15 real-world scenarios and 46 fine-grained categories. Explicitly named examples include Travel & Events, Sports, News & Politics, and Gaming. These were chosen to test heterogeneous scene changes, dynamic environments, fast motion, entity tracking, narrative coherence, persistent HUD-like overlays, and rapid commentary regimes. The stated rationale is to move beyond egocentric-only evaluation and to better reflect live streaming commentary, shopping, surveillance, driving, and meetings-like content (Yang et al., 16 Jun 2026).

Data collection for OmniStarPro-Live began from 120,598 short YouTube videos. Scenario targeting used YouTube category and tag metadata; geographic balance was enforced through stratified sampling by country, downsampling over-represented regions, and capping selection at no more than 20 videos per channel. A three-stage multimodal quality filter then retained only videos with at least 5,000 social interactions, applied Whisper-based ASR and a lexical-density threshold of at most 2 words per 10 seconds, and imposed visual-quality constraints using FFmpeg, optical-flow analysis, and an aesthetic scoring model based on the Open-Sora scoring tool. This reduced the candidate pool to 21,544 videos before manual annotation (Yang et al., 16 Jun 2026).

Temporal annotation operates at 1-second granularity. Qwen2-VL produces an initial caption for each second; cosine similarity between consecutive per-second captions is then computed, and adaptive thresholding with $\theta = 0.9$ and a LIFO stack merges adjacent seconds into coherent semantic segments. For each segment, Qwen2-VL generates a merged narration using fused captions and previous-segment narrations for context. Thirty domain experts subsequently refine and validate the output by checking AI-generated narration errors, missing temporal annotations, event-sequence errors, second-level timestamps, narration segmentation, violent content, and excessive dialogue. After multiple rounds and inter-annotator agreement checking, 20,137 high-quality video–text pairs remain, corresponding to a retention rate of 87.8% (Yang et al., 16 Jun 2026).

The long-form curation process applies the same filtering and expert verification to long videos. Per-second dense captions are generated, consolidated into coherent event segments using StreamingCoT’s dynamic fusion, and then used by experts to create memory-task queries with evidence timestamps or timestamp pairs and associated memory spans. This suggests that OmniStarPro-Long is not simply a longer version of the short-horizon dataset; it is explicitly instrumented to support controlled studies of retrieval beyond an active context window.

3. Task taxonomy and behavioral targets

OmniStarPro defines eight tasks divided into short-horizon online tasks and long-horizon memory tasks (Yang et al., 16 Jun 2026).

Family	Tasks	Target capability
OmniStarPro-Live	RNG, OTG, FDQ, COQ, MIQ	Proactive online narration, grounding, and QA
OmniStarPro-Long	LMR, CDQ, TBR	Long-term recall and backward retrieval

The five OmniStarPro-Live tasks are evaluated under an online streaming protocol in which frames arrive sequentially and models respond causally. Real-time Narration Generation (RNG) asks the model to function as a live commentator. The input is a continuous frame stream, optionally with an initial instruction such as “Describe events as they happen.” The output is a sequence of free-form text segments with timestamps chosen by the model. RNG penalizes latency, redundancy, and missed events, and matches model responses to ground-truth semantic clips with start and end times (Yang et al., 16 Jun 2026).

Online Temporal Grounding (OTG) evaluates event localization under causal access. The model receives streaming frames and event queries, and must output start and end timestamps for the event as soon as evidence appears, without future frames. Frame-level Dense QA (FDQ) focuses on fine-grained perception at dense temporal sampling by asking short questions at many time points about the current frame. Contextual Online QA (COQ) tests short-term context and causal reasoning by asking questions whose answers depend on recent history rather than only the current frame. Multi-turn Interactive QA (MIQ) extends this to sequences of user turns, requiring responses that remain consistent with prior answers and grounded in the ongoing stream (Yang et al., 16 Jun 2026).

The three OmniStarPro-Long tasks explicitly target memory beyond the active context window. Long-range Memory Recall (LMR) asks for details from much earlier in the stream; its queries have an average memory span of 18.6 minutes, a maximum of 71.3 minutes, and 73.4% of them exceed the model’s active context. Cross-event Difference Query (CDQ) requires retrieving two distant events and comparing them. Temporal Backtracking (TBR) asks for the most recent past occurrence of an event that has already left the active window, typically returning a timestamp or a tied attribute (Yang et al., 16 Jun 2026).

The task design makes a distinction between online perception and online agency. In RNG and OTG, timing is intrinsic to the problem definition, because the model chooses when to emit outputs. In COQ and MIQ, question times are externally triggered, so timing is less central than contextual fidelity. In LMR, CDQ, and TBR, the central variable is whether evicted information can be recalled or reconstructed after long delays. This suggests that OmniStarPro serves not only as a benchmark suite but also as a decomposition of streaming competence into proactive generation, causal grounding, short-context reasoning, dialog continuity, and explicit long-term retrieval.

4. Metrics and online evaluation protocol

OmniStarPro’s evaluation framework combines timing-sensitive metrics for streaming outputs with recall metrics for long-horizon memory (Yang et al., 16 Jun 2026). For tasks such as RNG, ground-truth semantic clips are denoted

$G = \{g_1, \dots, g_N\},$

where each $g_i$ has an interval $[t_{\text{start},i}, t_{\text{end},i}]$ and a caption. Model responses are

$R = \{r_1, \dots, r_M\},$

each associated with a response timestamp $t_{\text{resp},j}$. For each clip,

$$M_i = \{ r_j \in R \mid t_{\text{resp},j} \in [t_{\text{start},i}, t_{\text{end},i}] \}.$$

Timing Difference (TimDiff) measures lateness and redundancy:

$$\begin{aligned} \text{TimDiff} = \frac{1}{N} \sum_{i=1}^N \Big( & \mathbb{I}[|M_i| = 0] \cdot (t_{\text{end},i} - t_{\text{start},i}) \ & + \mathbb{I}[|M_i| > 0] \cdot \sum_{r_j \in M_i} (t_{\text{resp},j} - t_{\text{start},i}) \Big). \end{aligned}$$

If no response occurs within a clip, the penalty is the full clip duration; if responses are present, the metric sums delays relative to clip onset. Lower TimDiff indicates better temporal alignment (Yang et al., 16 Jun 2026).

Timing Redundancy (TimRedun) measures deviation from the ideal of exactly one response per clip:

$>1$0

Timing Coverage (TimCover) measures the proportion of clips receiving at least one response:

$>1$1

Taken together, these three metrics expose the trade-off between speaking too rarely, too often, or at poorly aligned times (Yang et al., 16 Jun 2026).

Semantic Correctness (SemCor) uses GPT-4o as a judge. For each clip, the earliest response in $>1$2 is selected when available, otherwise the response is treated as empty. GPT-4o scores Semantic Accuracy, Language Quality, and Information Completeness on 0–10 scales, and SemCor is the average across these dimensions and across clips. Summarization Fluency (SumFluen) instead evaluates the concatenated narrative generated over the full stream against the concatenated ground-truth narration, averaging GPT-4o scores over Writing Logicality, Language Fluency, Writing Conciseness, Semantic Consistency, and Narrative Completeness (Yang et al., 16 Jun 2026).

For OmniStarPro-Long, evaluation is organized by memory span buckets: short for spans under 10 minutes, medium for 10–30 minutes, and long for spans above 30 minutes. Recall accuracy within a bucket $>1$3 is defined as

$>1$4

where correctness is judged by GPT-4o against the ground-truth attribute or timestamp. This design directly links performance to temporal distance from the supporting evidence (Yang et al., 16 Jun 2026).

The benchmark also specifies offline metrics, including PPL, TokAcc, TimeDiff, Fluency, SemCor, and SumFluen, but its main protocol is online. In the online setting, models process video frame-by-frame or at fixed FPS, use causal attention without future access, and choose when to emit responses. The benchmark rules do not forbid external memory. Instead, they evaluate whether a system can recall evicted content by whatever mechanism it employs. In the LiveStarPro experiments, this permits direct comparison of streaming key-value caching and TSHM against baselines with no such mechanism (Yang et al., 16 Jun 2026).

5. Long-horizon memory stress and proactive streaming behavior

OmniStarPro-Long is constructed to make long-term memory a primary bottleneck. The benchmark contains 2,108 streams, many beyond 30 or 60 minutes, with an average duration of 34.7 minutes and a maximum exceeding 1 hour. Long-form queries have an average memory span of 18.6 minutes and a maximum of 71.3 minutes, while 73.4% of queries require evidence outside the model’s active context window (Yang et al., 16 Jun 2026). In the base LiveStarPro configuration, that window is 8K tokens.

The design intent is therefore not merely to evaluate performance degradation with longer videos but to force retrieval from evicted history. LMR, CDQ, and TBR isolate different retrieval operations: recalling a single past detail, comparing two distant events, and searching backward for a past occurrence. Because evidence often lies beyond the active window, simple sliding windows or FIFO-style memory mechanisms are reported to degrade sharply, whereas TSHM degrades more gradually (Yang et al., 16 Jun 2026).

The benchmark is equally explicit about proactivity. In RNG and related online tasks, models must autonomously choose when to speak. Ground truth consists of semantic clips with start and end times and associated captions, while predictions are free-form responses at arbitrary timestamps. TimDiff, TimRedun, and TimCover jointly reward speaking at appropriate times and penalize answering too early, too late, too frequently, or not at all (Yang et al., 16 Jun 2026).

This setting exposes the utility of the model-side mechanisms introduced alongside the benchmark. SVeD implements a verify-then-generate policy based on perplexity change:

$>1$5

where $>1$6 is a sensitivity threshold. SCAM is a training-time masking strategy that prevents future-caption leakage within semantic clips so that the per-frame probability distribution remains meaningful for SVeD’s gating. OmniStarPro’s online evaluation is the environment in which these mechanisms are assessed against EOS-based silence approaches and fixed-schedule offline baselines (Yang et al., 16 Jun 2026).

A plausible implication is that OmniStarPro reframes response timing as an inference problem over uncertainty calibration rather than as a simple end-of-sequence prediction problem. The benchmark’s annotation structure—semantic clips with aligned timestamps for narration, plus evidence timestamps and derived memory spans for recall tasks—supports that reframing by making both temporal decision quality and retrieval depth directly measurable.

6. Empirical findings, comparative position, and limitations

The benchmark is presented as more difficult and more discriminative than prior streaming datasets because of its longer horizon, broader scenario diversity, larger task set, and stricter online protocol (Yang et al., 16 Jun 2026). Earlier benchmarks are characterized as often using short clips, typically no more than 2–3 minutes, emphasizing Video QA, and sometimes operating in semi-online settings with fixed decoding times. OmniStarPro differs by including hour-scale streams, narration and grounding tasks, and fully online causal evaluation in which models autonomously decide when to respond (Yang et al., 16 Jun 2026).

On the RNG task, the reported online results show VideoLLM-online with SemCor 1.68 and TimDiff 2.67, VideoLLM-MoD with SemCor 1.66 and TimDiff 2.54, MMDuet with SemCor 1.93 and TimDiff 2.32, and LiveStarPro with SemCor 3.27, TimDiff 1.89, TimRedun 1.01, TimCover 0.78, and 3.96 FPS on a 5-minute video. The paper states that LiveStarPro achieves a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error over the best existing online Video-LLMs, while a streaming key-value cache yields a 1.58x inference speedup over the same model without caching (Yang et al., 16 Jun 2026).

On other short-horizon tasks, LiveStarPro is also reported as best or competitive. For FDQ, MMDuet obtains 4.78 / 2.65 on SemCor / TimDiff, LiveStar 6.44 / 1.80, and LiveStarPro 6.61 / 1.77. For COQ, MMDuet reaches 5.71 SemCor and LiveStarPro 5.97. For MIQ, MMDuet reaches 5.62 and LiveStarPro 5.81. These results are used to argue that the benchmark is sensitive both to proactive timing quality and to semantic accuracy (Yang et al., 16 Jun 2026).

The long-form results foreground memory structure. On LMR, reported recall for LiveStarPro is 63.4% for short spans, 49.7% for medium spans, and 37.2% for long spans, compared with 41.2%, 18.6%, and 6.4% for VideoLLM-online; 47.8%, 24.5%, and 9.1% for MMDuet; and 59.5%, 33.0%, and 21.1% for LiveStar. Similar patterns are reported for CDQ and TBR. In retrieval-structure ablations on LMR, a sliding window yields 41.2% short-span recall and 6.4% long-span recall, a flat k-NN memory bank yields 58.7% and 21.3%, and TSHM’s Recursive Event Tree yields 63.4% and 37.2%. Latency is reported as 38.6 ms for flat k-NN and 12.4 ms for the TSHM tree, consistent with sublinear retrieval (Yang et al., 16 Jun 2026).

Memory-strategy ablations on OmniStarPro-RNG compare FIFO, uniform dropout, Peak-End compression, and TSHM; TSHM yields the best SemCor at 3.27 and the lowest TimDiff at 1.89 while maintaining high FPS. Sensitivity analysis for the SVeD threshold $>1$7 shows trade-offs among TimDiff, TimRedun, and TimCover, with best performance reported in a narrow range around $>1$8–$>1$9 (Yang et al., 16 Jun 2026). This suggests that OmniStarPro functions not only as a benchmark leaderboard substrate but also as a controlled environment for analyzing gating thresholds, compression policies, and retrieval topologies.

The limitations noted or implied in the source are substantive. Despite its 15 scenarios, OmniStarPro remains YouTube-centric and may underrepresent specialized domains such as medical or industrial video, as well as non-public or low-engagement events. Because audio is filtered through a low lexical-density criterion and the evaluation primarily targets visual understanding, audio–visual reasoning is not fully covered. Annotation is expensive and partly subjective, and GPT-4o is used as an LLM-as-judge for SemCor, SumFluen, and long-form recall, which may introduce stylistic bias even if it is applied consistently across models. The benchmark is architecture-agnostic in principle, but some of the associated modeling strategies, especially perplexity-based SVeD, depend on calibrated language-model probabilities. It also does not cover action-level control for robotics or strong multimodal fusion with audio and sensors (Yang et al., 16 Jun 2026).

Taken together, OmniStarPro occupies a specific position in the evaluation landscape: it is a large-scale benchmark for proactive streaming video understanding in which temporal alignment, autonomous response timing, and memory beyond the active context window are jointly operationalized. Its empirical role is twofold. First, it provides a demanding testbed for online Video-LLMs. Second, it supplies the annotation structure and metric design needed to separate errors of perception, timing, redundancy, and recall under long-horizon causal streaming conditions (Yang et al., 16 Jun 2026).