VSAS-BENCH: Real-Time Evaluation of Visual Streaming Assistant Models

Abstract: Streaming vision-LLMs (VLMs) continuously generate responses given an instruction prompt and an online stream of input frames. This is a core mechanism for real-time visual assistants. Existing VLM frameworks predominantly assess models in offline settings. In contrast, the performance of a streaming VLM depends on additional metrics beyond pure video understanding, including proactiveness, which reflects the timeliness of the model's responses, and consistency, which captures the robustness of its responses over time. To address this limitation, we propose VSAS-Bench, a new framework and benchmark for Visual Streaming Assistants. In contrast to prior benchmarks that primarily employ single-turn question answering on video inputs, VSAS-Bench features temporally dense annotations with over 18,000 annotations across diverse input domains and task types. We introduce standardized synchronous and asynchronous evaluation protocols, along with metrics that isolate and measure distinct capabilities of streaming VLMs. Using this framework, we conduct large-scale evaluations of recent video and streaming VLMs, analyzing the accuracy-latency trade-off under key design factors such as memory buffer length, memory access policy, and input resolution, yielding several practical insights. Finally, we show empirically that conventional VLMs can be adapted to streaming settings without additional training, and demonstrate that these adapted models outperform recent streaming VLMs. For example, Qwen3-VL-4B surpasses Dispider, the best streaming VLM on our benchmark, by 3% under the asynchronous protocol. The benchmark and code will be available at https://github.com/apple/ml-vsas-bench.

• The paper introduces VSAS-Bench, a benchmark with 18,410 dense, free-form annotations across 92 videos and an asynchronous producer–consumer protocol that measures how latency, dropped frames, and stale context affect real-time visual assistance.
• The evaluation shows that training-free adaptation of Qwen3-VL-4B reaches 46.8% asynchronous accuracy, outperforming the strongest dedicated streaming baseline by 3 percentage points while preserving broader video-model generalization.
• The paper demonstrates that system design strongly affects performance: self-speculative decoding reduced OCR-task latency from 5.8 to 1.5 seconds while raising accuracy from 21.5% to 55.1% and consistency from 93.0% to 96.2%.

Motivation and positioning

Streaming vision-LLMs (VLMs) must process a continuous frame stream with bounded latency, respond selectively, and remain temporally consistent — capabilities that conventional offline video benchmarks do not measure. Existing streaming-oriented evaluations such as OVO-Bench, StreamingBench, and E.T. Bench rely on synchronous protocols in which frame generation is aligned to the model's processing rate, so slow models incur no penalty. They also reduce questions to multiple-choice or binary formats, which cannot capture the free-form, temporally grounded responses required of real-time visual assistants (2604.07634).

VSAS-Bench addresses both gaps. It provides temporally dense, free-form annotations (over 18,000 annotations at 1 FPS temporal resolution across 92 videos) and introduces an asynchronous evaluation protocol in which camera acquisition is decoupled from inference via a producer–consumer pattern with a bounded camera buffer. Under this protocol, delayed inference causes older frames to be dropped or stale context to be used, so reported accuracy explicitly reflects the trade-off between computational efficiency and temporal fidelity.

Benchmark design

The dataset comprises 92 videos of varying duration: 48 newly recorded videos covering cooking, urban human actions, and UI interaction, plus 44 sourced from STAR, PerceptionTest, and YouCook2. Annotations are generated by GPT-5 (medium reasoning) and verified by human experts; roughly 10% of the 18k annotations required correction, inter-annotator agreement is 96.7%, and GPT–human semantic agreement was validated on a 250-sample subset using all-mpnet-base-v2 embedding similarity. Three task types structure the evaluation:

• Present: queries about events currently occurring in the stream.
• Cumulative: queries requiring recall and reasoning over past events.
• Future: queries requiring prediction of upcoming events from ongoing visual cues.

Unlike OVO-Bench and StreamingBench, prompts specify task details explicitly to reduce ambiguity, and responses are free-form rather than discrete choices. The benchmark's annotation density is substantially higher than comparable streaming benchmarks: median interval between annotations of 1.0 s versus 61.8 s for RTV-Bench, and an average of 11.8 unique events per video versus 2.5 for RTV-Bench.

Evaluation protocol and metrics

The synchronous protocol mirrors prior work: frames arrive in lockstep with model inference, and latency does not affect scores. The asynchronous protocol simulates a camera process writing frames at fixed intervals into a queue-based buffer of configurable size $B$, while a separate VLM process consumes frames when idle. If inference lags, frames accumulate or are discarded once $|\mathcal{B}| > B$, altering the effective temporal context available to the model.

Because streaming VLMs may intentionally pause when their previous response remains valid, missing responses are extrapolated forward from the most recent available response before scoring. Two headline metrics are defined:

• Mean Average Accuracy ($\mathcal{A}$): per-timestep judgments by a GPT-5 LLM judge comparing each response against the ground-truth caption, averaged over timesteps within each task and then over tasks. The judge outputs a binary match decision plus a rubric score in $\{0,1,2,3\}$; judge variance across five independent runs is below 0.6%, and agreement with human experts on a held-out set of 450 samples yields Cohen's $\kappa = 0.91$.
• Mean Average Consistency ($\mathcal{C}$): one minus an LCS-based edit distance between consecutive responses, normalized by subtracting the ground truth's own edit distance to account for inherent annotation variability, then clipped to $[0,1]$ and averaged.

This consistency metric penalizes unnecessary or contradictory output updates — a failure mode specific to streaming assistants that accuracy alone does not capture.

Adapting video VLMs to streaming

Conventional video VLMs assume pre-segmented clips and cannot natively operate under the asynchronous protocol. The authors introduce two external mechanisms: a model memory buffer storing recent frames, and a memory policy selecting the working context of up to $k$ frames per inference step. Three policies are compared: Sliding Window (SW), Uniform sampling over all stored frames (U), and Sliding Window with Uniform tail (SW+U). SW is the default because it is the only feasible policy for long videos; U improves Cumulative tasks by admitting early frames, and SW+U performs best overall for GPT-5 (80.6 vs. 77.0 for SW) while yielding only marginal gains for Qwen3-VL-8B, which the authors note may depend on tuning the tail size.

Main results

Under the synchronous protocol, results follow the expected scaling pattern: larger models win, with GPT-5 reaching 77.1% overall accuracy and Qwen3-VL-32B reaching 59.5%. Under the asynchronous protocol, the ranking changes materially:

Model	Sync. Overall Acc.	Async. Overall Acc.
GPT-5 (high)	76.4	30.3*
Qwen3-VL-4B (adapted)	51.6	46.8
Qwen3-VL-32B (adapted)	59.5	41.0
Dispider (streaming)	46.8	43.4
StreamBridge (streaming)	36.8	21.7

*GPT-5 asynchronous results are grayed out in the paper because network/API latency dominates; its API latency ranges from 42.4 s (Present) to 65.8 s (Future) end-to-end, far exceeding locally deployed models whose latencies are typically 0.3–4.1 s per task.

The central empirical claim is that training-free adaptation of video VLMs outperforms purpose-built streaming VLMs: streaming-adapted Qwen3-VL-4B surpasses Dispider, the strongest streaming baseline, by 3% under the asynchronous protocol, and a 16-point gap separates it from GPT-5. The authors attribute the underperformance of dedicated streaming models (VideoLLM-Online, FlashVStream, Dispider, StreamBridge) to fine-tuning on narrow, task-specific domains, whereas adapted video VLMs retain their base models' generalization. Notably, VideoLLM-Online achieves only 7.9% async accuracy despite being designed for this setting.

Consistency shows the opposite trend synchronously: streaming VLMs outperform video models and GPT-5 because they are trained to produce stable outputs, whereas accurate video models tend toward verbose, rephrased responses. Under the asynchronous protocol, models respond less often, consistency rises across the board, and the gap narrows.

Ablations enabled by the asynchronous protocol

The protocol makes runtime design choices measurable in accuracy terms, which synchronous evaluation cannot do:

• Hardware: moving Qwen3-VL models from A100 to H100 improves async accuracy substantially (e.g., Qwen3-VL-4B: 43.3 → 46.8; Qwen3-VL-32B: 36.8 → 41.0), demonstrating that hardware-aware tuning directly affects measured streaming accuracy.
• Memory buffer size: for Qwen3-VL-8B, accuracy peaks at a buffer of 8 frames (47.0) and declines to 44.3 at 64, exposing a trade-off between temporal context richness and inference latency.
• Input resolution: increasing minimum pixel count from 16,384 to 262,144 raises accuracy from 35.5 to 46.3, but further increases to 409,600 degrade performance to 45.0 as prefill cost grows.
• Camera buffer size: with a realistic bounded camera buffer, reducing capacity to 16 frames drops Qwen3-VL-8B async accuracy by nearly 15 points relative to an effectively unbounded buffer, quantifying how constrained buffering alters effective temporal context.

The authors additionally propose a self-speculative decoding variant for streaming: the previous timestep's tokens serve as the draft and are verified against current visual input, with new tokens generated only on verification failure. On OCR cumulative-task videos, this reduces mean latency from 5.8 s to 1.5 s and improves accuracy from 21.5 to 55.1 (a 33.6% gain) and consistency from 93.0 to 96.2. They also recommend KV-cache reuse, visual token reuse, and concise-response prompt engineering as complementary optimizations.

Limitations and open questions

Several caveats bear directly on the results. First, the benchmark contains only 92 source videos; although dense annotations bring total annotations to 18,410, the authors acknowledge that not every video supports all task types (e.g., Future tasks are excluded from OCR scenarios where prediction is ill-defined). Inverse-category and inverse-task reweighting experiments show rankings are robust to this imbalance, but absolute numbers shift by up to ~2 points. Second, the LLM-judge methodology depends on GPT-5 both for annotation generation and evaluation; while human verification and high judge–human agreement ($\kappa = 0.91$) mitigate circularity concerns, judge bias toward GPT-family phrasing cannot be fully excluded. Third, GPT-5's asynchronous results conflate network latency with model capability, and the authors provide no mechanism to separate these factors. Fourth, the SW+U policy's benefit "may depend on tuning the tail size," leaving the optimal memory policy an open question. Finally, the finding that training-free adaptation beats specialized streaming VLMs is established on this benchmark's task distribution; whether it holds for other streaming domains is not tested here.

Conclusion

VSAS-Bench contributes a densely annotated, free-form streaming benchmark together with an asynchronous evaluation protocol and time-aware metrics that make latency–accuracy trade-offs measurable. Its principal empirical findings are that small, fast video VLMs adapted via a memory buffer outperform both larger offline models and dedicated streaming VLMs under realistic streaming conditions, and that runtime configuration choices (hardware, buffer sizes, resolution) produce accuracy differences large enough to be captured by the benchmark itself. The framework and code are released to support latency-aware development of interactive multimodal systems.