ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

Abstract: Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods largely focus on accelerating individual modules, such as visual encoding, token pruning, or KV-cache compression, but provide limited insight into whether the resulting system can sustain real-time streaming performance. We formulate streaming VideoLLM inference as a coordinated pipeline spanning visual preprocessing, visual encoding, token dropping, and LLM prefilling/decoding. Building on this formulation, we propose ViCoStream (Video Coordinated Streaming), a stage-wise coordinated streaming framework that combines chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs. We further provide a systematic study of bottleneck migration, revealing how chunk size, token retention, attention locality, and retrieval scope shape the throughput-accuracy trade-off. Experiments with Qwen2.5-VL-3B/7B-Instruct across multiple streaming benchmarks show that ViCoStream achieves 134 FPS video throughput and less than 50 ms TTFT on a single A100 GPU while maintaining accuracy close to full-history baselines.

• The paper introduces a stage-wise coordinated pipeline that overlaps preprocessing, chunk-wise ViT encoding, token dropping, and bounded LLM attention to address shifting system bottlenecks.
• ViCoStream reaches 134 FPS with 30% token retention and 101 FPS with 50% retention on a single A100, while query-driven retrieval keeps TTFT below 50 ms in compressed settings.
• The method preserves accuracy within roughly 3 points of full-history baselines at moderate compression, but aggressive dropping can harm backward tracing and long-range narrative understanding.

Motivation and problem formulation

Streaming VideoLLM deployment imposes two distinct system objectives: sustaining video-ingestion throughput that keeps pace with incoming frames, and minimizing query-time time-to-first-token (TTFT). The paper argues that prior work—sparse visual encoding, token pruning, KV-cache compression, memory banks—optimizes individual stages in isolation and offers little evidence about whether the resulting system can sustain real-time streaming. A key motivating observation is that token dropping alone does not stabilize latency: even with aggressive dropping, LLM prefill latency continues to grow with the streaming chunk index as visible visual history accumulates under full-history attention.

The paper formalizes streaming inference as a four-stage pipeline—visual preprocessing (VP), ViT encoding, token dropping, and LLM prefilling/decoding—executed over non-overlapping chunks of size $c$. Under a stage-wise resource model where each stage processes one chunk at a time but different stages operate on different chunks concurrently, steady-state throughput is governed by the slowest stage:

$$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$

This formulation reframes continuous streaming as a slowest-stage scheduling problem and makes bottleneck migration explicit: accelerating one stage merely shifts the ceiling elsewhere.

System design

ViCoStream coordinates controls at each stage. Visual preprocessing is assigned to an independent CUDA stream so GPU-side resizing, normalization, and host-to-device transfers overlap with later model stages; profiling shows roughly $8\times$ speedup over traditional CPU-based preprocessing for larger chunks (e.g., 1961 ms → 244 ms at $c=32$). Visual encoding uses chunk-wise ViT invocation to amortize per-frame scheduling overhead. Token dropping operates on two axes: spatial compression (average pooling, max pooling, or entropy-based top-$K$ selection) and chunk-level temporal compression that drops a spatial position only when all pairwise cosine similarities across frames in the chunk exceed a threshold $\tau$, always preserving the first frame of each chunk as an anchor. LLM inference combines vision-side bounded attention—each incoming chunk attends only to the most recent $acn$ chunks—with query-driven retrieval, where only the top-$ur$ chunks by query–chunk cosine similarity are exposed during decoding. An adaptive additive attention mask aligns training-time visibility with these two inference scopes: a staircase-shaped local band for vision tokens and sparse, non-contiguous visibility for query tokens.

Two empirical observations justify the design. First, dropping 78.1% of visual tokens preserves accuracy relative to the full-token baseline under full attention. Second, retaining only the nearest 4K visual tokens improves overall accuracy from 70.76 to 72.96 while discarding 64.5% of tokens, supporting a locality assumption for streaming video context.

Efficiency results

On a single A100 with Qwen2.5-VL backbones and simulated 720p input resized to 448×448, frame-level processing ($c=1$) caps throughput at 26 FPS due to frequent small ViT invocations. Chunk-wise processing raises maximum streaming FPS to 134 under 30% token retention ($c=16$) and 101 under 50% retention. Notably, larger chunks are not monotonically better: at $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$0, VP latency (248 ms) overtakes ViT encoding as the bottleneck, illustrating bottleneck migration from LLM prefilling at small chunks to front-end stages once LLM cost is bounded.

Long-duration stability follows directly from bounded attention: full-history prefill latency rises steadily with stream length, whereas $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$1 or $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$2 yields nearly flat per-chunk latency. On the query side, TTFT is governed by the product of retrieval scope and token retention—at 75–100% retention, TTFT rises sharply once queries attend to 16+ chunks, but with $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$3 or $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$4 at 25% retention, query-side prefill stays below 50 ms regardless of accumulated stream length. This decouples TTFT from streaming FPS and confirms that decoding will not become a new bottleneck under sufficiently aggressive token retention.

Accuracy analysis

The central claim is not an accuracy improvement but accuracy retention under real-time constraints, benchmarked against full-history streaming prefill with no compression on six streaming benchmarks. At moderate temporal dropping ($$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$5, ~48–58% drop rate), most configurations remain within 3 points of baseline across StreamingBench, OvO-Bench, StreamBench, and VStream, with several settings exceeding the baseline—for example, the 3B model reaches 70.68 on StreamingBench versus 70.32 for full history. More aggressive dropping ($$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$6, ~70–73%) exposes task-dependent fragility: OvO-Bench backward tracing and VStream-Movie degrade consistently, indicating sensitivity to fine temporal evidence and long-range narrative context respectively.

Ablations refine this picture. Temporal dropping produces a smooth trade-off, whereas spatial compression is strategy-sensitive: average and max pooling consistently hurt accuracy, entropy-based selection is more robust, and combining both axes yields the strongest compression but accumulates the largest loss. Frame-level temporal dropping is less robust than chunk-level dropping under aggressive thresholds, supporting the anchor-based chunk design. Attention locality is non-monotonic: $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$7 helps, $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$8 brings no further gain or slightly hurts, and query-side retrieval saturates around $$T_{\mathrm{pipe}} = \max\{T_{\mathrm{vp}}, T_{\mathrm{vit}}, T_{\mathrm{drop}}, T_{\mathrm{LLM}}\}, \qquad \mathrm{FPS}_{\max} = \frac{1000 \cdot c}{T_{\mathrm{pipe}}}$$9. On ETBench event-time tasks, the 3B model improves several temporal localization metrics despite bounded context, but the 7B model shows larger drops on ECA, TEM, and TVG—a notable inconsistency suggesting stronger sensitivity to compressed history at scale.

Limitations

The paper concedes several constraints explicitly. The throughput analysis assumes a single-replica-per-stage resource model and may not extend to heterogeneous accelerators or asynchronous multi-replica scheduling. The efficiency gains do not eliminate the underlying accuracy–latency trade-off: memory-heavy backward tracing and movie-level narrative reasoning remain sensitive to compressed or bounded history, and no adaptive policy currently allocates context according to task difficulty or query intent. Additionally, the main accuracy comparisons confound token dropping with bounded attention and retrieval—the controlled intra-chunk ablation shows neither attention scope is uniformly superior—so attribution of specific accuracy deltas to individual mechanisms remains partial.

Conclusion

ViCoStream demonstrates that treating streaming VideoLLM inference as a coordinated multi-stage pipeline, with per-stage bounds on computation and visible context, sustains over 130 FPS ingestion and sub-50 ms TTFT on a single A100 while preserving accuracy close to full-history baselines. Its principal contribution is analytical: identifying bottleneck migration across preprocessing, encoding, dropping, and LLM stages, and showing that LLM-side optimization alone cannot deliver long-duration real-time streaming.