Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding

Abstract: Online Video LLMs (Video-LLMs) have advanced toward seamless human-AI interaction through frame-by-frame processing and proactive responding. However, a critical challenge remains in streaming scenarios: existing models typically pause video perception while generating responses, breaking real-time video-language synchrony and causing stutters. To address this, we introduce a novel paradigm for online video understanding: Streaming Video-Language Synchrony (SVLS), and present LyraV, a live streaming assistant built upon a hierarchical control framework with two core innovations. First, the Frame-Driven Transition Controller (FDTC), a training-free verification-based finite-state machine, makes high-level semantic decisions on when to continue speaking, start a new response, or stay silent. Second, the Streaming Token Pacer (SToP), a plug-and-play lightweight predictive module, dynamically adapts the language generation rate to match the pace of the visual content. Concretely, LyraV performs \emph{per-frame incremental, sub-budget decoding}: within each frame interval it emits only a small chunk of tokens that fits the real-time budget, so perception is never blocked for a full sentence. Together, these components enable LyraV to seamlessly interleave incoming video frames with generated word tokens, achieving a fine-grained synchrony. Extensive experiments conducted on five online and three offline benchmarks demonstrate that LyraV preserves the backbone's general understanding ability while substantially improving streaming synchrony and narrative fluency, delivering a 98.29\% synchrony with video playback and a real-time processing speed of 3.89 FPS. Interestingly, we observe an empirical capability in LyraV: dynamic reasoning over streaming tokens, enabling continuous interpretation and "thinking" alongside visual input.

• The paper introduces LyraV, a framework that interleaves video frames and word tokens in real-time, achieving near-perfect synchrony (98.29% SR).
• It employs a hierarchical control architecture with FDTC and SToP modules to dynamically regulate semantic transitions and token pacing without interrupting video input.
• Experimental results show that LyraV outperforms baseline models in semantic coherence and response latency across multiple streaming video-language tasks.

Streaming Video-Language Synchrony for Online Video Understanding: An Essay on LyraV

Introduction

The proliferation of Video LLMs (Video-LLMs) has propelled advances in interactive video understanding, but real-world deployment in streaming scenarios remains bottlenecked by the inability to maintain tight synchrony between video perception and natural language generation. "Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding" (2606.06991) directly addresses this gap, presenting the LyraV control framework and formalizing the paradigm of Streaming Video-Language Synchrony (SVLS). LyraV fundamentally restructures online video comprehension, departing from the convention of halting visual perception during textual decoding and instituting an architecture where video frames and generated word tokens are interleaved at fine temporal granularity, subject to strict real-time constraints.

Figure 1: Illustration of Streaming Video-Language Synchrony. LyraV enables seamless interleaving of video frames with generated word tokens, in contrast to offline and conventional online models.

Related Work and Problem Formulation

Offline Video-LLMs (e.g., VideoLLaMA 2 [cheng2024videollama], Video-LLaVA [lin2023video]) have demonstrated strong video comprehension, but their interaction protocols assume complete video observation before response generation, precluding operation in environments with perpetual, never-pausing input streams (e.g., smart glasses, robotics). The emergence of online Video-LLMs, e.g., VideoLLM-Online [chen2024videollm], StreamMind [ding2025streammind], MMDuet [wang2024videollm], and LiveStar [yang2025livestar], has yielded incremental improvements, introducing response gating, EOS token prediction, and verification-based control. Nonetheless, these frameworks typically pause video intake to decode entire sentences, leading to perceptual stutter and poor human-level synchrony.

LyraV's SVLS reformulation departs from this design space. Rather than full-sentence emission at discrete trigger points, SVLS obligates the model to process each frame as it arrives and emit a small, potentially adaptive chunk of token(s), synchronizing perception and generation at the frame level. This paradigm directly operationalizes a "speak-while-watching" mode of online video understanding.

LyraV Framework

Hierarchical Control Architecture

LyraV's core is a lightweight, plug-and-play controller wrapped around an existing Video-LLM, introducing two modules: the Frame-Driven Transition Controller (FDTC) and the Streaming Token Pacer (SToP).

• FDTC is a training-free, verification-based finite state machine that governs when to continue an utterance, when to trigger a new response, and when to remain silent, based on instantaneous semantic alignment between the evolving video stream and the current narrative buffer.
• SToP is a content-aware token pacing module, implemented as a lightweight Transformer encoder, which predicts the optimal number of tokens to decode per frame in accordance with the density of visual events and constrained by a hard latency cutoff.

  Figure 2: Architecture of the proposed FDTC framework. The FDTC regulates state transitions based on evolving visual context.

Distinctively, FDTC extends prior verification-based methods—such as LiveStar's single-pass binary gating—by supporting prefix-level continuation, enabling LyraV to maintain ongoing utterances across temporally correlated events and only trigger new responses when semantic drift is detected via a perplexity threshold.

Figure 3: (a) FDTC’s three-state machine; (b) SToP’s architecture for token rate regulation.

Streaming Inference Pipeline

Each input frame updates the multimodal context. FDTC evaluates the utterance's perplexity conditioned on current and prior frames to dictate transitions among Silent, Continuing, and Triggered states. When in a generation state, SToP predicts the frame's token budget, which is then clipped by a real-time latency upper bound. Generation proceeds with this token budget, and the cycle iterates, ensuring that perception is never paused for the duration of a full sentence.

Experimental Results

A comprehensive evaluation suite spans eight video-language understanding tasks, with emphasis on real-time, streaming protocols. Metrics capture not only static understanding (e.g., semantic accuracy, narrative fluency) but, crucially, fine-grained synchrony (Sync Rate, SR) and response latency (RL), forcing models to balance perceptual awareness against narrative coherence without exceeding latency budgets.

Online Captioning and QA

On OmniStar-RNG, LyraV achieves a synchrony rate (SR) of 98.29% and a real-time FPS of 3.89, preserving semantic scores (SS) and narrative fluency (NF) on par with or exceeding its backbone LiveStar and significantly outpacing prior baselines such as Dispider and MMDuet. The gains are especially marked under strict real-time constraints with forced truncation, where all models have 100% SR but LyraV attains higher semantic and fluency scores, indicating superior scheduling and emission policy.

(Table excerpted for context, not displayed per instructions.)

On OVO-Bench [li2025ovo] and OVBench [huang2025online], LyraV matches or surpasses other open-source online models (average scores: 50.97 and 46.8), approaching the static accuracy of leading offline and even some proprietary systems (e.g., Gemini 1.5 Pro). Its robustness across visual perception, backward tracing, and real-time response is strong, despite operating strictly under streaming constraints.

Synchrony Analysis

The contribution of FDTC and SToP is directly validated via ablation. Removal of FDTC reverts LyraV to one-pass emission, causing a precipitous drop in coherence, semantic continuity, and response timing adherence. Fixed-rate token emission in lieu of SToP slightly reduces throughput efficiency but does not severely degrade content, consistent with SToP’s role as a pacing—not quality—regulator.

Table analysis reveals that baseline models, when forced to truncate all responses to the frame interval, attain perfect synchrony but at the cost of fragmented, incomplete sentences, confirming the necessity of LyraV's more expressive and context-sensitive control.

Qualitative Analysis

Case studies demonstrate incremental "reasoning" and refinement: LyraV partially decodes narratives as evidence accumulates, adapting and revising earlier hypotheses as new frames arrive, without committing prematurely.

Figure 4: Case study showing incremental updates and dynamic reasoning as LyraV generates only two tokens per frame.

Figure 5: Qualitative visualization of LyraV’s real-time narration process, displaying seamless frame-token interleaving.

Theoretical and Practical Implications

LyraV formalizes the contemporaneous processing of video and language—a theoretical advancement over prior architectures in which perception is functionally blocked during decoding. This formulation directly reflects human interaction patterns, enabling AI agents to engage in truly synchronous dialogue, narration, or virtual assistant tasks without stutter or temporal misalignment. The SVLS protocol raises the evaluation standard for online models, necessitating real-time joint optimization of perception and generation, and shifting emphasis away from static benchmarks toward dynamic, process-level evaluation.

Practically, LyraV's hierarchical framework is orthogonal to the underlying Video-LLM and can be deployed as a low-overhead wrapper, providing immediate improvements in streaming settings without retraining or model modification.

Figure 6: LyraV web demo interface for real-time, user-interactive streaming narrative generation.

Future Directions

Despite its strengths, the FDTC verification via perplexity remains grounded in a fixed scaling threshold; more robust or adaptive arbitration strategies may further enhance stability and narrative quality under extreme event densities or high frame rates. SToP currently uses human speaking rate as a weak supervision signal; incorporating multi-narrator references, richer event density annotations, or feedback-driven reinforcement learning-based adaptation could yield further improvements. Integration of SVLS-style streaming control with advanced memory and retrieval modules offers new directions for infinite context or long-horizon reasoning.

Conclusion

"Don't Pause: Streaming Video-Language Synchrony for Online Video Understanding" (2606.06991) redefines the streaming video-language modeling landscape, establishing a theoretical and practical protocol for synchronous, adaptive, and low-latency interaction. LyraV's SVLS-centric architecture is empirically and theoretically validated as a superior strategy for frame-accurate, stutter-free online video understanding, with broad implications for future multimodal AI systems in interactive, continuous environments.