Streaming Verification Decoding (SVeD)

• Streaming Verification Decoding (SVeD) is an inference framework for online video-language modeling that verifies the continuity of captions via single-pass perplexity evaluation.
• It replaces explicit silence-token prediction with a verification process that measures semantic drift, reducing over-speech and computational latency.
• SVeD integrates with a streaming key-value cache and hierarchical memory, enhancing both real-time efficiency and long-term narrative coherence.

Streaming Verification Decoding (SVeD) is an inference framework for online video-language modeling introduced as a core component of LiveStarPro, a live streaming assistant for proactive video understanding over long-horizon streams (Yang et al., 16 Jun 2026). SVeD addresses a central difficulty in streaming Video-LLMs: the system must continuously ingest frames, determine autonomously when to emit a response rather than remain silent, and preserve low latency under causality constraints. In LiveStarPro, SVeD replaces explicit silence-token prediction with a verification procedure based on single-pass perplexity evaluation of the most recently emitted caption under updated multimodal context. The method thereby reframes silence as a derived verification state rather than a generation target, with the stated aim of avoiding imbalance, temporal inconsistency, objective misalignment, and vocabulary degradation associated with EOS-driven online decoding (Yang et al., 16 Jun 2026).

1. Problem setting and motivation

SVeD is designed for continuous video streams in which a model must decide, at each newly arrived frame, whether the current visual state still supports the last emitted caption or whether sufficient semantic change has occurred to justify generating a new one (Yang et al., 16 Jun 2026). In this setting, prior online systems have typically relied on explicit EOS or silence tokens to represent non-response intervals. LiveStarPro identifies this paradigm as structurally flawed for continuous video streams (Yang et al., 16 Jun 2026).

The first stated issue is severe data imbalance. Most frames correspond to silence, while only a small subset warrant narration. The paper gives a concrete example: for a 90-second stream at 2 FPS, there are 180 frames, and if 6 events deserve narration, the ratio of response-to-silence is approximately $1:29$ (Yang et al., 16 Jun 2026). This imbalance is said to skew training and push the model to over-predict EOS. The second issue is temporal inconsistency: near-identical adjacent frames may have contradictory targets, with one requiring detailed narration and the next requiring EOS, which hinders convergence. The third issue is objective misalignment: pre-training aligns rich visual features with meaningful text, and mapping those features to a null EOS token conflicts with this cross-modal objective. The fourth issue is vocabulary degradation, because treating EOS as a frequent vocabulary entry contaminates the semantic space and distorts the natural distribution over meaningful tokens (Yang et al., 16 Jun 2026).

The same critique is extended to voice-activity-like gating in purely visual streams without speech. The paper states that there is no principled silence signal to learn in such settings, so the model learns non-semantic pauses rather than semantic responsiveness (Yang et al., 16 Jun 2026). Within this framing, SVeD is positioned as a mechanism that does not learn silence as a lexical output. Instead, it determines silence indirectly by checking whether the previously emitted caption remains semantically valid under newly arrived evidence.

2. Formal mechanism and decision criterion

SVeD operates on four central objects: the newly arrived frame $[Frm_{t_j}]$, the accumulated multimodal context $[Ctx_{< t_j}]$, the most recent emitted caption $[Dec]$, and the model’s conditional probability distribution $P(\cdot \mid \cdot)$ (Yang et al., 16 Jun 2026). The key signal is the perplexity of $[Dec]$ when reevaluated under the updated context.

For a tokenized caption $y_1, \ldots, y_N$, the per-token cross-entropy is defined as

$$CE = -\frac{1}{N}\sum_{i=1}^{N} \log p(y_i \,|\, [Ctx], y_{<i})$$

and the corresponding perplexity is

$$\mathrm{PPL} = \exp\left( -\frac{1}{N}\sum_{i=1}^{N} \log p(y_i \,|\, [Ctx], y_{<i}) \right).$$

In LiveStarPro, the verification score at time $t_j$ is

$[Frm_{t_j}]$0

with $[Frm_{t_j}]$1 denoting a sensitivity or scaling factor whose default value is $[Frm_{t_j}]$2 (Yang et al., 16 Jun 2026).

The decision rule compares the current verification perplexity with a scaled reference established at the last emission time $[Frm_{t_j}]$3. If

$[Frm_{t_j}]$4

the model concludes that the video has diverged sufficiently from the caption, activates decoding, and generates a new caption $[Frm_{t_j}]$5. Otherwise, generation is suppressed, and a logical “Swap” moves $[Frm_{t_j}]$6 to the end of the context buffer so that the active context reflects the continued validity of the caption at the current timestamp (Yang et al., 16 Jun 2026).

This verify-then-generate design is the defining property of SVeD. Silence is not emitted as a token. Instead, silence is the state in which the prior caption still explains the present context with sufficiently low degradation in likelihood. A plausible implication is that the gate is tied directly to the model’s semantic confidence in its previous statement rather than to a separately learned silence classifier.

3. Streaming algorithm and computational profile

The decoding procedure is causal and frame-synchronous. At each new frame, the model appends $[Frm_{t_j}]$7 to $[Frm_{t_j}]$8. If a caption already exists, it computes $[Frm_{t_j}]$9 via a single forward pass over the existing caption conditioned on $[Ctx_{< t_j}]$0. If the verification condition is exceeded, autoregressive decoding produces $[Ctx_{< t_j}]$1, which is then appended to context and used to update the reference time $[Ctx_{< t_j}]$2. If the condition is not exceeded, generation is suppressed and the Swap operation moves $[Ctx_{< t_j}]$3 to the end of $[Ctx_{< t_j}]$4. If no caption exists yet, the system performs an initial decode, appends the resulting caption, and sets $[Ctx_{< t_j}]$5 (Yang et al., 16 Jun 2026).

The paper emphasizes that verification occurs once per incoming frame and that the cost of this verification is a single forward pass to score the already emitted caption (Yang et al., 16 Jun 2026). This differs from token-by-token autoregressive generation, which incurs $[Ctx_{< t_j}]$6 latency per frame for caption length $[Ctx_{< t_j}]$7. SVeD therefore avoids full decoding at every step and maintains streaming causality without backtracking or multi-pass comparison. The reported experimental regime indicates that this matters for live streams running at $[Ctx_{< t_j}]$8–$[Ctx_{< t_j}]$9 FPS (Yang et al., 16 Jun 2026).

The contrastive baseline formalized in the paper is a generate-then-compare scheme. At each frame $[Dec]$0, the model would generate a candidate caption $[Dec]$1 and compare it with the previous caption $[Dec]$2 using a semantic similarity function $[Dec]$3 and threshold $[Dec]$4. Silence is selected if $[Dec]$5; otherwise, the candidate caption is emitted (Yang et al., 16 Jun 2026). This baseline requires full decoding per frame and is described as impractical for low-latency real-time streaming.

The computational characterization given in LiveStarPro is accordingly twofold. First, SVeD replaces per-frame autoregressive generation with a single forward verification pass plus occasional decoding only when needed. Second, when combined with the streaming key-value cache, it avoids recomputation under Swap and dynamic appends, yielding a reported $[Dec]$6 speedup, specifically $[Dec]$7 versus $[Dec]$8 FPS over 5-minute streams, with negligible loss in output quality (Yang et al., 16 Jun 2026).

4. Interaction with SCAM and the streaming key-value cache

SVeD is not presented as a standalone gate detached from training. Its efficacy depends on the model’s ability to assign semantically meaningful probabilities to captions under incrementally changing visual context. For this reason, LiveStarPro couples SVeD with Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams (Yang et al., 16 Jun 2026).

SCAM interleaves frames and captions by semantic clip $[Dec]$9, pairing a frame $P(\cdot \mid \cdot)$0 with a caption $P(\cdot \mid \cdot)$1 under stochastic paraphrase sampling. It masks intra-clip previously generated captions to prevent trivial copying across frames within the same event, while preserving visibility of terminal captions from preceding clips $P(\cdot \mid \cdot)$2 to maintain narrative coherence (Yang et al., 16 Jun 2026). The stated training objective is

$P(\cdot \mid \cdot)$3

The significance of this coupling is explicit in the paper: stream-oriented training gives the verification signal—the perplexity of $P(\cdot \mid \cdot)$4—“true semantic meaning,” aligning the gate to visual evidence rather than linguistic repetition (Yang et al., 16 Jun 2026). This suggests that SVeD depends on calibration not merely in the probabilistic sense but in the stronger sense that caption likelihood must track semantic persistence across temporally evolving visual inputs.

LiveStarPro also introduces a streaming KV cache because standard caches assume static sequences, whereas SVeD dynamically appends frames and logically repositions the last caption through Swap (Yang et al., 16 Jun 2026). The cache has two components: an intra-dialogue cache for frame-level processing and an inter-dialogue cache for long-term context across sessions. It is described as preserving sequence integrity under Swap without recomputation when a caption changes position and as accommodating dynamic length adaptation via Peak-End compression in short-term memory (Yang et al., 16 Jun 2026). Enabling both caching modes yields the reported $P(\cdot \mid \cdot)$5 speedup.

5. Coupling with hierarchical memory

SVeD also functions as a signal source for memory management in LiveStarPro. The paper states that the verification perplexity drives both short-term compression and long-term retrieval within Tree-Structured Hierarchical Memory (TSHM) (Yang et al., 16 Jun 2026).

In Short-Term Working Memory, Peak-End compression uses the verification perplexity as a semantic divergence score $P(\cdot \mid \cdot)$6. For each clip $P(\cdot \mid \cdot)$7, a dynamic threshold is computed as $P(\cdot \mid \cdot)$8, and the retained frames are

$P(\cdot \mid \cdot)$9

along with the clip’s summary caption (Yang et al., 16 Jun 2026). This is described as reducing redundancy while preserving salient peaks and clip-level summaries. A plausible implication is that low-perplexity regions mark temporally stable evidence that remains explanatory under the current caption, while thresholding around the median stabilizes memory size within each clip.

For Long-Term Retrieval Memory, when the gate activates decoding, LiveStarPro forms a query $[Dec]$0, explicit for QA and implicit from recent visual context for narration, and retrieves top-$[Dec]$1 event chains via hierarchical beam descent (Yang et al., 16 Jun 2026). Parent embeddings are maintained as subtree centroids via momentum:

$[Dec]$2

with $[Dec]$3 according to the reported hyperparameter choice (Yang et al., 16 Jun 2026). The retrieved history then augments generation when SVeD decides to speak.

This integration gives SVeD a dual role. It is both a response-timing mechanism and a semantic divergence measure used to organize memory. The paper’s broader claim is that SVeD together with TSHM sustains long-range coherence beyond the active window (Yang et al., 16 Jun 2026). That formulation is empirical rather than purely architectural, since the evidence comes from the long-form memory tasks.

6. Empirical evaluation, hyperparameters, and limitations

The empirical evidence for SVeD is reported within the evaluation of LiveStarPro on OmniStarPro, a benchmark spanning five live tasks and three long-form memory tasks (Yang et al., 16 Jun 2026). In fully online evaluation, where models choose autonomously when to speak or remain silent, LiveStarPro is reported to achieve on Real-time Narration Generation higher semantic correctness and lower timing error than prior online Video-LLMs while maintaining the fastest throughput (Yang et al., 16 Jun 2026). Relative to the second-best prior online model, the reported gains are $[Dec]$4 in SemCor, $[Dec]$5 in TimDiff, and $[Dec]$6 in FPS (Yang et al., 16 Jun 2026). Across RNG, OTG, FDQ, COQ, and MIQ, the paper reports consistent gains in semantic and timing metrics. On long-form memory tasks LMR, CDQ, and TBR, LiveStarPro is said to retain substantially higher recall at medium and long spans (Yang et al., 16 Jun 2026).

Ablation of the SVeD threshold $[Dec]$7 indicates that optimal performance lies in the interval $[Dec]$8, with default $[Dec]$9 balancing TimDiff, TimRedun, and TimCover (Yang et al., 16 Jun 2026). The practical guidance given is to calibrate $y_1, \ldots, y_N$0 on a small held-out stream, often fewer than 100 frames, for each domain so as to normalize relative drift in model perplexity scales (Yang et al., 16 Jun 2026). Verification frequency is per incoming frame, and Swap is always applied when suppressing generation so that temporal validity remains aligned (Yang et al., 16 Jun 2026). Additional reported settings include Peak-End pruning window $y_1, \ldots, y_N$1 frames, context token budget $y_1, \ldots, y_N$2, TSHM similarity threshold $y_1, \ldots, y_N$3, and momentum $y_1, \ldots, y_N$4 (Yang et al., 16 Jun 2026).

The paper also documents several failure modes. Threshold sensitivity is primary: if $y_1, \ldots, y_N$5 is too low, the gate may fire too often, producing over-speech and high TimRedun; if too high, it may suppress needed updates, leading to late responses and higher TimDiff (Yang et al., 16 Jun 2026). Rapid scene changes or flicker can make verification perplexity oscillate, and without hysteresis the gate may toggle frequently. LiveStarPro is said to rely on SCAM-aligned probabilities and Swap for stabilization, while the text identifies mild hysteresis or minimum-interval constraints as a potential extension (Yang et al., 16 Jun 2026). Under domain shift or low-confidence states, poorly calibrated probabilities may make perplexity unreliable. For TSHM, inappropriate $y_1, \ldots, y_N$6 values can induce tree degeneracy, either collapsing the tree into wide shallow branches or fragmenting it into many roots (Yang et al., 16 Jun 2026).

7. Conceptual significance and relation to alternative gating paradigms

SVeD’s central conceptual move is to treat silence not as a lexical class but as a verification outcome. The paper summarizes this as “silence as verification, not generation” (Yang et al., 16 Jun 2026). In the LiveStarPro formulation, the gate criterion is intentionally simple:

$y_1, \ldots, y_N$7

This simplicity is significant because it ties the response trigger to a quantity already native to autoregressive LLMs: conditional likelihood. Unlike EOS-based gating, SVeD does not require the model to devote vocabulary mass to a frequent non-semantic token. Unlike voice-activity-like heuristics, it does not assume a principled silence signal in visual-only streams. Unlike generate-then-compare, it does not require full candidate decoding at every frame (Yang et al., 16 Jun 2026).

The paper’s comparison with EOS/silence-token systems is explicit. Such systems are said to over-speak, achieving high TimCover but poor SemCor and high TimDiff (Yang et al., 16 Jun 2026). This is presented as empirical corroboration of the earlier critique concerning imbalance and misalignment. The comparison with generate-then-compare is computational rather than only statistical: SVeD reduces latency by using a single-pass verification of the previously emitted caption under updated context (Yang et al., 16 Jun 2026).

In broader methodological terms, SVeD may be interpreted as a streaming consistency test over evolving multimodal state. That interpretation is inferential, but it matches the paper’s formal structure: the current caption remains active unless its perplexity under the new context rises beyond a scaled reference. The resulting architecture delegates response timing to model-internal likelihood dynamics while preserving causality and low-latency operation. Within LiveStarPro, this mechanism is not isolated; its effectiveness is conditioned on SCAM training and reinforced by TSHM-based memory organization, making SVeD a central coordinating primitive for online narration, suppression, and retrieval (Yang et al., 16 Jun 2026).