FlashVID: Efficient Inference for Video LLMs

• FlashVID is a training-free acceleration framework for video LLMs that integrates attention and diversity-based token selection with tree-based spatiotemporal merging.
• It leverages joint spatiotemporal compression to drastically reduce visual tokens, preserving over 99% of performance at 10% token retention in experiments.
• The plug-and-play framework adapts across architectures, achieving up to 10× speedup and significant throughput improvements within a fixed computational budget.

FlashVID is a training-free inference acceleration framework for Video LLMs (VLLMs) that targets the computational inefficiency induced by high volumes of visual tokens. It combines Attention and Diversity-based Token Selection (ADTS) with Tree-based Spatiotemporal Token Merging (TSTM), then applies a light inner-LLM pruning stage to align computation to a target budget, while remaining plug-and-play across architectures such as LLaVA-OneVision, LLaVA-Video, and Qwen2.5-VL (Fan et al., 8 Feb 2026). In the reported experiments, retaining only 10% of visual tokens preserves 99.1% of the performance of LLaVA-OneVision, and the same framework enables a 10x increase in video frame input to Qwen2.5-VL, resulting in a relative improvement of 8.6% within the same computational budget (Fan et al., 8 Feb 2026).

1. Computational setting and design rationale

FlashVID is motivated by the fact that attention cost in VLLMs scales quadratically with sequence length and that the sequence length is dominated by visual tokens rather than textual tokens. The formulation reported for a transformer with $L$ layers is

$$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$

where $d$ is hidden size and $m$ is FFN intermediate size. The paper also gives a more precise form for Group Query Attention and SwiGLU,

$$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$

with $g$ the number of KV heads and $h$ the number of attention heads (Fan et al., 8 Feb 2026).

The framework is built around a critique of acceleration pipelines that compress spatial and temporal redundancy independently. According to the paper, such decoupling overlooks the intrinsic coupling between space and time in videos: tokens that are redundant within a frame often persist across frames, but their spatial position, scale, and orientation can change with motion. A rigid temporal correspondence at fixed spatial locations can therefore merge tokens that are not truly correlated under dynamic conditions, degrading representations (Fan et al., 8 Feb 2026).

FlashVID addresses this by performing joint spatiotemporal compression after the vision encoder and before the modality connector. The method first retains a compact set of salient, diverse tokens per frame, then merges highly correlated tokens across adjacent frames through tree-structured aggregation. The resulting design is neither purely spatial pruning nor purely temporal merging; it is a hybrid compression paradigm in which before-LLM compression is combined with inner-LLM pruning.

2. Attention and Diversity-based Token Selection

ADTS is the frame-level selection mechanism. Let the vision encoder output be

$E \in \mathbb{R}^{F \times N_v \times d},$

where $F$ is the number of frames, $N_v$ the number of tokens per frame, and $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$0 the feature dimension. For a given frame $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$1, the feature matrix is $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$2 and the $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$3-th token feature is $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$4 (Fan et al., 8 Feb 2026).

ADTS uses three signals: attention-derived importance, event relevance, and diversity. If the encoder has no explicit $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$5 token, token-level attention is computed from

$$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$6

and the attention weight for token $$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$7 is

$$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$8

With multiple heads, the score can be averaged across heads. Event relevance is defined by first computing a frame embedding via global average pooling,

$$\mathrm{FLOPs} = L \times (4 n d^2 + 2 n^2 d + 2 n d m),$$9

and then setting

$d$0

Diversity is measured by pairwise cosine distances within each frame,

$d$1

These distances are calibrated using attention and event relevance weights:

$d$2

$d$3

with defaults $d$4 (Fan et al., 8 Feb 2026).

Selection is then posed as a calibrated Max-Min Diversity Problem: choose a subset $d$5 of size $d$6 that maximizes the minimal calibrated pairwise distance among selected tokens. The paper uses a greedy procedure that seeds with the token maximizing its minimum calibrated distance to other tokens, then iteratively adds the token whose minimum distance to the selected set is largest. In effect, ADTS constructs a per-frame basis that is both representative and non-redundant.

The complexity profile reflects this design. Pairwise distances require $d$7 and greedy MMDP updates require $d$8. The method is therefore most useful because it drastically lowers downstream cost by reducing the token count from $d$9 to $m$0 per frame (Fan et al., 8 Feb 2026).

3. Tree-based Spatiotemporal Token Merging and hybrid compression

TSTM operates on the tokens retained by ADTS. For adjacent frames $m$1, it computes the cosine similarity matrix

$m$2

for $m$3 and $m$4. Each token in the first frame initializes a root node. For each subsequent token $m$5, the algorithm finds its most similar parent in the previous frame,

$m$6

and links $m$7 to $m$8 if the similarity exceeds the threshold $m$9; otherwise it starts a new root (Fan et al., 8 Feb 2026).

The resulting structures are spatiotemporal redundancy trees. Their purpose is to capture redundancy across time even when entities move, rescale, or reorient. A tree $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$0 is aggregated either by averaging or by similarity-weighted pooling:

$$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$1

The paper reports that explicit depth and breadth constraints showed negligible gains and that the threshold $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$2 alone suffices to maintain quality (Fan et al., 8 Feb 2026).

FlashVID inserts these operations after the vision encoder and before the modality connector. The compact representation passed to the LLM is the union

$$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$3

where $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$4 denotes the set of TSTM-aggregated tokens. A light inner-LLM pruning stage then aligns the token count to budget. The budget relation is written as

$$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$5

with average retained tokens per layer $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$6, tokens entering the LLM $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$7, pruning layer $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$8, total number of layers $$\mathrm{FLOPs} = L \times (2 n d^2 (1 + g/h) + 2 n^2 d + 3 n d m),$$9, and retained output tokens $g$0. Using an expansion factor $g$1 such that $g$2 gives

$g$3

and the inner-LLM pruning ratio

$g$4

The defaults reported as robust across models are $g$5 and $g$6 (Fan et al., 8 Feb 2026).

Component	Role	Reported default or best setting
ADTS	Select salient and diverse tokens per frame	$g$7
TSTM	Merge highly correlated tokens across adjacent frames	$g$8
Hybrid compression	Align before-LLM compression with inner-LLM pruning	$g$9, $h$0

This arrangement is central to FlashVID’s claim of being training-free and plug-and-play: no retraining or model modification is required, and the insertion point is localized to the path between the vision encoder and the modality connector (Fan et al., 8 Feb 2026).

4. Empirical performance and efficiency characteristics

The reported experiments cover three representative VLLMs—LLaVA-OneVision, LLaVA-Video, and Qwen2.5-VL—and five video understanding benchmarks: VideoMME, EgoSchema, LongVideoBench, MVBench, and MLVU (Fan et al., 8 Feb 2026).

For LLaVA-OneVision, retaining only 10% of visual tokens preserves 99.1% of vanilla performance. At 25%, 20%, and 15% retention, FlashVID surpasses the vanilla full-token baseline, which the paper describes as a “less-is-more” effect. For LLaVA-Video, FlashVID yields 5.3× prefilling and 1.9× TTFT speedups at 10% retention with 95.9% relative accuracy. For Qwen2.5-VL, FlashVID outperforms FastV, VisionZip, and FastVID across retention ratios; under a fixed token budget, it permits 10× frames and yields a relative improvement of 8.6% (Fan et al., 8 Feb 2026).

The efficiency profile is therefore twofold. First, token compression directly reduces the quadratic attention term and the KV cache footprint. Second, the framework uses light inner-LLM pruning rather than aggressive early-layer pruning, which the paper contrasts with alternatives such as FastV. In practical terms, the strongest headline numbers are 6.3× prefilling speedup and 2.1× TTFT speedup for LLaVA-OneVision at 10% retention, with most of the original accuracy preserved (Fan et al., 8 Feb 2026).

The paper also reports broad generalization. The method transfers across architectures that differ in tokenization conventions, including newline tokens in LLaVA-Video and MRoPE and Deepstack-style tokenization in Qwen2.5-VL. It also transfers across short, medium, and long videos and across tasks spanning temporal reasoning and spatial perception. This suggests that the framework’s benefits are tied more to its compression geometry than to any single backbone-specific assumption.

5. Ablation results and interpretive analyses

The ablation studies isolate the roles of ADTS, TSTM, and their interaction. ADTS significantly outperforms pure attention-based selection (ATS) and pure diversity selection (DTS). Both $h$1 attention and event relevance contribute, and their combination is optimal. When the two main modules are separated, ADTS-only ($h$2) outperforms TSTM-only ($h$3), but the best performance arises from combining them with $h$4, confirming the intended synergy (Fan et al., 8 Feb 2026).

The TSTM merge threshold is likewise sensitive. The reported best value is $h$5, which balances compression and quality; lower values risk merging less-correlated tokens, while higher values may under-compress. By contrast, explicit depth and breadth constraints on the redundancy trees show negligible gains (Fan et al., 8 Feb 2026).

These ablations support the paper’s broader argument that joint spatiotemporal merging is superior to rigid temporal token merging at fixed spatial locations. The advantage is not merely that fewer tokens are kept, but that the retained and merged tokens remain semantically coherent under motion. The paper states that TSTM can track correlated tokens across frames even when spatial positions shift, thereby increasing average inter-frame merging similarity versus rigid temporal token merging. A plausible implication is that FlashVID’s gains arise from better redundancy modeling rather than only from more aggressive compression.

The “less-is-more” phenomenon reported for LLaVA-OneVision also matters conceptually. Since the compressed models can exceed the full-token baseline at 15–25% retention, the framework is not only removing compute; under some conditions it is removing distracting or redundant visual content in a way that improves downstream reasoning (Fan et al., 8 Feb 2026).

6. Comparative position, limitations, and related uses of the term

FlashVID is positioned against several contemporary efficiency methods. FastV performs inner-LLM pruning at shallow layers and is described as unstable under early-layer pruning. VisionZip performs before-LLM pruning via $h$6 attention with primarily spatial compression and degrades sharply at low retention because temporal cues are lost. PruneVID combines attention-based selection and spatiotemporal merging inside the LLM and compresses the KV cache. FastVID is a density-based spatiotemporal pruning method. FlashVID is reported to produce stronger accuracy and speed under comparable retention than these baselines (Fan et al., 8 Feb 2026).

The framework also has explicit failure cases. Fine-grained motion cues at extreme compression can be lost if the retention ratio is too aggressive or if $h$7 is too low. Highly similar features from different entities may occasionally be merged, causing semantic confusion. The method relies on encoder feature cosine similarity, so heavy motion blur or drastic appearance changes can reduce similarity reliability. The paper proposes explicit motion modeling, adaptive thresholds, or structured diversity priors as possible future improvements (Fan et al., 8 Feb 2026).

A related development is ResAdapt, which targets the input side rather than the post-encoding stage. ResAdapt is described as complementary to FlashVID: it reduces the pixel and token load upfront by adaptive per-frame resizing, while FlashVID performs tree-based post-encoder merging. The ResAdapt paper states that this is “drop-in: you can keep your FlashVID post-encoder merging” and recommends “ResAdapt for input-side pixel control + FlashVID for post-encoder spatiotemporal merging” (Liao et al., 30 Mar 2026). This suggests a compositional efficiency stack in which input-side allocation and post-encoder compression are separated rather than treated as alternatives.

The term “FlashVID” is not entirely unambiguous in the literature. In the text-to-video generation paper “FlashVideo,” the authors state that FlashVID is simply a shorthand for FlashVideo and that there is no separate model distinct from FlashVideo (Lei et al., 2023). That usage refers to a RetNet-based framework for text-to-video generation with recurrent $h$8 inference and redundant-free frame interpolation, which is conceptually distinct from the training-free VLLM acceleration method described above (Lei et al., 2023). In current multimodal reasoning literature, however, “FlashVID” most prominently denotes the ADTS-plus-TSTM compression framework introduced for efficient video LLMs (Fan et al., 8 Feb 2026).