EarlyTom: Early Token Compression Completes Fast Video Understanding

Abstract: Video LLMs (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.

• The paper introduces a training-free framework that merges redundant video frames inside the vision encoder and applies decoupled spatial token selection to reduce both encoder and LLM computation.
• EarlyTom cuts time-to-first-token by up to 2.65× and FLOPs by 61% at 10% token retention on LLaVA-OneVision-7B, while preserving 96.2% of baseline accuracy.
• The method improves robustness under aggressive compression and achieves gains across multiple video benchmarks and backbones, though it requires architecture- and benchmark-specific tuning.

Motivation and problem statement

Video LLMs (Video-LLMs) incur substantial inference cost from processing hundreds to thousands of visual tokens per input. The authors of EarlyTom observe that prior training-free token compression methods—whether applied inside the LLM (FastV, SparseVLM, PyramidDrop), before the LLM (VisionZip, LLaVAPruMerge), or in hybrid pipelines (HoliTom, FastVID, DyCoke)—compress tokens only after the vision encoder, leaving the encoder itself as an unoptimized latency bottleneck. Their profiling on LLaVA-OneVision-7B shows that vision encoding accounts for 36.3% of time-to-first-token (TTFT) in the full-token baseline (323 ms), rising to 55.8% under HoliTom and 68.4% under VisionZip. HoliTom additionally introduces a 121.9% (+78 ms) overhead in the visual token processing stage. This motivates moving compression inside the vision encoder, at the earliest stage of the pipeline.

Method

EarlyTom is a training-free framework with two stages.

Stage I: inner vision encoder frame merging. The video is segmented in a streaming manner using EMA-smoothed cosine similarity between consecutive frames' token features; a smoothed score below a threshold $\tau_{\mathrm{seg}}$ marks a segment boundary. Within each segment, middle frames are merged pairwise under a local-optimal criterion: two frames merge only if their similarity exceeds $\tau_{\mathrm{merge}}$ and exceeds the similarity of the next adjacent pair. Merged representations use similarity-weighted fusion, concentrating features around semantically important content. Because merging occurs at selected encoder layers (empirically layers 6–23), it directly reduces both encoder compute and the number of tokens entering the LLM.

Stage II: decoupled spatial token selection. The authors identify a "video attention sink" phenomenon: certain spatial patch positions in SigLIP consistently attract disproportionately high attention across all frames regardless of content, because sink tokens have abnormally large query/key norms. Global Top-K selection therefore biases retained tokens toward these structural attractors and induces distribution shift across frames. EarlyTom decouples merged frames into dynamic sets (head and tail frames per segment, which carry the highest discriminative power) and static sets (middle frames). Dynamic frames undergo global Top-K selection by attention score; static frames use local-window Top-K, selecting one token per equal-sized window to preserve spatial distribution. A system co-design offloads static-frame selection to the CPU while the GPU handles dynamic selection, exploiting otherwise idle CPU capacity with a custom Triton kernel.

Main results

On LLaVA-OneVision-7B (single NVIDIA A100, 32 sampled frames), EarlyTom achieves state-of-the-art efficiency among training-free methods:

Retained ratio	FLOPs (T)	TTFT (ms)	Throughput (tok/s)	Avg. accuracy (% of baseline)
25%	36.5	426.3	32.9	99.7
20%	35.1	415.3	33.4	99.3
15%	33.6	390.6	30.4	98.1
10%	32.2	336.2	31.6	96.2

The baseline requires 82.6T FLOPs at 889.9 ms TTFT; HoliTom at 25% retention needs 49.0T and 661.3 ms. At 10% retention, EarlyTom reduces TTFT by up to 2.65× and FLOPs by up to 61%, with throughput reaching 32.9 tokens/s versus 24.4 for the baseline. Accuracy is evaluated on MVBench, EgoSchema, LongVideoBench, and VideoMME via LMMs-Eval. Notably, under aggressive compression VisionZip drops nearly 9% relative to baseline whereas EarlyTom drops only about 4%, indicating better robustness than late-stage pruning at extreme ratios. On the smaller LLaVA-OV-0.5B backbone, EarlyTom reaches 280.1 ms TTFT at 10% retention (1.48× speedup over baseline); here HoliTom's overhead actually makes it slower than the baseline (0.90× speedup), underscoring that compression overhead matters more when prefill savings shrink. Supplementary results extend the gains to LLaVA-Video-7B (6.8× TTFT speedup at 15% retention, 947.4 ms vs. 6429.3 ms) and Qwen2.5-VL-7B (12.2% FLOPs ratio, 62.2 average score vs. 59.3 for uniform subsampling).

Ablations

Module-wise, temporal merging alone retains ~73.9% of tokens at 98.8% of baseline accuracy; spatial selection alone achieves 20% retention at comparable accuracy; combining them yields 58.8 average versus 58.4 for either alone. Layer choice matters: merging starting at layer 6 gives the best accuracy–throughput balance (58.9 average), while layer 4 yields lower TTFT but degraded accuracy. For sampling strategy, random selection is fastest but least accurate (57.8), global Top-K is most accurate but slowest due to $O(N \log K)$ ranking (58.4), and local-window sampling achieves the best trade-off (58.8 at higher throughput than Top-K). On Qwen2.5-VL, removing spatial selection or weighted merging each costs roughly 0.8–0.9 points, confirming both components contribute.

Limitations and open questions

Several caveats bear on interpreting these results. First, hyperparameters ($\tau_{\mathrm{seg}}$, merge layer indices, EMA factor) are tuned per benchmark and per retention ratio—the appendix lists distinct thresholds for MVBench, EgoSchema, LongVideoBench, and VideoMME—which raises questions about out-of-the-box generalization to unseen distributions. Second, stage-I retention is sample-dependent since redundancy varies by input, so end-to-end compression ratios are not fixed a priori. Third, the method is evaluated only with SigLIP-based encoders and LLaVA-family/Qwen architectures; whether the sink-token analysis transfers to other vision encoders remains unverified. Fourth, the paper targets prefill-stage latency only; decoding acceleration for reasoning-style multimodal models is left open, as is principled joint algorithm–system co-design for heterogeneous inference rather than the ad hoc CPU offloading used here.

Conclusion

EarlyTom demonstrates that relocating token compression into the vision encoder—via streaming frame segmentation, local-optimal weighted frame merging, and bias-aware decoupled spatial selection—yields larger TTFT reductions (up to 2.65×) and FLOPs savings (up to 61%) than post-encoder methods, while retaining 96–100% of full-token accuracy across four video benchmarks and multiple backbones. The profiling evidence that vision encoding dominates residual TTFT in existing SoTA methods provides a clear diagnosis of where future Video-LLM efficiency work should focus.