Spatio-Temporal Token Scoring in Video VLMs

• Spatio-Temporal Token Scoring is a method that estimates token importance in video models by leveraging spatial and temporal redundancy.
• It employs a unified, end-to-end training approach that separates temporal scoring (with auxiliary losses) from spatial saliency derived via downstream task gradients.
• Empirical results demonstrate significant efficiency gains (up to 2.25× speedup) with minimal accuracy loss across video QA, autonomous driving, and 3D reconstruction tasks.

Spatio-Temporal Token Scoring (STTS) denotes a class of token-importance estimation methods that exploit redundancy across both space and time to reduce computation while preserving task-relevant information. In its explicit 2026 formulation for video vision-LLMs, STTS is a lightweight module that prunes vision tokens across both the ViT and the LLM, without text conditioning or token merging, and is fully compatible with end-to-end training (Zhang et al., 18 Mar 2026). Closely related ideas appear in earlier video-transformer token selection, semantic-aware temporal accumulation, event-based token sparsification, sparse attention for long-context LLMs, embodied navigation, autonomous driving, and streaming 3D reconstruction, where the common operation is to assign scores to tokens or token groups and use those scores to drive selection, pruning, masking, compression, or cache retention (Wang et al., 2021, Ding et al., 2023, Zhao et al., 26 Sep 2025, Xu et al., 15 May 2026, Wang et al., 6 Mar 2026, Sha et al., 21 Apr 2026, Wang et al., 18 Mar 2026).

1. Conceptual scope and terminology

A central premise of STTS is that dense token processing is wasteful because spatio-temporal inputs contain structured redundancy. In video VLMs, the redundancy is twofold: neighboring frames often repeat content temporally, and patches within a frame are unevenly informative spatially. The 2026 “Unified Spatio-Temporal Token Scoring for Efficient Video VLMs” paper formalizes this as a holistic pruning problem over the total number of patch tokens, with $N_{\text{total}} = T \times N$, rather than as a post hoc reduction applied only after visual encoding (Zhang et al., 18 Mar 2026).

A potential source of confusion is acronym overloading. In 2021, “STTS” denoted “Spatial-Temporal Token Selection,” a ranking-based framework for video transformers that dynamically selected informative tokens in temporal and spatial dimensions (Wang et al., 2021). By 2026, “STTS” denoted “Spatio-Temporal Token Scoring,” emphasizing learned token importance estimation rather than selection alone (Zhang et al., 18 Mar 2026). The underlying themes are related, but the later formulation places more emphasis on unified architecture-wide pruning and on how scores are learned.

Across the literature, token scoring may be learned or training-free, continuous or Top-$K$-based, and may act on frames, patches, anchors, attention entries, or KV-cache elements. This suggests that STTS is best understood not as a single operator, but as a family of mechanisms that convert spatio-temporal structure into computational sparsity.

2. Unified STTS for video VLMs

The 2026 unified STTS framework inserts a scorer into the ViT at a fixed depth $l$; in the main experiments, $l=3$ (Zhang et al., 18 Mar 2026). Given video tokens

$X \in \mathbb{R}^{T \times N \times D},$

the ViT processes tokens up to layer $l$, after which STTS applies a Token Pooler based on self-attention pooling and a 3-layer MLP to produce token scores. Because the Molmo2 backbone already uses $3 \times 3$ spatial pooling, the scorer pools features with window size $w=3$, reducing the spatial positions from $N$ to $N / w^2$. Temporal context is injected by concatenating pooled features from the current frame with those from the previous frame, yielding inputs in

$K$0

For the first frame, the previous-frame input is zero-padded, and the first frame is always kept intact (Zhang et al., 18 Mar 2026).

The paper separates temporal and spatial learning signals. Temporal scoring is supervised explicitly through neighboring-frame cosine similarity: $K$1 The auxiliary loss is

$K$2

and the full training loss is

$K$3

Spatial scoring is learned implicitly from downstream task gradients by injecting the score map into the next ViT attention layer: $K$4 In this formulation, temporal redundancy is learned through the auxiliary loss, while spatial saliency is shaped by the downstream VLM objective (Zhang et al., 18 Mar 2026).

After scoring, STTS performs hard pruning by removing the bottom-$K$5 of tokens according to the learned scores. Pruning occurs inside the ViT, so savings propagate through later ViT layers and the LLM. Because the surviving tokens are unevenly distributed across frames, the method uses a first-fit descending packing algorithm to repack them into dense compute-friendly tensors and builds a packed attention mask to prevent cross-frame leakage. The paper notes that the packing algorithm is $K$6, but $K$7, so the overhead is negligible (Zhang et al., 18 Mar 2026).

3. Antecedents in video transformers

The 2021 “Efficient Video Transformers with Spatial-Temporal Token Selection” paper established a direct antecedent for STTS-style pruning in video transformers (Wang et al., 2021). It formulates token selection as a ranking problem and uses a lightweight scorer network to estimate token importance. The scorer is a 2-layer MLP operating on token embeddings $K$8, with a reduced intermediate dimension $K$9, local projection, global average pooling, concatenation of local and global context, and min-max normalized scores. Temporal selection keeps the most relevant frames after spatial pooling, while spatial selection identifies the most discriminative region without breaking the hierarchical spatial context used in backbones such as MViT. Because hard Top-$l$0 is non-differentiable, the paper employs a perturbed-maximum based differentiable Top-$l$1 operator for end-to-end training, with $l$2 by default (Wang et al., 2021).

A major design point in that earlier framework is that naïve spatial token dropping can destroy structure. To avoid this, spatial selection is anchor-based: token scores are reshaped into a 2D map, split into overlapping anchors, averaged per anchor, and then reduced by Top-1 anchor selection. The stated rationale is to preserve contiguous spatial context, local modeling, hierarchical pooling, and temporal consistency. The paper reports that naïve token-based spatial selection causes a major accuracy drop, whereas anchor-based selection works much better (Wang et al., 2021).

The 2023 “Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation” paper shifted emphasis from learned scoring networks to a parameter-free score derived from temporal redundancy and semantic importance (Ding et al., 2023). It models video tokens as

$l$3

propagates an accumulative temporal score frame to frame through an affinity matrix between retained tokens and current-frame tokens, and combines that redundancy estimate with an activation-based semantic score: $l$4 The final pruning score is

$l$5

Tokens with larger scores are pruned. This design directly scores integral spatio-temporal tokens rather than separating frame selection from region selection, and it requires no extra learned parameters or retraining (Ding et al., 2023).

Taken together, these antecedents define two durable STTS principles: first, token importance should be conditioned on spatio-temporal structure rather than on frame-local saliency alone; second, sparsification must preserve the structural priors of the backbone, or the efficiency gain will be offset by representation damage.

4. Domain-specific variants and adjacent formulations

The same operational logic recurs in several adjacent literatures, even when the term STTS is not used explicitly.

Method	Domain	Core score or selection basis
PSTTS (Zhao et al., 26 Sep 2025)	Event-based representation learning	$l$6, $l$7, $l$8, $l$9, $l=3$0
STS (Xu et al., 15 May 2026)	Long-context LLM inference	Draft-model attention Top-$l=3$1 with offline head mapping
History-conditioned pruning (Wang et al., 6 Mar 2026)	Vision-Language Navigation	$l=3$2, A-MMR, $l=3$3, $l=3$4
ST-Prune (Sha et al., 21 Apr 2026)	Autonomous driving VLMs	$l=3$5 and $l=3$6
STAC (Wang et al., 18 Mar 2026)	Streaming 3D reconstruction	Decayed cumulative attention $l=3$7 and Top-$l=3$8 anchors

In event-based spatio-temporal representation learning, PSTTS introduces a parameter-free, plug-and-play module with two stages: Spatial Token Purification and Temporal Token Selection (Zhao et al., 26 Sep 2025). Spatial Token Purification computes temporal continuity using a Leaky Integrate-and-Fire spiking neuron, aggregates it with neighborhood-weighted spatial-temporal consistency into $l=3$9, downsamples to $X \in \mathbb{R}^{T \times N \times D},$0, and removes tokens below the adaptive threshold

$X \in \mathbb{R}^{T \times N \times D},$1

Temporal Token Selection then evaluates motion magnitude similarity, trajectory shape similarity, temporal motion redundancy

$X \in \mathbb{R}^{T \times N \times D},$2

and a semantic auxiliary cue based on the L2 norm of token embeddings, producing

$X \in \mathbb{R}^{T \times N \times D},$3

This is a progressive spatio-temporal token scoring pipeline grounded primarily in raw event statistics rather than intermediate RGB-style token embeddings (Zhao et al., 26 Sep 2025).

In long-context LLM inference, STS—“Speculative Token Sparsity”—reuses draft-model attention scores as token-importance signals for a larger target model (Xu et al., 15 May 2026). Its core insight is that for the same input, attention patterns of two related models are strongly correlated. The draft model runs first, its attention weights are extracted, Top-$X \in \mathbb{R}^{T \times N \times D},$4 tokens are selected per head, and an offline head mapping transfers these selections to the target model. For target head $X \in \mathbb{R}^{T \times N \times D},$5 and draft head $X \in \mathbb{R}^{T \times N \times D},$6,

$X \in \mathbb{R}^{T \times N \times D},$7

The resulting token-and-head-wise sparsity mask is then used by the target model for sparse attention. This is not called STTS in the paper, but it is an explicit token-importance estimation pipeline that scores which tokens matter head by head and token by token (Xu et al., 15 May 2026).

In Vision-Language Navigation, the 2026 history-conditioned pruning framework uses [CLS]-based attention saliency, Adaptive Maximal Marginal Relevance (A-MMR), and current-view-conditioned historical relevance (Wang et al., 6 Mar 2026). The current frame is pruned by combining base importance

$X \in \mathbb{R}^{T \times N \times D},$8

with an MMR distinctness term, while historical tokens are reweighted by

$X \in \mathbb{R}^{T \times N \times D},$9

with $l$0. The paper’s central claim is that token importance for historical memory cannot be scored in isolation; it must be conditioned on the current view (Wang et al., 6 Mar 2026).

In autonomous driving, ST-Prune applies the same principle to multi-view, multi-frame inputs (Sha et al., 21 Apr 2026). Motion-aware Temporal Pruning uses

$l$1

with $l$2, to encode motion volatility and temporal recency, while Ring-view Spatial Pruning uses

$l$3

to suppress redundancy across adjacent surround-view cameras. The selection engine is max-min diversity with temporal and spatial soft constraints (Sha et al., 21 Apr 2026).

In streaming 3D reconstruction, STAC interprets cache management itself as a spatio-temporal token scoring problem (Wang et al., 18 Mar 2026). It updates token importance through decayed cumulative attention: $l$4 then keeps Top-$l$5 anchors,

$l$6

while compressing spatially redundant historical tokens into voxel-aligned representatives. A plausible implication is that STTS has expanded from token pruning in encoder stacks to memory budgeting in persistent transformer state.

5. Empirical performance and efficiency trade-offs

In the unified video-VLM formulation, the headline result is that pruning 50% of vision tokens throughout the architecture yields a 62% improvement in efficiency during both training and inference with only a 0.7% drop in average performance across 13 short and long video QA tasks (Zhang et al., 18 Mar 2026). In the 128-frame setting, the paper reports a 1.62× training speedup and a 1.61× inference speedup; at 256 frames, the gains increase to 2.25× and 2.22×. The average benchmark score changes from 63.0 in the baseline to 62.3 at 50% pruning, and test-time scaling for long-video QA yields gains of 0.5–1% compared to the baseline (Zhang et al., 18 Mar 2026). The reported non-monotonicity—40% pruning gives 62.1 average while 50% gives 62.3—indicates that stronger pruning can sometimes remove borderline noisy tokens more effectively.

Earlier video-transformer evidence is consistent with this trade-off. The 2021 STTS paper reports similar results while requiring 20% less computation, and on Kinetics-400 with MViT-B16, keeping only 50% of input tokens reduces GFLOPs by more than 33% with accuracy drop within 0.7% (Wang et al., 2021). The 2023 STA paper reports over 30% computation reduction with a negligible $l$7 accuracy drop on Kinetics-400 and Something-Something V2, and gives examples such as ViT-L dropping from 597 GFLOPs to 308 GFLOPs with 0.1% accuracy loss (Ding et al., 2023).

Outside video QA, STTS-style scoring retains its efficiency-oriented character but adapts to domain-specific constraints. In long-context LLM inference, STS achieves a 2.67× speedup at approximately 90% sparsity on NarrativeQA while maintaining negligible accuracy degradation relative to dense attention, and reports 2.88× speedup at 100K context with a token budget of 512 (Xu et al., 15 May 2026). In event-based learning, PSTTS reduces FLOPs by 29–43.6% and increases FPS by 21.6–41.3% on DailyDVS-200 while maintaining task accuracy (Zhao et al., 26 Sep 2025). In autonomous driving, ST-Prune reports near-lossless performance even at 90% token reduction, with 2.84× end-to-end speedup on NuInstruct at 10% retention and 1.98× on OmniDrive at the same retention (Sha et al., 21 Apr 2026). In streaming 3D reconstruction, STAC reduces memory consumption by nearly $l$8 and accelerates inference by $l$9, while preserving reconstruction quality near the original models (Wang et al., 18 Mar 2026).

These results collectively support a recurring empirical pattern: efficiency gains become more pronounced as temporal horizon, frame count, sequence length, or memory footprint increases. This suggests that STTS methods are especially valuable when redundancy accumulates faster than task-relevant novelty.

6. Design constraints, failure modes, and open issues

A recurring limitation is that spatio-temporal sparsification can damage structure if applied too early or too aggressively. In the unified video-VLM STTS paper, insertions at $3 \times 3$0 and $3 \times 3$1 are clearly worse than $3 \times 3$2, indicating that very early pruning hurts because token representations are not yet stable enough (Zhang et al., 18 Mar 2026). The same paper also shows that heuristic pruning based directly on neighboring-frame cosine similarity underperforms the learned scorer, and that removing the auxiliary loss hurts substantially. This reflects a broader misconception: temporal similarity alone is not sufficient, because semantically critical regions can be temporally redundant.

The 2021 STTS work makes the structural issue explicit for spatial pruning: naïve token-based spatial selection can severely hurt performance, and for VideoSwin the authors use only temporal selection in the main comparison because spatial selection is challenging due to window shifting and structural issues in Swin-style architectures (Wang et al., 2021). The 2023 STA paper arrives at a different remedy—progressive token-level pruning using temporal accumulation and semantic suppression—but it also accepts that pruned tokens never re-enter later layers (Ding et al., 2023). That irreversible decision improves efficiency but reduces the possibility of later recovery.

Training-free variants introduce their own assumptions. STS for LLM inference works best when draft and target models are related, requires offline head mapping on representative traces, depends on a small draft model whose dense attention is cheap enough to compute, and has a $3 \times 3$3-dependent mapping, so different sparsity settings may require different mappings (Xu et al., 15 May 2026). The history-conditioned VLN method assumes that the current pruned tokens form a sufficiently informative query set for history filtering and notes that real-world performance can vary with lighting conditions and prompt wording (Wang et al., 6 Mar 2026). ST-Prune has not been fully validated on dedicated closed-loop VLA or action-generation benchmarks where temporal action ordering is critical (Sha et al., 21 Apr 2026). STAC relies on fixed voxel resolution, and highly dynamic scenes can destabilize the cache because token identity is partially lost through merging (Wang et al., 18 Mar 2026).

Across these works, one persistent trade-off is between plug-and-play deployment and task-adaptive learning. Unified STTS for video VLMs is explicitly end-to-end trainable (Zhang et al., 18 Mar 2026), whereas PSTTS is parameter-free, STS is training-free, ST-Prune requires no calibration data or architectural modification, and STAC operates directly on caches and attention outputs (Zhao et al., 26 Sep 2025, Xu et al., 15 May 2026, Sha et al., 21 Apr 2026, Wang et al., 18 Mar 2026). This suggests an unresolved design bifurcation: learned STTS variants can tailor scores to downstream objectives, while training-free variants prioritize deployability, backbone compatibility, and immediate inference-time gains.

A final point concerns what STTS is not. It is not synonymous with text-conditioned pruning, because unified STTS explicitly avoids text conditioning (Zhang et al., 18 Mar 2026). It is not synonymous with token merging, because several of the cited methods prune or mask tokens rather than merging them (Zhang et al., 18 Mar 2026, Xu et al., 15 May 2026). And it is not confined to video recognition, since the same scoring logic now appears in LLM sparse attention, embodied navigation, autonomous driving, and cache compression. The literature therefore indicates that STTS has become a general strategy for turning spatio-temporal relevance estimates into efficient computation.