Progressive Spatio-Temporal Token Selection

• PSTTS is a methodology that progressively refines token selection by leveraging spatial structures and temporal dynamics across model stages.
• It adapts token retention through context-aware scoring mechanisms, combining local spatial smoothing with temporal consistency to balance efficiency and performance.
• Empirical studies show that PSTTS significantly improves throughput and accuracy in diverse applications, from diffusion LLM decoding to video and multimodal grounding.

Progressive Spatio-Temporal Token Selection (PSTTS) denotes a family of token-efficient inference or representation-learning strategies that progressively retain, discard, or compress tokens using both spatial structure and temporal dynamics. Across the literature, the term is used in closely related but not identical ways: in diffusion LLM decoding, it refers to progressively committing tokens based on local decoded context and temporal ID consistency; in video and multimodal LLMs, it denotes query-guided or stability-guided selection of visual or audio-visual tokens under a budget; in event-based learning, GUI agents, autonomous driving, and navigation, it denotes staged pruning policies that suppress spatial noise, temporal redundancy, or cross-view duplication while preserving task-critical structure (Chen et al., 7 Apr 2026). A consistent thread is that token decisions are not made by a single global rule applied once, but are refined across denoising steps, temporal windows, model stages, or network depth, with the retained set evolving as additional spatial and temporal evidence becomes available.

1. Conceptual scope and historical lineage

The phrase “Progressive Spatio-Temporal Token Selection” is used explicitly in several 2026 works, but its conceptual lineage reaches earlier token-selection frameworks for video and multimodal reasoning. “Efficient Video Transformers with Spatial-Temporal Token Selection” formulates token selection as a ranking problem and dynamically selects informative tokens in both temporal and spatial dimensions, but does not use the term “progressive” as its primary label (Wang et al., 2021). “Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation” then introduces a progressive pruning schedule across backbone stages, where tokens are removed at three pruning stages according to temporal redundancy accumulation and semantic importance (Ding et al., 2023). In audio-visual question answering, “Progressive Spatio-temporal Perception for Audio-Visual Question Answering” organizes question-aware temporal segment selection, spatial region selection, and audio-guided refinement into a cascade that is naturally interpretable as progressive token selection across time and space (Li et al., 2023).

By 2025–2026, PSTTS becomes an explicit organizing term in multiple domains. In video question answering, “Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering” describes a two-stage “EXPLORE-THEN-SELECT” strategy that searches over different static/dynamic allocations under a token budget and uses a query-aware attention metric to choose the best candidate; the paper frames this as closely related to progressive spatio-temporal token selection, while noting that its own mechanism is not explicitly progressive in the layer-wise sense (Shi et al., 30 Apr 2025). In event-based representation learning, “PSTTS: A Plug-and-Play Token Selector for Efficient Event-based Spatio-temporal Representation Learning” uses the acronym directly for a two-stage module comprising Spatial Token Purification and Temporal Token Selection (Zhao et al., 26 Sep 2025).

The literature therefore does not present a single canonical PSTTS algorithm. Instead, it presents a recurring design pattern: temporal redundancy is estimated before or alongside spatial redundancy; token budgets are allocated adaptively rather than uniformly; and selection is updated progressively across steps, stages, or layers. This suggests that PSTTS functions less as a single method name than as a cross-domain methodological template.

2. Core design principles

Across the papers, “progressive” usually means that token decisions are revised or sharpened as computation proceeds. In STDec, the retained set grows during diffusion denoising because the neighborhood $N(i)$ and the threshold $\tau_t^{s}(i)$ evolve as $D_t$ grows, while the temporal consistency streak $K_t(i)$ increases when predictions remain stable; the resulting token-adaptive threshold $\tau_t^{st}(i)$ decreases for locally and temporally stable tokens, expanding the commit set step by step (Chen et al., 7 Apr 2026). In GroundVTS, the “progressive” component is a training curriculum rather than a per-layer inference schedule: Stage 1 warms up VTS alone, Stage 2 jointly tunes VTS and a projector with LoRA on the LLM, and Stage 3 fine-tunes for temporal grounding tasks (Fan et al., 2 Apr 2026). In SEATS, progression is explicitly depth-dependent: shallow blocks keep the pre-LLM retention ratio, middle blocks decay retention with an exponential schedule, and late blocks remove all remaining non-text tokens once cross-modal fusion is complete (Xin et al., 19 May 2026).

The “spatio-temporal” component is likewise instantiated differently across domains. Some methods encode spatial context through local neighborhoods or grids. STDec smooths a threshold map over decoded neighborhoods using Gaussian weights, with radius $r=2$ for 1D text and a local lattice neighborhood for 2D grids (Chen et al., 7 Apr 2026). GUIPruner uses Stratified Structure-aware Pruning to partition current-frame tokens into foreground and background, retain salient subsets from both, and then allocate the residual budget to a uniform structural grid so that the patch-lattice topology and M-RoPE positional encoding are preserved (Xu et al., 26 Feb 2026). V-CAST preserves original token coordinates under MRoPE by selecting Top-$b_t$ tokens with the largest dual-anchor scores and keeping them at their original $(t,h,w)$ coordinates rather than merging them (Lin et al., 29 Mar 2026).

Temporal reasoning is also heterogeneous. Some works use temporal stability or consistency, such as the ID-consistency streak $K_t(i)$ in STDec (Chen et al., 7 Apr 2026). Others use recency, motion, or inter-frame change. ST-Prune’s Motion-aware Temporal Pruning uses motion volatility $m_{v,t,i}$ and exponential temporal recency $\tau_t^{s}(i)$0 to prioritize dynamic trajectories and current-frame content (Sha et al., 21 Apr 2026). GUIPruner’s Temporal-Adaptive Resolution assigns more tokens to recent frames through a linear decay schedule aligned with the agent’s “fading memory” attention pattern (Xu et al., 26 Feb 2026). V-CAST treats the sequence of normalized frame summaries as a semantic trajectory on $\tau_t^{s}(i)$1 and routes per-frame budgets by curvature so that semantic turns and event boundaries receive more tokens (Lin et al., 29 Mar 2026).

A third recurring principle is that spatial and temporal signals are usually complementary rather than interchangeable. STDec’s ablations show that temporal-only and spatial-only decoding each improve throughput, while combining both gives the strongest overall result on HumanEval and GSM8K (Chen et al., 7 Apr 2026). ST-Prune similarly reports that adding MTP or RSP individually improves performance and that the combined gain is approximately $\tau_t^{s}(i)$2 on the NuInstruct+OmniDrive average (Sha et al., 21 Apr 2026). GUIPruner reports that TAR adaptive decay outperforms uniform scaling at the same budgets, while SSP’s uniform grid sampling outperforms random sampling, indicating that temporal prioritization and topology-preserving spatial retention solve different failure modes (Xu et al., 26 Feb 2026).

3. Representative formulations across domains

The term PSTTS spans several operational regimes rather than one homogeneous algorithm.

Family	Representative mechanism	Primary setting
Stability-guided commitment	Token-adaptive thresholds from spatial smoothing and temporal ID consistency	Diffusion LLM decoding
Query-guided top-$\tau_t^{s}(i)$3 sampling	Relevance scoring conditioned on query, then top-$\tau_t^{s}(i)$4 token retention	Video grounding and VQA
Training-free pruning	Motion, recency, geometry, or structural priors drive greedy or thresholded pruning	Driving, GUI, VLN, event data
Stage-adaptive depth pruning	Pre-LLM compression plus progressive inner-LLM pruning and late-token removal	Omni-modal LLMs

In diffusion LLM decoding, STDec formalizes a masked sequence $\tau_t^{s}(i)$5, with masked positions $\tau_t^{s}(i)$6 and decoded positions $\tau_t^{s}(i)$7, and replaces a single global confidence threshold with token-adaptive thresholds $\tau_t^{s}(i)$8 derived from spatial smoothing and temporal relaxation. The commit set is

$\tau_t^{s}(i)$9

with deadlock resolution by force-committing the most confident token (Chen et al., 7 Apr 2026). This is an instance of PSTTS in which “selection” means irreversible token commitment rather than pruning.

In video temporal grounding, GroundVTS performs query-guided Visual Token Sampling. It projects visual tokens and a pooled query into relevance subspaces, computes token weights by temperature-scaled dot products, applies differentiable top-$D_t$0 with Gumbel-Softmax and a Straight-Through Estimator, then reweights and masks the selected tokens:

$D_t$1

The selected tokens remain temporally aligned because GroundVTS reuses original positional encodings from dense sampling and masks only those of unselected visual tokens (Fan et al., 2 Apr 2026).

In autonomous driving, ST-Prune instantiates PSTTS with a temporal-first pipeline. Motion-aware Temporal Pruning selects $D_t$2 tokens per view from $D_t$3 candidates using a greedy max-min routine weighted by motion and recency, and Ring-view Spatial Pruning then reduces these to $D_t$4 tokens per view by penalizing bilateral cross-view similarity under ring-view adjacency (Sha et al., 21 Apr 2026). In GUI navigation, GUIPruner achieves source-level temporal compression by resizing history frames according to

$D_t$5

followed by stratified pruning of the current frame with foreground, salient background, and uniform grid components (Xu et al., 26 Feb 2026).

In omni-modal LLMs, SEATS combines window-local attention-weighted diversity selection before the LLM with stage-adaptive pruning inside the LLM. The pre-LLM retention ratios are $D_t$6 and $D_t$7, while middle-layer budgets are allocated across windows and modalities using cross-attention relevance from the last text token (Xin et al., 19 May 2026). In event-based learning, PSTTS consists of Spatial Token Purification and Temporal Token Selection driven by raw event statistics rather than learned scorers, specifically to avoid the unreliability of noisy intermediate event representations (Zhao et al., 26 Sep 2025).

4. Mathematical mechanisms and token-scoring strategies

A notable property of the PSTTS literature is that the scoring variable changes with the application.

In STDec, the key variables are the prediction $D_t$8, the confidence $D_t$9, the spatially smoothed threshold

$K_t(i)$0

and the temporal streak

$K_t(i)$1

Temporal relaxation is applied when $K_t(i)$2 and $K_t(i)$3, yielding $K_t(i)$4 (Chen et al., 7 Apr 2026). Here, the “score” is not an importance value but a thresholded confidence conditioned on local spatial and temporal stability.

In GroundVTS, the scoring function is explicitly query-conditioned. The paper defines

$K_t(i)$5

then samples a hard top-$K_t(i)$6 mask through Gumbel-Softmax and the STE, and forms

$K_t(i)$7

This scoring is global over all visual tokens rather than per-frame, which allows simultaneous spatial and temporal redundancy removal (Fan et al., 2 Apr 2026).

Other methods rely on geometry- or trajectory-based priors. V-CAST computes normalized frame summaries $K_t(i)$8, directional differences, and curvature

$K_t(i)$9

then solves an entropy-regularized allocation problem whose closed-form solution is

$\tau_t^{st}(i)$0

Within each frame, tokens receive a dual-anchor score

$\tau_t^{st}(i)$1

which combines contextual diversity and feature activation saliency (Lin et al., 29 Mar 2026).

ST-Prune’s temporal score is based on motion volatility, temporal recency, and content salience. The paper gives

$\tau_t^{st}(i)$2

and uses these inside a greedy diversity objective implemented by a max-min routine (Sha et al., 21 Apr 2026). GUIPruner instead fixes budgets analytically via decay-based resizing and then uses TopK selection inside current-frame strata, with

$\tau_t^{st}(i)$3

and residual uniform grid sampling to preserve the layout skeleton (Xu et al., 26 Feb 2026).

A distinct branch uses current-context-conditioned memory selection. In VLN, history-conditioned spatio-temporal pruning computes base importance from cosine similarity between a [CLS] token and each patch token,

$\tau_t^{st}(i)$4

then applies Adaptive Maximal Marginal Relevance to the current frame and reweights historical tokens by relevance to the selected current tokens:

$\tau_t^{st}(i)$5

This yields a progressive memory pool $\tau_t^{st}(i)$6 updated at every navigation step (Wang et al., 6 Mar 2026).

These differences show that PSTTS does not imply a single scoring functional. It implies a structured decomposition of token utility into spatial, temporal, and often task-conditional components.

5. Empirical behavior and reported gains

Reported gains are substantial but domain-specific. In STDec, the primary effect is decoding throughput. On MBPP with LLaDA-8B-Instruct, throughput improves from $\tau_t^{st}(i)$7 to $\tau_t^{st}(i)$8 TPS, a $\tau_t^{st}(i)$9 speedup, while the score changes from $r=2$0 to $r=2$1; on HumanEval with the same model, throughput improves from $r=2$2 to $r=2$3 TPS with score $r=2$4 (Chen et al., 7 Apr 2026). STDec also stacks effectively with caching: LLaDA + dKV-Cache + STDec yields additional $r=2$5 on GPQA, $r=2$6 on GSM8K, and $r=2$7 on MBPP over cache-only baselines (Chen et al., 7 Apr 2026).

GroundVTS emphasizes token efficiency and grounding accuracy. On Charades-STA, GroundVTS-Q reaches $r=2$8 [email protected]/0.5/0.7 and $r=2$9 mIoU, improving over Qwen2.5VL-7B-G by $b_t$0 [email protected] and $b_t$1 mIoU. On QVHighlights highlight detection, GroundVTS-I reaches $b_t$2 mAP and $b_t$3 Hit@1, improving over InternVL3.5-8B-G by $b_t$4 and $b_t$5 (Fan et al., 2 Apr 2026). The abstract-level summary reports a $b_t$6-point improvement in mIoU for moment retrieval and $b_t$7-point improvement in mAP for highlight detection (Fan et al., 2 Apr 2026).

Training-free pruning methods in embodied or multimodal settings often emphasize near-lossless retention under high pruning. ST-Prune reports that even at $b_t$8 token reduction it achieves near-lossless performance, with some metrics surpassing the full-model baseline; at $b_t$9 retention, end-to-end speedups are approximately $(t,h,w)$0 on NuInstruct and approximately $(t,h,w)$1 on OmniDrive (Sha et al., 21 Apr 2026). GUIPruner reports that on Qwen2-VL-2B with AITW and $(t,h,w)$2 history frames, tokens drop from $(t,h,w)$3 to $(t,h,w)$4, total FLOPs from $(t,h,w)$5T to $(t,h,w)$6T, and vision encoder latency from $(t,h,w)$7 ms to $(t,h,w)$8 ms, with over $(t,h,w)$9 of the original performance retained under aggressive history retention (Xu et al., 26 Feb 2026). In omni-modal LLMs, SEATS at $K_t(i)$0 on Qwen2.5-Omni-7B reduces TFLOPs from $K_t(i)$1 to $K_t(i)$2, improves prefill from $K_t(i)$3 s to $K_t(i)$4 s, and retains $K_t(i)$5 of the original performance; the paper summarizes this as a $K_t(i)$6 FLOPs reduction and a $K_t(i)$7 prefill speedup while preserving $K_t(i)$8 of performance (Xin et al., 19 May 2026).

PSTTS-like methods also show strong behavior in longer-context video settings. V-CAST on Qwen3-VL-8B-Instruct at $K_t(i)$9 and Max $m_{v,t,i}$0 frames achieves $m_{v,t,i}$1 average versus $m_{v,t,i}$2 baseline, which the paper describes as $m_{v,t,i}$3 of the original performance, while reducing peak memory and total latency to $m_{v,t,i}$4 and $m_{v,t,i}$5 of vanilla Qwen3-VL-8B-Instruct (Lin et al., 29 Mar 2026). Unified Spatio-Temporal Token Scoring, although not described as progressive, reports $m_{v,t,i}$6 pruning across both ViT and LLM, a $m_{v,t,i}$7 efficiency improvement during training and inference, and only a $m_{v,t,i}$8 drop in average performance across $m_{v,t,i}$9 short and long video QA tasks (Zhang et al., 18 Mar 2026). PVC reports that with a fixed budget of $\tau_t^{s}(i)$00 tokens per frame, its unified progressive compression preserves image performance while achieving state-of-the-art results across multiple video benchmarks (Yang et al., 2024).

These results do not imply that every PSTTS variant dominates every baseline on every task. Several papers report small drops under aggressive compression, such as the $\tau_t^{s}(i)$01 drop on MBPP long-sequence decoding for STDec and the modest top-1 drops reported by event-based PSTTS under FLOPs reductions of $\tau_t^{s}(i)$02–$\tau_t^{s}(i)$03 (Chen et al., 7 Apr 2026). The consistent empirical pattern is not zero-loss compression, but a favorable efficiency–accuracy frontier.

6. Ambiguities, limitations, and open directions

One source of ambiguity is terminological. Some papers explicitly use PSTTS; others are mapped to PSTTS only “through the lens” of progressive spatio-temporal token selection. “Unified Spatio-Temporal Token Scoring for Efficient Video VLMs” states directly that it “does not use the term ‘progressive,’ nor does it claim layer-wise or iterative pruning over multiple stages,” even though its unified scorer and packing algorithm could, in principle, be adapted into a progressive scheme (Zhang et al., 18 Mar 2026). “Static or Dynamic: Towards Query-Adaptive Token Selection for Video Question Answering” similarly notes that its method is “two-stage but not explicitly progressive in the sense of layer-wise or time-window gradual selection” (Shi et al., 30 Apr 2025). An objective reading is therefore that PSTTS names a broader design family, not a universally agreed formalism.

A second recurring issue is coverage failure under aggressive sparsification. GroundVTS notes that if critical evidence is extremely subtle or distant, top-$\tau_t^{s}(i)$04 selection may under-sample it, especially when $\tau_t^{s}(i)$05 is small, and that the absence of explicit continuity constraints may occasionally fragment long moments (Fan et al., 2 Apr 2026). GUIPruner notes that extremely aggressive compression can lose fine-grained pixel cues and risk misclicks on small widgets (Xu et al., 26 Feb 2026). ST-Prune notes that minimal-motion scenes, heavy occlusions, or camera miscalibration can weaken the reliability of motion volatility or ring-view similarity, requiring more conservative $\tau_t^{s}(i)$06, larger $\tau_t^{s}(i)$07, or even disabling RSP in extreme cases (Sha et al., 21 Apr 2026). Event-based PSTTS identifies low-motion scenes and sensor-noise spikes as failure cases where adaptive thresholds may need retuning (Zhao et al., 26 Sep 2025).

A third issue concerns positional integrity. Several papers argue that naive token dropping or merging can damage positional reasoning. GUIPruner attributes collapse under unstructured pruning to broken patch-lattice topology and M-RoPE grounding (Xu et al., 26 Feb 2026). V-CAST makes this point especially strongly for MRoPE-based VideoLLMs, arguing that token merging shifts tokens off the discrete $\tau_t^{s}(i)$08 grid and weakens spatio-temporal alignment, whereas pruning at original coordinates preserves it (Lin et al., 29 Mar 2026). This suggests that a central controversy in PSTTS-like design is not simply how many tokens to keep, but whether compression may alter the coordinate system on which later reasoning depends.

Open directions recur across the surveyed papers. STDec points to learned or adaptive neighborhood graphs, dynamic $\tau_t^{s}(i)$09 and adaptive $\tau_t^{s}(i)$10, and reinforcement learning or bandit optimization for threshold scheduling (Chen et al., 7 Apr 2026). GroundVTS suggests sparse gates such as sparsemax or entmax, reinforcement-learning-based selection, multi-stage sampling, and segment-level grouping (Fan et al., 2 Apr 2026). SEATS identifies automatic adaptation of stage boundaries, streaming inference, learned relevance estimators, and adaptive fusion detection as open problems (Xin et al., 19 May 2026). More generally, this suggests that future PSTTS research will likely focus on replacing hand-designed schedules and fixed heuristics with policies that remain plug-and-play enough to preserve deployment efficiency while being more adaptive to input difficulty, modality balance, and architectural idiosyncrasies.