Spatial Effective Token Selection (SETS)

• Spatial Effective Token Selection (SETS) is a framework that retains only the most useful visual tokens by balancing saliency, diversity, and spatial coherence.
• It reduces computational load in vision-language models and video transformers by eliminating redundant and spatially ineffective tokens.
• Empirical studies show that SETS improves performance in anomaly detection and multimodal reasoning tasks through efficient token pruning.

Spatial Effective Token Selection (SETS) denotes a family of mechanisms for retaining only those visual tokens that are most useful for downstream multimodal or video reasoning while suppressing redundant, irrelevant, or spatially ineffective tokens. In the strictest usage within the cited literature, SETS is the spatial module of VA-GPT, where patch tokens are selected by ranking inter-frame DINOv2 feature differences and keeping the top-$K$ ratio per frame (Chen et al., 8 Aug 2025). In a broader and increasingly useful sense, SETS functions as an organizing concept for methods that jointly address saliency, redundancy, and spatial coherence in visual token reduction, especially in vision-LLMs (VLMs), video transformers, and multimodal LLMs (Wu et al., 2 Dec 2025).

1. Conceptual basis and problem setting

SETS arises from a common efficiency bottleneck: visual encoders generate many more tokens than text encoders, and these tokens are then processed by self-attention whose cost scales quadratically with sequence length. The resulting burden appears in compute, memory, latency, and context-length pressure across VLMs, video transformers, and MLLMs (Wu et al., 2 Dec 2025). The motivating observation is not merely that many tokens are unimportant, but that visual token redundancy has multiple forms. The literature repeatedly distinguishes irrelevant regions, duplicated evidence around the same object part, and selections that are diverse in feature space yet spatially ineffective for object coverage (Wu et al., 2 Dec 2025).

A central SETS claim is therefore that importance-only pruning is insufficient. Attention or saliency scores can repeatedly preserve similar tokens around the same locally important region, wasting token budget on near-duplicate evidence. The converse extreme is also inadequate: diversity-only selection can become overly dispersed, preserving dissimilar but spatially remote tokens that fail to densely cover target objects or preserve fine-grained local detail (Wu et al., 2 Dec 2025). In the most explicit formulation of this view, a good selector must simultaneously identify salient content, avoid selecting duplicates, and maintain spatially coherent coverage of target regions (Wu et al., 2 Dec 2025).

This concern with spatial integrity is sharpened by work on grounding-sensitive VLMs. Nüwa argues that pruning cannot be judged only by semantic redundancy or attention importance, because grounding depends on preserving a “Global Spatial Reference Frame” derived from the integrity of the retained token set and their positional interactions (Huang et al., 3 Feb 2026). TOPS provides a complementary first-principles account: effective visual token preservation should balance Task Relevance, Information Coverage, and Semantic Diversity, even though TOPS itself is not explicitly geometry-aware (Wang et al., 25 Jun 2026). Taken together, these works define SETS less as a single algorithm than as a design problem: selecting a compact subset of visual tokens that remains semantically informative, non-redundant, and spatially usable for reasoning.

2. Canonical formulation in VA-GPT

The clearest explicit definition of SETS appears in VA-GPT, which addresses video anomaly understanding under the premise that abnormal events are sparse in both space and time (Chen et al., 8 Aug 2025). For each frame $V^t$, a frozen CLIP encoder produces visual tokens

$X^t=\{x_i^t\}_{i=1,\dots,N},$

and a DINOv2 feature extractor $\mathcal{F}_E$ produces patch embeddings

$$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$

SETS then computes a patch-wise inter-frame difference map

$D^t = dis(F^t, F^{t-1}),$

where $dis(\cdot)$ is the Manhattan distance. A binary mask

$M^t = [m_1^t,m_2^t,\dots,m_N^t]$

is formed by assigning value $1$ to the top $K$ ratio of patches with largest distances and $V^t$0 elsewhere, and the selected Spatial Effective Tokens are

$V^t$1

In this formulation, a token is “effective” if its patch representation changes strongly between adjacent frames (Chen et al., 8 Aug 2025).

Several properties of this canonical SETS definition are notable. The mechanism is deterministic, non-parametric, and patch-level. It does not rely on learned gating, clustering, saliency supervision, or cross-attention to text. It introduces no trainable parameters for SETS itself and no auxiliary loss for the selection mask. Instead, it pre-filters the visual token set so that downstream modality alignment and reasoning operate on anomaly-relevant spatial content rather than the full frame (Chen et al., 8 Aug 2025).

Empirical evidence in the same paper shows that SETS contributes materially to anomaly understanding. On UCF-Crime, the baseline without fine-tuning and without SETS/TETG yields $V^t$2 total accuracy and $V^t$3 temporal accuracy; adding only SETS raises these to $V^t$4 and $V^t$5. Under Stage Two fine-tuning with SETS alignment, SETS reaches $V^t$6 total and $V^t$7 temporal accuracy, while the full SETS+TETG system reaches $V^t$8 and $V^t$9. The best reported selection ratio is $X^t=\{x_i^t\}_{i=1,\dots,N},$0, with too small a ratio losing useful information and too large a ratio reintroducing noise and redundancy (Chen et al., 8 Aug 2025).

3. Methodological families in SETS-related research

The literature surrounding SETS spans learned selection, training-free pruning, geometry-guided diversification, and spatially preserving merging. Although these methods do not share a single formalism, they converge on the problem of selecting or compressing spatial tokens without destroying the information needed for downstream reasoning.

Method	Core selection principle	Spatial mechanism
TS2-Net (Liu et al., 2022)	Per-frame top-$X^t=\{x_i^t\}_{i=1,\dots,N},$1 token selection conditioned on $X^t=\{x_i^t\}_{i=1,\dots,N},$2 context	Selects tokens that “contribute most to local spatial semantics”
STTS (Wang et al., 2021)	Ranking plus Top-K with differentiable perturbed maximum	Anchor-based region selection to preserve coherent spatial context
LaVida Drive (Jiao et al., 2024)	Query-aware hard top-$X^t=\{x_i^t\}_{i=1,\dots,N},$3 over high-resolution image tokens	Text-conditioned spatial pruning with post-selection enhancement
VLM-Pruner (Wu et al., 2 Dec 2025)	Training-free centrifugal pruning with redundancy control	Buffering for Spatial Sparsity (BSS) enforces near-to-far spatial growth
SeGPruner (Li et al., 31 Mar 2026)	Attention-based saliency plus geometry-guided diversification	Uses token-level 3D coordinates and geometric distance
Nüwa (Huang et al., 3 Feb 2026)	Two-stage pruning with region-wise anchors and text-guided filtering	Preserves global spatial anchors and spatial integrity
SPA / SPT (Zhang et al., 2024)	Supervised dynamic token gating and packing	Multi-scale spatial labels from masks or boxes
ToSA (Huang et al., 24 Jun 2025)	Token merging with fused semantic and spatial affinity	Uses depth-derived pseudo spatial tokens
CubistMerge (Gong et al., 26 Sep 2025)	Local token merging under 2D structural constraints	Preserves valid 2D token geometry for spatial backbones
Perceptio (Li et al., 19 Mar 2026)	Generates explicit spatial tokens rather than pruning	Emits segmentation and depth tokens before answering

Two broad distinctions are especially important. First, some methods are explicitly query-conditioned, such as LaVida Drive, which scores image tokens by image-text similarity and applies hard top-$X^t=\{x_i^t\}_{i=1,\dots,N},$4 selection, whereas others are query-agnostic at the selection stage, such as SeGPruner’s visual self-attention saliency and geometry-guided diversification (Jiao et al., 2024, Li et al., 31 Mar 2026). Second, some methods prune or select original tokens, whereas others merge or generate new structured tokens. ToSA is merge-based and uses depth-derived pseudo spatial tokens to guide reduction, while Perceptio sidesteps pruning by forcing the LVLM to generate explicit segmentation and depth tokens inside the autoregressive sequence (Huang et al., 24 Jun 2025, Li et al., 19 Mar 2026).

A further systems-level branch is represented by SPA, which couples dynamic token selection with token packing so that variable token counts remain efficient under GPU batch processing. Its significance for SETS lies not only in the gating criterion but in the realization that variable-cardinality spatial token selection must also be operationally efficient during training and inference (Zhang et al., 2024).

4. Spatial coherence, integrity, and geometry

The decisive issue in SETS is not only which tokens are important, but whether the retained set preserves a usable spatial structure. VLM-Pruner provides one of the clearest formulations of this requirement. It defines a visual token grid with coordinates

$X^t=\{x_i^t\}_{i=1,\dots,N},$5

and spatial distance

$X^t=\{x_i^t\}_{i=1,\dots,N},$6

Its Buffering for Spatial Sparsity criterion modulates feature similarity using grid-space distance so that selection expands in a near-to-far manner, preserving local object detail before reaching farther regions (Wu et al., 2 Dec 2025). This is an explicit response to the failure mode in which redundancy-aware methods become too globally dispersed.

Nüwa generalizes the same concern under the term “spatial integrity.” It argues that grounding depends on a coherent global spatial reference frame, and that token-token interactions, not only token coordinates, are essential for localization. The paper reports that blocking vision-to-vision interactions in critical LLM middle layers causes grounding performance on RefCOCO-test to collapse from $X^t=\{x_i^t\}_{i=1,\dots,N},$7 to $X^t=\{x_i^t\}_{i=1,\dots,N},$8, which directly supports the claim that spatial reasoning requires structurally meaningful retained visual tokens rather than isolated semantics (Huang et al., 3 Feb 2026).

STTS reaches a similar conclusion in hierarchical video transformers. Rather than prune arbitrary patch tokens, it selects a contiguous anchor region. The reported contrast is severe: random token-based spatial selection yields $X^t=\{x_i^t\}_{i=1,\dots,N},$9, whereas random anchor-based spatial selection yields $\mathcal{F}_E$0, demonstrating that preserving a coherent local region is radically different from selecting scattered tokens in flattened token space (Wang et al., 2021). Good Token Hunting extends this logic to multi-view geometry transformers by separating inter-frame diversity selection from intra-frame sparsification and by showing that layer-aware sparsification is necessary because the entropy of global attention changes across depth (Zheng et al., 22 May 2026).

Geometry-aware SETS variants make this spatial prior explicit. SeGPruner assigns each multi-view token a 3D coordinate

$\mathcal{F}_E$1

via depth maps and camera poses, then combines a Saliency-aware Token Selector with a Geometry-aware Token Diversifier so that token reduction preserves both object-critical evidence and scene-level spatial coverage (Li et al., 31 Mar 2026). This suggests that, once 3D structure is available, spatial effectiveness can be defined not only in image-grid terms but in a unified world coordinate frame.

5. Applications and empirical behavior

SETS-style mechanisms have been applied across anomaly understanding, VLM acceleration, 3D question answering, omni-modal inference, event-based video understanding, and first-person spatial reasoning. In anomaly video analysis, the canonical SETS module of VA-GPT improves total anomaly understanding accuracy substantially while operating on only the top-$\mathcal{F}_E$2-ratio spatial tokens per frame (Chen et al., 8 Aug 2025). In efficient VLM pruning, VLM-Pruner reports consistent gains across five VLMs at an $\mathcal{F}_E$3 pruning rate while still delivering end-to-end inference speedup, which places spatially coherent selection in a regime of very aggressive token reduction (Wu et al., 2 Dec 2025).

In 3D question answering, SeGPruner reduces the visual token budget by $\mathcal{F}_E$4 and inference latency by $\mathcal{F}_E$5 while maintaining competitive performance, showing that spatial coverage remains critical even when the underlying visual representation is assembled from many overlapping 2D views (Li et al., 31 Mar 2026). TOPS, although not explicitly spatial in formulation, demonstrates that strong retention can be achieved at high compression rates: on LLaVA-NeXT it removes $\mathcal{F}_E$6 of visual tokens while preserving $\mathcal{F}_E$7 and $\mathcal{F}_E$8 performance on the 7B and 13B models, respectively (Wang et al., 25 Jun 2026).

Related omni-modal work further broadens the scope of SETS-like design. SEATS, a distinct acronym from SETS, applies stage-adaptive token selection to audio-visual om-LLMs and reports that retaining only $\mathcal{F}_E$9 of visual and audio tokens yields a $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$0 FLOPs reduction and a $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$1 prefill speedup while preserving $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$2 of the original performance (Xin et al., 19 May 2026). In event-based representation learning, PSTTS removes spatio-temporally redundant tokens using raw event statistics, reducing FLOPs by $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$3–$$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$4 and increasing FPS by $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$5–$$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$6 on DailyDVS-200 while maintaining task accuracy (Zhao et al., 26 Sep 2025). EventPrune reaches a similar conclusion in first-person spatial reasoning: with $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$7 visual token reduction, it surpasses the full-token baseline ($$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$8 versus $$F^t, F^{t-1} = \mathcal{F}_E(V^t), \mathcal{F}_E(V^{t-1}), \qquad F^t, F^{t-1} \in \mathbb{R}^{N\times C}.$$9), achieves $D^t = dis(F^t, F^{t-1}),$0 inference speedup, and reduces GFLOPs by $D^t = dis(F^t, F^{t-1}),$1 (Ma et al., 19 May 2026).

Across these results, a recurrent empirical pattern appears. Spatially informed pruning can match or surpass full-token baselines not only by reducing cost but by suppressing noisy, duplicated, or structurally misleading tokens. This pattern is explicit in VLM-Pruner, TOPS, SEATS, and EventPrune, even though each paper defines token effectiveness differently (Wu et al., 2 Dec 2025, Wang et al., 25 Jun 2026, Xin et al., 19 May 2026, Ma et al., 19 May 2026).

6. Misconceptions, boundaries, and likely directions

A common misconception is that SETS is synonymous with any token pruning method. The literature does not support this. Some methods rely only on importance ranking, some on diversity, some on geometry, some on query conditioning, and some on explicit preservation of spatial integrity. SETS is therefore better understood as a stricter requirement: token reduction must remain spatially effective for the target task, not merely sparse.

A second misconception is that SETS is always a hard selection problem over existing patch tokens. Perceptio is an explicit counterexample: it improves spatial grounding by generating a special segmentation token and a sequence of depth tokens inside the autoregressive output, rather than selecting a subset of visual encoder tokens (Li et al., 19 Mar 2026). “Tokenize Image as a Set” is another adjacent but conceptually different departure: it abandons fixed spatial ordering and treats image tokens as an unordered multiset, redistributing coding capacity according to semantic complexity rather than explicit keep/drop decisions (Geng et al., 20 Mar 2025). These works imply that “effective spatial tokens” can arise through generation or representation change, not only through pruning.

A third misconception is that irreversible pruning is always desirable. Token Sparse Attention in long-context LLMs shows a different design point: per-head token subsets are selected inside attention, dense attention is run on the compressed set, and the result is then scattered back so that skipped tokens remain available in later layers (Jo et al., 3 Feb 2026). A plausible implication is that future SETS variants for multimodal transformers may combine spatial priors with reversible, interleaved token selection rather than committing to one-shot deletion.

The boundary between SETS and spatially preserving merging also remains important. CubistMerge and ToSA show that some tasks, especially dense prediction and early-layer spatial reasoning, benefit from structure-preserving merging rather than hard deletion (Gong et al., 26 Sep 2025, Huang et al., 24 Jun 2025). SPA shows the complementary lesson that supervised spatial selection can be effective but depends on annotation-derived selection labels and systems support for packed execution (Zhang et al., 2024). PSTTS, by contrast, demonstrates that a modality-native spatial selector can be parameter-free and grounded in raw event statistics rather than learned saliency (Zhao et al., 26 Sep 2025).

This suggests that the future of SETS is unlikely to converge on a single criterion. The evidence instead points toward hybrid selectors that combine task relevance, redundancy control, spatial integrity, and execution-aware design. In the literature considered here, the decisive question is no longer whether tokens should be reduced, but how to reduce them without sacrificing the spatial reference frame, object coverage, and local structure on which multimodal reasoning depends.