Sparsity-Aware Event Token Selection

• Sparsity-aware event token selection is a method that dynamically decides which event-derived tokens to create, retain, merge, or skip, based on motion cues and temporal novelty.
• It leverages diverse saliency signals—such as spike activity, spatio-temporal continuity, and event density—to guide adaptive token creation and pruning within transformer architectures.
• This approach enhances efficiency by reducing FLOPs and boosting throughput while preserving accuracy in both event-only and multimodal data processing.

Sparsity-aware event token selection is the problem of deciding which event-derived tokens should be created, retained, merged, skipped, or revisited so that computation concentrates on motion-informative, temporally novel, or structurally reliable regions rather than on empty background, noise, or redundant repetitions. In the recent literature, the topic spans native event-token formulations, threshold-triggered token emission, spike-activity-guided pruning, temporal-window aggregation, density-guided spatial sparsification, event-assisted pruning of RGB tokens, and training-free attention-time filtering inside large event or multimodal transformers (Jiang et al., 2022, Øhrstrøm et al., 30 Oct 2025, Liu et al., 2023, Liu et al., 3 Feb 2026).

1. Scope and conceptual boundaries

Work in this area does not use a single token ontology. “Event Transformer” treats each raw event $e_i=(\vec{u}_i,t_i,p_i)$ as an event-token and embeds an input of size $N\times 4$ into $N\times C$, preserving event-level spatiotemporal structure before local temporal, sparse spatial, and global processing (Jiang et al., 2022). “Spiking Patches” instead makes token existence conditional on local event accumulation: each spatial patch behaves like an integrate-and-fire unit, and a token is emitted only when the patch potential reaches threshold $\sigma$; an optional refractory period $T$ further suppresses redundant token emissions (Øhrstrøm et al., 30 Oct 2025).

Later systems often begin from temporally binned or voxelized event inputs and then sparsify those higher-level tokens. EventFlash first divides the stream into fine-grained temporal bins and later merges adjacent bins into meta event windows $M_i$; Match-Any-Events uses multi-timescale voxel tokens $\mathbf t\in\mathbb R^{T\times H\times W\times D}$; FocusMamba tokenizes RGB and event voxel inputs before stage-wise sparsification; PSTTS starts from sequences of event frames and prunes patch tokens inside them (Liu et al., 3 Feb 2026, Zhang et al., 20 Apr 2026, Yang et al., 4 Sep 2025, Zhao et al., 26 Sep 2025).

Not every neighboring method is a selector in the strict sense. EvRainDrop is best interpreted as a sparse token completion / relation modeling method rather than a token selection method: the supplied text supports “HyperGraph-guided Completion” and “frame and event stream aggregation,” but not an explicit top-$k$, adaptive dropping, or sparsity-aware routing module (Wang et al., 26 Nov 2025).

2. Selection signals and decision rules

The literature uses several distinct saliency signals: spike activity, raw-event spatio-temporal continuity, event-density or occupancy statistics, learned halting probabilities, and motion saliency projected from events onto visual tokens. These signals differ not only in what they score, but also in when the decision is made: before token creation, before attention, inside attention, or after temporal aggregation.

In spiking transformers, the salient signal is often the token’s own spike activity. SparseSpikformer inserts a Spiking Token Selector before self-attention, after the 2nd, 3rd, and 4th encoder layers. Given $X\in\mathbb R^{T\times N\times D}$, it computes $X'=\mathrm{GAP}(X)\in\mathbb R^{N\times D}$, then predicts keep/drop scores with $N\times 4$0, and obtains a hard binary decision with Gumbel-Softmax. The cumulative mask is updated by $N\times 4$1, so a token dropped once remains dropped in later stages; the paper describes progressive keep ratios such as $N\times 4$2 (Liu et al., 2023).

In RGB-event detection, FocusMamba defines event saliency through spatiotemporal continuity rather than raw count alone. It first accumulates timestamps at each spatial location, $N\times 4$3, aligns the result to the current token grid with max-pooling, and then applies Gaussian neighborhood aggregation to obtain $N\times 4$4. A sample-level event spatial ratio $N\times 4$5 controls both score sharpening and thresholding through $N\times 4$6, $N\times 4$7, and $N\times 4$8; the final hard masks $N\times 4$9 and $N\times C$0 are produced by thresholding normalized RGB and event scores against $N\times C$1 (Yang et al., 4 Sep 2025).

PSTTS deliberately avoids intermediate-feature token scores in favor of raw-event spatio-temporal statistics. Spatial Token Purification subdivides each event segment into $N\times C$2 fine temporal bins, builds an event density sequence $N\times C$3, drives a Leaky Integrate-and-Fire neuron to obtain temporal continuity $N\times C$4, and then computes a neighborhood-weighted spatio-temporal continuity map $N\times C$5, which is pooled to patch resolution as $N\times C$6. Temporal Token Selection compares retained patches across adjacent frames with motion magnitude similarity $N\times C$7 and trajectory shape similarity $N\times C$8, combines them as $N\times C$9, and ranks tokens by $\sigma$0 (Zhao et al., 26 Sep 2025).

3. Temporal sparsification, halting, and token creation

Some methods move selection to the token-creation stage itself. Spiking Patches partitions the sensor plane into non-overlapping $\sigma$1 patches, treats each patch as an integrate-and-fire unit with $\sigma$2, $\sigma$3, and $\sigma$4, sets $\sigma$5 and $\sigma$6, and emits a token only when local activity crosses threshold $\sigma$7. With an absolute refractory period $\sigma$8, token rate can be reduced further; on GEN1, $\sigma$9 ms reduces input size by $T$0 while preserving 96.7% of the original mAP (Øhrstrøm et al., 30 Oct 2025).

EventFlash performs temporal sparsification as adaptive merging of short bins before language modeling. For bin $T$1, it defines an asynchronous spatiotemporal intensity $T$2, compares adjacent bins with $T$3, merges bins whose distance is below threshold $T$4, and then refines the merged windows with a semantic-and-density score $T$5. The resulting compressed temporal sequence is passed to spatial sparsification; the full system reports a 12.4× throughput improvement over EventFlash-Zero, while temporal sparsification alone gives 6.1× and spatial sparsification alone gives 2.3× (Liu et al., 3 Feb 2026).

Match-Any-Events introduces Sparsity-aware Event Token Selection as an adaptive halting mechanism over multi-timescale voxel tokens $T$6. For each token $T$7, an MLP plus sigmoid predicts a halting score $T$8, a remainder term $T$9 preserves differentiability, and the bias injected into spatial attention is $M_i$0 for active steps and $M_i$1 after halting. The paper states that this reduces spatial attention from $M_i$2 to $M_i$3, and reports a 21.5% reduction in FLOPs for spatial attention operations, from 62.22G to 48.87G, with minimal impact on overall performance (Zhang et al., 20 Apr 2026).

4. Integration into attention architectures

Event-based selectors are often integrated before attention rather than as post-hoc masks. SAST performs window-token co-sparsification inside each sparse transformer layer: the scoring module computes $M_i$4, with $M_i$5 and $M_i$6 derived from the scene-level event sparsity descriptor $M_i$7; a competition unit turns these into token and window scores $M_i$8 and $M_i$9, and selection uses thresholds $\mathbf t\in\mathbb R^{T\times H\times W\times D}$0 and $\mathbf t\in\mathbb R^{T\times H\times W\times D}$1. Only selected windows and selected tokens are sent to Masked Sparse Window Self-Attention, $\mathbf t\in\mathbb R^{T\times H\times W\times D}$2, so sparsity reduces actual attention and MLP computation rather than only suppressing outputs (Peng et al., 2024).

EventPrune moves event guidance into a multimodal cascade for first-person spatial reasoning. Event-Triggered Causal Sampling uses event activity flux $\mathbf t\in\mathbb R^{T\times H\times W\times D}$3 and temporal change $\mathbf t\in\mathbb R^{T\times H\times W\times D}$4 to anchor keyframes; Event-guided Motion Saliency Filtering maps filtered events onto RGB patches via $\mathbf t\in\mathbb R^{T\times H\times W\times D}$5 and retains the top $\mathbf t\in\mathbb R^{T\times H\times W\times D}$6 tokens; Event-Attention Ranking Fusion then combines text-conditioned attention rank and event-saliency rank as $\mathbf t\in\mathbb R^{T\times H\times W\times D}$7 (Ma et al., 19 May 2026).

In FocusMamba, Event-Guided Multimodal Sparsification is inserted before each backbone stage, Sparse VSS layers execute only on tokens with $\mathbf t\in\mathbb R^{T\times H\times W\times D}$8 or $\mathbf t\in\mathbb R^{T\times H\times W\times D}$9, and Cross-Modality Focus Fusion restricts fusion to the union mask $k$0; stage 1 even uses the event modality’s sparsification map for the image branch (Yang et al., 4 Sep 2025).

Not all event-aware systems prune individual tokens. The tracking framework “Dynamic Pondering Sparsity-aware Mixture-of-Experts Transformer for Event Stream based Visual Object Tracking” progressively injects sparse, medium-density, and dense search tokens into a three-stage Vision Transformer, routes features through density-specialized experts, and adaptively halts inference depth. Its “sparsity-aware” behavior is therefore group-wise inclusion and routing over density-specific token sets rather than fine-grained token pruning (Wang et al., 7 May 2026). This distinction matters because the literature uses “selection” to describe event-native token emission, hard pruning, reversible skipping, and density-conditioned routing, even when the retained unit is a token group rather than a single token.

5. Empirical behavior and reported trade-offs

Representative quantitative results reported in the literature are summarized below (Liu et al., 2023, Peng et al., 2024, Yang et al., 4 Sep 2025, Zhao et al., 26 Sep 2025, Liu et al., 3 Feb 2026, Zhang et al., 20 Apr 2026, Ma et al., 19 May 2026).

Method	Reported effect	Setting
SparseSpikformer	22.49%–26.47% reduction in floating-point cost; 21.98%–66.96% throughput acceleration	CIFAR10, CIFAR100, CIFAR10-DVS, DVS-128
SAST	48.3 mAP with 1.8 G A-FLOPs versus 47.4 mAP with 6.5 A-FLOPs	1Mpx
FocusMamba	32.6 mAP to 34.6 mAP; 87.2G to 60.8G FLOPs	DSEC-Det
PSTTS	29–43.6% FLOPs reduction; 21.6–41.3% FPS increase	DailyDVS-200
EventFlash	2.3 to 28.5 tokens/s; up to 1,000 bins	EventFlash-Zero vs EventFlash-3B
Match-Any-Events	62.22G to 48.87G spatial-attention FLOPs	wide-baseline event matching
EventPrune	80% visual token reduction; 37.62% vs. 36.31%; 1.89x inference speedup; 52% GFLOPs reduction	first-person dynamic spatial reasoning

These comparisons make several recurring patterns visible. First, selector quality matters: sparsity obtained from informative spike activity, spatio-temporal continuity, or event-guided ranking behaves differently from random or fixed-budget dropping. Second, temporal compression often dominates raw throughput gains, whereas spatial or token-level ranking determines how much accuracy is preserved under aggressive sparsity. Third, co-sparsification or multimodal guidance can improve both accuracy and efficiency rather than forcing a strict trade-off. This suggests that the decisive issue is rarely whether sparsity is used at all; it is which signal defines token importance, at what stage the decision is made, and whether the architecture permits later recovery or complementary fusion.

6. Relation to broader sparse attention and execution literature

Beyond event-camera papers, recent sparse-attention work supplies design motifs that transfer to event tokens. Token Sparse Attention proposes reversible, interleaved token selection: for each head $k$1, select $k$2, gather $k$3, compute attention in compressed space, and scatter the output back to full length so skipped tokens remain available through the residual path. This suggests a clear alternative to irreversible event-token eviction when token salience is strongly layer-dependent or head-dependent (Jo et al., 3 Feb 2026).

Long-context inference work makes the same point from a different angle: relevance is query-conditioned rather than global. “Exploiting Sparsity for Long Context Inference” retrieves the top-$k$4 keys by dot product $k$5, attends to less than 2% of input tokens, and achieves over 95% of model performance on common benchmarks. A plausible implication is that event-memory systems should prefer per-query retrieval over one-shot pruning when rare events may become relevant later (Synk et al., 10 Feb 2025).

“Training-free sparse attention based on cumulative energy filtering” argues that fixed recall is sufficient for ensuring accuracy, whereas a fixed threshold is suboptimal for reducing computational cost, and replaces static token filters with the online criterion $k$6. SATA, by contrast, assumes the sparse query-key pattern already exists and optimizes execution through key sorting, HEAD/TAIL/GLOB query classification, and locality-centric scheduling. Together these results suggest that event-token research now spans not only token scoring but also threshold adaptation and sparse execution once selection has been made (Li et al., 15 Jun 2026, Fan et al., 28 Jan 2026).

7. Limitations and unresolved issues

Several influential methods leave nontrivial ambiguities. SparseSpikformer motivates selection with “average spike firing rate,” yet its equations use temporally averaged token features followed by an MLP and Gumbel-Softmax rather than a direct scalar firing-rate ranking; the same paper alternates between “top $k$7” wording and a learned keep/drop classifier, and does not specify an exact attention masking equation or Gumbel-Softmax temperature (Liu et al., 2023). EventFlash does not provide the exact temporal merge threshold $k$8, the stopping rule for iterative merging, or the exact TokenSelector implementation, while Match-Any-Events omits the halting MLP architecture, exact thresholds, and a standalone no-SETS comparison (Liu et al., 3 Feb 2026, Zhang et al., 20 Apr 2026).

Failure modes are also method-specific. FocusMamba notes that when both the scene and objects are stationary relative to the camera, the event spatial ratio $k$9 may no longer reflect object information well, so event-guided threshold regulation loses effectiveness. Spiking Patches exposes the complementary risk: too small $X\in\mathbb R^{T\times N\times D}$0 yields many tokens and less sparsity, whereas too large $X\in\mathbb R^{T\times N\times D}$1 increases delay and may harm responsiveness; events inside refractory periods are discarded (Yang et al., 4 Sep 2025, Øhrstrøm et al., 30 Oct 2025).

The field also contains a persistent categorical ambiguity between selection, completion, routing, and reversible skipping. EvRainDrop is framed by the supplied text as completion rather than explicit selection; the tracking MoE paper is density-conditioned inclusion and adaptive depth rather than fine-grained pruning; EventPrune uses events to prune RGB visual tokens; and Token Sparse Attention keeps skipped tokens alive in the residual stream. This suggests that “sparsity-aware event token selection” is best treated as an umbrella term for a set of closely related operations—event-native token creation, sample-adaptive token pruning, temporal merging, reversible in-attention skipping, and event-guided multimodal routing—whose common aim is to align transformer computation with the intrinsic sparsity, temporal density, and motion structure of event data (Wang et al., 26 Nov 2025, Wang et al., 7 May 2026, Ma et al., 19 May 2026, Jo et al., 3 Feb 2026).