Event Sparsity-Aware Tracking Framework

Updated 5 July 2026

Event Sparsity-Aware Tracking Framework is a design paradigm that leverages asynchronous, spatially sparse event data to improve state estimation and energy efficiency.
It integrates methods from direct geometric optimizers to learned matchers, addressing challenges like temporal density variation, modality incompleteness, and computational sparsity.
Implementations across single-object, multi-object, and RGB-event fusion tasks demonstrate reduced computational load and enhanced low-latency tracking performance.

Searching arXiv for recent and foundational papers on event sparsity-aware tracking, event-based tracking, and RGB-event tracking. An Event Sparsity-Aware Tracking Framework denotes a family of tracking systems built around the fact that event cameras emit asynchronous, spatially sparse, polarity-coded brightness changes rather than dense frames. In these frameworks, sparsity is treated not merely as a nuisance but as a first-class modeling condition: it shapes the event representation, the temporal update rule, the fusion policy, and often the computational pathway itself. Across the literature, the term spans single-object tracking, Tracking Any Point (TAP), feature tracking, camera tracking, and multi-object tracking (MOT), with outputs ranging from boxes and point trajectories to camera pose and track identities. The shared technical problem is that event streams are informative precisely because they are sparse, motion-sensitive, and high-rate, yet those same properties create ambiguity, temporal density variation, and modality incompleteness that dense frame-centric trackers do not handle naturally (Shan et al., 9 Mar 2025, Han et al., 2024, Wang et al., 2024, Reinbacher et al., 2017).

1. Scope, task formulations, and the meaning of sparsity-awareness

Event-based tracking methods typically start from the standard event-camera measurement

$e=\{x,y,t,p\},$

where $x,y$ are image coordinates, $t$ is timestamp, and $p$ is polarity. What changes across tasks is the tracked state. In event-only single-object tracking, the state is usually a standard box parameterization inferred from event inputs, as in SDTrack, which predicts target position and scale with classification, $\mathcal{L}_1$ , and GIoU supervision (Shan et al., 9 Mar 2025). In TAP, the state is a trajectory of arbitrary image points, often with visibility, as in ETAP and the motion-augmented temporal-consistency framework (Hamann et al., 2024, Han et al., 2024). In asynchronous feature tracking, the state may be $(x,y,\theta)$ for a local patch, as in event-ECC (Zafeiri et al., 2024). In panoramic camera tracking, the state is 3-DoF orientation relative to a cylindrical map (Reinbacher et al., 2017). In event-based MOT, the state includes positions, extents, and identities, with tracklet management and association, as in SpikeMOT (Wang et al., 2023).

Within this literature, “sparsity-aware” has several distinct meanings. In some works, sparsity is treated as an informative measurement process: event positions themselves are the selected high-information subset, so direct geometric alignment can be done without reconstructing intensity or appearance descriptors (Reinbacher et al., 2017). In others, sparsity is treated as a matching failure mode: local event support may be spatially discontinuous, velocity-dependent, or absent, so trackers must inject motion cues, temporal consistency, or memory to stabilize correspondence (Han et al., 2024, Wang et al., 2024). A third usage is computational sparsity: the framework seeks to reduce energy or latency by maintaining sparse spike activations, active-patch token sets, per-event updates, or adaptive depth (Shan et al., 9 Mar 2025, Sabater et al., 2022, Zafeiri et al., 2024).

This diversity implies that an event sparsity-aware framework is not a single architecture class. It is better understood as a design stance in which the tracker explicitly accounts for at least one of the following: spatial sparsity, temporal density variation, asynchrony, or modality incompleteness. Some systems address all four; many address only one or two.

2. Event representations and temporal aggregation

The first decisive choice is how raw events are converted into a form usable by the tracker. The literature spans a continuum from direct event-native alignment to heavily regularized image-like tensors.

At one end, some methods operate on event positions or local event densities with minimal densification. The panoramic tracker of Gallego et al. aligns projected event positions directly to a panoramic event-occurrence map $M(v)=O(v)/N(v)$ , where $O$ counts event occurrences and $N$ counts path-length-normalized visibility (Reinbacher et al., 2017). Event-ECC similarly builds model and template windows as event-density maps updated by bilinear interpolation and optimizes one continuous warp update per event, rather than per packet (Zafeiri et al., 2024). The event-based mean-shift tracker clusters events directly in a joint space of position, polarity, and decayed temporal recency $f_\delta(t)=e^{-\Delta t/\tau}$ , then smooths cluster centers with Kalman filters (Barranco et al., 2018).

Most recent learning-based trackers, however, use structured aggregation. Time Surface (TS) representations are central in motion-augmented TAP: each pixel stores the timestamp of its most recent event, yielding a recency-sensitive surface whose gradients support local kinematic estimation (Han et al., 2024). SDTrack groups events within duration $x,y$ 0 into three-channel event images: one channel accumulates positive events, one accumulates negative events, and a third stores a decayed trajectory history from previous windows. Its Global Trajectory Prompt (GTP) was introduced precisely because single event frames lose directional evidence when motion reverses within a short interval (Shan et al., 9 Mar 2025). ETAP instead uses constant-event-count windows and a 10-channel mixed-density event stack, where channel $x,y$ 1 aggregates

$x,y$ 2

events, so early channels are dense and later channels are more recent. This stabilizes information content across varying event rates while preserving a hierarchy of temporal recency (Hamann et al., 2024).

RGB-event tracking has largely favored voxelized or event-image inputs. FELT/AMTTrack converts raw event streams into $x,y$ 3 and then keeps the densest event voxels—top 4096 for the search region and top 1024 for the template region—before unified tokenization (Wang et al., 2024). APMTrack also uses event voxels $x,y$ 4 with $x,y$ 5, then derives motion-enhanced difference features from temporally aligned bins (Wang et al., 3 Jan 2026). DS-MESA aggregates the event stream between the start and end exposure timestamps of an RGB frame into an $x,y$ 6-bin voxel grid, then splits it into $x,y$ 7 temporal subframes; the best reported configuration uses 3 subframes, not a single accumulated event frame (Shao et al., 2024). MambaTrack temporally aligns events to RGB timestamps by interpolation,

$x,y$ 8

and then computes a normalized event density

$x,y$ 9

to modulate the event branch dynamics (You et al., 15 Apr 2026).

A distinct line of work emphasizes causal and low-latency binning. The eye-tracking network constructs a causal event volume of shape $t$ 0 with polarity separation and a one-sided temporal kernel $t$ 1, so preprocessing does not require future events (Pei et al., 2024). EvT aggregates events into $t$ 2, applies $t$ 3, then keeps only activated patches whose nonzero ratio exceeds $t$ 4; if too few patches are active, it expands the temporal window until at least $t$ 5 active patches are obtained (Sabater et al., 2022).

Taken together, these choices expose a recurrent trade-off. Event-native methods preserve more of the asynchronous measurement structure, while image-like and voxelized methods preserve compatibility with pretrained backbones and standard dense tracking heads. This suggests that “sparsity-aware” front ends are often only partially event-native: they usually preserve some motion chronology, polarity, or density structure, but they rarely maintain full event-by-event precision throughout the entire pipeline.

3. Event-only tracking architectures

Event-only tracking frameworks divide broadly into direct geometric optimizers, motion-aware learned matchers, and spike-driven box trackers.

Direct methods treat sparse events as sufficient geometric evidence. In panoramic tracking, the map is not an appearance map but a probability map of event occurrence. The tracker minimizes a direct reprojection residual

$t$ 6

over event packets, with a smoothness prior on pose updates, and achieves mean angular error below $t$ 7 on all four tested rotational sequences with packet sizes between 1000 and 2000 events (Reinbacher et al., 2017). Event-ECC pursues a similar philosophy at the feature level: it aligns two normalized event-density windows with an ECC criterion and performs exactly one continuous optimization iteration per event. Its lightweight incremental version reduces runtime from 43.2 $t$ 8/event to 11.8 $t$ 9/event while improving tracking accuracy and feature age over HASTE-style asynchronous baselines (Zafeiri et al., 2024).

Learned point trackers attack sparsity by augmenting local appearance matching with motion or descriptor regularization. The motion-augmented TAP framework introduces a Motion-Guidance Module (MGM) and a Variable Motion Aware Module (VMAM). MGM fits local tangent planes to the time surface to derive kinematic vectors; VMAM fuses those cues with temporally sampled reference features using cross-attention, 1D temporal convolution, and temporal attention, thereby stabilizing correlation responses under variable event density and nonlinear motion. Its full VMAM improves $p$ 0, MTE, and $p$ 1 from $p$ 2 for the appearance-only baseline to $p$ 3 on Ev-PointOdyssey (Han et al., 2024). ETAP reaches the same problem from another angle: it adapts CoTracker-style iterative point refinement to event stacks and introduces a feature-alignment loss

$p$ 4

built from time-inverted and rotated event observations, in order to make event descriptors less motion-direction-dependent. Without this loss, the gap between intra-motion and inter-motion cosine similarity is $p$ 5; with it, the gap drops to $p$ 6 (Hamann et al., 2024).

Event-only box tracking has recently been pushed by SNN-based architectures. SDTrack is notable because it is mostly spike-driven rather than ANN-SNN hybrid. It converts events to GTP event images, diagonally concatenates template and search via Intrinsic Position Learning (IPL), and processes them with a spiking MetaFormer built from I-LIF neurons, depthwise separable convolutions, and spiking self-attention. On FE108, SDTrack-Tiny reaches 59.0 AUC / 91.3 PR, while SDTrack-Base reaches 59.9 / 91.5, and the system remains almost fully spike-driven except for the final regression convolution (Shan et al., 9 Mar 2025).

For MOT, SpikeMOT exemplifies a hybrid detector-tracker strategy specialized for sparse motion evidence. It uses spiking Siamese tracking at 10 ms intervals, event voxels of 100 slices, and detector refresh at 50 ms, so sparse high-frequency motion is handled locally while a lower-rate detector stabilizes identities and spatial extent (Wang et al., 2023).

4. RGB-event fusion and multimodal incompleteness

RGB-event tracking turns sparsity-awareness into a multimodal problem: the event branch is sparse, motion-sensitive, and sometimes empty, while the RGB branch is dense but can be degraded by blur, low light, or occlusion. The most explicit articulation of this regime appears in FELT, which frames long-term RGB-event tracking as tracking with incomplete information: RGB frames can be weakened, while event data can be spatially sparse or carry no target signal at all (Wang et al., 2024).

A representative cross-section of fusion strategies is shown below.

Method	Modality/task	Sparsity-aware mechanism
AMTTrack	RGB-E SOT	Hopfield associative memory for incomplete modalities
TENet	RGB-E SOT	Pooler with Multi-Scale Pooling on sparse event images
DS-MESA	RGB-E SOT	Dynamic event subframe splitting and Top- $p$ 7 sparse attention
APMTrack	RGB-E SOT	Frequency-domain event injection and motion-guided Top- $p$ 8 sparsification
MambaTrack	RGB-E SOT	Event-adaptive state transition and gated fusion
SpikeFET	Frame-event SOT	Fully spiking fusion and response-map alignment

AMTTrack addresses incomplete modalities with associative memory. After cropping template and search regions from both modalities and selecting the densest event voxels, RGB and event tokens are concatenated into a unified sequence and passed through Transformer layers augmented with modern Hopfield memory,

$p$ 9

The intention is to retrieve multimodal prototypes from partial inputs, so that degraded event or RGB observations can be reinforced by stored associations (Wang et al., 2024).

TENet builds sparsity-awareness directly into the event encoder. Its Pooler replaces RGB-oriented event backbones with pooling-centric stages and Multi-Scale Pooling (MSP) to capture motion trends across kernel sizes $\mathcal{L}_1$ 0. It then fuses event and RGB via bidirectional cross-attention in Mutually-Guided Fusion (MGF). On COESOT, TENet reaches 68.4% SR and 76.8% PR, outperforming HRCeutrack by 4.9% PR and 5.2% SR; on VisEvent it reaches 76.5% PR and 60.1% SR (Shao et al., 2024).

DS-MESA is organized around the claim that event data should not be collapsed into a single frame and should not be processed with dense all-pairs attention. Its Dynamic Event Subframe Splitting (DES) divides the event stream over an RGB exposure interval into multiple temporal subframes, and its Event-based Sparse Attention (ESA) keeps only Top- $\mathcal{L}_1$ 1 attention interactions at multiple sparsity levels $\mathcal{L}_1$ 2. Replacing dense ViT attention with ESA improves FE240 performance from 89.3 / 62.1 to 91.9 / 63.8 in PR/SR, while multiple event subframes outperform a single event frame on both FE240 and COESOT (Shao et al., 2024).

APMTrack pushes sparsity-awareness further upstream and downstream. Upstream, it performs frequency-domain early fusion: RGB and event inputs are transformed by FFT, amplitude and phase are decoupled, and high-frequency event information is injected into RGB through amplitude and phase attention before the backbone. Downstream, Motion-Guided Spatial Sparsification (MGSS) converts event-derived motion features into a spatial probability distribution over search tokens and retains only an adaptive Top- $\mathcal{L}_1$ 3 set, with

$\mathcal{L}_1$ 4

On COESOT, APMTrack reaches 68.0 SR / 83.3 PR, and the component analysis reports a reduction from 1167.4G FLOPs for concatenation to 701.0G FLOPs for the full model (Wang et al., 3 Jan 2026).

MambaTrack introduces sparsity-awareness inside the state-space model itself. In its event branch, the transition matrix is modulated by normalized event density $\mathcal{L}_1$ 5, a projected scaling factor $\mathcal{L}_1$ 6, and a learnable evolution rate $\mathcal{L}_1$ 7, yielding

$\mathcal{L}_1$ 8

Fusion is then gated by event density and RGB confidence,

$\mathcal{L}_1$ 9

On FELT, MambaTrack reaches 42.5 SR / 54.0 PR, improving over ViPT by +6.8 SR and +9.9 PR (You et al., 15 Apr 2026).

SpikeFET takes a different route by moving the entire frame-event pipeline into the spiking domain. It uses dual modality-specific SNN branches, spiking Transformer fusion, a Random Patchwork Module (RPM) to remove translation bias, and Spatial-Temporal Regularization (STR) to align adjacent template embeddings. Its frame-event fusion is not event-native in the raw asynchronous sense—the event stream is first converted into event temporal frames—but it preserves sparse spike computation throughout the network. On FE108, SpikeFET-Tiny reaches 68.5 AUC / 96.5 PR at 18.36 mJ, while SpikeFET-Base reaches 68.7 / 97.0 at 102.61 mJ (Yang et al., 27 May 2025).

5. Computational sparsity, energy, and online operation

A central distinction in this literature is between representational sparsity and computational sparsity. Many trackers preserve some event sparsity in the input representation but still perform dense tensor processing. Others carry sparsity into the update rule, token set, or neuron activity.

SDTrack provides one of the clearest energy formulations. Its theoretical energy model distinguishes MAC-heavy non-spiking layers from AC-dominated spike-driven layers and explicitly ties energy to firing rate $(x,y,\theta)$ 0. Under 32-bit floating-point assumptions in 45 nm, it sets $(x,y,\theta)$ 1 pJ and $(x,y,\theta)$ 2 pJ. In its comparison table, SDTrack-Tiny has 19.61M parameters and 8.16 mJ theoretical energy, while SDTrack-Base has 107.26M and 30.52 mJ (Shan et al., 9 Mar 2025). SpikeFET makes the same argument from the frame-event side: on FE108, SpikeFET-Tiny uses 18.36 mJ and SpikeFET-Base 102.61 mJ, compared with 719.10 mJ for SDSTrack (Yang et al., 27 May 2025).

Other frameworks exploit sparsity through adaptive computation rather than spikes. PSMTrack explicitly models sparse, medium-density, and dense event windows, routes them through a sparsity-aware Mixture-of-Experts, and uses a Dynamic Pondering Strategy (DPS) that predicts whether to halt at deeper layers. On EventVOT, the model with DPS improves speed from 56 FPS to 79 FPS with only about 0.3% average accuracy drop, and the paper reports that about 50% of scenarios terminate early (Wang et al., 7 May 2026).

Patch- and event-level incremental schemes push even closer to native sparsity. EvT discards inactive patches before the transformer backbone, with cross-attention complexity $(x,y,\theta)$ 3 and self-attention complexity $(x,y,\theta)$ 4 over latent memory. On N-Caltech-101, it reports 0.20 GFLOPs and 0.48M parameters, far below dense baselines (Sabater et al., 2022). The eye-tracking network adds $(x,y,\theta)$ 5 regularization on ReLU activations and reports greater than 90% activation sparsity with minimal performance drop; on processors that fully exploit sparsity, this corresponds to a 5x inference speedup relative to the default configuration (Pei et al., 2024). Event-ECC’s lightweight formulation reduces asynchronous tracking to 11.8 $(x,y,\theta)$ 6/event, while the event-based mean-shift tracker reports 88% computational reduction in the best case, 83% on average, and 2.5 pixels average tracking error (Zafeiri et al., 2024, Barranco et al., 2018).

These results also clarify a frequent misconception. A framework can be “event sparsity-aware” without being fully event-by-event or fully asynchronous. Many of the most accurate methods—SDTrack, ETAP, TENet, APMTrack, MambaTrack, SpikeFET—still begin from event images, voxels, or temporal frames. Their sparsity-awareness lies in what they preserve from the event stream and how they compute internally, not necessarily in raw sensor-to-prediction asynchrony.

6. Benchmarks, misconceptions, and open directions

The empirical landscape is fragmented by task. Event-only SOT largely uses AUC and PR on FE108, FELT, and VisEvent (Shan et al., 9 Mar 2025). RGB-event trackers report SR, PR, and sometimes NPR on COESOT, FELT, EventVOT, FE240 or FE240hz, and VisEvent (Shao et al., 2024, Wang et al., 3 Jan 2026, Wang et al., 7 May 2026). TAP methods use $(x,y,\theta)$ 7, MTE, $(x,y,\theta)$ 8, FA/EFA, OA, AJ, and standard TAP-Vid thresholds on datasets such as Ev-PointOdyssey, EventKubric, EC, EDS, and E2D2 (Han et al., 2024, Hamann et al., 2024). MOT introduces HOTA, DetA, AssA, MOTA, and IDF1, as in DSEC-MOT, where SpikeMOT reports HOTA 52.5, IDF1 62.9, and only 7 identity switches on the test set (Wang et al., 2023). Long-term multimodal SOT uses FELT, which contains 742 videos and 1,594,474 RGB-frame/event-stream pairs and was explicitly designed around “Incomplete Information” (Wang et al., 2024).

Several misconceptions recur. The first is that event-only tracking should automatically dominate RGB. FELT shows the opposite: event-only trackers are generally weaker than RGB-only trackers, and naive RGB+Event fusion can even degrade performance for OSTrack, ATOM, and KYS (Wang et al., 2024). The second is that sparsity-aware means fully asynchronous. SDTrack explicitly groups events within a duration $(x,y,\theta)$ 9 into event images, ETAP uses constant-event-count stacks, and SpikeFET consumes event temporal frames rather than raw events (Shan et al., 9 Mar 2025, Hamann et al., 2024, Yang et al., 27 May 2025). The third is that one temporal scale suffices. DS-MESA shows that multiple event subframes outperform a single event frame, PSMTrack reports that dense $M(v)=O(v)/N(v)$ 0 medium $M(v)=O(v)/N(v)$ 1 sparse is the best search-input order, and ETAP’s mixed-density stacks slightly outperform uniform voxel grids (Shao et al., 2024, Wang et al., 7 May 2026, Hamann et al., 2024).

The strongest limitations are equally consistent. Many frameworks still lose intra-window temporal order through aggregation; many event branches inherit frame-based assumptions because they rely on 3-channel compatibility or ImageNet pretraining; direct event-native methods often remain restricted in motion model or task scope; and multimodal methods frequently lack explicit reliability estimation beyond learned fusion. This suggests that the next phase of event sparsity-aware tracking will likely focus on more adaptive front ends and more explicit sparse computation. The literature already points in that direction: DS-MESA identifies adaptive subframe granularity as future work, PSMTrack points to learned temporal windows and token-level adaptive depth, APMTrack notes modality imbalance and challenge-specific adaptation, and MambaTrack emphasizes event-density-conditioned temporal evolution rather than static dynamics (Shao et al., 2024, Wang et al., 7 May 2026, Wang et al., 3 Jan 2026, You et al., 15 Apr 2026).

In this sense, the mature definition of an Event Sparsity-Aware Tracking Framework is narrower than a generic event tracker but broader than a raw asynchronous optimizer. It refers to a tracking system that explicitly models the consequences of sparse, asynchronous, density-varying event measurements—whether through event-native geometry, motion-guided matching, modality-aware memory, sparse attention, adaptive state-space dynamics, spike-driven inference, or active-token computation—and treats those consequences as central, not incidental, to the design of the tracker.