Event Transformer: Sparse-Aware Processing

• Event Transformer (EvT) is a family of event-oriented transformer architectures that efficiently process asynchronous event data using sparse-aware techniques.
• It employs patch-based and raw-event tokenizations to focus computation only on informative spatial regions, preserving data sparsity and enabling low-latency inference.
• Variations such as EvT+ and alternative raw-event models offer diverse design tradeoffs tailored to tasks like event classification, dense depth estimation, and rare-event forecasting.

Event Transformer (EvT) denotes, in its narrowest and most historically specific sense, a sparse-aware framework for efficient event-camera data processing introduced by Sabater, Montesano, and Murillo, whose central idea is to preserve event-data sparsity by processing only informative spatial regions with a compact transformer-like latent-memory backbone (Sabater et al., 2022). The term is also used more broadly for architectures that apply transformer machinery to event-camera inputs, raw event tokens, grouped event representations, or even non-visual event abstractions in language, audio, and rare-event forecasting. As a result, “Event Transformer” names not one canonical architecture but a family of event-oriented transformer formulations whose main point of divergence is the definition of an event and the degree to which asynchronous structure is preserved (Sabater et al., 2022).

1. Terminological scope and historical usage

Within event-camera vision, the original EvT is a patch-based, sparse-aware architecture that first builds a frame-like representation of events, discards inactive patches, and routes only activated patches through a compact backbone (Sabater et al., 2022). Its direct extension, Event Transformer$^+$, refines the event representation, strengthens the backbone, supports multiple modalities, and adds task-specific heads for event-stream classification and dense depth estimation (Sabater et al., 2022).

The same label, however, has been reused for materially different designs. A distinct paper also titled “Event Transformer” treats each raw event as an event-token and builds a hierarchical backbone around local temporal attention, sparse local spatial attention, and global representative attention (Jiang et al., 2022). Later work such as GET, EVSTr, Force-EvT, EventTransAct, eMoE-Tracker, and SEST further broadened the space by using grouped tokens, voxel sets, event frames, stacked event frames, or voxel-grid tensors rather than a single shared notion of event-native processing (Peng et al., 2023).

This variation is easiest to understand in terms of the event unit presented to the transformer.

Model family	Event unit	Characterization
EvT (Sabater et al., 2022)	Activated patches from $F^{H \times W \times B \times 2}$	Sparse-aware latent-memory transformer
“Event Transformer” (Jiang et al., 2022)	Raw event-tokens $e_i=(x_i,y_i,t_i,p_i)$	ETB with LXformer, SCformer, GXformer
GET (Peng et al., 2023)	Group Tokens grouped by timestamps and polarities	Hierarchical event-native backbone
EVSTr (Xie et al., 2023)	Sparse voxel sets $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$	Voxel-set transformer encoder
Frame-adapted variants (Blegiers et al., 2023)	Event frames	Video-style temporal transformer over frame embeddings

A recurrent misconception is that any model carrying the EvT label is necessarily a raw-event transformer. Several works explicitly do not operate directly on asynchronous event tuples. Force-EvT is best understood as a task-specific adaptation of a Vision Transformer pipeline to event-camera data for robotic force regression, rather than as a fundamentally new event-native transformer architecture (Guo et al., 2024).

2. Original EvT: sparse-aware patch processing for event streams

The original EvT starts from the standard event tuple $e=\{x,y,t,p\}$ and defines an event stream within a time-window $\Delta t$ as $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$ (Sabater et al., 2022). Rather than processing raw events directly, it first aggregates them into a frame-like tensor

$F^{H \times W \times B \times 2},$

where $H \times W$ is the sensor resolution, $B$ is the number of temporal bins, and the last dimension corresponds to positive and negative polarities. Each bin stores per-pixel event counts over a subinterval $F^{H \times W \times B \times 2}$0, and a log transform

$F^{H \times W \times B \times 2}$1

is applied to compress large activation values.

The defining representational move is then to split $F^{H \times W \times B \times 2}$2 into non-overlapping $F^{H \times W \times B \times 2}$3 patches and keep only activated patches. A patch is retained if at least $F^{H \times W \times B \times 2}$4 of its pixels are non-zero. Activated patches are flattened into tokens of size

$F^{H \times W \times B \times 2}$5

while non-activated patches are discarded. This gives EvT its sparse-aware character: the network does not process the whole frame, only event-active regions. The paper also introduces adaptive temporal accumulation. If a window yields fewer than $F^{H \times W \times B \times 2}$6 activated patches, $F^{H \times W \times B \times 2}$7 is expanded, newly arriving events are added, and the representation is recomputed until the frame contains at least $F^{H \times W \times B \times 2}$8 activated patches.

Architecturally, EvT embeds these activated patches into a constant-dimensional space $F^{H \times W \times B \times 2}$9 through an initial single-layer feed-forward network, concatenation with 2D-aware positional embeddings, and a final single-layer feed-forward network. Before the main backbone, a double-layer feed-forward network with a skip connection further refines local token features. Positional information is provided by 2D Fourier Features initialized with 16 bands.

The backbone uses $e_i=(x_i,y_i,t_i,p_i)$0 learned latent memory vectors. These vectors act both as a computational bottleneck and as a stream-level memory. Processing consists of one Cross-Attention module followed by $e_i=(x_i,y_i,t_i,p_i)$1 Self-Attention modules. In cross-attention, latent memory vectors provide queries and patch tokens provide keys and values; in self-attention, latent vectors refine themselves. After each time-window, the new latent output is combined with the existing memory by simple addition. Classification is performed from the final latent memory using two feed-forward layers and Global Average Pooling. The reported implementation uses $e_i=(x_i,y_i,t_i,p_i)$2, $e_i=(x_i,y_i,t_i,p_i)$3, $e_i=(x_i,y_i,t_i,p_i)$4, $e_i=(x_i,y_i,t_i,p_i)$5, $e_i=(x_i,y_i,t_i,p_i)$6, one Multi-Head Cross-Attention layer, two Multi-Head Self-Attention layers, and four attention heads (Sabater et al., 2022).

The efficiency argument follows directly from this design. Cross-attention scales as $e_i=(x_i,y_i,t_i,p_i)$7, where $e_i=(x_i,y_i,t_i,p_i)$8 is the number of activated patches, while latent self-attention scales as $e_i=(x_i,y_i,t_i,p_i)$9 instead of $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$0. This makes EvT sparse-aware in two senses: it reduces the number of inputs by keeping only activated patches, and it avoids full token-token self-attention over all patches by routing computation through a fixed latent bottleneck.

3. Event Transformer$\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$1: refined temporal representation and multi-purpose extension

Event Transformer$\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$2 generalizes the original EvT in three directions: a refined patch-based event representation, a more robust backbone, and support for both event-stream classification and dense depth estimation (Sabater et al., 2022). The principal representational change is the replacement of the original histogram-like encoding with a FIFO-based temporal representation. For each pixel $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$3, polarity $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$4, and queue slot $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$5, the method stores events in $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$6, but not simply the last $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$7 events: it stores the last $\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$8 events that are separated by at least

$\mathcal{V}=\{(\mathbf{f}_i,\mathbf{c}_i)\}$9

The resulting intermediate tensor is

$e=\{x,y,t,p\}$0

and timestamps are normalized by

$e=\{x,y,t,p\}$1

This representation preserves a compact temporal signature per pixel while retaining sparse patch selection. A patch is activated if the percentage of pixels whose events were triggered within $e=\{x,y,t,p\}$2 exceeds threshold $e=\{x,y,t,p\}$3. Older events up to maximum age $e=\{x,y,t,p\}$4 enrich the token content but do not affect activation. Each activated patch is flattened to size

$e=\{x,y,t,p\}$5

For classification, the reported settings are $e=\{x,y,t,p\}$6, $e=\{x,y,t,p\}$7, $e=\{x,y,t,p\}$8 ms, $e=\{x,y,t,p\}$9, latent dimension $\Delta t$0, and $\Delta t$1 latent vectors. For dense depth estimation, $\Delta t$2 is used.

EvT$\Delta t$3 also changes the backbone order. Before interacting with latent memory, activated patches are embedded, augmented with learned 2D positional encodings initialized as 2D Fourier Features with 6 bands, and passed through $\Delta t$4 self-attention blocks that model spatial dependencies among sparse patches within a single time-window. Cross-attention then fuses the processed patches with latent memory, followed by $\Delta t$5 self-attention layers on memory. Memory is updated incrementally by

$\Delta t$6

For classification, the final memory is passed through $\Delta t$7 self-attention modules, two feed-forward layers, and Global Average Pooling. For dense prediction, sparse tokens are restored to a dense patch grid by inserting dummy tokens at filtered-out locations, then updated through cross-attention using latent memory as keys and values.

Empirically, EvT$\Delta t$8 improves on EvT in event-stream classification while retaining low latency. On DVS128 Gesture it reaches $\Delta t$9 for 10 classes and $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$0 for 11 classes, compared with EvT’s $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$1 and $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$2; on SL-Animals-DVS it reaches $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$3, compared with EvT’s $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$4 (Sabater et al., 2022). In dense prediction, it reports average absolute depth error improvements on MVSEC for both event-only and event-plus-grayscale settings, and also evaluates on EV-IMO2. The efficiency profile changes, however. For classification the dominant cost becomes the initial self-attention over sparse input tokens, $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$5, whereas dense prediction incurs a dense-head cost scaling with the total number of patches rather than the sparse activated subset.

4. Alternative event-camera transformer formulations

A separate line of work uses the Event Transformer name for a fundamentally different architecture: a raw-event token model in which each event is treated as the basic processing unit (Jiang et al., 2022). Its event stream is

$$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$6

and the backbone is a 4-stage hierarchy of Event Transformer Blocks (ETBs). Each ETB serially composes LXformer for local temporal attention, SCformer for sparse local spatial attention, and GXformer for global attention over a downsampled representative set. This design explicitly models temporal neighbors, sparse 2D neighborhoods, and global nonlocal interactions. On the reported classification benchmarks it achieves $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$7 on N-MNIST, $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$8 on N-Caltech101, $$\varepsilon = \{e^1, e^2, e^3, ...\} \mid e_t^i \epsilon \Delta t$$9 on CIFAR10-DVS, $F^{H \times W \times B \times 2},$0 on N-Cars, and $F^{H \times W \times B \times 2},$1 on ASL-DVS (Jiang et al., 2022). According to the provided details, the work is demonstrated on classification rather than on a broader dense-task suite.

GET and EVSTr represent two additional event-native directions. GET introduces Group Tokens that group asynchronous events by timestamp intervals and polarity, then processes them with Event Dual Self-Attention and Group Token Aggregation, thereby preserving temporal-polarity structure throughout a hierarchical backbone (Peng et al., 2023). EVSTr instead converts event streams into sparse voxel sets, uses a Multi-Scale Neighbor Embedding Layer for local aggregation, a Voxel Self-Attention Layer for global interaction, and a segment modeling strategy for long-range action dynamics (Xie et al., 2023). Both approaches remain closer to event-native processing than frame-based ViT adaptations, but they still regularize the raw stream through grouping or voxelization before self-attention.

By contrast, a substantial subfamily of event-camera transformers is frame-adapted rather than event-native. EventTransAct converts event streams into event frames, uses ViT-B as a spatial encoder, a 3-layer, 8-head LongFormer as a temporal encoder, and augments the model with Event-Contrastive Loss and event-specific augmentations (Blegiers et al., 2023). Force-EvT converts asynchronous events from a DVXplorer Lite series camera into event frames over a fixed time interval $F^{H \times W \times B \times 2},$2 ms and feeds them to a ViT backbone for scalar force regression, achieving $F^{H \times W \times B \times 2},$3 N and $F^{H \times W \times B \times 2},$4 on its held-out test set (Guo et al., 2024). eMoE-Tracker uses stacked event frames in a one-stream RGB-event tracker, while SEST converts event streams into voxel grids of shape $F^{H \times W \times B \times 2},$5 and applies an event-pretrained Swin Transformer to each temporal bin independently before 3D convolutional decoding (Chen et al., 2024, Mazna et al., 22 May 2026). These models are transformer-based event systems, but they do not preserve asynchronous event tuples as first-class transformer tokens.

5. Event Transformer beyond event-camera vision

Outside event-camera vision, “event transformer” refers to still broader abstractions in which an event is semantic or temporal rather than sensor-driven. ClarET is an event-centric encoder–decoder Transformer built by continual pre-training from BART-large, where an event is a text span, usually composed of a predicate and its arguments, and pre-training is cast as context-to-event transformation with objectives for whole event recovering, contrastive event-correlation encoding, and prompt-based event locating (Zhou et al., 2022). Its event-centricity is implemented through masking granularity, event extraction, contrastive alignment of masked-event states with event mentions, and prompt-based event coherence objectives, not through a novel transformer block.

Sound Event Detection Transformer (SEDT) applies DETR-style set prediction to audio by treating a clip as an unordered set of sound events, each with a class and temporal extent. It uses event queries, bipartite matching, and direct event-level outputs rather than framewise sequence labeling followed by post-processing (Ye et al., 2021). In this setting, the event is a temporal interval in audio, not a neuromorphic sensor event.

EVEREST extends the notion further still. It is a transformer-based architecture for binary rare-event forecasting on multivariate time series, with a learnable single-query attention bottleneck, an evidential head over the logit, an extreme-value head in logit space, and a precursor head for early-event detection (Zilinskas et al., 26 Jan 2026). Here, “event” denotes occurrence within a forecast horizon rather than a discrete input token. This suggests that “Event Transformer” has become a design pattern in which transformer learning is specialized around event occurrence, event coherence, or event-set prediction, even when the underlying modality is text, audio, or tabular time series rather than event-camera data.

6. Conceptual distinctions, limitations, and open directions

The central conceptual distinction in EvT research is between event-native and frame-derived processing. Original EvT and EvT$F^{H \times W \times B \times 2},$6 exploit sparsity aggressively, but they still build frame-like intermediate tensors before tokenization (Sabater et al., 2022, Sabater et al., 2022). EventTransAct, Force-EvT, eMoE-Tracker, and SEST go further in the same direction by converting events into event frames, stacked event frames, or voxel-grid tensors specifically to reuse standard image or video transformer machinery (Blegiers et al., 2023, Guo et al., 2024, Mazna et al., 22 May 2026). This is practical and often effective, but it partially sacrifices asynchronous fidelity, timestamp precision, and event-level sparsity.

Raw-event and grouped-token models attempt to preserve more of the native signal, yet they introduce their own tradeoffs. The raw-event Event Transformer relies on design choices such as the number of closest temporal neighbors, the spatial window size, and the downsampling rate (Jiang et al., 2022). GET depends on discretized time grouping and group count $F^{H \times W \times B \times 2},$7, with the optimal value varying by dataset temporal extent (Peng et al., 2023). EVSTr uses voxel size, number of selected voxels, number of segments, and temporal normalization coefficients that are likewise dataset-specific (Xie et al., 2023). This suggests that event-native fidelity does not remove the need for representational design; it relocates that design to grouping, neighborhood construction, or sparse abstraction.

A second misconception is that transformer usage alone implies superior exploitation of event sensors. Several successful models are closer to “transformer adaptation for event data” than to a distinct event-transformer architecture. Force-EvT explicitly fits this description, and EventTransAct is best described as a frame-based event video transformer rather than a raw-event transformer (Guo et al., 2024, Blegiers et al., 2023). Conversely, some of the strongest event-native proposals derive their inductive bias less from new attention equations than from how events are represented and aggregated before self-attention.

Open directions follow directly from these limitations. The literature repeatedly points toward comparisons between frame-based baselines and more explicitly event-native transformers, improved preservation of timestamp information, stronger deployment-oriented sparse computation, and architectures that maintain event-aware efficiency while supporting dense outputs and multimodal fusion. In that sense, Event Transformer remains less a settled architecture than an active research category spanning sparse-aware patch transformers, raw event-token models, grouped spatiotemporal backbones, voxel-set transformers, and task-specific transformer adaptations.