Event-Voxel Embeddings

• Event-voxel embeddings are learned representations generated by discretizing asynchronous event streams into structured spatiotemporal voxel grids that capture essential local event features.
• They leverage Transformer-based and GNN-based architectures to extract robust features using self-attention and graph message passing, enabling effective multi-scale analysis.
• Fusion of event-voxel data with RGB and event-frame modalities, combined with contrastive and reconstruction training, significantly enhances performance in recognition, tracking, and reconstruction tasks.

Event-voxel embeddings are learned representations derived from discretizing asynchronous event streams—typically output by neuromorphic vision sensors—into structured spatiotemporal “voxel” elements. These embeddings serve as the foundation for modern event-based perception architectures, enabling efficient, robust, and multi-scale analysis on recognition, reconstruction, and fusion tasks across diverse domains such as classification, tracking, and action recognition (Wu et al., 17 Apr 2025Yuan et al., 2023Huang et al., 2024Xie et al., 2023Lou et al., 22 May 2025). This article reviews the theoretical underpinnings, common preprocessing steps, embedding architectures, integration with other modalities, fusion and contrastive mechanisms, and empirical impact of event-voxel embeddings.

1. Spatiotemporal Voxelization from Event Streams

The foundational step in event-voxel embedding pipelines is the discretization of raw event streams $\mathcal{E} = \{(x_k, y_k, t_k, p_k)\}$ into a structured spatial-temporal grid of voxels. This is accomplished by quantizing pixel locations and timestamps into 3D grid cells of size $(\Delta_x, \Delta_y, \Delta_t)$, producing a volume where each voxel accumulates polarity counts or other statistics from events falling within its bounds (Chen et al., 2024Yuan et al., 2023Jiang et al., 2023Xie et al., 2023Huang et al., 2024Lou et al., 22 May 2025).

Typical approaches include:

• Uniform spatiotemporal binning: Partitioning the input space into $$(H/\Delta_x) \times (W/\Delta_y) \times (T/\Delta_t)$$ voxels, with features formed by summing event polarities or aggregating local histograms (Chen et al., 2024Yuan et al., 2023Xie et al., 2023).
• Adaptive selection: Selecting the top-K voxels by activity magnitude to form a sparse set of meaningful units, biasing representation toward informative regions (Yuan et al., 2023Jiang et al., 2023Huang et al., 2024).
• Video-to-voxel simulation: Directly synthesizing event voxels from conventional video frames by thresholding log-intensity differences and modeling camera noise, yielding dense event-voxel tensors with substantial storage and generation advantages (Lou et al., 22 May 2025).

Mathematical Formulations

Let $V_{i,j,k}$ denote the occupancy or aggregated feature in voxel $(i,j,k)$: $$V_{i,j,k} = \sum_{e \in \mathcal{E}} \delta\left( \left\lfloor\frac{x_e}{\Delta_x}\right\rfloor = i, \left\lfloor\frac{y_e}{\Delta_y}\right\rfloor = j, \left\lfloor\frac{t_e}{\Delta_t}\right\rfloor = k \right) p_e$$ where $\delta(\cdot)$ is the indicator and $p_e$ is the event polarity (Chen et al., 2024).

Parameters such as the spatiotemporal bin size, number of voxels, and whether polarity is one or two channels are generally selected empirically for task and dataset (Lou et al., 22 May 2025).

2. Embedding Architectures for Event Voxels

Once voxels or voxel sets are constructed, neural network architectures are deployed to extract discriminative embeddings. Two architectural families dominate:

Transformer-based Embedding

Transformer encoders treat flattened voxel groups as sequences of tokens, optionally with positional encodings (Wu et al., 17 Apr 2025Xie et al., 2023Huang et al., 2024). Each token—the output of flattening or concatenating features within a voxel group—typically passes through:

• Linear projection layers
• Addition of learnable CLS tokens and positional encodings
• Stacks of Multi-Head Self-Attention (MHSA) and MLP residual blocks
• Final dimensionality reduction by linear projection

Formally, for a voxel-token sequence $[v_1, \dots, v_{N}]$, the encoder outputs $$z_v = f_\theta({\bf v}) \in \mathbb{R}^{N \times d}$$ (Wu et al., 17 Apr 2025).

Graph Neural Network (GNN)-based Embedding

Voxels are mapped to nodes in a spatiotemporal geometric graph $(\Delta_x, \Delta_y, \Delta_t)$0, with edges defined by proximity in 3D event space. Node features are learned through message passing:

• Gaussian-mixture-kernel or GMM-weighted graph convolution (Yuan et al., 2023Jiang et al., 2023Chen et al., 2024)
• Multiple GNN layers with ReLU activations and residual connections
• Aggregation (mean or via “absorbing nodes”) for global descriptors

A general GNN update for voxel node $(\Delta_x, \Delta_y, \Delta_t)$1 is: $(\Delta_x, \Delta_y, \Delta_t)$2 with GMM-attention variants further modulating the neighbor summation (Chen et al., 2024Yuan et al., 2023).

Absorbing Node Mechanism

Some approaches augment the graph with a global “absorbing” node connected to all others, enabling more effective aggregation of local-to-global information (Jiang et al., 2023).

3. Fusion with Image and Event-Frame Modalities

Integrative multi-modal event perception fuses event-voxel embeddings with representations from RGB or event-frame streams. Fusion mechanisms include:

• Concatenation and Transformer fusion: At matched spatial positions, event-, RGB-, and voxel-tokens are concatenated and processed by Transformer blocks for cross-modal feature alignment (Wu et al., 17 Apr 2025).
• Bottleneck Transformer: To reduce quadratic attention complexity, bottleneck tokens mediate cross-attention between image and voxel branches over multi-stage fusion (Yuan et al., 2023Chen et al., 2024).
• Graph-level fusion: At the embedding (graph-global) level, point and voxel graph embeddings are merged, often after “absorbing node” readout (Jiang et al., 2023).

Some frameworks further differentiate feature fusion by retaining, blending, or exchanging features based on learned per-voxel quality measures (the Retain–Blend–Exchange, or RBE scheme) (Chen et al., 2024).

4. Supervised, Self-Supervised, and Contrastive Training

Event-voxel embeddings are optimized via several complementary objectives:

• Masked autoencoding: Predicting masked event voxels or pixel patches from visible tokens using decoder heads or reconstruction modules (Wu et al., 17 Apr 2025Huang et al., 2024).
• Multi-modal contrastive losses: Aligning global embeddings from different modalities (e.g., RGB⇄voxel, event-frame⇄voxel) in a shared latent space with InfoNCE-style losses (Wu et al., 17 Apr 2025).
• Disentangled masked modeling: Enforcing local spatiotemporal and global semantic reconstruction via parallel transformer decoders, with regionally uniform random masking for unbiased learning (Huang et al., 2024).
• Standard cross-entropy classification: For downstream tasks, pooled or fused embeddings are classified through shallow MLPs with softmax (Jiang et al., 2023Yuan et al., 2023).

5. Empirical Evaluation and Benchmark Performance

Recent works report strong empirical gains for event-voxel embeddings across classification, detection, tracking, and reconstruction tasks. Example results include:

Framework	Task	Dataset	Modality	Top-1/Metric	Gain vs. Baseline
CM3AE (Wu et al., 17 Apr 2025)	Action Recog., Detection, Tracking	HARDVS, EvDET200K, COESOT	Voxel fusion	53% (Top-1), AP=49.0	+2–3%
EFV++ (Chen et al., 2024)	Classification	Bullying10k	Event+Voxel	90.51%	+2.21%
EVSTr (Xie et al., 2023)	ObjCls / ActRec	N-Caltech101 / DailyAction	Voxel set	79.7% / 99.6%	SOTA
Bottleneck Transf. (Yuan et al., 2023)	Classification	ASL-DVS, N-MNIST	Image+Voxel	~99%	+0.8–3.5%
AGCN (Jiang et al., 2023)	Classification	DVS128-Gait	Point+Voxel	99.7%	+1–3%
V2V (Lou et al., 22 May 2025)	Video reconstr./flow	WebVid10K	Video→Voxel	SSIM ↑, AEE ↓	Outperforms prior

These results consistently indicate that (a) introducing explicit event-voxel branches improves performance over image-only or event-frame-only backbones, (b) the joint exploitation of spatial and temporal locality is essential, and (c) fusion and/or contrastive learning yields additional robustness and accuracy.

6. Design Considerations and Methodological Variants

Distinct architectural and preprocessing choices are found in the literature:

• Voxel size and density: (e.g., 4×4×4, 10×10×1) are selected to balance sparsity and information content. Top-K selection (K=512, 2048) is typical for scalable embedding (Yuan et al., 2023Jiang et al., 2023).
• Embedding depth and width: GNNs of 3 layers and Transformers with 6–12 layers are standard; feature dims typically 128–768 (Wu et al., 17 Apr 2025Xie et al., 2023Yuan et al., 2023).
• Quality-aware selection: Differentiating between high- and low-quality nodes via self-attention scores and selectively fusing, exchanging, or blending features improves expressivity (Chen et al., 2024).
• Synthetic to real adaptation: Video-to-voxel simulation enables large-scale synthetic pretraining, generalizing robustly to real-world event data (Lou et al., 22 May 2025).
• Pooling/Readout strategies: Use of both average and max pooling, absorbing nodes, and GRU hybrid interaction modules to maximize downstream discriminative power (Chen et al., 2024Jiang et al., 2023).

7. Applications and Future Impact

Event-voxel embeddings have rapidly become foundational for event-based perception systems:

• Action, object, and gesture recognition: Enhanced performance via spatiotemporal feature capture (Yuan et al., 2023Xie et al., 2023Huang et al., 2024Chen et al., 2024).
• Multi-modal fusion with RGB: Robust downstream adaptation by explicit fusion and alignment (Wu et al., 17 Apr 2025).
• Video reconstruction and optical flow: High precision and efficiency through large-scale video-to-voxel pipelines (Lou et al., 22 May 2025).
• Privacy-preserving analytics: Outperforming frame-based competitors on datasets such as Bullying10k (Chen et al., 2024).

A plausible implication is that advances in voxelization procedures, graph construction, and multi-modal masked/contrastive training will further improve the scale, efficiency, and robustness of event-based neural systems across real-world scenarios.