Hypergraph-Guided Spatio-Temporal Event Completion

• The paper presents a novel framework, EvRainDrop, that leverages hypergraph-based message passing to integrate spatial, temporal, and RGB modalities for event stream completion.
• It employs cross-modal aggregation and self-attention to fuse information, significantly enhancing classification and attribute recognition performance.
• Experimental validations reveal notable accuracy gains over baselines, confirming the method’s effectiveness in addressing event undersampling challenges.

Hypergraph-guided spatio-temporal event stream completion is an approach addressing the sparsity and undersampling inherent in event camera data by leveraging hypergraph-based relational modeling and multi-modal data integration. This paradigm, instantiated concretely in the EvRainDrop framework, constructs spatio-temporal hypergraphs where nodes and hyperedges encode fine-grained spatial, temporal, and cross-modal relationships between asynchronous event tokens and optionally co-occurring RGB frame tokens. The core methodology includes contextual message passing on the hypergraph, multi-modal aggregation, and self-attention-based temporal fusion, yielding substantial improvements across event stream classification and attribute recognition tasks (Wang et al., 26 Nov 2025).

1. Spatio-Temporal Hypergraph Modeling

A hypergraph $\Gcal = (\Vcal, \Ecal)$ is constructed where the node set $\Vcal$ encompasses both event tokens $e_{t,i}$—representing activity at pixel $i$ and event-packet index $t$—and optionally, RGB frame tokens $f_t$ for keyframes at time $t$. Thus,

$\Vcal = \{e_{t,i}\mid t=1..T,\; i=1..N_{\mathrm{pix}}\} \cup \{f_t\mid t\in\Tfr\}$

Three hyperedge types encode the relational structure:

• Spatial hyperedges $\Ecal^S$: For each time $t$ and pixel $i$, spatial neighborhood $\Ncal(i)$ induces a hyperedge $e^S_{t,\Ncal(i)} = \{e_{t,j} \mid j \in \Ncal(i)\}$.
• Temporal hyperedges $\Ecal^T$: For pixel $i$ over a temporal window $\tau$, $$e^T_{i,(t-\tau+1):t} = \{e_{t',i} \mid t' = t-\tau+1,\dots,t\}$$.
• Cross-modal hyperedges $\Ecal^C$: Each frame token $f_t$ forms $e^C_t = \{f_t\} \cup \{e_{t,i}\}$ for all $i$ within the same frame period.

The comprehensive hyperedge set $\Ecal = \Ecal^S \cup \Ecal^T \cup \Ecal^C$ enables integrated spatial, temporal, and modality-aware reasoning. The structural information is encoded via incidence matrix $H$, node-degree matrix $D_v$, and hyperedge-degree matrix $D_e$.

2. Contextual Message Passing Mechanisms

Each hypergraph convolutional layer propagates contextual information throughout the spatio-temporal hypergraph. The node feature matrix $\Xb\in\R^{|\Vcal|\times d}$ is updated as follows:

$\Xb' = \sigma\left(D_v^{-1/2} H W D_e^{-1} H^\top D_v^{-1/2} \Xb \right)$

where $\sigma(\cdot)$ is a pointwise nonlinearity (e.g., ReLU), and $W$ is a learnable filter over hyperedges. Separate filters $W^S$, $W^T$, $W^C$ are typically employed for different hyperedge types to enable edge-type-specific parameterization, with $W = \mathrm{blockdiag}(W^S, W^T, W^C)$.

This formulation can be equivalently viewed as a two-step process: aggregation of edge messages and scattering back to nodes, supporting flexible information flow across spatial, temporal, and cross-modal links.

3. Multi-Modal Event and RGB Frame Integration

EvRainDrop incorporates both event tokens and RGB frame tokens as nodes within the hypergraph, enabling multi-modal information propagation. Feature initialization employs two encoder backbones:

• Event encoder: $$x_{e_{t,i}}^{(0)} = \phi_{\mathrm{event}}(\Delta E_{t,i})$$
• Frame encoder: $x_{f_t}^{(0)} = \phi_{\mathrm{frame}}(I_t)$

Cross-modal hyperedges $\Ecal^C$ directly couple RGB information to event nodes, facilitating the completion of missing or sparse information due to event undersampling. Modality-specific weights for the three hyperedge types further enhance the ability to model complex interactions between modalities. Optional post-layer gating MLPs adaptively reweight within-modality and cross-modality contributions via learned gating parameters.

4. Temporal Aggregation with Self-Attention

After L layers of hypergraph-based message passing, each time step $t$ yields a pooled representation:

$$z_t = \mathrm{Pool}(\{x_{e_{t,i}}^{(L)}\}_i \cup \{x_{f_t}^{(L)}\})$$

Temporal aggregation across sequence length $T$ is then performed using standard scaled dot-product self-attention:

$$\begin{align*} Q &= Z W^Q\ K &= Z W^K\ V &= Z W^V\ \mathrm{Attention}(Q, K, V) &= \mathrm{softmax}\left(\frac{Q K^\top}{\sqrt{d_k}}\right) V \end{align*}$$

where $Z = [z_1; \dots; z_T ]$. The attended temporal representations support downstream classification or attribute recognition, reinforcing temporal coherence and global context exploitation.

5. Algorithmic Workflow

The EvRainDrop procedure follows the sequence below:

Input: - Event packets E₁…E_T - Key RGB frames I_{t} for t∈T_fr - Spatial neighborhood size k - Temporal window τ - # layers L 1. Construct node set: V ← { e_{t,i} for all t,i } ∪ { f_t for t∈T_fr }. 2. Build hyperedges: E^S ← for each (t,i): hyperedge connecting {e_{t,j}: j in N_k(i)} E^T ← for each i,t: hyperedge connecting {e_{t',i}: t'∈[t-τ+1,…,t]} E^C ← for each t∈T_fr: hyperedge {f_t}∪{e_{t,i}: all i} E ← E^S ∪ E^T ∪ E^C. 3. Compute incidence H, degrees D_v, D_e. 4. Initialization: for every event‐node e_{t,i}: x^{(0)} ← φ_event(E_t,i) for every frame‐node f_t : x^{(0)} ← φ_frame(I_t) 5. for ℓ=1…L do – X ← [x_v^{(ℓ-1)}]_{v∈V} ∈R^{|V|×d} – M ← W · D_e^{-1} · Hᵀ · D_v^{-½} · X – X' ← σ( D_v^{-½} · H · M ) – x_v^{(ℓ)} ← X' row-v end for 6. Temporal pooling: for t=1…T do z_t ← Pool( {x_{e_{t,i}^{(L)}_i ∪ {x_{f_t}^{(L)} ) end for 7. Self-attention over [z₁…z_T] produce Ẑ. 8. Classifier head: ŷ ← softmax(W_cls · Ẑ + b_cls). Output: predictions ŷ.

6. Experimental Validation and Performance

EvRainDrop is validated on diverse tasks involving event-stream and multi-modal event+frame data:

Dataset	Task	Metric	EvRainDrop	Best Baseline	Gain
PokerEvent	114-way single-label classification	Top-1 Acc	89.7%	84.2%	+5.5%
HARDVS	300-way single-label human activity	Top-1 Acc	76.3%	71.0%	+5.3%
MARS-Attribute	Multi-label pedestrian attr. (43 labels)	mAP	68.5%	64.0%	+4.5%
DukeMTMC-VID-Attribute	Multi-label pedestrian attr. (36 labels)	mAP	70.2%	65.1%	+5.1%

Ablation studies reveal:

• w/o hypergraph (simple Transformer): –3.8% on PokerEvent
• w/o multi-modal edges (no $\Ecal^C$): –2.9% on HARDVS
• w/o self-attention (mean pool instead): –1.7% decrease overall

These results empirically confirm that spatio-temporal hypergraph guidance, explicit cross-modal coupling, and temporal self-attention are each critical to attained performance attributes (Wang et al., 26 Nov 2025).

7. Significance and Outlook

Hypergraph-guided spatio-temporal completion addresses the persistent undersampling in event camera streams by contextualizing sparse event tokens with both spatial and temporal neighborhood structure and, where available, RGB frame information. The introduction of modality-adaptive message passing, hyperedge partitioning, and global temporal self-attention enables significant gains over non-relational and unimodal baselines. A plausible implication is that further advances in multi-modal spatio-temporal graph construction and flexible message-passing mechanisms may unlock even broader applicability for tasks involving inherently sparse or underdetermined perceptual streams.