Event-Prior Sparsification in Multimodal Detection

• Event-Prior Sparsification is an adaptive mechanism that discards low-information tokens by leveraging spatio-temporal cues from event cameras.
• It computes dynamic token importance using the event-pixel ratio and softmax scaling, enabling efficient token scoring and adaptive thresholding.
• Integrated in the FocusMamba architecture, this approach boosts detection metrics and reduces computational FLOPs by up to 30–40% in challenging environments.

Event-prior sparsification is an adaptive mechanism for discarding low-information tokens in multimodal object detection pipelines, principally those combining RGB images and event camera data. Unlike prior methods that apply uniform or fixed-threshold token pruning, event-prior sparsification leverages spatio-temporal cues directly derived from event cameras, guiding token selection in both image and event domains according to scene complexity. The approach was introduced and systematically developed in the context of the FocusMamba architecture for RGB-event object detection, delivering substantial computational savings while enhancing detection metrics on established benchmarks (Yang et al., 4 Sep 2025).

1. Principles and Motivations

Traditional RGB-event fusion networks process both high- and low-information spatial regions uniformly in feature extraction and modality fusion, leading to excessive computational expenditure and suboptimal accuracy due to the dominance of background or noise tokens. Preceding sparsification approaches for both modalities rely on global keep rates or static thresholds, which fail to accommodate variation in informational density across scenes. Event-prior sparsification instead introduces an adaptive, content-driven sparsification mechanism exploiting the instantaneous event-pixel ratio—an intrinsic dynamic statistic from event cameras quantifying spatial activity—to modulate both token importance scoring and the pruning threshold at inference time.

2. Event-Guided Multimodal Sparsification (EGMS)

Event-guided multimodal sparsification (EGMS) forms the central event-prior sparsification mechanism. It operates in two key stages: modality-specific token scoring and an event-guided binarization process.

Token Representation:

• RGB images ($H\times W$) are divided into $P\times P$ patches, forming $N\approx HW/P^2$ linear-projected image tokens ($X_I \in\mathbb{R}^{N\times C}$).
• Event streams are discretized over a window $\Delta t$ into $V$ temporal bins; the same spatial partition produces event tokens ($X_E \in\mathbb{R}^{N\times C}$).

Scoring:

• Image token scores: $\ell_2$ norm of token feature, $S_I\in\mathbb{R}^{N}$, where $(S_I)_i = \|X_{I,i}\|_2$.
• Event token scores: event timestamp accumulation per spatial bin, followed by Gaussian smoothing:

$$S_E^T[x,y]=\sum_{k: (x_k,y_k)=(x,y)}t_k;\quad S_E[c]=\frac{\sum_{q\in\Omega}\exp(-\|q-c\|^2/(2\sigma^2))S_E^T[q]}{\sum_{q\in\Omega}\exp(-\|q-c\|^2/(2\sigma^2))}$$

Event-Guided Control Mechanism (EGCM):

• Event-pixel ratio $$r = \# \text{(pixels with at least one event in } \Delta t) / (H \times W)$$ quantifies scene sparsity/activity.
• Scaling factor $\alpha_s = r^{1/\rho}$ (with $\rho > 0$ hyperparameter) modulates contrast:

$$\widetilde{S}_I = \mathrm{softmax}(S_I/\alpha_s),\quad \widetilde{S}_E = \mathrm{softmax}(S_E/\alpha_s)$$

• Token-selection threshold $\tau = {(1-r)^{1/\rho}}/N$ prunes more aggressively with sparser event activity.
• Sparsification masks:

$$M_I[i]= \begin{cases} 1, & \widetilde{S}_I[i]\geq\tau\ 0, & \text{otherwise} \end{cases},\quad M_E[i]= \begin{cases} 1, & \widetilde{S}_E[i]\geq\tau\ 0, & \text{otherwise} \end{cases}$$

Masked tokens ($M_I, M_E$) subsequently gate both the selective scan and MLP computations in the sparse Vision Mamba (VSS) backbone, yielding $30$–$40$\% reduction in computational FLOPs (see Table IV in (Yang et al., 4 Sep 2025)).

3. Mathematical Workflow and Pseudocode

The process can be formalized as follows:

• Image scoring: $S_I = \|X_I\|_2$
• Event scoring: $S_E = \mathrm{GaussianFilter}(\sum_t t,\,\sigma)$
• Softmax scaling: $\widetilde{S}_I = \mathrm{softmax}(S_I/\alpha_s)$, $\widetilde{S}_E = \mathrm{softmax}(S_E/\alpha_s)$
• Adaptive threshold: $\tau = {(1-r)^{1/\rho}}/N$
• Mask generation as above

Pseudocode excerpt:

def EGMS(X_I, X_E, event_stream, ρ, σ, P): # 1) Compute raw scores S_I = [l2_norm(x) for x in X_I] S_E_temp = accumulate_timestamps(event_stream, P) S_E = gaussian_smooth(S_E_temp, σ) # 2) Event-pixel ratio r = (num_event_pixels) / (H*W) # 3) Scaling and softmax α_s = r**(1/ρ) S_I = softmax(S_I / α_s) S_E = softmax(S_E / α_s) # 4) Threshold Control = (1-r)**(1/ρ) τ = Control / N # 5) Masks M_I = [1 if S_I[i] >= τ else 0 for i in range(N)] M_E = [1 if S_E[i] >= τ else 0 for i in range(N)] return M_I, M_E

This workflow ensures the token selection is both scene- and modality-adaptive, discarding more tokens in low-activity backgrounds and retaining salient tokens as complexity increases.

4. Cross-Modality Focus Fusion (CMFF)

Following sparsification, only tokens designated as important by $M_I$ and $M_E$ are submitted to cross-modality fusion via the Cross-Modality Focus Fusion (CMFF) module. CMFF comprises:

• Complementarity-Aware Enhancement (CAE): Identifies regions with unique visibility in one modality (e.g., $\Delta_I = M_E~\mathrm{xor}~(M_E \wedge M_I)$); corresponding feature channels are selectively amplified by a factor $\beta > 1$.
• Focused Interlaced Mamba (FI-Mamba): Aggregates tokens where either $M_I$ or $M_E$ is active; tokens are interleaved and processed with a bi-directional state-space scan (bidirectional SSM) to capture inter- and intra-modal dependencies. Enhanced features $F_I^{\mathrm{enh}}$ and $F_E^{\mathrm{enh}}$ are then summed and fused via a SparseMLP restricted to the selected tokens.

This two-stage fusion suppresses background, accentuates complementary structure between the RGB and event streams, and confines expensive global reasoning to joint "focus" regions.

5. Experimental Characterization

Ablation studies on DSEC-Det (640×640) and PKU-DAVIS-SOD datasets establish the quantitative impact and trade-offs of event-prior sparsification:

Configuration	mAP50/mAP	FLOPs (G)
Baseline (no sparsification)	50.3 / 31.6	81.5
+EGMS only	51.2 / 32.4	54.5 (−33.1%)
+EGMS + CAE	53.6 / 33.6	54.2
+EGMS + CAE + FI-Mamba	55.3 / 34.6	60.8

Comparative analysis with state-of-the-art token pruning networks demonstrates that EGMS delivers higher mAP at lower FLOPs (e.g., FocusMamba: 55.3/34.6 @ 60.8G FLOPs versus AS-ViT: 53.7/33.0 @ 65.3G FLOPs). The approach achieves an improvement of 4.2% mAP on DSEC-Det while utilizing only 29% of typical SOTA FLOPs.

Event-guided control factors (scaling and threshold modulation) are crucial; ablations removing either degrade mAP by 1–3 percentage points and are less efficient.

6. Comparative Perspective and Implications

Event-prior sparsification, as instantiated in FocusMamba, supersedes fixed-rate or purely image-based sparsification by dynamically allocating compute to "focus" regions according to semantic and physical scene activity. Benchmark results reveal that scene adaptivity, as measured via the event-pixel ratio, is essential for maintaining detection accuracy while reducing compute loads. Integration of this sparsification regime with cross-modality fusion provides further gains, enabling competitive or superior performance to prior fusion-centric and sparsification-centric architectures at a fraction of the computational cost.

A plausible implication is that event-driven paradigms for network sparsification may generalize to broader settings in multimodal vision, especially where temporal structure or sensor activity is naturally informative.

7. Applications and Extensions

Event-prior sparsification is especially well-suited to resource-constrained object detection where low latency and high efficiency are central—such as robotics, automotive perception, and edge deployment of sensor fusion models. The method exhibits robust adaptability to scenes of varying complexity, making it relevant for datasets and domains characterized by dynamic scene changes and structured background clutter.

Further research may examine the utility of event-prior mechanisms for other tasks (e.g., segmentation, tracking) or for alternative sensor modalities, and investigate alternative statistics beyond the event-pixel ratio for guiding adaptive sparsification in multimodal pipelines (Yang et al., 4 Sep 2025).