Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

Abstract: Integrating frame-based RGB cameras with event streams offers a promising solution for robust object detection under challenging dynamic conditions. However, the inherent heterogeneity and data redundancy of these modalities often lead to prohibitive computational overhead or suboptimal feature fusion. In this paper, we propose Hyper-FEOD, a high-performance and efficient detection framework, which synergistically optimizes multi-modal interaction through two core components. First, we introduce Sparse Hypergraph-enhanced Cross-Modal Fusion (S-HCF), which leverages the inherent sparsity of event streams to construct an event-guided activity map. By performing high-order hypergraph modeling exclusively on selected motion-critical sparse tokens, S-HCF captures complex non-local dependencies between RGB and event data while overcoming the traditional complexity bottlenecks of hypergraph computation. Second, we design a Fine-Grained Mixture of Experts (FG-MoE) Enhancement module to address the diverse semantic requirements of different image regions. This module employs specialized hypergraph experts tailored for object boundaries, internal textures, and backgrounds, utilizing a pixel-level spatial gating mechanism to adaptively route and enhance features. Combined with a load-balancing loss and zero-initialization strategy, FG-MoE ensures stable training and precise feature refinement without disrupting the pre-trained backbone's distribution. Experimental results on mainstream RGB-Event benchmarks demonstrate that Hyper-FEOD achieves a superior accuracy-efficiency trade-off, outperforming state-of-the-art methods while maintaining a lightweight footprint suitable for real-time edge deployment.

• The paper introduces Hyper-FEOD, employing sparse hypergraph cross-modal fusion and a fine-grained mixture-of-experts to enhance frame-event detection.
• It reduces computational complexity by focusing on motion-critical tokens and efficiently modeling high-order dependencies across modalities.
• Experimental results on the DSEC benchmark show a significant +7.7 mAP improvement with only 13.4M parameters and 10ms inference latency.

Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE

Introduction and Problem Significance

This paper introduces Hyper-FEOD, a computationally efficient and high-performance framework for frame-event object detection that unifies RGB frames and event streams through sparse hypergraph-based feature fusion and fine-grained expert modeling (2604.11140). Traditional FEOD systems either rely on dense pairwise attention for cross-modal interaction or neglect modality-aware feature enhancement, resulting in high computational cost or incomplete exploitation of complementary cues. Hyper-FEOD addresses these deficiencies by focusing computation on motion-critical regions and adaptively refining modality-specific features.

Figure 1: Hyper-FEOD restricts high-order non-local dependency modeling to sparse motion-critical tokens while adaptively refining region-aware features via a novel mixture-of-experts strategy.

Hyper-FEOD Architecture

Hyper-FEOD takes paired RGB images and event streams as input, employing a dual-stream encoder to extract modality-specific features. The centerpiece is the Sparse Hypergraph-enhanced Cross-Modal Fusion (S-HCF) module, which utilizes event-guided sparsity to select motion-salient tokens for high-order non-local interaction using hypergraph computation. This enables efficient modeling of complex dependencies across modalities with substantially lower computational complexity compared to dense attention.

Post-fusion, a Fine-Grained Mixture of Experts (FG-MoE) module is introduced. Unlike conventional MoE, this module employs specialized hypergraph-based experts for different image regions, combined with pixel-level spatial gating. This structure allows adaptive routing of features to region-specific experts (boundary, texture, background), facilitating precise region-wise refinement and stable feature distribution.

Figure 2: Workflow of Hyper-FEOD, highlighting two-stream encoding, event-guided sparse cross-modal fusion (S-HCF), and fine-grained mixture-of-experts (FG-MoE) for region-adaptive feature enhancement.

Sparse Hypergraph Cross-Modal Fusion

Conventional cross-modal attention is fundamentally limited to low-order pairwise interactions. S-HCF transcends this by constructing hyperedges that aggregate and distribute information across both intra- and inter-modal tokens, derived from global adaptive pooling and optimized using fuzzy C-means clustering. The process comprises:

1. Cross-modal Hyperedges Construct (CHC): Spatially structured hyperedges are generated from each modality and concatenated, then refined iteratively to encode discriminative latent correlations.
2. Adaptive Hypergraph Attention (AHA): Feature aggregation and dissemination operate bidirectionally between tokens and hyperedges, implemented with a multi-head mechanism to diversify relational modeling.

The computation is focused exclusively on motion-critical tokens suggested by sparse event cues, which sharply reduces the combinatorial explosion typical in dense hypergraph frameworks.

Figure 3: Detailed structure of S-HCF, showing the construction of cross-modal hyperedges and bidirectional high-order message passing.

Fine-Grained Mixture-of-Experts (FG-MoE) Enhancement

FG-MoE adopts a unique, fine-grained strategy compared to traditional expert models. Each region (object boundary, texture, background) is adaptively routed to a dedicated hypergraph expert using a spatial gating mechanism. Training employs both load-balancing and zero-initialization regularization, ensuring stable optimization without distribution shift from the pretrained backbone. Furthermore, a dual-branch self-feature distillation mechanism is integrated, which uses the stronger fused features as targets to supervise the unimodal branches. This mutually reinforces both frame and event representations, optimizing cross-modal information transfer in a cost-free manner (i.e., no additional inference overhead).

Experimental Evaluation

Hyper-FEOD demonstrates substantial advances on three large-scale RGB-event object detection datasets, achieving state-of-the-art accuracy with minimal model size and latency. Notably, on the DSEC benchmark, Hyper-FEOD surpasses the FAOD baseline by +7.7 mAP with a parameter budget of only 13.4M and an inference latency of 10ms. Ablation studies confirm the complementary contribution of S-HCF (high-order sparse modeling) and FG-MoE (region-specialized adaptive enhancement). The effectiveness of bilateral self-distillation is substantiated, with unilateral distillation resulting in limited gains due to introduced modality imbalance.

Figure 4: Qualitative comparison on DSEC—Hyper-FEOD achieves more complete detection under challenging scenarios, reducing miss rates compared to a strong baseline.

Theoretical and Practical Implications

The sparse event-guided hypergraph paradigm sets a new computational standard for cross-modal aggregation—enabling non-local high-order dependency capture at a fraction of the complexity of conventional frameworks. This design is amenable to real-time edge deployment, facilitating robust detection under high dynamic range and rapid scene motion. The region-aware FG-MoE module highlights the efficacy of combining MoE routing with hypergraph relational priors, suggesting potential for broader application in region-adaptive perception tasks.

The bilateral self-distillation mechanism implies further theoretical synergy between cross-modal fusion and unimodal feature learning, where quality of fusion provides a learning signal that stabilizes and enhances backbone feature extractors.

Future Directions

Future work should extend the sparse hypergraph mechanism to temporal modeling, further capturing dynamic spatiotemporal correlations inherent in event-based data. Exploring end-to-end differentiable hyperedge construction, higher-level abstraction for expert specialization, and more structured balancing in multi-expert routing could further enhance model robustness and efficiency.

Conclusion

Hyper-FEOD advances the state-of-the-art in FEOD by integrating event-guided sparse high-order modeling and fine-grained expert enhancement, validated by strong empirical results and efficiency. The method presents a compelling direction for efficient, adaptive multi-modal perception under challenging, dynamic vision conditions.

(2604.11140)