EIFNet: Multi-modal Fusion in Event Segmentation

• EIFNet is a multi-modal fusion network that integrates sparse event streams and dense RGB frames for robust, state-of-the-art semantic segmentation.
• It employs a dual-encoder/single-decoder design with modules like AEFRM and MARM to address modality differences and enhance feature extraction.
• Experimental results on DDD17 and DSEC demonstrate significant mIoU improvements, validating its effective hierarchical fusion and attention strategy.

EIFNet is a multi-modal fusion network for event-based semantic segmentation that combines synchronized event streams $E$ and RGB frames $I$ in a dual-encoder / single-decoder design. It is motivated by two stated difficulties in the task: extracting reliable features from sparse and noisy event streams, and effectively fusing them with dense, semantically rich image data that differ in structure and representation. The model addresses these issues through three named components: the Adaptive Event Feature Refinement Module (AEFRM), the Modality-Adaptive Recalibration Module (MARM), and the Multi-Head Attention Gated Fusion Module (MGFM). On DDD17-Semantic and DSEC-Semantic, EIFNet is reported to achieve state-of-the-art performance, with 76.56 mIoU and 96.18 PA on DDD17, and 74.64 mIoU and 95.61 PA on DSEC (Li et al., 29 Jul 2025).

1. Problem formulation and modality rationale

Event-based semantic segmentation explores the use of event cameras, which offer high dynamic range and fine temporal resolution, for robust scene understanding in challenging environments. In EIFNet, the input consists of synchronized event streams and frame-based RGB imagery. Raw events are represented as tuples $\{(x_i,y_i,t_i,p_i)\}$ over a time window $\Delta T$, while the image branch receives RGB frames directly.

The design premise is explicitly multimodal. Event inputs are sparse and noisy, but they preserve temporal activity structure; RGB frames are dense and semantically rich, but they differ from event data in representation and structure. EIFNet therefore does not treat either modality as auxiliary. Instead, it constructs an event branch and an image branch, then recalibrates and fuses their features at four encoder stages. This organization places the model within the fused setting reported in the quantitative comparison, alongside EDCNet-S2D, HALSIE, CMX, CMNeXt, and EISNet (Li et al., 29 Jul 2025).

2. System architecture

EIFNet adopts a dual-encoder / single-decoder design. Events are first encoded into a 4D tensor via AEFRM, and images are processed directly by a Transformer backbone. The event branch uses MiT-B0, and the image branch uses MiT-B2. At each of four encoder stages, with feature maps at resolutions $1/4$, $1/8$, $1/16$, and $1/32$, the features from both branches are first recalibrated independently by MARM and then fused via MGFM.

The four fused feature maps $\{M_1,M_2,M_3,M_4\}$ are aggregated by concatenation followed by $1\times1$ convolution. The aggregated representation is then passed to a lightweight Transformer decoder that upsamples to full resolution and outputs per-pixel class probabilities. This arrangement makes the fusion process explicitly hierarchical rather than deferred to a single late-fusion block, and the paper attributes the final segmentation performance to the combined effect of event refinement, modality-specific recalibration, and gated cross-modal fusion (Li et al., 29 Jul 2025).

3. Adaptive Event Feature Refinement Module (AEFRM)

AEFRM takes raw event tuples $I$0 over a time window $I$1 and produces an enhanced event feature tensor $I$2. Its first step is polarity-aware projection and activity-map construction. With

$I$3

the polarity-aware event volume is defined as

$I$4

and the activity count map as

$I$5

These are discretized into $I$6 temporal bins, yielding a tensor of shape $I$7, with $I$8 for polarity.

AEFRM then performs multi-scale spatial modeling on $I$9 using three parallel branches:

$\{(x_i,y_i,t_i,p_i)\}$0

$\{(x_i,y_i,t_i,p_i)\}$1

$\{(x_i,y_i,t_i,p_i)\}$2

These branches are fused and refined as

$\{(x_i,y_i,t_i,p_i)\}$3

where $\{(x_i,y_i,t_i,p_i)\}$4.

The module then applies channel-wise attention:

$\{(x_i,y_i,t_i,p_i)\}$5

$\{(x_i,y_i,t_i,p_i)\}$6

followed by a spatial attention mask

$\{(x_i,y_i,t_i,p_i)\}$7

The enhanced event representation is finally computed as

$\{(x_i,y_i,t_i,p_i)\}$8

Within the reported architecture, AEFRM is the event-specific mechanism for transforming sparse event streams into features suitable for stage-wise cross-modal interaction (Li et al., 29 Jul 2025).

4. Modality-adaptive recalibration and gated fusion

MARM is applied separately at each stage to event features $\{(x_i,y_i,t_i,p_i)\}$9 and image features $\Delta T$0. Its channel recalibration step is

$\Delta T$1

$\Delta T$2

$\Delta T$3

Spatial recalibration then uses

$\Delta T$4

$\Delta T$5

followed by a split into $\Delta T$6 and residual recalibration:

$\Delta T$7

$\Delta T$8

where $\Delta T$9 and $1/4$0 are learnable scalars.

MGFM then takes $1/4$1 and $1/4$2 as inputs. Its first component is bidirectional attention. For events, Differential Attention is used in sparse-data focus form:

$1/4$3

For images, Efficient Cross-Attention is used in dense focus form:

$1/4$4

Each attention is implemented in multi-head form, with head $1/4$5 defined by

$1/4$6

$1/4$7

and

$1/4$8

The second component is gated attention generation. With

$1/4$9

the channel and spatial gate terms are

$1/8$0

$1/8$1

The gate is

$1/8$2

which is split into $1/8$3. Fusion and feed-forward are then given by

$1/8$4

$1/8$5

Taken together, MARM and MGFM define EIFNet’s modality interaction scheme: recalibration occurs before fusion, and fusion itself is both attention-based and gate-controlled (Li et al., 29 Jul 2025).

5. Optimization protocol and empirical performance

EIFNet is trained with per-pixel cross-entropy,

$1/8$6

The optimizer is AdamW with initial learning rate $1/8$7 and weight decay $1/8$8. Training runs for 60 epochs, with no learning-rate warmup and cosine decay to 0. The batch size is 16 with 8 data-loader workers. Data augmentation consists of random horizontal flip, resize, and center-crop to $1/8$9 for DDD17 and $1/16$0 for DSEC. Event encoding uses AEFRM with $1/16$1 temporal bins and $1/16$2. The backbones are initialized with ImageNet weights.

In the reported comparison on DDD17 and DSEC, the image-only baselines SegFormer-B2 and SegNeXt-B reach 71.05 and 71.46 mIoU on DDD17, and 71.99 and 71.55 mIoU on DSEC. The event-only baselines EV-SegNet and ESS report 54.81 and 61.37 mIoU on DDD17, and 51.76 and 51.57 mIoU on DSEC. Among fused methods, EDCNet-S2D, HALSIE, CMX, CMNeXt, and EISNet report DDD17 mIoU values of 61.99, 60.66, 71.88, 72.67, and 73.41, respectively, while their DSEC mIoU values are 56.75, 52.43, 72.42, 72.54, and 73.07. EIFNet reports 76.56 mIoU and 96.18 PA on DDD17, and 74.64 mIoU and 95.61 PA on DSEC, and is stated to set a new state-of-the-art on both datasets with +3.15 mIoU over the prior best (Li et al., 29 Jul 2025).

6. Ablation, efficiency, and interpretive significance

The ablation study is conducted on DDD17 with EISNet as the baseline. Adding AEFRM alone yields 74.69 mIoU and 96.03 PA. Adding MARM alone yields 74.11 mIoU and 96.09 PA. Adding MGFM alone yields 74.56 mIoU and 96.09 PA. The two-module variants report 75.58 mIoU and 96.10 PA for AEFRM + MARM, 74.94 mIoU and 95.68 PA for AEFRM + MGFM, and 76.36 mIoU and 96.08 PA for MARM + MGFM. The full model, EIFNet (AEFRM + MARM + MGFM), reaches 76.55 mIoU and 96.19 PA. The reported gains over baseline are +1.28 mIoU for AEFRM, +0.70 mIoU for MARM, +1.15 mIoU for MGFM, and +3.14 mIoU for the full synergy.

The efficiency comparison reports 66.56M parameters and 16.29G FLOPs for CMX (2×MiT-B2), 34.39M parameters and 17.30G FLOPs for EISNet (MiT-B0+B2), and 35.48M parameters and 17.89G FLOPs for EIFNet (MiT-B0+B2). Despite multi-stage attention, EIFNet is reported to run at $1/16$3 FPS on RTX 3090 and to maintain similar complexity to EISNet.

A common simplification is to conflate event-based semantic segmentation with event-only processing. The reported benchmark does not support that simplification: it includes image-only, event-only, and fused methods, and EIFNet is evaluated in the fused category. Another simplification is to view event encoding and fusion as separable engineering steps. The ablation pattern suggests otherwise: the largest result is obtained by the full combination of AEFRM, MARM, and MGFM rather than by any single component in isolation. A plausible implication is that EIFNet’s contribution lies less in any one module than in the staged coupling of event refinement, modality-specific recalibration, and gated cross-modal attention under real-time constraints (Li et al., 29 Jul 2025).