EventSSEG: Event-based Semantic Segmentation

• EventSSEG is an event-based semantic segmentation framework that assigns per-pixel labels from asynchronous, high-dynamic-range outputs of neuromorphic cameras.
• It employs diverse representations—from voxel grids and event volumes to 3D event graphs—allowing both event-only and multi-modal fusion architectures to capture motion and structure.
• Researchers address challenges in supervision and label scarcity using unsupervised adaptation, weak labeling, and annotation-free methods to enhance segmentation performance.

Event-based semantic segmentation, often abbreviated EventSSEG, denotes the problem of assigning scene labels from the asynchronous output of neuromorphic cameras rather than from conventional frames. In its strict sense, the task is dense per-pixel semantic labeling; in closely related work built on the same sensing model, the scope extends to motion-layer segmentation, panoptic and instance segmentation, part segmentation, and structural parsing such as line-segment extraction. The field is motivated by the fact that event cameras emit brightness-change tuples $e_i=(x_i,y_i,t_i,p_i)$ with microsecond temporal precision and very high dynamic range, making them well suited to high-speed motion, low light, and HDR scenes where frame-based perception degrades (Sun et al., 2022, Stoffregen et al., 2019).

1. Sensor model and task formulations

The basic event-camera measurement model is shared across the literature. An event is triggered when the change in log-intensity exceeds a threshold, for example

$$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$

with $p$ denoting polarity and $C$ the contrast threshold (Zhang et al., 2020). In another common notation, the same mechanism is written as

$$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$

with each event stored as $e_k=(x_k,t_k,s_k)$ (Stoffregen et al., 2019). These sensors sample at the rate of scene dynamics rather than at a fixed exposure schedule.

For dense EventSSEG, the computational objective is typically to map an event representation $E$ or $V$ to per-pixel class probabilities $P(x,y,c)$. One explicit formulation writes event-based semantic segmentation as learning a mapping

$E \mapsto P(x,y,c),$

where $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$0 is the predicted probability that pixel $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$1 belongs to class $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$2 (Zhu et al., 2024). This formulation underlies event-only segmentation systems such as ESS, SLTNet, and EvSegSNN, as well as hybrid RGB-event systems such as HALSIE and ISSAFE (Sun et al., 2022, Zhu et al., 2024, Hareb et al., 2024, Biswas et al., 2022, Zhang et al., 2020).

Closely related variants broaden the segmentation target. Motion segmentation assigns each event to one of several motion layers and jointly estimates layer motion parameters, yielding per-event rather than per-pixel labels (Stoffregen et al., 2019). Panoptic and instance-oriented work segments objects or object parts from event streams, sometimes in open vocabulary with free-form language prompts (Kachole et al., 2023, Lee et al., 30 Jan 2026). Structural segmentation uses the same event/frame fusion principle to recover line segments and junctions from motion-blurred imagery (Yu et al., 2022). Taken together, these formulations show that EventSSEG is best understood as a family of segmentation problems whose common substrate is asynchronous event sensing.

2. Event representations and computational primitives

Because raw events are sparse and asynchronous, most EventSSEG systems first convert them into network-friendly tensors. A canonical representation is the discretized event volume used by ISSAFE, where timestamps are normalized into bins and positive and negative events are accumulated separately: $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$3

$$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$4

with final volume $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$5, $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$6 (Zhang et al., 2020). ESS, EV-WSSS, SEAL, and SLTNet likewise rely on voxel-grid-like encodings, differing mainly in bin count, polarity handling, and downstream architecture (Sun et al., 2022, Cho et al., 2024, Lee et al., 30 Jan 2026, Zhu et al., 2024).

SLTNet writes a voxel grid over a time interval $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$7 as

$$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$8

and then feeds an event tensor $$L(x,y,t)-L(x,y,t-\Delta t)\ge pC,\quad p\in\{-1,+1\},$$9 to a spiking encoder (Zhu et al., 2024). SEAL also uses voxel grids and describes them as one of several event representations alongside E2VID reconstructed frames and spike tensors (Lee et al., 30 Jan 2026). For line-structure extraction, FE-LSD uses an Event Spike Tensor with $p$0 temporal bins and separate accumulation for positive and negative polarities (Yu et al., 2022).

Not all EventSSEG work rasterizes events into images. GMNN constructs 3D event graphs over nodes $p$1, with edges defined by spatiotemporal $p$2-nearest neighbors in normalized space-time; its Collaborative Contextual Mixing layer then aggregates features across $p$3-neighbor pyramids (Kachole et al., 2023). Event-based motion segmentation by motion compensation operates directly on warped events via weighted Images of Warped Events (IWEs),

$p$4

and maximizes the sum of their variances,

$p$5

to obtain motion-consistent event partitions (Stoffregen et al., 2019). These alternatives show that the field is not tied to a single representation; voxel grids, event volumes, event graphs, and warped-event images each support different segmentation regimes.

3. Architectural families

A first major family uses multi-modal fusion of events with frames. ESS aligns a recurrent, motion-invariant event embedding from a pre-trained E2VID encoder with an image embedding from a ResNet-18-based image encoder, then applies a shared task decoder to both domains (Sun et al., 2022). HALSIE adopts a dual-encoder design: a spiking Temporal Feature Extractor processes event bins sequentially, a conventional Spatial Feature Extractor processes synchronized grayscale frames, and a Multi-Scale Mixer fuses the resulting embeddings before a lightweight semantic head predicts the mask (Biswas et al., 2022). ISSAFE specializes multi-modal fusion to accidental scenarios through Event-aware Fusion and Event-aware Domain Adaptation; its more effective Dense-to-Sparse variant supervises an auxiliary event-prediction branch with BCE while the segmentation branch uses CE, combining them as

$p$6

(Zhang et al., 2020). These systems share the premise that events contribute motion-sensitive, HDR-robust cues while frames contribute dense spatial context.

A second family is event-only dense segmentation. EvSegSNN uses a biologically inspired U-shaped encoder-decoder with Parametric Leaky Integrate-and-Fire neurons and avoids batch normalization to improve biological plausibility and neuromorphic deployability (Hareb et al., 2024). SLTNet pursues the same direction with a single-branch spike-driven lightweight transformer-based SNN. Its Spiking Lightweight Dilated modules handle local multi-scale features, while its Spike-driven Multi-head Self-Attention uses

$p$7

to reduce standard transformer-style dense operations to masked spike-driven accumulation (Zhu et al., 2024). This line of work treats event sparsity not as a nuisance to be converted away, but as a computational prior.

A third family preserves event asynchrony more explicitly. GMNN performs asynchronous panoptic-style segmentation on 3D event graphs and propagates contextual information over multi-scale neighborhoods using its Collaborative Contextual Mixing layer (Kachole et al., 2023). Motion segmentation by motion compensation similarly stays close to the raw event stream, jointly inferring motion parameters and per-event cluster probabilities by aligning event packets in space-time (Stoffregen et al., 2019). These methods are particularly relevant when the segmentation target is dynamic-object separation or motion-layer reasoning rather than dense semantic maps alone.

A fourth family moves toward open-vocabulary segmentation. SEAL builds on EventSAM and introduces a multimodal fusion network that combines event features, SAM mask tokens, and CLIP-space text features, thereby supporting both event segmentation and open-vocabulary mask classification at instance and part granularity (Lee et al., 30 Jan 2026). This extends EventSSEG from closed-set semantic parsing into prompt-based, language-aware scene understanding.

4. Supervision, adaptation, and weak labels

Label scarcity is a persistent structural problem in EventSSEG. ESS addresses it through unsupervised domain adaptation from labeled still images to unlabeled event streams. Its total objective combines supervised source-domain segmentation with embedding, prediction, and task-feature consistency: $p$8 (Sun et al., 2022). The key idea is to align recurrent event embeddings with image embeddings without requiring paired video, per-pixel alignment, or synthetic motion hallucination.

HPL-ESS also targets unlabeled events, but treats pseudo-label quality itself as the central problem. It combines self-training on event inputs with pseudo labels obtained from event-to-image reconstruction and mixes the two by noisy-label learning: $p$9 It further introduces a Soft Prototypical Alignment module based on Jensen–Shannon divergence between prototype-similarity distributions, improving consistency between reconstructed-image features and event features as well as between different target-domain event subsets (Jing et al., 2024). This formulation directly addresses confirmation bias from single-source pseudo labeling.

Weak supervision offers a different route. EV-WSSS assumes only sparse point annotations under a “1-Class-1-Click” protocol and trains asymmetric forward and backward event branches with a point-supervision loss

$C$0

a cross-branch pseudo-label loss, class-wise prototype contrastive learning, and feature-space distillation for prototype exchange (Cho et al., 2024). The result is a dense prediction framework that uses neither pixel-wise event labels nor RGB guidance at annotation time.

SEAL pushes label efficiency further through annotation-free domain adaptation. During training it uses event–image pairs and automatically generated hierarchical masks and captions rather than manual event annotations, then aligns event mask features to both visual and text guidance through a cosine-similarity distillation loss over semantic-, instance-, and part-level masks (Lee et al., 30 Jan 2026). This suggests that future EventSSEG systems may increasingly depend on foundation-model supervision rather than dense manual masks.

5. Benchmarks, datasets, and representative results

The empirical landscape of EventSSEG is organized around a diverse set of benchmarks spanning driving, accidents, indoor clutter, open-vocabulary segmentation, and structural parsing.

Benchmark	Task emphasis	Stated properties
DDD17	Driving semantic segmentation	15,950 training samples, 3,890 testing samples, 6 classes
DSEC-Semantic	Large-scale driving semantic segmentation	8,082 training samples, 2,809 testing samples, 11 classes, $C$1 labels
DADA-seg	Accident semantic segmentation	313 accident sequences, 40 frames each, 313 labeled frames and 12,207 unlabeled frames
ESD	Indoor cluttered object segmentation	145 sequences, 14,166 RGB frames, stereo event cameras, depth information
DDD17-Ins / DSEC11-Ins / DSEC19-Ins / DSEC-Part	Open-vocabulary instance and part segmentation	Four benchmarks spanning coarse-to-fine label granularity and instance-to-part semantics
FE-Wireframe / FE-Blurframe	Structural line segmentation with events	Pairwise motion-blurred images and events; synthetic and real datasets

These benchmarks are introduced and used across ESS, ISSAFE, ESD, SEAL, and FE-LSD (Sun et al., 2022, Zhang et al., 2020, Huang et al., 2023, Lee et al., 30 Jan 2026, Yu et al., 2022).

Several results have become reference points. On DADA-seg, ISSAFE-D2S with $C$2 event bins improves a SwiftNet RGB-only baseline from $C$3 mIoU to $C$4, a headline gain of $C$5 mIoU, while also maintaining Cityscapes performance (Zhang et al., 2020). On DDD17 and DSEC-Semantic, SLTNet reports $C$6 and $C$7 mIoU, outperforms state-of-the-art SNN-based methods by at most $C$8 and $C$9 mIoU, respectively, achieves $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$0 FPS, and is reported with extremely $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$1 lower energy consumption (Zhu et al., 2024). On DSEC-Semantic, HPL-ESS reaches $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$2 accuracy and $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$3 mIoU, improving over ESS-UDA by $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$4 accuracy and $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$5 mIoU and even surpassing several supervised methods (Jing et al., 2024).

Weak supervision has also become competitive. EV-WSSS reaches $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$6 mIoU on DSEC-Semantic with 1C1C point labels and $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$7 with 1C10C, while on DDD17 it reports $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$8 and $$\Delta L(x_k,t_k)\doteq L(x_k,t_k)-L(x_k,t_k-\Delta t_k)=s_k\,C,$$9 mIoU under the same two settings (Cho et al., 2024). In asynchronous graph-based segmentation on ESD known objects, GMNN reports $e_k=(x_k,t_k,s_k)$0 mIoU and $e_k=(x_k,t_k,s_k)$1 accuracy, and on unknown objects $e_k=(x_k,t_k,s_k)$2 mIoU and $e_k=(x_k,t_k,s_k)$3 accuracy (Kachole et al., 2023). In open-vocabulary event instance segmentation, SEAL reports, for example, about $e_k=(x_k,t_k,s_k)$4 AP on DDD17-Ins with box prompts and $e_k=(x_k,t_k,s_k)$5 box AP on DSEC-Part while remaining markedly more parameter- and runtime-efficient than hybrid baselines (Lee et al., 30 Jan 2026). In structural parsing, FE-LSD achieves $e_k=(x_k,t_k,s_k)$6 msAP on FE-Blurframe after pre-training on FE-Wireframe and fine-tuning on FE-Blurframe (Yu et al., 2022). Finally, motion-only event segmentation achieves around $e_k=(x_k,t_k,s_k)$7 per-event accuracy at about $e_k=(x_k,t_k,s_k)$8 pixels relative displacement in ESIM-based evaluation (Stoffregen et al., 2019). The overall empirical picture is that event information is most valuable when motion blur, HDR, low light, or unusual object dynamics make frame-only perception unreliable.

6. Limitations, misconceptions, and research directions

A recurring misconception is that event cameras alone solve segmentation. The literature instead shows a sharper trade-off. Event streams are highly informative for fast motion and HDR scenes, but they remain sparse, asynchronous, and weak in static texture; several works explicitly note that event-only representations can lack dense spatial context, especially for interiors of objects or slowly moving regions (Biswas et al., 2022, Sun et al., 2022). This is why multi-modal fusion remains strong in accidents, driving, and structural parsing, and why event-only models often emphasize specialized encoders or spiking dynamics rather than naïve frame-style CNN processing (Zhang et al., 2020, Biswas et al., 2022, Zhu et al., 2024).

Another recurrent limitation is supervision. Dense event labels are expensive, pseudo labels can be noisy, and cross-domain transfer can reinforce errors if it trusts a single label source. HPL-ESS, EV-WSSS, and SEAL all respond to this problem with different mechanisms—hybrid pseudo labeling, sparse point supervision with prototype exchange, and annotation-free visual/text guidance—but none remove the need for careful representation and confidence modeling (Jing et al., 2024, Cho et al., 2024, Lee et al., 30 Jan 2026). This suggests that future progress will continue to depend on better uncertainty-aware training and on more principled use of auxiliary modalities.

The literature also identifies task-specific open problems. Motion-layer methods assume piecewise rigid motion and a known number of clusters, and they can fail when only inappropriate motion models are used (Stoffregen et al., 2019). Open-vocabulary systems remain sensitive to low event-rate regions and domain shifts outside driving scenes (Lee et al., 30 Jan 2026). Indoor clutter benchmarks reveal that models trained on driving data transfer poorly to static tabletop settings, especially under occlusion and unseen objects (Huang et al., 2023). Accident-focused segmentation exposes catastrophic robustness gaps between nominal driving datasets and extreme scenes, even for strong frame-based baselines (Zhang et al., 2020).

Research directions already present in the literature are correspondingly diverse. Several works point toward deeper spatiotemporal modeling, stronger event-native backbones, and expansion from semantic to instance, part, panoptic, and 3D reasoning (Lee et al., 30 Jan 2026, Zhu et al., 2024, Huang et al., 2023). Neuromorphic efficiency remains central: SLTNet and EvSegSNN indicate that spiking models can make event-only segmentation more practical on resource-constrained hardware (Zhu et al., 2024, Hareb et al., 2024). Domain adaptation from normal to accident scenes, events as a cross-domain bridge, and open-vocabulary language grounding all appear especially promising because they address exactly the regimes where event sensing is most distinctive (Zhang et al., 2020, Jing et al., 2024, Lee et al., 30 Jan 2026). Overall, EventSSEG is evolving from a narrow dense-labeling problem into a broader event-centric scene understanding stack in which representation, supervision, efficiency, and robustness are inseparable.