---
title: 'OV-EIS: Open-Vocabulary Event Segmentation'
url: https://www.emergentmind.com/topics/open-vocabulary-event-instance-segmentation-ov-eis
type: topic
---

# OV-EIS: Open-Vocabulary Event Segmentation

Open-Vocabulary Event Instance Segmentation (OV-EIS) denotes the task of segmenting instances in event camera streams and assigning open-set, language-grounded labels to each predicted mask. In OV-EIS, the system receives spatiotemporal event data from event-based sensors and a set of arbitrary text queries describing potential categories or parts. The objective is to produce segmentation masks delineating individual event instances and to select, for each instance, a semantically consistent label from the potentially unbounded vocabulary of input queries. This is performed without restricting classification to a fixed, pre-defined taxonomy, enabling both semantic and part-level scene understanding at arbitrary levels of granularity [2601.23159].

## 1. Formal Task Definition and Notation

Given a stream from an event camera as $e = \{(x_i, y_i, t_i, p_i)\}_{i=1}^N$, where $(x_i, y_i)$ are spatial coordinates, $t_i$ is the timestamp, and $p_i \in \{+1, -1\}$ represents polarity, events are aggregated into a regular representation $I_{\rm evt} \in \mathbb{R}^{C \times H \times W}$ (commonly using spatiotemporal voxel grids, frame reconstructions, or spike encoding). The system is provided with a set of free-form language queries $\{q_j\}_{j=1}^M$ and visual prompts $P$ (points or boxes). The OV-EIS task is to predict binary masks $\{M_k \in \{0, 1\}^{H \times W}\}_{k=1}^K$ for each instance and assign a label $c_k$ from the open vocabulary of queries.

The functional pipeline is as follows:
- $M = \text{MaskGen}(I_{\rm evt}, P)$ (class-agnostic mask generation)
- $s_k = \text{ClassScore}(\text{MaskFeat}(M_k, I_{\rm evt}), \text{TextEmbed}(\{q_j\}))$ (similarity scoring)
- $c_k = \arg\max_j s_k[j]$ (selecting the top scoring query for each segment)

The core requirement of OV-EIS is that mask generation is class-agnostic, and label assignment leverages open-vocabulary classification through cross-modal feature alignment.

## 2. The SEAL Framework for OV-EIS

The SEAL ("Semantic-aware Segment Any Events with Language") architecture is designed to unify event-domain instance segmentation and open-vocabulary classification via a parameter-efficient model. The main components are:

- **Input Representation**: Events are discretized into voxel grids ($I_{\rm evt} \in \mathbb{R}^{3 \times H \times W}$ with three temporal bins), sampled over fixed windows (e.g., 25 ms for DSEC, 15 ms for DDD17). SEAL does not require frame-level reconstruction at inference and operates directly on event voxel grids for computational efficiency.

- **Backbone**: SEAL uses a ViT-B-based EventSAM backbone ($F_{\rm evt}$), pretrained and then frozen, to encode $I_{\rm evt}$ into a token map $T_{\rm evt} \in \mathbb{R}^{D \times H/32 \times W/32}$. The mask decoder, adapted from SAM, generates class-agnostic binary masks $\{M_k\}$ based on visual prompts $P$.

- **Multimodal Hierarchical Semantic Guidance (MHSG)**: During training, hierarchical supervision is derived from paired RGB images and captions generated at three semantic levels: semantic, instance, and part. Visual guidance employs SAM to segment images; CLIP is used to extract visual and text features for alignment.

- **Fusion Network**: A multimodal fusion network is layered atop the backbone and consists of: 
  - a backbone feature enhancer (Transformer layers with cross-attention to text anchors),
  - a spatial encoding module combining mask tokens and pooled semantic features,
  - and a mask feature enhancer using cross-attention masked to the predicted region.

- **Classification Head**: For each candidate mask, cosine similarity is computed between the mask’s feature embedding and embeddings of user-provided queries, with the maximum determining the mask’s label.

## 3. Training Paradigm and Loss Functions

SEAL employs a two-stage training regime:

- **Stage 1**: EventSAM pretraining via distillation from SAM image features, using mixed event-image pairs (no human annotations). The backbone is pretrained using cross-entropy and Dice losses.

- **Stage 2**: Multimodal fusion training (MHSG) aligns event-based mask features to both CLIP-derived visual features and text features across three levels (semantic, instance, part). The objective is to minimize

$$
\mathcal{L}_{\rm distill} = \sum_{l\in\{s,i,p\}} \sum_{k=1}^{K_l} \left[ (1-\cos(m_{l,k}^e, v_{l,k}^I)) + \alpha (1-\cos(m_{l,k}^e, v_{l,k}^T)) \right]
$$

where $\cos(a, b)$ is the cosine similarity, $\alpha$ weights text/visual alignment, and $K_l$ is the number of masks at level $l$. After stage 1, the EventSAM backbone and decoder are frozen; only fusion modules and projection layers are updated in stage 2 [2601.23159].

## 4. Benchmark Datasets and Evaluation Protocols

Four OV-EIS benchmarks were curated from the DDD17-Seg and DSEC-Semantic datasets by generating SAM masks and assigning them semantic or part-level labels. Benchmarks differ in class granularity, sequence count, spatial resolution, and segmentation level:

| Benchmark      | Classes           | Events/Seq | Resolution | Levels                   |
|----------------|-------------------|------------|------------|--------------------------|
| DDD17-Ins      | 5                 | 3,890      | 352×200    | semantic + instance      |
| DSEC11-Ins     | 7                 | 2,809      | 640×440    | semantic + instance      |
| DSEC19-Ins     | 14                | 2,809      | 640×440    | semantic + instance      |
| DSEC-Part      | 9 (parts)         | 2,809      | 640×440    | part-level only          |

The principal metric is Average Precision (AP), computed as the mean of AP@τ over $\tau \in \{0.5 : 0.05 : 0.95\}$ (area under the precision-recall curve at each IoU threshold). Auxiliary reporting includes AP50 and AP25.

## 5. Empirical Performance and Efficiency

SEAL achieves state-of-the-art results compared to prior open-vocabulary or event segmentation methods. On the DDD17-Ins, DSEC11-Ins, DSEC19-Ins, and DSEC-Part benchmarks, SEAL demonstrates gains in AP, AP50, and AP25 over all provided baselines, as summarized:

| Benchmark   | Metric | Best Baseline      | SEAL     | Δ    | Time (ms) | Params (M) |
|-------------|--------|--------------------|----------|------|-----------|------------|
| DDD17-Ins   | AP     | 29.8 (OpenSeg)     | 32.3     | +2.5 | 22.3      | 99.1       |
| DSEC11-Ins  | AP50   | 40.3 (frame2recon) | 55.1     | +14.8| –         | –          |
| DSEC19-Ins  | AP25   | 26.9 (frame2recon) | 36.3     | +9.4 | –         | –          |
| DSEC-Part   | AP     | 12.9 (VLPart)      | 13.6     | +0.7 | –         | –          |

SEAL attains 45 FPS inference speed (22 ms per frame) on a single RTX 6000 Ada, with a memory footprint of 99.1M parameters. This is substantially smaller than alternatives requiring 300–530M parameters (dual-backbone designs).

## 6. Prompt-Free OV-EIS and SEAL⁺⁺

SEAL⁺⁺ extends SEAL to prompt-free, generic, spatiotemporal OV-EIS. It augments the backbone with a lightweight, class-agnostic event detector (RVT+FPN). At inference, detected boxes provide automatic prompts for mask decoding and classification. SEAL⁺⁺ is trained with IoU loss, objectness BCE, and regression on DSEC-Detection and achieves:

| Method              | AP   | AP50 | AP75 | Time (ms) |
|---------------------|------|------|------|-----------|
| DEOE+Best Baseline  | 16.3 | 21.8 | 18.5 | 400       |
| DEOE+SEAL           | 16.4 | 22.1 | 18.8 | 28        |
| SEAL⁺⁺              | 17.8 | 23.6 | 19.2 | 24        |

SEAL⁺⁺ outperforms naive detection+segmentation pipelining and prior closed-set active object detection systems, retaining real-time performance.

## 7. Significance and Future Prospects

SEAL establishes a rigorous formulation for OV-EIS, demonstrating that a single, parameter-efficient backbone can integrate event-based instance segmentation and open-vocabulary mask classification at multiple semantic granularities. Its empirical advances—across accuracy, inference speed, and model parsimoniousness—suggest that OV-EIS is tractable at scale and for real-time robotics applications. The introduction of hierarchical semantic supervision and prompt-free extensions underscores the extensibility and generalizability of the framework. A plausible implication is that future research will build on open-vocabulary approaches for embodied event perception in unconstrained environments [2601.23159].

Source: https://www.emergentmind.com/topics/open-vocabulary-event-instance-segmentation-ov-eis