Talk2Event: Event-Based Visual Grounding

• Talk2Event is a benchmark for language-driven object grounding in event-camera data, using dynamic driving scenes with synchronized event streams and RGB frames.
• It employs an attribute-centric annotation scheme with 30,690 validated referring expressions over 5,567 scenes to support multimodal grounding.
• The EventRefer framework integrates attribute-specific experts, achieving superior mAcc and mIoU in event-only, frame-only, and fusion settings.

Searching arXiv for the Talk2Event benchmark and closely related event-camera grounding work. Talk2Event is a benchmark and task formulation for language-driven object grounding in event-camera data, introduced as, to the authors’ knowledge, the first large-scale benchmark of its kind. It defines a setting in which a natural-language referring expression is grounded to a 2D bounding box in a dynamic scene observed through an event stream, optionally together with a synchronized RGB frame. Across its benchmark papers, Talk2Event couples a DSEC-derived dataset of 5,567 scenes, 13,458 annotated objects, and 30,690 validated referring expressions with an attribute-centric annotation scheme, and later pairs that benchmark with the EventRefer model for attribute-aware multimodal grounding (Kong et al., 11 Sep 2025, Kong et al., 23 Jul 2025).

1. Task definition and sensing model

Talk2Event studies event-based visual grounding. The raw event stream is written as $\mathcal{E} = \{e_k\}_{k=1}^N$, where each event is $e_k = (x_k, y_k, t_k, p_k)$ and the polarity satisfies $p_k \in \{-1,+1\}$. Given a voxelized event representation $\mathbf{E}$, a referring expression $\mathcal{S} = \{w_1, w_2, \dots, w_C\}$, and optionally a synchronized RGB frame $\mathbf{F}$, the task is to predict a bounding box $\hat{\mathbf{b}} = (x, y, w, h)$ for the referred object (Kong et al., 11 Sep 2025).

The event tensor preserves polarity and discretized temporal structure, with the general representation $$\mathbf{E} \in \mathbb{R}^{2 \times T \times H \times W}$$. In the EventRefer implementation, the event input is voxelized with $T = 9$, $H = 128$, and $e_k = (x_k, y_k, t_k, p_k)$0, giving $e_k = (x_k, y_k, t_k, p_k)$1 (Kong et al., 23 Jul 2025).

The benchmark supports three evaluation settings: event-only grounding, frame-only grounding, and event-frame fusion. This structure is central to Talk2Event’s design because event streams are motion-centric, asynchronous, and sparse, whereas frames retain richer appearance cues. The benchmark is therefore not merely a modality substitution for standard referring expression grounding; it is explicitly organized around the interaction between dynamic sensing and language.

2. Dataset construction and benchmark composition

Talk2Event is built from DSEC, a real-world driving dataset with synchronized event streams and RGB frames recorded in urban driving environments in Switzerland. Construction proceeds by sampling keyframes at 5 Hz and, for each selected timestamp $e_k = (x_k, y_k, t_k, p_k)$2, extracting a centered event volume over $e_k = (x_k, y_k, t_k, p_k)$3, i.e. a 200 ms event window, together with synchronized frame context (Kong et al., 23 Jul 2025).

The benchmark contains 5,567 scenes, 13,458 annotated objects, and 30,690 validated referring expressions, with an average referring-expression length of 34.1 words. It covers seven traffic-related categories: Car, Pedestrian, Bus, Truck, Bike / Bicycle, Motorcycle, and Rider. The split reported in the EventRefer paper consists of 4,433 training scenes with 10,321 unique objects and 23,025 captions, and 1,134 test scenes with 3,137 unique objects and 7,665 captions (Kong et al., 23 Jul 2025).

A concise summary of the benchmark scale is as follows.

Component	Value	Source
Scenes	5,567	DSEC-derived benchmark
Annotated objects	13,458	Human-verified targets
Validated referring expressions	30,690	Three expressions per object after filtering

The annotation pipeline is explicitly context-aware. For each target object at $e_k = (x_k, y_k, t_k, p_k)$4, two neighboring RGB frames at $e_k = (x_k, y_k, t_k, p_k)$5 and $e_k = (x_k, y_k, t_k, p_k)$6, with $e_k = (x_k, y_k, t_k, p_k)$7, are provided to Qwen2-VL-72B to elicit descriptions that can mention appearance, motion, displacement, interactions, and scene relations. For each object, three distinct captions are generated, then manually verified. Human verification covers all 30,690 expressions over 13,458 objects; the reported annotation effort is 8,612 minutes for the training set and 3,370 minutes for the test set, carried out by five trained annotators with calibration sessions and spot-check audits (Kong et al., 23 Jul 2025).

Quality control includes visibility filtering for small, occluded, or ambiguous objects, redundancy filtering to enforce linguistic diversity across the three captions per object, and attribute validation. The stricter human-verification criteria in the later paper require referential correctness, linguistic fluency, and the presence of at least two of the four grounding attributes without hallucinated unsupported attributes. This suggests that Talk2Event was designed not simply as a caption-and-box corpus, but as a controlled grounding benchmark whose language is intentionally compositional and discriminative.

3. Attribute-centric annotation scheme

A defining feature of Talk2Event is its four-attribute annotation schema. Each expression is decomposed into Appearance, Status, Relation-to-Viewer, and Relation-to-Others, and the benchmark is explicitly presented as attribute-centric rather than as a plain caption-and-box dataset (Kong et al., 11 Sep 2025).

Attribute	Content	Function
Appearance	category, color, size, shape	appearance-based disambiguation
Status	moving, stopped, turning, crossing	temporal reasoning
Relation-to-Viewer	left, right, front, near, far	egocentric spatial grounding
Relation-to-Others	next to, behind, together with	contextual and relational grounding

Appearance covers static and object-centric properties such as category, size, shape, and color. Status covers dynamic state, including whether an object is moving, stopped, turning, approaching, or crossing. Relation-to-Viewer encodes egocentric positioning relative to the observer or ego vehicle, such as in front, on the left, nearby, or far. Relation-to-Others describes contextual relations involving nearby entities, such as being behind a bus or next to a car (Kong et al., 23 Jul 2025).

This schema is intended to support compositional grounding, interpretable grounding, spatial reasoning, temporal reasoning, and relational reasoning. In the event-camera setting, the distinction is methodologically important: status aligns naturally with event streams, appearance aligns more strongly with RGB frames, and viewer-relative or inter-object relations require more global context. A plausible implication is that the benchmark’s attribute design was chosen to make modality-specific strengths measurable rather than latent.

Talk2Event’s language is correspondingly verbose. The paper emphasizes that expressions are long and context-rich because dynamic driving scenes often contain multiple similar objects, making short category-style phrases insufficient. Representative examples include descriptions such as “The red motorcycle ahead is turning right next to a white car,” together with rewrite variants that preserve the same grounding content while varying lexical form (Kong et al., 23 Jul 2025).

4. EventRefer and attribute-aware grounding

The later Talk2Event paper introduces EventRefer, an attribute-aware grounding framework that uses Positive Word Matching, Multi-Attribute Fusion, and a Mixture of Event-Attribute Experts. The model uses RoBERTa-base for language encoding, a transformer-based event backbone initialized from FlexEvent for event-only and fusion settings, ResNet-101 for frame-only and fusion settings, and a shared DETR-style encoder-decoder initialized from BUTD-DETR (Kong et al., 23 Jul 2025).

Positive Word Matching constructs attribute-specific token maps by fuzzy matching cue words in the referring expression to the four grounding attributes. These token maps are then used to build attribute-aware masks over fused multimodal hidden states. EventRefer therefore does not treat the sentence as a single undifferentiated text embedding; it explicitly creates appearance-aware, status-aware, viewer-relation-aware, and relation-to-others-aware streams.

The Mixture of Event-Attribute Experts is the core fusion mechanism. Each attribute-specific stream is refined by an expert, summarized into a pooled descriptor, and then routed through a softmax gate with a learnable noise term $e_k = (x_k, y_k, t_k, p_k)$8, where $e_k = (x_k, y_k, t_k, p_k)$9. The final fused representation is a weighted sum of the expert outputs. In the paper’s interpretation, this allows appearance to dominate when appearance cues are strong, status to dominate in highly dynamic scenes, and relational cues to receive higher weight in cluttered settings.

Training uses a DETR-style Hungarian matching procedure, but with a distinctive multi-attribute construction: a single ground-truth box is duplicated into four co-located pseudo-targets, one for each attribute token map. The matching cost combines $p_k \in \{-1,+1\}$0 box regression, GIoU, and an attribute-alignment cross-entropy term. At inference time, decoder queries are scored by token-distribution similarity against the attribute token maps, and the highest-scoring query determines the final box (Kong et al., 23 Jul 2025).

The architecture is deliberately modality-flexible. The same grounding framework is used in event-only, frame-only, and event-frame fusion settings, with the main change being which visual token sources are present. This is one of the paper’s main methodological claims: attribute decomposition and dynamic fusion are useful even outside pure event-only grounding.

5. Empirical performance and ablations

In the reported benchmark results, EventRefer achieves 55.47 mAcc and 85.76 mIoU in the frame-only setting, 31.96 mAcc and 76.46 mIoU in the event-only setting, and 61.82 mAcc and 87.32 mIoU in the event-frame fusion setting. These results exceed the strongest baselines listed in the paper for all three settings: BUTD-DETR in frame-only, EvRT-DETR in event-only, and FlexEvent in event-frame fusion (Kong et al., 23 Jul 2025).

A compact summary of the headline results is given below.

Setting	Strongest baseline	EventRefer
Frame-only	48.91 mAcc / 84.30 mIoU	55.47 mAcc / 85.76 mIoU
Event-only	29.34 mAcc / 75.66 mIoU	31.96 mAcc / 76.46 mIoU
Event-frame fusion	59.40 mAcc / 86.83 mIoU	61.82 mAcc / 87.32 mIoU

The paper also reports class-wise gains, especially in dynamic or difficult categories. In event-frame fusion, the highlighted top-1 gains are particularly strong for rider ($p_k \in \{-1,+1\}$1), bicycle ($p_k \in \{-1,+1\}$2), and truck ($p_k \in \{-1,+1\}$3). In frame-only grounding, the paper notes gains of about $p_k \in \{-1,+1\}$4 for pedestrian and about $p_k \in \{-1,+1\}$5 for rider relative to the best baseline mAcc. Scene-complexity analysis further shows that EventRefer remains stronger than prior systems as the number of objects per scene increases, including heavily cluttered settings (Kong et al., 23 Jul 2025).

The ablation studies are central to the technical narrative. In the event-only setting, the base system scores 22.07 mAcc; adding Positive Word Matching raises this to 26.38, adding Multi-Attribute Fusion yields 27.01, using both gives 29.66, and the full model with MoEE reaches 31.96. Single-attribute ablations show that status alone yields 28.90 mAcc, slightly above appearance alone at 27.98, while all four attributes together reach 31.96. This is consistent with the event-camera premise that motion-related descriptions carry special weight in event-only perception (Kong et al., 23 Jul 2025).

The paper’s interpretability analyses make this explicit. In event-only grounding, Rider and Bike rely more on status, while Bus and Truck rely more on appearance and viewer relation. A second analysis bins scenes by event response strength and reports that low event response favors appearance and viewer-centric cues, whereas high event response increases the importance of status and relation-to-others. This suggests that EventRefer’s gains are tied not only to better text conditioning, but to adaptive redistribution of attribute emphasis under changing scene dynamics.

6. Scope, limitations, and significance

Talk2Event is domain-specific. It is built from driving scenes sourced from DSEC rather than from general-purpose imagery, and the object distribution is class-imbalanced, with cars heavily dominating the benchmark. The task output is a 2D bounding box, so the benchmark does not exploit richer 3D grounding possibilities despite DSEC’s multimodal nature. The reported failure cases include highly cluttered scenes, overlapping instances, very similar objects, occluded or viewpoint-limited targets, expressions emphasizing abstract or implicit relations such as “approaching an intersection,” and extreme low-light scenes where event signals are sparse (Kong et al., 23 Jul 2025).

The benchmark’s significance lies in the specific gap it addresses. Event-camera research had largely emphasized detection, segmentation, optical flow, odometry, and related low-level or mid-level perception tasks, while visual grounding benchmarks had remained centered on RGB, RGB-D, LiDAR, or static 3D scans. Talk2Event bridges those two literatures by making event streams compatible with free-form referring expressions and by explicitly structuring the language side around spatial, temporal, and relational cues (Kong et al., 11 Sep 2025).

Its broader role is therefore methodological rather than merely dataset-scaling. Talk2Event formalizes a language-grounding problem for asynchronous sensing, provides a large real-world benchmark with validated expressions, and shows that attribute-aware multimodal fusion improves grounding across event-only, frame-only, and fusion regimes. This suggests a foundation for multimodal, temporally-aware perception in robotics, autonomy, and human-AI interaction, especially in environments where motion blur, rapid ego-motion, low light, and high dynamic range degrade conventional frame-based vision (Kong et al., 11 Sep 2025)